Regarding the log4j slowdown, perhaps have a look at https://github.com/line/armeria/issues/2306 which showed similar symptoms and was due to specific calls in the client code (not log4j itself) that used a long-deprecated method.
For what it's worth, we saw the opposite problem: before depstar added the Multi-Release: true detection/setting, log4j2 for us was painfully slow on Java 11 (but fine on Java 8) because it was falling back to reflection and a stack-walker, as I recall.
Regarding :manifest configuration: that's something already built into depstar which now has an API offering drop-in replacements for tools.build's jar and uber functions (with the exact same inputs and returns).
I'd be happy to add an option to disable MR detection to depstar if you want to try it with tools.build? See https://cljdoc.org/d/com.github.seancorfield/depstar/2.1.267/doc/tools-build-usage for details.