For a Kotlin/JVM project that uses Gradle's application plugin and Kotest as the test framework, is it somehow possible to build a self-contained "executable" that does not run the main class of the application, but runs the tests? I'd like to create a "distribution" of my application that can be used to run the application's test easily in different environments without the need to ship the test source code and / or Gradle as a build tool.

gradle uses junit-platform-launcher directly; if you want to run tests in your own binary, you can use the https://github.com/junit-team/junit5/blob/main/junit-platform-console/src/main/java/org/junit/platform/console/ConsoleLauncher.java entrypoint