I'm using the Open Policy Agent in a project. It has its own non-JVM testing framework. See https://www.openpolicyagent.org/docs/policy-testing. I can create custom tasks to run the tests, but the tests do not show up in the Gradle build scan. I believe I must create a custom Test task for that but the Test task expects JVM classes or else it says NO-SOURCE and doesn't run the tests. What is the recommended way to run non-JVM tests and have them show up in the Gradle build scan?

There is an incubating API for this: https://docs.gradle.org/current/userguide/test_reporting_api.html#test_reporting_api

That takes care of basic reporting. What about develocity features like predictive test selection, test distribution, or flaky test reporting? Is it possible to integrate those features with non-JVM tests?

No, and there are no current plans to support that.

if your custom test task creates a junit xml you might be able to use this helper from the develocity plugin.

import com.gradle.develocity.agent.gradle.test.ImportJUnitXmlReports
import com.gradle.develocity.agent.gradle.test.JUnitXmlDialect

ImportJUnitXmlReports.register(project.tasks, customTask, JUnitXmlDialect.GENERIC)

Making the

Test

task run and not have no-source is trivial, just register another skip-when-empty input using the runtime API and make sure it is not empty but contains anything like the build script or a temporary file. A task is only skipped with no-source if all those inputs are empty. You could also implement a JUnit Platform engine that "translates". A JUnit engine can also report test out of thin air and for Gradle they are just tests that come from JUnit Platform. You might need to follow some restrictions if the predictive test selection needs them or similar. But it should be well doable I guess, even without that new Gradle API.

Thanks everyone for the input!