I am trying to add a jasper related task like below,

tasks.register('compileJasperReports') {
    dependsOn classes

    doLast {
        def jasperOut = jasperOutDir.get().asFile
        jasperOut.mkdirs()

        ant.taskdef(
                name: 'jasper',
                classname: 'net.sf.jasperreports.ant.JRAntCompileTask',
                classpath: jasperCompileClasspath.asPath
        )

        ant.jasper(
                destdir: jasperOut,
                srcdir: jasperSrcDir,
                includes: jasperIncludes
        ) {
            classpath {
                pathelement(path: mainOutputClassesDirs.asPath)
                pathelement(path: mainOutputResourcesDir)
                pathelement(path: compileClasspath.asPath)
                pathelement(path: jasperCompileClasspath.asPath)
            }
        }
    }
}

but the build is failing with

14:08:04 FAILURE: Build failed with an exception.
14:08:04 
14:08:04 * What went wrong:
14:08:04 Java heap space

I have added this in gradle.properties

org.gradle.jvmargs=-Xmx12g -XX:MaxMetaspaceSize=2g -XX:+HeapDumpOnOutOfMemoryError

What is cousing this issue?

its possible that the heap space runs out in the client, not the deamon, Your can try to set the environament variable GRADLE_OPTS to

-Xmx12g

Is this not the same? org.gradle.jvmargs=-Xmx12g -XX:MaxMetaspaceSize=2g -XX:+HeapDumpOnOutOfMemoryError

org.gradle.jvmargs is affecting the gradle daemon. GRADLE_OPTS is affecting the gradle client. its not quite clear from where your heap space error is comming from, given your very brief failure log. Could be Deamon, could be client, could be some task

Could be similar to this: https://github.com/gradle/gradle/issues/34796 The message is slightly different, but I also got the message from here when trying to reproduce that. So yes, CLI process OOM could also look like the shown error.

I think it's a CLI issue

for now your best bet is to use GRADLE_OPTS to raise memory until it works

where to add this GRADLE_OPTS in a fresstyle gradle jenkins job

Wherever you define environment variables

Now there I am getting another issue while using Gradle 9+ when using mode to set the executable permissions. Could not set unknown property 'mode' for object of type org.gradle.api.internal.file.DefaultConfigurableFilePermissions. It was working with 8.13. What I need to use for this in gradle 9+?

Thanks @TheGoesen it worked need to implement this way

filePermissions {
     user {
         read = true
         execute = true
     }
     other.execute = false
 }

Don't ignore deprecation messages and this will not happen. ;-) With at latest 8.14.3 that produced a deprecation warning with the hint that it will fail the build in Gradle 9.