This message was deleted.

My team has worked with macrobenchmarks for a similar project. Android documentation and nowinandroid were good samples for us. Can you share what errors you are facing?

Thanks I’ll check out the nowinandroid repo shortly. In AS Dolphin, using the New Module… benchmark wizard didn’t work out of the box:

Build file '/path/to/repo/MyProject/build.gradle.kts' line: 26

Plugin [id: 'com.android.test', version: '7.3.1', artifact: 'com.android.tools.build:gradle:7.3.1', apply: false] was not found in any of the following sources:

* Try:
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Exception is:
org.gradle.api.plugins.UnknownPluginException: Plugin [id: 'com.android.test', version: '7.3.1', artifact: 'com.android.tools.build:gradle:7.3.1', apply: false] was not found in any of the following sources:

- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Plugin Repositories (could not resolve plugin artifact 'com.android.tools.build:gradle:7.3.1')
  Searched in the following repositories:
    Gradle Central Plugin Repository
	at org.gradle.plugin.use.internal.DefaultPluginRequestApplicator.resolveToFoundResult(DefaultPluginRequestApplicator.java:238)

<clipped>

I can get past this error by removing the automatic changes to the project level

build.gradle.kts

but then I run into other errors. These were the extra lines added by the wizard:

plugins {
    ...
    id("com.android.test") version "7.3.1" apply false
    id("org.jetbrains.kotlin.android") version "1.7.20" apply false
}

I guess

id("com.android.test") version "7.3.1" apply false

is a newer convention for applying android plugins instead of adding buildscript.dependencies.classpath. It looks fine to change it and keep it consistent with how "com.android.application" is getting applied in your project. What error do you get after deleting the extra lines?

id("com.android.test")

is the same convention I’m using for com.android.application (except that one is applied in the app-level

build.gradle.kts

. Once I remove those 2 new lines, the project successfully syncs but when I try to build and deploy the app, I see there are a bunch of errors in the AndroidManifest.xml caused by the wizard mangling the

<profileable>

tag. When I manually fix that (side note: should I ensure the tools:targetApi matches the app targetSdkVersion?) then I can build the app. However, when switching to the new benchmark variant

fooBarBenchmark

I get a bunch of dependency errors like:

:lib-base has variant "release" select, but the modules … depend on variant "debug"

In the build output I see

Unable to resolve dependency for ':benchmark@benchmark/testTarget'

which perhaps is due to not including

com.android.test

?

I was only referring to project level build.gradle.kts for

id("com.android.test") version "7.3.1" apply false

convention. module level build.gradle will have

id("com.android.application")

,

id("com.android.library")

,

id("com.android.test")

etc. You can remove

id("com.android.test") version "7.3.1" apply false

from root build.gradle if there is no

id("com.android.application") version "7.3.1" apply false

. But that seems sorted now. Not sure what part of <profileable> was mangled, but you should fix any syntax errors manually. The tools:targetApi inside <profileable> should be "q" or 29, not dependent on your targetSdkVersion. This can later be moved to AndroidManifest in benchmark sourceset, to leave your release builds unaffected. Are you using AS "select build variant" to switch? If so, check all modules are switched to correct variant. If you app has custom flavors, you'll have specify which dimension to use in your benchmark build.gradle using missingDimensionStrategy

When I switch to the `fooBarBenchmark`variant, some of the dependency modules switch to release but others do not (they stay on debug). So I get the feeling something is not set up correctly. It shouldn’t be that I have to manually switch some dependencies but not others. I have another app module (which I don’t want to benchmark) and I need to manually switch that to “release” in order for certain other dependencies to also be switched to “release”. Then all is fine regarding the dependency build variants. One thing I found confusing is when the benchmark docs refer to “package name of the app” do they mean the packageName of the app module or the applicationId of the app (i.e. for whichever variant is currently selected)?

packageName in tests should be the applicationId of your target app. My applicationId is same as package, so I havent faced any issue. The manual variant switching issue may not be related to benchmark, hope you can debug it. Did you get more errors after manually switching all module variants?

When attempting to launch the Benchmark startup method I get:

Could not determine the dependencies of task ':benchmark:connectedBenchmarkAndroidTest'.
> Could not determine the dependencies of null.
   > Could not resolve all task dependencies for configuration ':benchmark:benchmarkTestedApks'.
      > Could not resolve project :app-main.
        Required by:
            project :benchmark
         > The consumer was configured to find a runtime of a component, as well as attribute 'com.android.build.api.attributes.BuildTypeAttr' with value 'benchmark', attribute 'com.android.build.api.attributes.AgpVersionAttr' with value '7.3.1'. However we cannot choose between the following variants of project :app-main:
             - foo1Bar1BenchmarkRuntimeElements
             - foo1Bar2BenchmarkRuntimeElements
             - foo2Bar2BenchmarkRuntimeElements
           All of them match the consumer attributes:
             - Variant 'foo1Bar1BenchmarkRuntimeElements' capability MyProject:app-main:unspecified declares a runtime of a component, as well as attribute 'com.android.build.api.attributes.AgpVersionAttr' with value '7.3.1', attribute 'com.android.build.api.attributes.BuildTypeAttr' with value 'benchmark':
                 - Unmatched attributes:
                     - Provides attribute 'com.android.build.api.attributes.ProductFlavor:lang' with value 'foo1' but the consumer didn't ask for it
                     - Provides attribute 'com.android.build.api.attributes.ProductFlavor:tier' with value 'bar1' but the consumer didn't ask for it
                     - Provides attribute 'com.android.build.gradle.internal.attributes.VariantAttr' with value 'foo1Bar1Benchmark' but the consumer didn't ask for it
                     - Provides attribute 'foodimension' with value 'foo1' but the consumer didn't ask for it
                     - Provides a library but the consumer didn't ask for it
                     - Provides attribute 'org.gradle.jvm.environment' with value 'android' but the consumer didn't ask for it
                     - Provides attribute 'org.jetbrains.kotlin.platform.type' with value 'androidJvm' but the consumer didn't ask for it
                     - Provides attribute 'tier' with value 'bar1' but the consumer didn't ask for it
<clipped>

Yeah this is why you'll need to define missingDimensionStrategy

missingDimensionStrategy("lang", "foo1")
missingDimensionStrategy("tier", "bar1")

This will tell benchmark to use foo1Bar1 variant of app-main

Thanks very much, that really helps! Looks like I’m almost there. I’ve set up the managedDevices in the gradle script, but when I execute

./gradlew :benchmark:pixel6Api31BenchmarkAndroidTest

I get:

> Task :benchmark:pixel6Api31BenchmarkAndroidTest
Exception thrown during onBeforeAll invocation of plugin com.google.testing.platform.plugin.android.AndroidDevicePlugin.
Failed to install APK: /path/to/repo/MyProject/app/build/outputs/apk/fooBar/benchmark/app-main-foo-bar-x86_64-benchmark.apk, with option -r,-t
com.google.testing.platform.api.plugin.PluginException: Failed to install APK: /path/to/repo/MyProject/app/build/outputs/apk/fooBar/benchmark/app-main-foo-bar-x86_64-benchmark.apk, with option -r,-t
        at com.google.testing.platform.plugin.android.AndroidDevicePlugin.beforeAll(AndroidDevicePlugin.kt:109)

My app is fairly large so perhaps it’s a storage issue, however I can’t see anything in the error regarding this. Also, where to find the available configuration options for setting up a managedDevice? Update: I can generate the

baseline-prof.txt

by running the BaselineProfileGenerator from AS, but it would be nice to be able to use a managedDevice.

I havent tried managed devices before, but if Android studio is failing to install, you can try building and installing through command line. If you plan to use benchmark with multiple app variants, then you'll need to create same variants in your benchmark module too. If your app and benchmark modules have the same variants then you dont need missingDimensionStrategy at all.

Thanks again, yes that’s working well. However, in AS Build Variants pane, when I change the app module variant the benchmark module variant does not automatically change so I need to do it manually (same vice-versa). Presumably because neither modules depend on each other. Is there a way to set up the dependencies without causing side effects? Note: I changed the benchmark module namespace to be the app module namespace + “.benchmark”. I also had to add logic to the benchmark gradle script to generate the applicationId from the flavors (just like what needs to be done in the app gradle script). Would be nice to share that code, but it’s not important and I don’t think it’s worth getting side tracked.