https://gradle.com/ logo
Join Slack
Powered by
This message was deleted.
# community-support
s
This message was deleted.
s
multiple upload attempts from the Gradle client simultaneously - the upload attempts have the same timestamp, with no time interval between them, which is odd, as we would expect the client to wait for a response and not send multiple upload attempts of the same file at the same time.
v
Are you using the artifactory Gradle plugin, or ure 
maven-publish
? I don't think Gradle should try multiple uploads concurrently . If that really is the case, you should probably open a GitHub issue in the Gradle repository, optimally with an MCVE.
s
We're using 
maven-publish
plugin
v
More like the opposite I'd say
s
How should we debug this? We captured the tcpdump but could not find something meaningful, we re using AKS not sure whether can it be related with VMs also
Do you have any suggestion?
v
Maybe you have multiple projects that publish to the same coordinates? Try to find from the debug lot which publish tasks the identical times are coming from
s
can you see anything meaningful here?
v
Didn't you say it tries concurrently? There are 10 minutes before the tries. And no, from this small excerpt no.
s
yeah that hypothesis dead, somehow we saw some duplicate logs which misguided
v
Well, scroll up from both upload tries and see which publish task is currently executed
s
there are tons of tasks uploaded
Copy code
Run ./gradleWrapper.sh \

+ ./gradlew -s '-Dorg.gradle.jvmargs=-Xms4g -Xmx32g -XX:MaxMetaspaceSize=4g -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParallelGC -Dfile.encoding=UTF-8' -Dorg.gradle.workers.max=15 publish -PpublishRepo=<https://artifactory.xxxx.com/artifactory/as-navapp-maven-release/> -PpublishUser=svc-ar-as-navapp-release-publisher -Dorg.gradle.internal.http.socketTimeout=900000 -Dorg.gradle.internal.http.connectionTimeout=900000
Type-safe project accessors is an incubating feature.
> Task :buildSrc:compileKotlin UP-TO-DATE
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy NO-SOURCE
> Task :buildSrc:pluginDescriptors UP-TO-DATE
> Task :buildSrc:processResources NO-SOURCE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar UP-TO-DATE
> Task :buildSrc:inspectClassesForKotlinIC UP-TO-DATE
> Task :buildPlugins:compileKotlin UP-TO-DATE
> Task :buildPlugins:compileJava NO-SOURCE
> Task :buildPlugins:pluginDescriptors UP-TO-DATE
> Task :buildPlugins:processResources UP-TO-DATE
> Task :buildPlugins:classes UP-TO-DATE
> Task :buildPlugins:jar UP-TO-DATE
> Task :buildPlugins:inspectClassesForKotlinIC UP-TO-DATE
which one should I point out for reference?
v
In what you show is no single publishing task. Scroll in the debug log from the two upload lines up to find which publish task is currently executed, to hopefully find which projects have conflicting coordinates.
s
@Vampire is default timeout 
10m
for Gradle?
v
What timeout?
s
For retry.. I mean why it is retrying again after 
10m
I could not find this in src code of gradle somehow
v
I have no idea, but my suspection was a different one
s
Why it is not 5 min for instance, trying to understand the relation between whether it's related our setup or not
Long story short, we see client retries (after 10 minutes) for requests that took 2s for artifactory to process
v
Again, I don't this is an actual retry. Are you sure it is not coming from different tasks?
s
What do you mean by actualy retry?
v
As I said multiple times already, my suspicion is, that you have multiple tasks / projects that try to publish to the same coordinates, not a retry to upload from one task. You didn't yet falsify that suspicion.
s
I could not catch this tbh
v
Then ask about what you don't understand 🙂
s
What I meant i could find a way to falsify your suspicion through debug logs
v
As i said, scroll up from the upload requests to find the task that is causing this upload and compare whether it is the same task or different tasks.
For example you can use
Copy code
grep 'HTTP PUT\|Executing actions for task.*publish' theOutputOfRunningGradleWithDebug.txt
Then please share the result
s
Copy code
$ grep 'HTTP PUT\|Executing actions for task.*publish' itsm-35683-p1-failure-mode-annotated-client-logging.txt
2023-06-09T08:16:04.9936065Z 2023-06-09T08:16:00.447+0000 [DEBUG] [org.gradle.api.internal.tasks.execution.TaskExecution] Executing actions for task ':data-management:navigation-tile-store-binding:publishJvm-veda-x86-64PublicationToArtifactoryGoSDKDevRepository'.
2023-06-09T08:16:04.9967041Z 2023-06-09T08:16:00.554+0000 [DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP PUT: ***/artifactory/sdk-maven-dev/com/tomtom/sdk/navigation-tile-store-binding-internal-jvm-veda-x86-64/0.2.%233ea1b93-SNAPSHOT/maven-metadata.xml
2023-06-09T08:16:04.9974690Z 2023-06-09T08:16:00.580+0000 [DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP PUT: ***/artifactory/sdk-maven-dev/com/tomtom/sdk/navigation-tile-store-binding-internal-jvm-veda-x86-64/0.2.%233ea1b93-SNAPSHOT/maven-metadata.xml.sha1
2023-06-09T08:16:04.9982480Z 2023-06-09T08:16:00.605+0000 [DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP PUT: ***/artifactory/sdk-maven-dev/com/tomtom/sdk/navigation-tile-store-binding-internal-jvm-veda-x86-64/0.2.%233ea1b93-SNAPSHOT/maven-metadata.xml.md5
2023-06-09T08:16:04.9990346Z 2023-06-09T08:16:00.631+0000 [DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP PUT: ***/artifactory/sdk-maven-dev/com/tomtom/sdk/navigation-tile-store-binding-internal-jvm-veda-x86-64/0.2.%233ea1b93-SNAPSHOT/maven-metadata.xml.sha256
2023-06-09T08:16:04.9998158Z 2023-06-09T08:16:00.658+0000 [DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP PUT: ***/artifactory/sdk-maven-dev/com/tomtom/sdk/navigation-tile-store-binding-internal-jvm-veda-x86-64/0.2.%233ea1b93-SNAPSHOT/maven-metadata.xml.sha512
2023-06-09T08:16:05.0006846Z 2023-06-09T08:16:00.842+0000 [DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP PUT: ***/artifactory/sdk-maven-dev/com/tomtom/sdk/navigation-tile-store-binding-internal-jvm-veda-x86-64/0.2.%233ea1b93-SNAPSHOT/navigation-tile-store-binding-internal-jvm-veda-x86-64-0.2.%233ea1b93-20230609.074212-1.jar
2023-06-09T08:26:03.4930157Z 2023-06-09T08:26:02.724+0000 [DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP PUT: ***/artifactory/sdk-maven-dev/com/tomtom/sdk/navigation-tile-store-binding-internal-jvm-veda-x86-64/0.2.%233ea1b93-SNAPSHOT/navigation-tile-store-binding-internal-jvm-veda-x86-64-0.2.%233ea1b93-20230609.074212-1.jar
v
Do you get results when grepping for 
NetworkOperationBackOffAndRetry
?
s
yes
Copy code
$ grep 'NetworkOperationBackOffAndRetry' itsm-35683-p1-failure-mode-annotated-client-logging.txt
2023-06-09T08:26:01.7934357Z 2023-06-09T08:26:01.722+0000 [INFO] [org.gradle.api.internal.artifacts.repositories.transport.NetworkOperationBackOffAndRetry] Error in 'PUT ***/artifactory/sdk-maven-dev/com/tomtom/sdk/navigation-tile-store-binding-internal-jvm-veda-x86-64/0.2.#3ea1b93-SNAPSHOT/navigation-tile-store-binding-internal-jvm-veda-x86-64-0.2.#3ea1b93-20230609.074212-1.jar'. Waiting 1000ms before next retry, 2 retries left
2023-06-09T08:26:01.7935644Z 2023-06-09T08:26:01.723+0000 [DEBUG] [org.gradle.api.internal.artifacts.repositories.transport.NetworkOperationBackOffAndRetry] Network operation failed
2023-06-09T08:26:01.7940544Z 	at org.gradle.api.internal.artifacts.repositories.transport.NetworkOperationBackOffAndRetry.withBackoffAndRetry(NetworkOperationBackOffAndRetry.java:50)
v
Yeah, as I assumed from reading the code. It retries up to 3 times with growing time between them, starting with 1 second, then 2 seconds between the retries.
So yes, it is indeed retrying to upload
I guess something in your infrastructure does not properly return the success reply when doing the first upload for 10 minutes, then gives an error or just closes, or a timeout is hit and then Gradle retries, failing due to the artifact already being present.
So you have to find out (probably on a network or infrastructure level) why the success response is not coming through
If the 10 minutes is a timeout value in Gradle, reducing it would not help, it would just fail faster
You could maybe also have a look at the stacktrace where you see the first line in the last grep output, maybe the full stacktrace has some information on what went wrong, but I slightly doubt it
s
If the 10 minutes is a timeout value in Gradle, reducing it would not help, it would just fail faster
We are using Azure DevOps and traffic goes away NAT Gateway
10min might be causing some SNAT exhaustion
v
I don't really get what you are saying, but you said artifactory processed the first PUT successfully in 2 seconds. So the success return should be there after 2 seconds too, so the 10 minutes should be irrelevant, once you sorted out why the success return is not coming through.
s
that's the tricky part 😕
We tried to enable tcpdump
v
Yeah, but nothing the Gradle community is likely to be able to help with. 🙂
s
but it's also a bit useless due to SSL
thanks @Vampire, at least i am sure it's not a gradle bug
👌 1
2 Views
Open in Slack
PreviousNext
Powered by