409 conflict when publishing many artifacts via maven-publish (gradle)

[glureau]

Select Topic Area

Bug

Body

* What went wrong:
For more on this, please refer to https://docs.gradle.org/8.8/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
Execution failed for task ':aPrivateModule:publishIosSimulatorArm64PublicationToGitHubPackagesRepository'.
> Failed to publish publication 'iosSimulatorArm64' to repository 'GitHubPackages'
1098 actionable tasks: 759 executed, 339 from cache
   > Could not PUT 'https://maven.pkg.github.com/aPrivateCompany/aPrivateRepository/com/foo/bar/aPrivateModule-iossimulatorarm64/13.0.0-20250117090733/aPrivateModule-iossimulatorarm64-13.0.0-20250117090733.pom.md5'. Received status code 409 from server: Conflict

I've 2 main projects, each one containing more than 100 gradle modules. Using the standard maven-publish plugin, we were able to publish those packages for a while. Then a few months ago we notice that sometime there was this kind of error. There's clearly no conflict and restarting the build was "fixing" the problem. (Mostly snapshot builds which contains the date+time to avoid conflicts, ex: 13.0.0-20250117090733 from the error above.)

There days it's getting way worse, I've to restart the build multiple times, taking half a day or more of just checking CI and restarting this.

My hypothesis / uderstanding so far:

• maven-publish starts to publish
• github packages at some points times out (maybe a security feature to avoid DDOS?)
• maven-publish after a timeout attempt to upload again the same file
• then github packages returns the 409, because yes now there is a real duplicate

Thanks for your attention.

Similar topics

https://discuss.gradle.org/t/maven-publish-plugin-to-github-repo/39503/9

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[JMoserCricut]

@glureau We're also facing this same issue and i'm wondering if it might be related to the number of active versions that a package has. How many active versions do you have? on our failing repos it seems to be at 600+ active versions, which we honestly don't really need, but we shouldn't have to maintain a way of deleting old unneeded versions I wouldn't think.

[glureau]

Damn, we are at 646 packages (entire company). I'm building ~150 of them on this specific build.

I created a specific desktop app to be able to clean packages and versions, as the current UI doesn't offer bulk edition, it's private for now (and ugly). But I agree we shouldn't have to maintain a way of deleting old unneeded versions...

[JMoserCricut]

Yea... I did find this github action, but it's not that straight forward either (we'd need to iterate over all possible package names), a retention policy would be preferred
https://github.com/marketplace/actions/delete-package-versions

[michelau]

We have also been getting bit by this intermittently for quite some time, and we also have a very large number of packages and versions.

I wrote a cron action that implements a retention policy for us to do regular cleanup. I'll see if I can post it for sharing.

I'm wondering if you are doing parallel builds in Gradle. I've considered turning them off for just the publishin step to see if it goes away.

[michelau]

https://gist.github.com/michelau/94cc01fb6545d4303cdd15a939eced74

[glureau]

No parallel builds here.
Also the problem is not always there, 3 weeks ago when I posted it was a nightmare for 1~2 days. Since then we got only <10 build failed for this issue. Maybe there were github load issues (not visible in github status), no idea if github package developers are reading those conversations, hopefully they can investigate it. 🤞🏻

[gobetti]

Just had this issue too, and I appreciate the OP's hypothesis and further discussion here as it's been helpful to unblock me. Deleting the successfully uploaded packages and re-running the job succeeded (but I guess that was by chance and the more packages you have, the more likely you are to run into the issue again).

I genuinely don't know how feasible it is to fix or improve the chances that a package upload doesn't time out, but alternatively, it would probably be good if gradle publish could gracefully handle a "good" conflict, i.e. when retrying due to an unknown status on the first attempt, somehow tell that an eventual conflict just means that the first attempt succeeded. We do something similar in a chat application where, oversimplifying it here, we generate a unique id for each request, and if the response returns a conflict with that very id, we know it came from this session and we can proceed with that.

[menjoo]

Same here:

> Task :hal:testsuite:publishAndroidDebugPublicationToGitHubPackagesRepository FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':hal:testsuite:publishAndroidDebugPublicationToGitHubPackagesRepository'.
> Failed to publish publication 'androidDebug' to repository 'GitHubPackages'
   > Could not PUT 'https://maven.pkg.github.com/<our-repo>/hal/testsuite-android-debug/4.1.0/testsuite-android-debug-4.1.0.module.sha1'. Received status code 409 from server: Conflict

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org./

BUILD FAILED in 22m 24s
1998 actionable tasks: 1308 executed, 161 from cache, 529 up-to-date
Error: Process completed with exit code 1.

Can happen at any random moment when publishing artifacts to GitHub Package Registry. We get this 1 out of 10 times, and always a different artifacts that we get the 409 for.

[syslogic]

Not a bug. HTTP verb PUT will result in HTTP 409 Conflict, when there indeed is a conflict.
This conflict persists, unless deleting the package one tries to PUT there twice ...

Instead of PUT 409 Conflict the proper sequence would ge:

• GET 200 OK look up the numeric package version ID - or 404 Not Found.
• DELETE 400 Bad Request cannot delete the only/last version - or 204 No Content version deleted.
• DELETE 204 No Content delete the whole package, when config deleteLastVersion = true.
• GET 200 OK list all packages (optional).
• PUT 200 OK publish package.

Meanwhile I've wrote a gpr-maintenance-gradle-plugin, which manages packages and versions.

[suhtai]

I think may have figured out why this is happening. Given these problems, we decided to create a task to upload all the artifacts from a local maven repository. We have about 15k files in total (over 500 modules with md5 hash files and the rest). When we tried to launch 15k connections, we kept getting random errors (403, 400, 500) not in a consistent manner. However, the artifacts were deployed despite the error returned from GHP. We reduced the number of parallel connections to 500 and the problem persisted. Only when we reduced it to 100 the problem stopped happening.

Our hypothesis is that under relatively medium load, the Github Packages starts misbehaving and running into all sorts of problems. For the Gradle plugin, we think that it retries to upload the artifact but it's already published, and that's when we get the 409.

Need to run more tests to validate this though

[syslogic]

These artififacts are immutable and unless moving them out of the way, this will ultimately result in 409.
And this behavior is quite reproducible with one single artififact already; the rest may count as API abuse.

[suhtai]

Trying to upload 15k artifacts was a development oopsies, but it make it more apparent that the service is not working as intended. The expectations from an HTTP PUT failure is that the resource is not created. Respecting this contract should fix the underlying problem Gradle publish finds, when it re-tries the upload when it fails.

Gradle will try at least 3 times to upload an artifact if there's any failure:
https://github.com/gradle/gradle/blob/91152477041e661e543be55ab3954bc397c575a1/platforms/software/maven/src/main/java/org/gradle/api/publish/maven/internal/publisher/AbstractMavenPublisher.java#L328