Page MenuHomePhabricator
Create Task
Maniphest T426114

Archiva (archiva.wikimedia.org) returning HTTP 504 Gateway Timeout or no responses, breaking production Maven builds
Closed, ResolvedPublic
Actions

Description

Archiva (archiva.wikimedia.org) is intermittently returning HTTP 504 Gateway Timeout responses, breaking production Maven builds across WMF JVM projects.

Why 504 is worse than a slow Archiva

Maven treats any 5xx response from a configured repository as a hard "transfer failure" and does *not* fall through to other repositories (e.g. Maven Central). This means that when Archiva 504s on an artifact it does not even host — such as org.apache.spark:spark-sql, io.circe:circe-core, or org.scalatest:scalatest — Maven marks those artifacts as unresolvable and the entire build fails, even though they are available on Maven Central.

A 404 (artifact not here) would be harmless; Maven would move on to the next repository. A 504 is not.

Additionally, Maven caches the failure in a .lastUpdated marker file in the local repository. This means developer builds stay broken even after Archiva recovers — developers must manually purge ~/.m2/**/*.lastUpdated to unblock themselves.

Impact

  • Production CI builds fail whenever Archiva 504s, including for modules with no dependency on any Archiva-hosted artifact — the 504 poisons the resolution of unrelated Central artifacts.
  • Developer local builds stay broken after an incident until .lastUpdated files are manually purged.
  • The wmf-jvm-parent-pom injects repository references inherited by all WMF JVM projects, so the blast radius of an Archiva outage extends beyond projects that directly configure Archiva repos.

Details

Other Assignee
RKemper
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
archiva: block scraper UAs at nginxRyan Kemperoperations/puppetproduction+38 -8
Remove wmf-analytics-old-uploads Archiva repositoryXcollazoanalytics/refinery/sourcemaster+0 -16
Upgrade graphframes to 0.11.0 from Maven Central, drop Archiva reposXcollazoanalytics/refinery/sourcemaster+55 -64
Upgrade graphframes to 0.11.0 from Maven Central, drop Archiva reposXcollazoanalytics/refinery/sourcemaster+55 -64
Customize query in gerrit

Related Objects

Event Timeline

xcollazo created this task.May 12 2026, 6:42 PM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptMay 12 2026, 6:42 PM
xcollazo triaged this task as Unbreak Now! priority.May 12 2026, 6:42 PM
xcollazo added a subscriber: amastilovic.May 12 2026, 6:45 PM

airflow-dags is also having issues (slack thread), although those can definitely be ameliorated by changing unit test behavior as per @amastilovic

bking subscribed.May 12 2026, 6:49 PM
xcollazo added a comment.May 12 2026, 6:53 PM

Archiva now stopped responding at all:

curl -s --max-time 10 "https://archiva.wikimedia.org/repository/mirror-spark/graphframes/graphframes/0.8.1-spark3.0-s_2.12/graphframes-0.8.1-spark3.0-s_2.12.pom" -o /dev/null -w "%{http_code}\n"
000

echo $?
28
xcollazo renamed this task from Archiva (archiva.wikimedia.org) is intermittently returning HTTP 504 Gateway Timeout responses, breaking production Maven builds to Archiva (archiva.wikimedia.org) returning HTTP 504 Gateway Timeout or no responses, breaking production Maven builds.May 12 2026, 6:54 PM
BTullis subscribed.May 12 2026, 8:13 PM

We have observed that a bot identifying itself with this UA is attempting to get lots of URLs.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/147.0.7727.137 Safari/537.36

This did have a legitimate reverse DNS to crawl-66-249-72-2.googlebot.com.
We have temporarily added a block for this in /etc/nginx/sites-enabled/archiva and restarted both the nginx and archiva services.

Let's see if it behaves any better.

gerritbot added a comment.May 12 2026, 8:42 PM

Change #1286481 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[analytics/refinery/source@master] Upgrade graphframes to 0.11.0 from Maven Central, drop Archiva repos

https://gerrit.wikimedia.org/r/1286481

gerritbot added a project: Patch-For-Review.May 12 2026, 8:42 PM
xcollazo lowered the priority of this task from Unbreak Now! to High.May 12 2026, 8:43 PM

Builds are now able to move forward.

We are going to make refinery source less dependent on Archiva as well via https://gerrit.wikimedia.org/r/1286481.

xcollazo added a comment.May 12 2026, 9:25 PM

Unfortunately now gitlab has an issue:

From https://integration.wikimedia.org/ci/job/analytics-refinery-maven-java8-site-publish/133/console:

17:17:07 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.12.1:site (default-site) on project refinery: SiteToolException: The site descriptor cannot be resolved from the repository: ArtifactResolutionException: Unable to locate site descriptor: The following artifacts could not be resolved: org.wikimedia:wmf-jvm-parent-pom:xml:site_en:1.96 (absent): Could not transfer artifact org.wikimedia:wmf-jvm-parent-pom:xml:site_en:1.96 from/to gitlab-maven (https://gitlab.wikimedia.org/api/v4/groups/186/-/packages/maven): status code: 401, reason phrase: Unauthorized (401)
17:17:07 [ERROR]   org.wikimedia:wmf-jvm-parent-pom:xml:1.96
gerritbot added a comment.May 13 2026, 12:54 AM

Change #1286530 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[analytics/refinery/source@master] Upgrade graphframes to 0.11.0 from Maven Central, drop Archiva repos

https://gerrit.wikimedia.org/r/1286530

gerritbot added a comment.May 13 2026, 2:50 AM

Change #1286536 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] archiva: block scraper UAs at nginx

https://gerrit.wikimedia.org/r/1286536

gerritbot added a comment.May 13 2026, 12:37 PM

Change #1286481 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[analytics/refinery/source@master] Upgrade graphframes to 0.11.0 from Maven Central, drop Archiva repos

https://gerrit.wikimedia.org/r/1286481

gerritbot added a comment.May 13 2026, 1:07 PM

Change #1286481 merged by jenkins-bot:

[analytics/refinery/source@master] Upgrade graphframes to 0.11.0 from Maven Central, drop Archiva repos

https://gerrit.wikimedia.org/r/1286481

xcollazo added a subscriber: pfischer.May 13 2026, 1:25 PM
In T426114#11914940, @xcollazo wrote:

Unfortunately now gitlab has an issue:

From https://integration.wikimedia.org/ci/job/analytics-refinery-maven-java8-site-publish/133/console:

17:17:07 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.12.1:site (default-site) on project refinery: SiteToolException: The site descriptor cannot be resolved from the repository: ArtifactResolutionException: Unable to locate site descriptor: The following artifacts could not be resolved: org.wikimedia:wmf-jvm-parent-pom:xml:site_en:1.96 (absent): Could not transfer artifact org.wikimedia:wmf-jvm-parent-pom:xml:site_en:1.96 from/to gitlab-maven (https://gitlab.wikimedia.org/api/v4/groups/186/-/packages/maven): status code: 401, reason phrase: Unauthorized (401)
17:17:07 [ERROR]   org.wikimedia:wmf-jvm-parent-pom:xml:1.96

This continues today. It only happens on post-merge builds.

New example at https://integration.wikimedia.org/ci/job/analytics-refinery-maven-java8-site-publish/134/console

@pfischer do you know what needs updated? Could you please point me in the right direction?

CodeReviewBot added a comment.May 13 2026, 1:32 PM

xcollazo opened https://gitlab.wikimedia.org/repos/wmf-packages/-/merge_requests/2

Import org.pentaho:pentaho-aggdesigner-algorithm from Archiva

xcollazo added a subscriber: jnuche.May 13 2026, 1:56 PM
In T426114#11917372, @xcollazo wrote:
In T426114#11914940, @xcollazo wrote:

Unfortunately now gitlab has an issue:

From https://integration.wikimedia.org/ci/job/analytics-refinery-maven-java8-site-publish/133/console:

17:17:07 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.12.1:site (default-site) on project refinery: SiteToolException: The site descriptor cannot be resolved from the repository: ArtifactResolutionException: Unable to locate site descriptor: The following artifacts could not be resolved: org.wikimedia:wmf-jvm-parent-pom:xml:site_en:1.96 (absent): Could not transfer artifact org.wikimedia:wmf-jvm-parent-pom:xml:site_en:1.96 from/to gitlab-maven (https://gitlab.wikimedia.org/api/v4/groups/186/-/packages/maven): status code: 401, reason phrase: Unauthorized (401)
17:17:07 [ERROR]   org.wikimedia:wmf-jvm-parent-pom:xml:1.96

This continues today. It only happens on post-merge builds.

New example at https://integration.wikimedia.org/ci/job/analytics-refinery-maven-java8-site-publish/134/console

@pfischer do you know what needs updated? Could you please point me in the right direction?

Synced up with @pfischer. Stack points at a potentially expired token at https://gitlab.wikimedia.org/repos/wmf-packages/-/settings/access_tokens. However, the single token there appears to have 5 months before it expires.

@jnuche could you help us here?

gerritbot added a comment.May 13 2026, 2:03 PM

Change #1286530 abandoned by Xcollazo:

[analytics/refinery/source@master] Upgrade graphframes to 0.11.0 from Maven Central, drop Archiva repos

Reason:

Superseded by 1286481

https://gerrit.wikimedia.org/r/1286530

jnuche added a comment.May 13 2026, 2:20 PM
In T426114#11917593, @xcollazo wrote:
In T426114#11917372, @xcollazo wrote:
In T426114#11914940, @xcollazo wrote:

Unfortunately now gitlab has an issue:

From https://integration.wikimedia.org/ci/job/analytics-refinery-maven-java8-site-publish/133/console:

17:17:07 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.12.1:site (default-site) on project refinery: SiteToolException: The site descriptor cannot be resolved from the repository: ArtifactResolutionException: Unable to locate site descriptor: The following artifacts could not be resolved: org.wikimedia:wmf-jvm-parent-pom:xml:site_en:1.96 (absent): Could not transfer artifact org.wikimedia:wmf-jvm-parent-pom:xml:site_en:1.96 from/to gitlab-maven (https://gitlab.wikimedia.org/api/v4/groups/186/-/packages/maven): status code: 401, reason phrase: Unauthorized (401)
17:17:07 [ERROR]   org.wikimedia:wmf-jvm-parent-pom:xml:1.96

This continues today. It only happens on post-merge builds.

New example at https://integration.wikimedia.org/ci/job/analytics-refinery-maven-java8-site-publish/134/console

@pfischer do you know what needs updated? Could you please point me in the right direction?

Synced up with @pfischer. Stack points at a potentially expired token at https://gitlab.wikimedia.org/repos/wmf-packages/-/settings/access_tokens. However, the single token there appears to have 5 months before it expires.

@jnuche could you help us here?

@xcollazo I'm not sure how I can help here, as you pointed out that token should still be valid. However in its description it says "Token used by jenkins to upload packages" and your error message mentions:

The following artifacts could not be resolved: org.wikimedia:wmf-jvm-parent-pom:xml:site_en:1.96 (absent): Could not transfer artifact org.wikimedia:wmf-jvm-parent-pom:xml:site_en:1.96 from/to gitlab-maven (https://gitlab.wikimedia.org/api/v4/groups/186/-/packages/maven)

which makes me think your job is failing when trying to download the parent pom. So maybe that token is not involved in the issue.

I see that the problem in the job started at the same time that this config change happened: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-java8-site-publish/jobConfigHistory/showDiffFiles?timestamp1=2026-02-20_09-40-04&timestamp2=2026-04-20_18-10-45

I have no idea how these jobs are wired, but that config change started using a new image (castor:0.4.1) Could that be related to the problem?

CodeReviewBot added a comment.May 13 2026, 2:22 PM

pfischer merged https://gitlab.wikimedia.org/repos/wmf-packages/-/merge_requests/2

Import org.pentaho:pentaho-aggdesigner-algorithm from Archiva

gerritbot added a comment.May 13 2026, 5:11 PM

Change #1286989 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[analytics/refinery/source@master] Remove wmf-analytics-old-uploads Archiva repository

https://gerrit.wikimedia.org/r/1286989

xcollazo added a comment.May 13 2026, 5:13 PM

@jnuche:

I see that the problem in the job started at the same time that this config change happened: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-java8-site-publish/jobConfigHistory/showDiffFiles?timestamp1=2026-02-20_09-40-04&timestamp2=2026-04-20_18-10-45

I get access denied "xcollazo is missing the Job/Configure permission". Could I get that permission?

gerritbot added a comment.May 13 2026, 6:09 PM

Change #1286989 merged by jenkins-bot:

[analytics/refinery/source@master] Remove wmf-analytics-old-uploads Archiva repository

https://gerrit.wikimedia.org/r/1286989

RKemper subscribed.May 13 2026, 8:10 PM
In T426114#11914719, @BTullis wrote:

We have observed that a bot identifying itself with this UA is attempting to get lots of URLs.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/147.0.7727.137 Safari/537.36

This did have a legitimate reverse DNS to crawl-66-249-72-2.googlebot.com.
We have temporarily added a block for this in /etc/nginx/sites-enabled/archiva and restarted both the nginx and archiva services.

Let's see if it behaves any better.

To add some color, here's a jstack dump from before we added this rule: P92513

Out of 250 Jetty worker threads, 246 were blocked on AbstractConnPool.getPoolEntryBlocking, and the other 4 threads were 2 Acceptor and 2 Selector threads, so that means literally every worker thread tied up waiting on a connection pool lease. Here's one representative trace from the (massive) dump:

"qtp271254268-349" #349 prio=5 os_prio=0 tid=0x00007f58c011e800 nid=0x1016c waiting on condition [0x00007f4fc62b0000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000cda5c950> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:393)
	at org.apache.http.pool.AbstractConnPool.access$300(AbstractConnPool.java:70)
	at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:253)
	- locked <0x00000000cf0f4728> (a org.apache.http.pool.AbstractConnPool$2)
	at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:198)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:306)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.execute(AbstractHttpClientWagon.java:832)
	at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:983)
	at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:960)
	at org.apache.maven.wagon.StreamWagon.getInputStream(StreamWagon.java:116)
	at org.apache.maven.wagon.StreamWagon.getIfNewer(StreamWagon.java:88)
	at org.apache.maven.wagon.StreamWagon.get(StreamWagon.java:61)
	at org.apache.archiva.proxy.DefaultRepositoryProxyConnectors.transferSimpleFile(DefaultRepositoryProxyConnectors.java:930)
	at org.apache.archiva.proxy.DefaultRepositoryProxyConnectors.transferArtifact(DefaultRepositoryProxyConnectors.java:631)
	at org.apache.archiva.proxy.DefaultRepositoryProxyConnectors.transferResources(DefaultRepositoryProxyConnectors.java:579)
	at org.apache.archiva.proxy.DefaultRepositoryProxyConnectors.transferFile(DefaultRepositoryProxyConnectors.java:775)
	at org.apache.archiva.proxy.DefaultRepositoryProxyConnectors.fetchMetadataFromProxies(DefaultRepositoryProxyConnectors.java:475)
	at org.apache.archiva.webdav.ArchivaDavResourceFactory.fetchContentFromProxies(ArchivaDavResourceFactory.java:792)
	at org.apache.archiva.webdav.ArchivaDavResourceFactory.processRepository(ArchivaDavResourceFactory.java:627)
	at org.apache.archiva.webdav.ArchivaDavResourceFactory.processRepositoryGroup(ArchivaDavResourceFactory.java:490)
	at org.apache.archiva.webdav.ArchivaDavResourceFactory.createResource(ArchivaDavResourceFactory.java:262)
	at org.apache.archiva.webdav.RepositoryServlet.service(RepositoryServlet.java:126)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:370)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:750)

   Locked ownable synchronizers:
	- None

A second jstack dump (not included for brevity) that I grabbed 30 seconds after the first one showed the same story, and the thread IDs were unchanged, meaning none of those 246 threads were doing any actual work.

xcollazo mentioned this in T425986: Add log_id to wmf.mediawiki_history.May 14 2026, 7:38 PM
xcollazo added a comment.May 15 2026, 2:21 AM

Archiva has not been an issue for a bit now, so seems like the fix worked.

Additionally, analytics-refinery-source no longer depends on Archiva.

The gitlab issue discussed in T426114#11914940 remains, but I will split that off as it is a different issue.

Thanks @BTullis and @RKemper for the help getting that fixed.

xcollazo closed this task as Resolved.May 15 2026, 2:22 AM
xcollazo assigned this task to BTullis.
xcollazo updated Other Assignee, added: RKemper.
xcollazo mentioned this in T426369: analytics-refinery-source repo fails post-merge builds, releases succeed.May 15 2026, 2:25 AM
xcollazo added a project: Data-Engineering (Q4 FS25/26 April 1st - June 30st).May 15 2026, 2:40 PM
gerritbot added a comment.May 15 2026, 10:12 PM

Change #1286536 merged by Ryan Kemper:

[operations/puppet@production] archiva: block scraper UAs at nginx

https://gerrit.wikimedia.org/r/1286536

Gehel edited projects, added: Data-Platform-SRE (2026-04-24 - 2026-05-15); removed: Data-Platform-SRE.May 19 2026, 12:52 PM
Gehel moved this task from Backlog - project to Done on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.May 22 2026, 9:49 AM
Ahoelzl moved this task from Next Up to Done on the Data-Engineering (Q4 FS25/26 April 1st - June 30st) board.Wed, Jul 1, 11:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits