I used to be able to upload artifacts to archiva.wikimedia.org from my user (joal) either manually or through maven.
Now I can't do it anymore, neither through the UI (upload section is invisible) not through Maven (401 is answered when trying).
Interestingly, the jenkins user can still release jars.
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| archiva: fix 400s when proxying requests with spaces in the URL | operations/puppet | production | +2 -2 |
Related Objects
Related Objects
Event Timeline
Comment Actions
I've ssh -ed onto archiva1002.wikimedia.org and had a look at the archiva logs.
I'm seeing many occurences of these logs in /var/log/archiva/wrapper.log:
INFO | jvm 1 | 2024/01/22 03:06:38 | [INFO] Failed to parse Maven artifact /var/lib/archiva/repositories/releases/com/googlesource/gerrit/plugins/wikimedia/2.15.2/wikimedia-2.15.2.jar due to error in opening zip file INFO | jvm 1 | 2024/01/22 03:06:38 | [INFO] Failed to parse Maven artifact /var/lib/archiva/repositories/releases/com/googlesource/gerrit/plugins/quota/2.15.2/quota-2.15.2.jar due to error in opening zip file
and indeed, these files seem tiny in size:
brouberol@archiva1002:~$ ls -alh /var/lib/archiva/repositories/releases/com/googlesource/gerrit/plugins/wikimedia/2.16.16/wikimedia-2.16.16.jar -rw-r--r-- 1 archiva archiva 217K Feb 21 2020 /var/lib/archiva/repositories/releases/com/googlesource/gerrit/plugins/wikimedia/2.16.16/wikimedia-2.16.16.jar brouberol@archiva1002:~$ ls -alh /var/lib/archiva/repositories/releases/com/googlesource/gerrit/plugins/quota/2.15.2/quota-2.15.2.jar -rw-r--r-- 1 archiva archiva 74 Jun 7 2018 /var/lib/archiva/repositories/releases/com/googlesource/gerrit/plugins/quota/2.15.2/quota-2.15.2.jar
Could they be corrupted?
I checked, and these are regular files, not symlinks
brouberol@archiva1002:~$ stat /var/lib/archiva/repositories/releases/com/googlesource/gerrit/plugins/wikimedia/2.15.2/wikimedia-2.15.2.jar File: /var/lib/archiva/repositories/releases/com/googlesource/gerrit/plugins/wikimedia/2.15.2/wikimedia-2.15.2.jar Size: 74 Blocks: 8 IO Block: 4096 regular file Device: fe11h/65041d Inode: 5378818 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 112/ archiva) Gid: ( 119/ archiva) Access: 2022-07-24 20:59:48.860010437 +0000 Modify: 2018-06-07 22:46:53.933146869 +0000 Change: 2022-07-24 20:59:48.860010437 +0000 Birth: - brouberol@archiva1002:~$ stat /var/lib/archiva/repositories/releases/com/googlesource/gerrit/plugins/quota/2.15.2/quota-2.15.2.jar File: /var/lib/archiva/repositories/releases/com/googlesource/gerrit/plugins/quota/2.15.2/quota-2.15.2.jar Size: 74 Blocks: 8 IO Block: 4096 regular file Device: fe11h/65041d Inode: 5378497 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 112/ archiva) Gid: ( 119/ archiva) Access: 2022-07-24 20:59:48.644009325 +0000 Modify: 2018-06-07 22:46:01.744872087 +0000 Change: 2022-07-24 20:59:48.644009325 +0000 Birth: -
They do point to git-fat though:
brouberol@archiva1002:~$ cat /var/lib/archiva/repositories/releases/com/googlesource/gerrit/plugins/wikimedia/2.15.2/wikimedia-2.15.2.jar #$# git-fat c97c489fe61891fca04ad715b4a55547a4d74cd3 219919
and git-fat does seem to be struggling:
brouberol@archiva1002:~$ sudo systemctl status archiva-gitfat-link.service ● archiva-gitfat-link.service - Archiva tool to create jar symlinks using their sha1 checksum as filename. Loaded: loaded (/lib/systemd/system/archiva-gitfat-link.service; static; vendor preset: enabled) Active: activating (start) since Mon 2024-01-22 07:35:01 UTC; 3min 27s ago Docs: https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state Main PID: 12817 (archiva-gitfat-) Tasks: 4 (limit: 4700) Memory: 353.7M CGroup: /system.slice/archiva-gitfat-link.service ├─12817 /bin/sh /usr/local/bin/archiva-gitfat-link ../repositories . ├─16620 /bin/sh /usr/local/bin/archiva-gitfat-link ../repositories . ├─16621 /bin/sh /usr/local/bin/archiva-gitfat-link ../repositories . └─16622 awk {print $1} Jan 22 07:35:01 archiva1002 systemd[1]: Starting Archiva tool to create jar symlinks using their sha1 checksum as filename.... brouberol@archiva1002:~$ sudo systemctl status archiva-gitfat-link.service ● archiva-gitfat-link.service - Archiva tool to create jar symlinks using their sha1 checksum as filename. Loaded: loaded (/lib/systemd/system/archiva-gitfat-link.service; static; vendor preset: enabled) Active: inactive (dead) since Mon 2024-01-22 07:38:43 UTC; 2s ago Docs: https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state Process: 12817 ExecStart=/bin/bash -c cd /var/lib/archiva/git-fat && /usr/local/bin/archiva-gitfat-link ../repositories . (code=exited, status=0/SUCCESS) Main PID: 12817 (code=exited, status=0/SUCCESS) Jan 22 07:35:01 archiva1002 systemd[1]: Starting Archiva tool to create jar symlinks using their sha1 checksum as filename.... Jan 22 07:38:43 archiva1002 systemd[1]: archiva-gitfat-link.service: Succeeded. Jan 22 07:38:43 archiva1002 systemd[1]: Started Archiva tool to create jar symlinks using their sha1 checksum as filename..
Comment Actions
I'm also seeing the following traceback in /var/log/archiva/wrapper.log
INFO | jvm 1 | 2024/01/22 02:30:02 | 2024-01-22 02:30:02.519:WARN:oejs.ServletHandler:/repository/mirrored/io/undertow/undertow-core/maven-metadata.xml INFO | jvm 1 | 2024/01/22 02:30:02 | java.lang.IllegalArgumentException: Comparison method violates its general contract! INFO | jvm 1 | 2024/01/22 02:30:02 | at java.util.TimSort.mergeHi(TimSort.java:899) INFO | jvm 1 | 2024/01/22 02:30:02 | at java.util.TimSort.mergeAt(TimSort.java:516) INFO | jvm 1 | 2024/01/22 02:30:02 | at java.util.TimSort.mergeForceCollapse(TimSort.java:457) INFO | jvm 1 | 2024/01/22 02:30:02 | at java.util.TimSort.sort(TimSort.java:254) INFO | jvm 1 | 2024/01/22 02:30:02 | at java.util.Arrays.sort(Arrays.java:1512) INFO | jvm 1 | 2024/01/22 02:30:02 | at java.util.ArrayList.sort(ArrayList.java:1464) INFO | jvm 1 | 2024/01/22 02:30:02 | at java.util.Collections.sort(Collections.java:177) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.apache.archiva.repository.metadata.MetadataTools.updateMetadataVersions(MetadataTools.java:649) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.apache.archiva.repository.metadata.MetadataTools.updateMetadata(MetadataTools.java:471) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.apache.archiva.proxy.DefaultRepositoryProxyConnectors.fetchMetadataFromProxies(DefaultRepositoryProxyConnectors.java:515) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.apache.archiva.webdav.ArchivaDavResourceFactory.fetchContentFromProxies(ArchivaDavResourceFactory.java:792) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.apache.archiva.webdav.ArchivaDavResourceFactory.processRepository(ArchivaDavResourceFactory.java:627) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.apache.archiva.webdav.ArchivaDavResourceFactory.processRepositoryGroup(ArchivaDavResourceFactory.java:490) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.apache.archiva.webdav.ArchivaDavResourceFactory.createResource(ArchivaDavResourceFactory.java:262) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.apache.archiva.webdav.RepositoryServlet.service(RepositoryServlet.java:126) INFO | jvm 1 | 2024/01/22 02:30:02 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.server.Server.handle(Server.java:370) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) INFO | jvm 1 | 2024/01/22 02:30:02 | at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) INFO | jvm 1 | 2024/01/22 02:30:02 | at java.lang.Thread.run(Thread.java:750)
Comment Actions
What I'm seeing points to an issue with git-fat, and I'll continue investigating after baby-duty.
Comment Actions
Ah wait, no, git-fat is a oneshot service, so it's *supposed* to start regularly
brouberol@archiva1002:~$ sudo systemctl cat archiva-gitfat-link.service # /lib/systemd/system/archiva-gitfat-link.service [Unit] Description=Archiva tool to create jar symlinks using their sha1 checksum as filename. Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [Service] Type=oneshot User=archiva SyslogIdentifier=archiva-gitfat-link ExecStart=/bin/bash -c 'cd /var/lib/archiva/git-fat && /usr/local/bin/archiva-gitfat-link ../repositories .'
Comment Actions
Change 992089 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] archiva: fix 400s when proxying requests with spaces in the URL
Comment Actions
All my previous comments related to git-fat were invalid. When digging in the Archiva UI, we realized that XHR calls to https://archiva.wikimedia.org/restServices/redbackServices/roleManagementService/getRole/<any-role-name-with-spaces-in-them> failed. if the group didn't have any space in it, the call succeeded.
We believed that fixing this bug might fix the permission issue, as archiva is trying to assess what permission are associated with the LDAP user, and can't get the permission associated to the Archiva group, mapped to the LDAP user cn.
Comment Actions
We checked with @BTullis that this patch didn't reopen the XSS vulnerability documented in T318962, which it did not.
Comment Actions
Change 992089 merged by Brouberol:
[operations/puppet@production] archiva: fix 400s when proxying requests with spaces in the URL
Comment Actions
It turns out the issue was in LDAP/Archiva. The archiva-deployers had an extra cn attribute with value cn=releng. Somehow, both groups were shown in the Archiva list, and only the permission applied to the releng group were taken into account.
We cleaned up the extra LDAP cn, and permission issues have now gone away.
Comment Actions
In case it helps, there is a long Slack thread about this incident here: https://wikimedia.slack.com/archives/C055QGPTC69/p1705609783973829