On a CI build or on my local machine (on an idle connection), maven takes long to download material from Archiva compared to Maven Central. Seems like something is throttling it?
For a project having just few dependencies it takes 7 seconds from Maven Central compared to 25 seconds with Archiva.
That can be reproduced by picking a random Java project and configure the pom to point to Archiva instead of Maven Central ( https://wikitech.wikimedia.org/wiki/Archiva#Using_pom.xml )
Change 659025 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Raise timeout for gearman-java jobs
Change 659025 merged by jenkins-bot:
[integration/config@master] Raise timeout for gearman-java jobs
I have never deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/608812 but it may help in this case. Archiva runs on a ganeti VM, and currently all the artifacts are streamed through the java daemon even if they are on disk. Allowing nginx to control those files directly would surely improve performances in my opinion.. I'll find some time to go through this again, reviews welcome :)
To keep archives happy: we deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/659236 to disable proxy buffering in nginx, pending verification of perf improvements. Depending on the results we'll think about applying https://gerrit.wikimedia.org/r/c/operations/puppet/+/608812 as well.
For a project having just few dependencies it takes 7 seconds from Maven Central compared to 25 seconds with Archiva.
After the Nginx buffering has been disabled, the same test case went from 25 to 17 seconds. That is definitely an improvement.
I use integration/gearman-java.git and change Ib687675b71dd0e875df88a1b1d27d9e5027f0a3a then issue mvn dependency:purge-local-repository. It force download the few dependencies of that project.
Alternatively:
curl -w "
| time_namelookup | %{time_namelookup}
| time_connect | %{time_connect}
| time_appconnect | %{time_appconnect}
| time_pretransfer | %{time_pretransfer}
| time_starttransfer | %{time_starttransfer}
| time_total | %{time_total}
|--|--
" -o /dev/null -s https://archiva.wikimedia.org/repository/mirrored/junit/junit/4.13.1/junit-4.13.1.jarFrom Maven Central it can be fetched via https://repo1.maven.org/maven2/junit/junit/4.13.1/junit-4.13.1.jar
| Metric | Archiva | Maven Central |
|---|---|---|
| time_namelookup | 0,018565 | 0,106797 |
| time_connect | 0,148440 | 0,169125 |
| time_appconnect | 0,322394 | 0,292053 |
| time_pretransfer | 0,322491 | 0,292272 |
| time_starttransfer | 1,433891 | 0,357669 |
| time_total | 3,622132 | 0,902485 |
So 300 ms until fully connected (tcp connect + TLS handshake) and 1.5 seconds before transfers start. Archiva is not that fast :]
Fetching https://archiva.wikimedia.org/repository/mirrored/junit/junit/4.13.1/junit-4.13.1.jar it still takes a while until the transfer starts:
| time_namelookup | 0,106173 |
| time_connect | 0,303668 |
| time_appconnect | 0,494282 |
| time_pretransfer | 0,494402 |
| time_starttransfer | 1,516547 |
| time_total | 2,646278 |
|---|---|
When using wget it is twice faster over ipv4 compared to ipv6 for some reason, but that might be my ISP.
From integration/gearman-java.git using mvn dependency:purge-local-repository that is ~ 22 seconds. So I can't say it is any faster hitting directly from disk?!?
I think that we should make tests inside the wikimedia network, testing from home is not reliable (as you said there are too many variables, one above all the ISP connection). Also the fleet power available for Maven Central is surely way bigger than Archiva, I wouldn't expect our endpoint to be faster (single tiny VM on Ganeti). The important part in my opinion is that it is fast inside our network, where I'd expect it to be faster than Maven central.
I tested a build of analytics refinery source from stat1004 and I noticed a 4 minute improvement compared to the last time that I did it, so something has improved in my opinion. We'll see :)
From my connection something else is broken download a 2.13M javamelody-3.2.2-97-gcaf5020db1.jar, it takes almost a minute (at 40KB/s !). But since that works super fast from our infra I guess that is an entirely different problem.
On our infra (contint2001, using a Docker container) the mvn dependency:purge-local-repository takes 12 seconds \o/ I guess cause of the low latency and the "instant" downloads. And the curl command gives:
| time_namelookup | 0.003073 |
| time_connect | 0.033640 |
| time_appconnect | 0.068507 |
| time_pretransfer | 0.068828 |
| time_starttransfer | 0.968723 |
| time_total | 1.061788 |
|---|---|
So looks fast enough (beside the 900ms delay before the transfer starts). That delay is what has lead me to suspect the Nginx buffering and turning it off did improve thing. Now that nginx change to bypass Jetty is applied I guess it is another issue with Nginx or the Ganeti VM is strangely super slow?
For the context, I filed that task cause a CI job started running three time slower when switched from Maven Central to Archiva so I felt like maybe we can improve it. I have raised the timeout and overall this slowness is not a blocker for me :]
To keep archives happy - I had to revert the patch since some maven build jobs issue HTTP PUT to the /repository path, meanwhile my assumption was that it was only GET traffic.
We had Nginx buffering disabled but there are still unreasonable delay to start a transfer. There are Java CI builds failing randomly as a result when they reach the build timeout and on my local machine I often have several seconds delay when fetching artifacts (even with plain curl). Can we enable some logging in Archiva / Nginx and try to find which of the two is the cause?
That is still hitting us from time to time, specially when the CI cache is not warmed up or has been wiped.
Change 708301 had a related patch set uploaded (by Hashar; author: Hashar):
[integration/config@master] jjb: raise timeout wikidata query rdf release job
Change 708301 merged by jenkins-bot:
[integration/config@master] jjb: raise timeout wikidata query rdf release job
The performance are currently severely degraded, seems each request made to archiva has a 3-4 seconds delay before starting the transfer. From the previous test case:
curl -w "
> | time_namelookup | %{time_namelookup}
> | time_connect | %{time_connect}
> | time_appconnect | %{time_appconnect}
> | time_pretransfer | %{time_pretransfer}
> | time_starttransfer | %{time_starttransfer}
> | time_total | %{time_total}
> |--|--
> " -o /dev/null -s https://archiva.wikimedia.org/repository/mirrored/junit/junit/4.13.1/junit-4.13.1.jar| time_namelookup | 0.004282 |
| time_connect | 0.005089 |
| time_appconnect | 0.019017 |
| time_pretransfer | 0.019082 |
| time_starttransfer | > 3.449087 < |
| time_total | 3.462073 |
|---|---|
Note that uploading is fast. Here for a file named service-0.3.78-dist.tar.gz
01:42:07.283 [INFO] [INFO] Uploaded to archiva.releases: https://archiva.wikimedia.org/repository/releases/org/wikidata/query/rdf/service/0.3.78/service-0.3.78.pom (3.3 kB at 61 kB/s) 01:42:07.338 [INFO] [INFO] Uploading to archiva.releases: https://archiva.wikimedia.org/repository/releases/org/wikidata/query/rdf/service/0.3.78/service-0.3.78-dist.tar.gz 01:42:07.339 [INFO] [INFO] Uploaded to archiva.releases: https://archiva.wikimedia.org/repository/releases/org/wikidata/query/rdf/service/0.3.78/service-0.3.78-dist.tar.gz (165 MB at 43 MB/s) 01:42:11.137 [INFO] [INFO] Downloading from archiva.releases: https://archiva.wikimedia.org/repository/releases/org/wikidata/query/rdf/service/maven-metadata.xml
Change 764380 had a related patch set uploaded (by Hashar; author: Hashar):
[integration/config@master] Raise timeout for gearman-java jobs
Change 764380 merged by jenkins-bot:
[integration/config@master] Raise timeout for gearman-java jobs
@hashar Is this latency with Archiva still apparent? I guess it probably is, since you had to increase the timeouts again in February of this year.
@BTullis yes archiva is still rather slow. From the verbose curl commands above T273086#6783722, there is a large delay (1+ seconds) before the transfer start and that pills up quickly when Maven download hundred if not thousands of dependencies. On CI it is more or less worked around by saving and restoring the local Maven repository.
From some quick debugging with @elukey nginx buffering to disabled via https://gerrit.wikimedia.org/r/c/operations/puppet/+/659236 . Some change got tried to serve files directly from the disk ( https://gerrit.wikimedia.org/r/c/operations/puppet/+/608812 ) but it ended up causing some problem (no clue what) and that got reverted.
There is most probably an issue in the nginx configuration which might still be buffering the files entirely before even starting the transfer, and Archiva itself surely has some source of slowness. Then I don't have access to the system to debug it nor do I know anything about Nginx or Archiva :-)
Change 976802 had a related patch set uploaded (by Hashar; author: Hashar):
[integration/config@master] Further raise timeout for gearman-java jobs
Change 976802 merged by jenkins-bot:
[integration/config@master] Further raise timeout for gearman-java jobs
(Just passing by to +1 this issue since it hit me quite badly recently while trying to build a maven based project on a stat machine)
Change 987434 had a related patch set uploaded (by Hashar; author: Hashar):
[operations/software/gerrit@deploy/wmf/stable-3.5] Remove archiva.wikimedia.org
Change 987434 merged by jenkins-bot:
[operations/software/gerrit@deploy/wmf/stable-3.5] Remove archiva.wikimedia.org
We will migrate away from Archiva, so working on this ticket does not make sense anymore.