Time to move archiva1001 to Debian Buster, and it seems easier to just create a new VM and point the ATS' config to it.
Couple of notes:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | elukey | T234629 Move the Analytics infrastructure to Debian Buster | |||
| Resolved | elukey | T252767 Move Archiva to Debian Buster | |||
| Resolved | elukey | T254849 Purge old files on Archiva to free some space | |||
| Resolved | elukey | T254890 Create archiva1002 as replacement of archiva1001 |
Change 596425 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::archiva: move to profile::java::analytics
The disk usage is very interesting:
In the past 5 months we increased the /var/lib/archiva usage by 8%, that should be ~8G. Now we are at ~90% usage, broken down in:
elukey@archiva1001:/var/lib/archiva/repositories$ sudo du -hs * 68K internal 16G mirrored 665M python 33G releases 34G snapshots Filesystem Size Used Avail Use% Mounted on /dev/vdb1 98G 85G 8.9G 91% /var/lib/archiva
Before creating a new VM, I would check if we can clean up some data or if we need to keep it all.
@Ottomata @JAllemandou thoughts?
Change 596460 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Factor out java 8 installation into java_8 class
Change 596460 merged by Ottomata:
[operations/puppet@production] Factor out java 8 installation into java_8 class
Change 596425 merged by Elukey:
[operations/puppet@production] role::archiva: move to profile::java::analytics
Some info about the move from meitnerium to archiva1001 for Stretch: https://phabricator.wikimedia.org/T192639
Change 603809 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add piwik overrides for matomo1002 to ease testing
Change 603809 merged by Elukey:
[operations/puppet@production] Add piwik overrides for matomo1002 to ease testing
Change 604066 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] archiva: move archiva-gitfat-link to systemd timer
Change 604066 merged by Elukey:
[operations/puppet@production] archiva: move archiva-gitfat-link to systemd timer
Some notes from the other migration:
Change 604691 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Assign role archiva to archiva1002
Change 604698 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] archiva::proxy: raise TLS ciphersuite requirements
Change 604691 merged by Elukey:
[operations/puppet@production] Assign role archiva to archiva1002
Change 604734 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Add archiva-new.wikimedia.org as CNAME to archiva1002
Change 605136 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/homer/public@master] Add archiva1002 IPs to analytics-in4/6 filters
Change 604734 merged by Elukey:
[operations/dns@master] Add archiva-new.wikimedia.org as CNAME to archiva1002
Change 605136 merged by Elukey:
[operations/homer/public@master] Add archiva1002 IPs to analytics-in4/6 filters
Mentioned in SAL (#wikimedia-operations) [2020-06-12T08:48:37Z] <elukey> update cr1/cr2 analyitics filters for T252767 and T252675
Change 605186 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] archiva: assign archiva-new.wikimedia.org to archiva1002
Change 605186 merged by Elukey:
[operations/puppet@production] archiva: assign archiva-new.wikimedia.org to archiva1002
Change 605187 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add archiva-new configuration for Acme Chief
Change 605187 merged by Elukey:
[operations/puppet@production] Add archiva-new configuration for Acme Chief
Change 605203 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::archiva::proxy: use certificate_name for monitoring
Change 605203 merged by Elukey:
[operations/puppet@production] profile::archiva::proxy: use certificate_name for monitoring
Change 607989 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Move archiva.wikimedia.org from archiva1001 to archiva1002
Change 607989 merged by Elukey:
[operations/puppet@production] Move archiva.wikimedia.org from archiva1001 to archiva1002
Change 608308 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Update CNAMEs for archiva
archiva.wikimedia.org now points to archiva1002, and archiva-old.wikimedia.org points to archiva1001. Will keep the latter around for a couple of days in case a quick rollback is needed.
Some changes:
The changes should be transparent for analytics, but will allow other teams to use proxy repos with more narrow scope.
I think we might be missing some referenced artifacts in refinery.
./hive-jdbc-1.1.0-cdh5.10.0.jar:#$# git-fat 08067db8f8120d408a324159ba981905f041bcfc 96774 ./hive-service-1.1.0-cdh5.10.0.jar:#$# git-fat 192b8c280dcd132e6eaa1037e9c655b738925f5a 2067290 ./article-recommender/venv-0.0.1.zip:#$# git-fat 167d4c5c0eeb07deb66ca84f61d032b6186605fc 54952360 ./article-recommender/venv-0.0.2.zip:#$# git-fat 80e06f209b5a687ab694199ac6ed0d5f9925f1ad 54957315
Deploying refinery is failing because these files are missing.
Lovely :(
./repositories/mirrored/org/apache/hive/hive-jdbc/1.1.0-cdh5.10.0/hive-jdbc-1.1.0-cdh5.10.0.jar.sha1 ./repositories/mirrored/org/apache/hive/hive-jdbc/1.1.0-cdh5.10.0/hive-jdbc-1.1.0-cdh5.10.0.jar.md5 ./repositories/mirrored/org/apache/hive/hive-jdbc/1.1.0-cdh5.10.0/hive-jdbc-1.1.0-cdh5.10.0.jar
The above files are on archiva1001, so I guess that either we uploaded it manually or the cloudera repository doesn't have it anymore..
Very weird:
elukey@archiva1002:/var/lib/archiva/repositories$ ls mirror-cloudera/org/apache/hive/hive-jdbc 1.1.0-cdh5.15.0 maven-metadata.xml maven-metadata.xml.md5 maven-metadata.xml.sha
So on archvia1002 we have 5.15, not 5.10 as requested but git-fat. Moreover, the cloudera repo still shows 5.10 available:
https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hive/hive-jdbc/1.1.0-cdh5.10.0/
I downloaded the 2 article-recommender venvs from archiva-old and uploaded them to archiva:
Artifacts for 'article-recommender:venv:0.0.1', packaged as 'zip', with no POM Generated, were uploaded and saved on Server side to 'releases' repository. Artifacts for 'article-recommender:venv:0.0.2', packaged as 'zip', with no POM Generated, were uploaded and saved on Server side to 'releases' repository.
We do indeed need the cdh5.10.0 versions of those hive jars:
https://github.com/wikimedia/analytics-refinery/blob/master/artifacts/hive-cdh5.10.0.README
I downloaded them and uploaded them to the analytics-old-uploads repository:
Artifacts for 'org.apache.hive:hive-service:1.1.0-cdh5.10.0', packaged as 'jar', with no POM Generated, were uploaded and saved on Server side to 'analytics-old-uploads' repository. Artifacts for 'org.apache.hive:hive-jdbc:1.1.0-cdh5.10.0', packaged as 'jar', with no POM Generated, were uploaded and saved on Server side to 'analytics-old-uploads' repository.
I then forced a run of archiva-gitfat-link and was able to successfully deploy refinery.
Ok I think I know what happened:
Hm, I spoke too soon, I think the upload didn't quite work in the way I expected? Even though I uploaded all the files, there was a .jar and a -standalone.jar, and it looks like the -standalone.jar took precedence over the regular .jar. will try again.
We had to trash my uploads, and then I re-uploaded ONLY the .jar files we needed, not the extraneous sources or javadocs or poms.
Change 608812 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] archiva::proxy: allow nginx to serve content from repositories
Summary of my understanding so far:
Possible solutions:
I'd prefer 3) to be honest, seems cleaner and compatible with https://gerrit.wikimedia.org/r/c/operations/puppet/608812
Change 608872 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery/source@master] Explicit archiva mirrored repositories in pom.xml
https://gerrit.wikimedia.org/r/c/analytics/refinery/source/ /608872
Change 608879 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] maven: remove mirrored repository from main settings
Change 608879 merged by Elukey:
[operations/puppet@production] maven: remove main /etc/maven/settings.xml
Change 608872 merged by jenkins-bot:
[analytics/refinery/source@master] Explicit archiva mirrored repositories in pom.xml
https://gerrit.wikimedia.org/r/c/analytics/refinery/source/ /608872
We moved refinery to use specific repos (mirror-cloudera/mirror-spark/mirror-maven-central/analytics-old-uploads) and the build went fine, everything seems finally working. The repo group mirrored is still needed for the Discovery team so we'll leave it there.
Verified on backup1001 that archiva1002 is correctly using bacula:
Terminated Jobs: JobId Level Files Bytes Status Finished Name =================================================================== 240718 Full 319,197 24.28 G OK 30-Jun-20 04:54 archiva1002.wikimedia.org-Monthly-1st-Thu-production-var-lib-archiva 240865 Incr 34,354 952.2 M OK 01-Jul-20 04:12 archiva1002.wikimedia.org-Monthly-1st-Thu-production-var-lib-archiva 240979 Full 376,398 25.84 G OK 02-Jul-20 02:57 archiva1002.wikimedia.org-Monthly-1st-Thu-production-var-lib-archiva 241008 Incr 2,032 248.9 M OK 02-Jul-20 04:09 archiva1002.wikimedia.org-Monthly-1st-Thu-production-var-lib-archiva 241156 Incr 38,298 926.4 M OK 03-Jul-20 04:09 archiva1002.wikimedia.org-Monthly-1st-Thu-production-var-lib-archiva 241297 Incr 20,139 974.9 M OK 04-Jul-20 04:11 archiva1002.wikimedia.org-Monthly-1st-Thu-production-var-lib-archiva 241438 Incr 18,633 755.6 M OK 05-Jul-20 09:38 archiva1002.wikimedia.org-Monthly-1st-Thu-production-var-lib-archiva 241576 Incr 10,181 753.4 M OK 06-Jul-20 04:09 archiva1002.wikimedia.org-Monthly-1st-Thu-production-var-lib-archiva 242784 Incr 14,228 918.0 M OK 07-Jul-20 08:00 archiva1002.wikimedia.org-Monthly-1st-Thu-production-var-lib-archiva
Change 610006 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Remove archiva1001 from puppet
Change 610006 merged by Elukey:
[operations/puppet@production] Remove archiva1001 from puppet
cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: archiva1001.wikimedia.org
Tried to run the decom cookbook and got:
elukey@cumin1001:~$ sudo cookbook sre.hosts.decommission -t T252767 archiva1001.wikimedia.org START - Cookbook sre.hosts.decommission ATTENTION: destructive action for 1 hosts: archiva1001.wikimedia.org Are you sure to proceed? Type "done" to proceed > done Looking for matches in puppetmaster1001.eqiad.wmnet:/var/lib/git/operations/puppet hieradata/common/lvs/interfaces.yaml: 'lvs1013': 'enp4s0f1:208.80.154.167' hieradata/common/lvs/interfaces.yaml: 'lvs1014': 'enp4s0f0:208.80.154.168' hieradata/common/lvs/interfaces.yaml: 'lvs1015': 'enp4s0f1:208.80.154.169' hieradata/role/eqiad/wmcs/openstack/eqiad1/labweb.yaml: host: 208.80.154.160 hieradata/role/eqiad/wmcs/openstack/eqiad1/labweb.yaml: host: 208.80.154.160 modules/role/templates/mariadb/grants/production-m5.sql.erb:GRANT SELECT, INSERT, UPDATE, DELETE ON striker.* TO 'striker'@'208.80.154.160' modules/role/templates/mariadb/grants/production-m5.sql.erb:GRANT ALL ON striker.* TO 'striker_admin'@'208.80.154.160' modules/role/templates/mariadb/grants/production-m5.sql.erb:GRANT ALL ON `labswiki`.* TO 'wikiadmin'@'208.80.154.160' modules/role/templates/mariadb/grants/production-m5.sql.erb:GRANT DELETE, INSERT, SELECT, UPDATE ON `labswiki`.* TO 'wikiuser'@'208.80.154.160' Looking for matches in puppetmaster1001.eqiad.wmnet:/srv/private Looking for matches in deploy1001.eqiad.wmnet:/srv/mediawiki-staging
The above is interesting since 208.80.154.16 (note the missing zero at the end) is the IP of archiva1001 :D
Change 610011 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Decommission archiva1001
Change 659236 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] archiva: disable nginx proxy buffering
Change 659236 merged by Elukey:
[operations/puppet@production] archiva: disable nginx proxy buffering
Mentioned in SAL (#wikimedia-operations) [2021-01-28T11:30:18Z] <elukey> disable nginx proxy buffering on archiva.wikimedia.org for a perf test - T252767
Change 608812 merged by Elukey:
[operations/puppet@production] archiva::proxy: allow nginx to serve content from repositories
Change 604698 abandoned by Elukey:
[operations/puppet@production] archiva::proxy: raise TLS ciphersuite requirements
Reason: