Page MenuHomePhabricator
Create Task
Maniphest T327068

Bullseye upgrade for remaining Collab hosts
Closed, ResolvedPublic
Actions

Description

ServiceOps-Collab systems still running Buster: https://os-reports.wikimedia.org/os-report-todo-2023-05-15-buster.html

  • role::ci::master (2 host(s)) (T334517)
  • role::releases (2 host(s)) (T334435)
  • role::gerrit (2 hosts(s)) (T326368, T334521) decom gerrit1001: T336427
  • role::miscweb (2 host(s)) (Racktables should be removed first: T327405)
  • role::requesttracker (1 host(s))
  • role::planet (2 host(s))
  • Requires upgrading the RSS aggregator (T281219), alternatively may be a decom target
  • role::phabricator (2 host(s)) (T334519)
  • role::aphlict (1 host(s)) (T333452)
  • role::vrts (1 host(s)) (T295416)
  • role::doc (2 host(s)) (T319477)
  • role::lists (1 host) (T331706)

Details

SubjectRepoBranchLines +/-
phabricator: install python3-phabricator if bullseye or neweroperations/puppetproduction+7 -1
php: add templates to support php8.2 on bookwormoperations/puppetproduction+9 -0
phabricator::main: add support for PHP versions other than 7.3operations/puppetproduction+6 -1
phabricator/httpd: add support for bullseye/bookworm PHP versionsoperations/puppetproduction+5 -3
sre: update planned quarters and tickets for collab servicesoperations/puppetproduction+17 -16
planet: if on bullseye, install package contents via puppetoperations/puppetproduction+2 K -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT291916 Tracking task for Bullseye migrations in production
 ResolvedDzahnT327068 Bullseye upgrade for remaining Collab hosts
 ResolvedArnoldokothT295416 upgrade/replace VRTS (formerly OTRS) buster to bullseye
 ResolvedArnoldokothT338418 VRTS eqiad replacement
 ResolvedArnoldokothT339253 Decom otrs1001
 ResolvedeoghanT319477 Migrate doc hosts to Bullseye
 Resolvedandrea.denisseT332812 Site: 1 VM request for doc1003
 Resolvedandrea.denisseT332819 Site: 1 VM request for doc2002
 ResolvedhasharT333294 MediaWiki periodic Doxygen Jenkins job fails to publish
 ResolvedDzahnT327405 Decommission Racktables
 ResolvedDzahnT331896 upgrade miscweb VMs to bullseye
 ResolvedDzahnT332102 Site: eqiad/codfw 2 VM request for miscweb
 ResolvedRequestDzahnT334024 decommission miscweb2002.codfw.wmnet / miscweb1002.eqiad.wmnet
 ResolvedDzahnT332952 upgrade moscovium (RT) to bullseye
 ResolvedeoghanT334435 upgrade releases hosts to bullseye
 ResolvedeoghanT337349 eqiad+codfw: 1 VM each site request for releases.wikimedia.org
ResolvedeoghanT333452 Replace existing aphlict1001 with puppet-managed bullseye host
 ResolvedDzahnT334519 upgrade phab (phorge) hosts to bullseye
 ResolvedDzahnT354221 automate data syncing between phabricator servers
 ResolvedDzahnT334517 upgrade contint servers to bullseye
 ResolvedNoneT342346 Refresh integration/zuul/deploy to work on Debian Bullseye
 ResolvedhasharT259611 The python-build images regenerate wheels even when matching ones are already available
 ResolvedDzahnT358237 Ganeti VM for contint migration
 ResolvedhasharT361224 Test zuul under Bullseye with contint1003
 DeclinedNoneT157818 zuul-merger fails when repository names overlaps
 ResolvedhasharT158406 operations/software repo commits always getting -1 from Jenkins-bot
 ResolvedDzahnT334521 upgrade gerrit servers to bullseye
 ResolvedDzahnT326368 gerrit1003 service implementation task
 ResolvedDzahnT334524 add new and remove old Gerrit service IP from netbox and DNS
 ResolvedDzahnT336427 decom gerrit1001
 ResolvedJclark-ctrT340077 decommission gerrit1001.wikimedia.org (dcops, netbox)
 ResolvedBUG REPORThasharT336524 IPv6 SSH to gerrit.wikimedia.org hangs (blackhole route?)
 ResolvedJeltoT345577 Investigate the phabricator-prod-1001 alert
 ResolvedDzahnT348392 Migrate planet servers to bullseye or bookworm
 ResolvedLegoktmT281219 Find a replacement for RSS aggregator for planet.wikimedia.org
 ResolvedDzahnT351849 Site: 2 VMs %request for planet
 ResolvedMoritzMuehlenhoffT353379 spicerack.ganeti.GanetiError: Error while performing request to RAPI
 InvalidNoneT351333 build python-phabricator package for bullseye (and bookworm?)
 ResolvedeoghanT331706 Migrate Mailman/lists to Bullseye/Bookworm
 ResolvedeoghanT363572 Reimage physical lists hosts to have public IPs
 ResolvedeoghanT283615 Make mailman3 work in the standby host (lists2001.wikimedia.org)
 ResolvedeoghanT365714 PuppetFailure - lists2001
 ResolvedeoghanT365698 SystemdUnitFailed - lists2001
 ResolvedeoghanT365781 SystemdUnitFailed - gitlab1003, gitlab1004, lists2001
 ResolvedeoghanT365804 SystemdUnitFailed - lists2001
 ResolvedeoghanT365789 SystemdUnitFailed - gitlab1003, gitlab1004, lists2001
 ResolvedeoghanT365999 SystemdUnitFailed - lists1004
 ResolvedeoghanT367521 Mailman Downtime: Migrate mailman from lists1001 to lists1004
 ResolvedLadsgroupT367833 Update grants for mailman

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Dzahn mentioned this in T334517: upgrade contint servers to bullseye.Apr 11 2023, 8:05 PM
Dzahn updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)Apr 11 2023, 8:16 PM
Dzahn updated the task description. (Show Details)Apr 11 2023, 8:25 PM
Dzahn added a subtask: T333452: Replace existing aphlict1001 with puppet-managed bullseye host.
Dzahn added subtasks: T334519: upgrade phab (phorge) hosts to bullseye, T334517: upgrade contint servers to bullseye.Apr 11 2023, 8:31 PM
Dzahn mentioned this in T334521: upgrade gerrit servers to bullseye.Apr 11 2023, 8:35 PM
Dzahn changed the task status from Open to In Progress.Apr 11 2023, 8:45 PM
Dzahn updated the task description. (Show Details)
Dzahn added subtasks: T334521: upgrade gerrit servers to bullseye, T326368: gerrit1003 service implementation task.
Dzahn changed the status of subtask T334521: upgrade gerrit servers to bullseye from Open to In Progress.Apr 20 2023, 4:56 PM
gerritbot added a comment.Apr 24 2023, 5:42 PM

Change 908644 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] sre: update planned quarters and tickets for collab services

https://gerrit.wikimedia.org/r/908644

gerritbot added a project: Patch-For-Review.Apr 24 2023, 5:42 PM
gerritbot added a comment.Apr 24 2023, 5:48 PM

Change 908644 merged by Dzahn:

[operations/puppet@production] sre: update planned quarters and tickets for collab services

https://gerrit.wikimedia.org/r/908644

Maintenance_bot removed a project: Patch-For-Review.Apr 24 2023, 6:16 PM
eoghan closed subtask T333452: Replace existing aphlict1001 with puppet-managed bullseye host as Resolved.May 10 2023, 11:03 AM
Dzahn closed subtask T326368: gerrit1003 service implementation task as Resolved.May 12 2023, 12:12 AM
LSobanski updated the task description. (Show Details)May 15 2023, 2:04 PM
LSobanski updated the task description. (Show Details)May 15 2023, 2:10 PM
Dzahn added a comment.EditedMay 15 2023, 5:19 PM

re: gerrit, gerrit1003 is now production and bullseye. gerrit1001 is in grace period before being shut down in T336427 and reimaging gerrit2002 with bullseye remains to be done but is part of T334521

Dzahn moved this task from Backlog to Work in Progress on the collaboration-services board.May 17 2023, 6:54 PM
Dzahn added a comment.May 19 2023, 5:07 PM

upgrade of gerrit2002 scheduled for May 25th

contint - in progress - just merged changes to reserve UID for zuul etc.. prep for migration

doc - in progress - Eoghan working on it after Andrea created new VMs

Dzahn added a comment.May 25 2023, 6:07 PM

gerrit2002 is now on bullseye, number of buster hosts reduced by 1 without name change.

eoghan closed subtask T319477: Migrate doc hosts to Bullseye as Resolved.May 31 2023, 11:46 AM
eoghan updated the task description. (Show Details)Jun 1 2023, 10:33 PM
LSobanski assigned this task to Dzahn.Jun 5 2023, 1:52 PM
Dzahn updated the task description. (Show Details)Jun 8 2023, 10:48 PM
Dzahn updated the task description. (Show Details)
Dzahn closed subtask T334521: upgrade gerrit servers to bullseye as Resolved.
Dzahn updated the task description. (Show Details)Jun 14 2023, 9:03 PM
Arnoldokoth closed subtask T295416: upgrade/replace VRTS (formerly OTRS) buster to bullseye as Resolved.Jun 20 2023, 1:18 PM
Dzahn added a comment.Jun 21 2023, 10:33 PM

gerrit1001 has been destroyed today

Dzahn removed Dzahn as the assignee of this task.Jun 22 2023, 8:21 PM
LSobanski moved this task from Work in Progress to Work in Progress (Tracking tasks) on the collaboration-services board.Jun 26 2023, 10:33 AM
eoghan closed subtask T334435: upgrade releases hosts to bullseye as Resolved.Jul 26 2023, 2:46 PM
eoghan updated the task description. (Show Details)Jul 31 2023, 9:05 PM
LSobanski added a subtask: T345577: Investigate the phabricator-prod-1001 alert.Sep 8 2023, 11:16 AM
Jelto closed subtask T345577: Investigate the phabricator-prod-1001 alert as Resolved.Sep 11 2023, 9:44 AM
gerritbot added a comment.Nov 9 2023, 6:02 PM

Change 973213 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator/httpd: add support for bullseye/bookworm PHP versions

https://gerrit.wikimedia.org/r/973213

gerritbot added a project: Patch-For-Review.Nov 9 2023, 6:02 PM
gerritbot added a comment.Nov 14 2023, 10:47 PM

Change 973213 merged by Dzahn:

[operations/puppet@production] phabricator/httpd: add support for bullseye/bookworm PHP versions

https://gerrit.wikimedia.org/r/973213

Maintenance_bot removed a project: Patch-For-Review.Nov 14 2023, 11:10 PM
gerritbot added a comment.Nov 14 2023, 11:11 PM

Change 974280 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator::main: add support for PHP versions other than 7.3

https://gerrit.wikimedia.org/r/974280

gerritbot added a project: Patch-For-Review.Nov 14 2023, 11:11 PM
gerritbot added a comment.Nov 14 2023, 11:35 PM

Change 974280 merged by Dzahn:

[operations/puppet@production] phabricator::main: add support for PHP versions other than 7.3

https://gerrit.wikimedia.org/r/974280

gerritbot added a comment.Nov 14 2023, 11:47 PM

Change 974286 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] php: add templates to support php8.2 on bookworm

https://gerrit.wikimedia.org/r/974286

Dzahn added a comment.Nov 15 2023, 12:09 AM

after patches above 7.4 can be installed on phab hosts / phab role can be applied on bullseye. tested in cloud VPS.

remaining issues:

  • Unable to locate package python-phabricator
  • Needs scap deployment but "Provider scap3 is not functional on this host"
  • something with SSL certs
gerritbot added a comment.Nov 15 2023, 5:32 PM

Change 974286 merged by Dzahn:

[operations/puppet@production] php: add templates to support php8.2 on bookworm

https://gerrit.wikimedia.org/r/974286

Dzahn added a subtask: T351333: build python-phabricator package for bullseye (and bookworm?).Nov 15 2023, 5:48 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 15 2023, 6:10 PM
Volans closed subtask T351333: build python-phabricator package for bullseye (and bookworm?) as Invalid.Nov 15 2023, 6:17 PM
gerritbot added a comment.Nov 15 2023, 6:33 PM

Change 974660 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator: install python3-phabricator if bullseye or newer

https://gerrit.wikimedia.org/r/974660

gerritbot added a project: Patch-For-Review.Nov 15 2023, 6:33 PM
gerritbot added a comment.Nov 17 2023, 4:24 PM

Change 974660 merged by Dzahn:

[operations/puppet@production] phabricator: install python3-phabricator if bullseye or newer

https://gerrit.wikimedia.org/r/974660

Maintenance_bot removed a project: Patch-For-Review.Nov 17 2023, 4:31 PM
Stashbot mentioned this in T328595: Create proof of concept Phorge instance in devtools project in wmcs (phorge.wmcloud.org).Nov 21 2023, 9:00 PM

Mentioned in SAL (#wikimedia-cloud) [2023-11-21T21:00:34Z] <mutante> - deleted instance phorge-1001 to get quota back and allow for creting new phabricator-on-bullseye instance T328595 T327068

Stashbot added a comment.Nov 21 2023, 9:05 PM

Mentioned in SAL (#wikimedia-cloud) [2023-11-21T21:05:17Z] <mutante> - creating instance phabricator-bullseye g3.cores2.ram4.disk20 T327068

Stashbot added a comment.Nov 21 2023, 9:24 PM

Mentioned in SAL (#wikimedia-cloud) [2023-11-21T21:24:21Z] <mutante> - initial puppet run on newly created VM fails with "SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain): [self signed certificate in certificate chain for /CN=Puppet CA: puppetmaster-1001.devtools.eqiad.wmflabs" T327068

Stashbot added a comment.Nov 21 2023, 9:41 PM

Mentioned in SAL (#wikimedia-cloud) [2023-11-21T21:41:52Z] <mutante> - cert issue on new machine related to having local puppetmaster, like T349937#9288547 except "rm -rf /var/lib/puppet/ssl" was enough since puppetmaster did auto-sign new CSR - T327068

Stashbot mentioned this in T349937: New Cloud VPS instance has "Failed to start Execute cloud user/final script." in its error log.Nov 21 2023, 9:41 PM
Stashbot added a comment.Nov 21 2023, 9:52 PM

Mentioned in SAL (#wikimedia-cloud) [2023-11-21T21:52:05Z] <mutante> - commit fake key for phabricator-bullseye host in git /var/lib/git/labs/private/modules/secret/secrets/ssl on puppetmaster-1001.devtools T327068

Dzahn added a subscriber: brennen.EditedNov 21 2023, 9:59 PM

@brennen Per our meeting chat today:

  • phorge-1001.devtools deleted
  • new VM created: phabricator-bullseye.devtools
  • added Hiera keys needed for phabricator
  • fixed cert signing issue and added fake SSL key on local puppetmaster, fixed initial puppet run
  • added prod prabricator puppet role (it installed php7.4 modules etc after previous puppet code changes)

Now apache is installed and all that and things work up to:

Package[phabricator/deployment]: Provider scap3 is not functional on this host
and "Dependency Package[phabricator/deployment] has failures: true"

The local deployment server should be deploy-1004.devtools.

edit: Sorry Brennen, wrong ticket. We should use T334519 instead.

Dzahn mentioned this in T334519: upgrade phab (phorge) hosts to bullseye.Nov 21 2023, 10:01 PM
Dzahn removed a subscriber: brennen.Nov 21 2023, 10:06 PM
Dzahn changed the status of subtask T348392: Migrate planet servers to bullseye or bookworm from Open to In Progress.Nov 27 2023, 10:39 PM
Stashbot added a comment.Dec 12 2023, 6:10 PM

Mentioned in SAL (#wikimedia-operations) [2023-12-12T18:10:02Z] <mutante> reimaging phab2002 (stand-by phorge server with bullseye - T327068

ops-monitoring-bot added a comment.Dec 12 2023, 6:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1001 for host phab2002.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 12 2023, 6:55 PM

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1001 for host phab2002.codfw.wmnet with OS bullseye completed:

  • phab2002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312121832_dzahn_1780890_phab2002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Dzahn added a subscriber: brennen.Dec 12 2023, 7:01 PM

[phab2002:~] $ lsb_release -c
Codename: bullseye

phab2002 is now on bullseye and ready for a deployment @brennen

Dzahn updated the task description. (Show Details)Dec 14 2023, 3:45 PM

planet VMs: buster replaced with bookworm machines, done!

Dzahn closed subtask T348392: Migrate planet servers to bullseye or bookworm as Resolved.Dec 14 2023, 3:58 PM
Dzahn closed subtask T334519: upgrade phab (phorge) hosts to bullseye as Resolved.Jan 21 2024, 12:28 AM
Dzahn updated the task description. (Show Details)
Dzahn claimed this task.Apr 16 2024, 5:25 PM
Dzahn changed the status of subtask T334517: upgrade contint servers to bullseye from Open to In Progress.
Dzahn added a comment.Apr 17 2024, 9:23 PM

contint1002 upgraded

Dzahn closed subtask T334517: upgrade contint servers to bullseye as Resolved.May 16 2024, 8:38 PM
Dzahn updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)
Dzahn added a comment.EditedMay 16 2024, 8:41 PM

contint2002 upgraded

@LSobanski was resolved but also added lists1001 now

hashar reopened subtask T334517: upgrade contint servers to bullseye as Open.May 22 2024, 7:33 AM
hashar closed subtask T334517: upgrade contint servers to bullseye as Resolved.May 29 2024, 12:45 PM
Dzahn updated the task description. (Show Details)Jul 3 2024, 10:32 PM
Dzahn closed this task as Resolved.Jul 3 2024, 10:35 PM
Dzahn added a subscriber: eoghan.

Since @eoghan shut down lists1001 in T331706#9944135 this epic task is now completed.

[cumin2002:~] $ sudo cumin 'A:owner-collaboration-services' 'lsb_release -c'
37 hosts will be targeted:
...
(10) .. Codename: bookworm
(27) .. Codename: bullseye
Dzahn added a subtask: T331706: Migrate Mailman/lists to Bullseye/Bookworm.Jul 3 2024, 11:32 PM
eoghan closed subtask T331706: Migrate Mailman/lists to Bullseye/Bookworm as Resolved.Jul 9 2024, 3:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits