Page MenuHomePhabricator

Migrate apt repository to bookworm
Closed, ResolvedPublic

Description

It's currently on Buster. While moving to a new OS, we should also address T317178 alongside.

In Bookworm reprepro was updated to a fork which has merged various patches. It might also make sense to migrate to Bookworm rightaway (even though it's not yet final)

Event Timeline

MoritzMuehlenhoff renamed this task from Migrate apt repository to bullseye or bookworm to Migrate apt repository to bookworm.Oct 5 2023, 7:18 AM
MoritzMuehlenhoff claimed this task.
MoritzMuehlenhoff triaged this task as Medium priority.

Change 964844 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] aptrepo::rsync: Don't setup rsync for empty list of secondary servers

https://gerrit.wikimedia.org/r/964844

Change 964844 merged by Muehlenhoff:

[operations/puppet@production] aptrepo::rsync: Don't setup rsync for empty list of secondary servers

https://gerrit.wikimedia.org/r/964844

Change 964900 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] Add dummy keytabs for apt1002/apt2002

https://gerrit.wikimedia.org/r/964900

Change 964900 merged by Muehlenhoff:

[labs/private@master] Add dummy keytabs for apt1002/apt2002

https://gerrit.wikimedia.org/r/964900

Change 965101 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Assign apt_repo role to apt1002

https://gerrit.wikimedia.org/r/965101

Change 965101 merged by Muehlenhoff:

[operations/puppet@production] Assign apt_repo role to apt1002

https://gerrit.wikimedia.org/r/965101

Change 965170 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Extend acmechief config for new apt hosts

https://gerrit.wikimedia.org/r/965170

Change 965170 merged by Muehlenhoff:

[operations/puppet@production] Extend acmechief config for new apt hosts

https://gerrit.wikimedia.org/r/965170

Change 971117 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Setup rsync between apt1001/apt1002

https://gerrit.wikimedia.org/r/971117

Change 971117 merged by Muehlenhoff:

[operations/puppet@production] Setup rsync between apt1001/apt1002

https://gerrit.wikimedia.org/r/971117

Change 1005498 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] acmechief: Remove obsolete entries from apt record

https://gerrit.wikimedia.org/r/1005498

Change 1005498 merged by Muehlenhoff:

[operations/puppet@production] acmechief: Remove obsolete entries from apt record

https://gerrit.wikimedia.org/r/1005498

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host apt2002.wikimedia.org with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host apt2002.wikimedia.org with OS bookworm executed with errors:

  • apt2002 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402211334_jmm_2872487_apt2002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" apt2002.wikimedia.org to get a root shellbut depending on the failure this may not work.

Mentioned in SAL (#wikimedia-operations) [2024-02-21T14:20:53Z] <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "new apt server in codfw - jmm@cumin2002 - T331613"

Mentioned in SAL (#wikimedia-operations) [2024-02-21T14:22:04Z] <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "new apt server in codfw - jmm@cumin2002 - T331613"

Change 1005524 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Explicitly configure apt2002 for Puppet 7

https://gerrit.wikimedia.org/r/1005524

Change 1005524 merged by Muehlenhoff:

[operations/puppet@production] Explicitly configure apt2002 for Puppet 7

https://gerrit.wikimedia.org/r/1005524

Change 1005731 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Make apt2002 a repository server

https://gerrit.wikimedia.org/r/1005731

Change 1005731 merged by Muehlenhoff:

[operations/puppet@production] Make apt2002 a repository server

https://gerrit.wikimedia.org/r/1005731

Change 1008893 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Point apt discovery records to apt1002/apt2002 (new bookworm hosts)

https://gerrit.wikimedia.org/r/1008893

Change 1008893 merged by Muehlenhoff:

[operations/puppet@production] Point apt discovery records to apt1002/apt2002 (new bookworm hosts)

https://gerrit.wikimedia.org/r/1008893

Change 1009199 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add an motd for the old buster reposority server

https://gerrit.wikimedia.org/r/1009199

Change 1009199 merged by Muehlenhoff:

[operations/puppet@production] Add an motd for the old buster reposority server

https://gerrit.wikimedia.org/r/1009199

Change 1009281 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Move the old apt servers to insetup::buster role

https://gerrit.wikimedia.org/r/1009281

Change 1009282 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Move nginx/Puppet settings for new apt hosts to the role Hiera data

https://gerrit.wikimedia.org/r/1009282

Change 1009281 merged by Muehlenhoff:

[operations/puppet@production] Move the old apt servers to insetup::buster role

https://gerrit.wikimedia.org/r/1009281

Mentioned in SAL (#wikimedia-operations) [2024-03-07T07:06:17Z] <moritzm> revoke Kerberos host principals for apt1001/apt2001 T331613

Change 1009282 merged by Muehlenhoff:

[operations/puppet@production] Move nginx/Puppet settings for new apt hosts to the role Hiera data

https://gerrit.wikimedia.org/r/1009282

Change 1012996 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] aptrepo: Remove obsolete migration code

https://gerrit.wikimedia.org/r/1012996

Change 1012996 merged by Muehlenhoff:

[operations/puppet@production] aptrepo: Remove obsolete migration code

https://gerrit.wikimedia.org/r/1012996

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: apt2001.wikimedia.org

  • apt2001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox

Change #1013509 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove puppet references to apt1001/2001

https://gerrit.wikimedia.org/r/1013509

Change #1013511 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] Remove obsolete dummy key tabs

https://gerrit.wikimedia.org/r/1013511

Change #1013511 merged by Muehlenhoff:

[labs/private@master] Remove obsolete dummy key tabs

https://gerrit.wikimedia.org/r/1013511

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: apt1001.wikimedia.org

  • apt1001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox

Change #1013509 merged by Muehlenhoff:

[operations/puppet@production] Remove puppet references to apt1001/2001

https://gerrit.wikimedia.org/r/1013509

apt.wikimedia.org is now running on two Bookworm VMs (apt1002 and apt2002), using the new/forked reprepro. The repository metadata/database got imported into the new setup. Wiki documentation was updated and the old VMs decommissioned.