Page MenuHomePhabricator
Create Task
Maniphest T340721

Build Debian packages for Bookworm
Closed, ResolvedPublic
Actions

Description

Bitu should be deployed to production using deb packages.
Staging will remain on Git deployments.

Details

SubjectRepoBranchLines +/-
P:IDM: Failover Redisoperations/puppetproduction+2 -2
IDM Switchoveroperations/dnsmaster+1 -1
P:idm switch idm2001 to Debian packageoperations/puppetproduction+1 -0
P:idm allow for installation via Debian packages.operations/puppetproduction+179 -151
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT189531 All Wikimedia developer services should use single sign-on
 OpenNoneT363125 sustainability of wikitech.wikimedia.org
 OpenNoneT161859 Make Wikitech an SUL wiki
 OpenSLyngshede-WMFT163478 Allow viewing/searching LDAP account creations including date
 ResolvedNoneT153821 wikitech.wikimedia.org missing from pageviews API
 OpenNoneT237773 Move Wikitech onto the production MW cluster
 ResolvedMarosteguiT167973 Move database for wikitech (labswiki) to a main cluster section
 ResolvedAndrewT188029 Move labswiki database to m5
ResolvedMarosteguiT282074 Audit labswiki grants
 ResolvedAndrewT282209 Enable connection from labweb1001/labweb1002 to s6
 DeclinedAndrewT282573 Move labweb/cloudweb hosts to private IPs
 ResolvedAndrewT284106 Replicate & sanitize wikitech data
 ResolvedAndrewT284108 Bring labswiki database tables up to date
 ResolvedLadsgroupT269348 wikitech database has almost all of its varbinary fields wrong
 DuplicateNoneT284736 Create views on labswiki on clouddb* hosts
 Resolved• BstormT287442 Create views for labswiki (wikitech)
 DeclinedNoneT237889 Install php-ldap on all MW appservers
 ResolvedtaaviT237890 Remove and empty useless user groups
 ResolvedJdforrester-WMFT196466 Remove 'shell user' right on wikitech
 DeclinedNoneT246405 On wikitech, make sysops unable to edit site JS, and then retire contentadmin
 OpenjijikiT292707 Migrate Wikitech to Kubernetes
 OpenNoneT359551 Replace wikitech as source of two-factor auth protection for developer accounts
 OpenFeatureNoneT359554 Use IDP for authentication in Striker
 StalledFeatureNoneT359590 Use IDP for authentication in Horizon
 OpenFeatureSLyngshede-WMFT359552 Enable self-service IDP two-factor authentication management
 OpenNoneT359820 Add developer account (un)blocking support to Bitu
 OpenAndrewT360795 Set up a bitu instance for codfw1dev
 OpenSLyngshede-WMFT362128 Site: 1 VM for codfw1dev bitu deployment
 ResolvedABran-WMFT362619 Database request for Bitu Cloud DEV installation
 ResolvedtaaviT360883 Disable email address changes in Wikitech
 OpenNoneT364605 Move Striker to Bitu username validation API
 OpenSLyngshede-WMFT362318 Add Bitu container to Striker development environment
 ResolvedPRODUCTION ERRORTgrT195253 Special:Notifications gives a consistent PHP exception on load ("The trash icon is not registered") for users with OpenStackManager notifications
 OpenNoneT106123 Extensions needing to be removed from Wikimedia wikis
 Resolvedbd808T53642 Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org
 ResolvedyuvipandaT103995 Build a simple tool to query which instances have which roles / puppet variables
 Resolvedbd808T161662 Replace SMW data on project instances with a tool using the OpenStack APIs
 Resolvedbd808T162508 Implement Tool Labs membership application and processing in Striker
 Resolvedbd808T164787 Keystone permissions exception when trying to approve Tool Labs membership request via Striker
 OpentaaviT161553 Remove OpenStackManager from Wikitech
 ResolvedtaaviT196171 Developer account creation without OpenStackManager
 DeclinedNoneT179463 Create a single application to provision and manage developer (LDAP) accounts
 OpenNoneT319405 Create an IDM for Wikimedia developer accounts
 ResolvedSLyngshede-WMFT320603 IDM milestone 2 "Initial limited deployment"
 OpenNoneT148048 Store Wikimedia unified account name (SUL) in LDAP directory
 OpentaaviT359428 Striker should use ID instead of username to identify SUL accounts
 ResolvedSLyngshede-WMFT340721 Build Debian packages for Bookworm

Event Timeline

SLyngshede-WMF created this task.Jun 29 2023, 10:10 AM
SLyngshede-WMF triaged this task as Medium priority.Jun 29 2023, 11:00 AM
gerritbot added a comment.Sep 14 2023, 8:01 AM

Change 956836 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] P:idm allow for installation via Debian packages.

https://gerrit.wikimedia.org/r/956836

gerritbot added a project: Patch-For-Review.Sep 14 2023, 8:01 AM
gerritbot added a comment.Sep 14 2023, 8:19 AM

Change 957669 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] WIP: P:idm switch idm2001 to Debian package

https://gerrit.wikimedia.org/r/957669

SLyngshede-WMF added a comment.EditedSep 14 2023, 8:31 AM

Plan for testing rollout of Debian packages:

Upgrade test to Bookworm:

Pre-update:

Reimage idm-test100 

ssh cumin1001.eqiad.wmnet
sudo cookbook sre.hosts.reimage --os bookworm -t T340721 idm-test1001

IDM2001 upgrade

ssh cumin1001.eqiad.wmnet
sudo cumin 'idm2001.wikimedia.org' "disable-puppet 'bitu deb install - slyngshede'"
sudo cookbook sre.hosts.reimage --os bookworm -t T340721 idm2001

Switch over to IDM2001

WAIT AND ALLOW ANY BUG TO REVEAL THEMSELVES

IDM1001 Upgrade

ssh cumin1001.eqiad.wmnet
sudo cumin 'idm1001.wikimedia.org' "disable-puppet 'bitu deb install - slyngshede'"
sudo cookbook sre.hosts.reimage --os bookworm -t T340721 idm1001
gerritbot added a comment.Sep 14 2023, 8:50 AM

Change 957674 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/dns@master] IDM Switchover

https://gerrit.wikimedia.org/r/957674

gerritbot added a comment.Sep 14 2023, 8:55 AM

Change 957676 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] IDM: Deploy deb to idm1001.

https://gerrit.wikimedia.org/r/957676

MoritzMuehlenhoff renamed this task from Build Debian packages for Bookwork to Build Debian packages for Bookworm.Sep 14 2023, 9:25 AM

Plan looks good to me.

gerritbot added a comment.Sep 14 2023, 11:37 AM

Change 956836 merged by Slyngshede:

[operations/puppet@production] P:idm allow for installation via Debian packages.

https://gerrit.wikimedia.org/r/956836

ops-monitoring-bot added a comment.Sep 14 2023, 11:54 AM

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1001 for host idm-test1001.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Sep 14 2023, 12:56 PM

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1001 for host idm-test1001.wikimedia.org with OS bookworm completed:

  • idm-test1001 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309141210_slyngshede_31369_idm-test1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 15 2023, 1:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1001 for host idm-test1001.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Sep 15 2023, 1:41 PM

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1001 for host idm-test1001.wikimedia.org with OS bookworm completed:

  • idm-test1001 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309151319_slyngshede_376144_idm-test1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Sep 19 2023, 8:10 AM

Change 957669 merged by Slyngshede:

[operations/puppet@production] P:idm switch idm2001 to Debian package

https://gerrit.wikimedia.org/r/957669

ops-monitoring-bot added a comment.Sep 19 2023, 8:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1001 for host idm2001.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Sep 19 2023, 9:48 AM

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1001 for host idm2001.wikimedia.org with OS bookworm completed:

  • idm2001 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202309190829_slyngshede_3921709_idm2001.out, asking the operator what to do
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202309190831_slyngshede_3921709_idm2001.out, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309190845_slyngshede_3921709_idm2001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Sep 19 2023, 1:00 PM

Change 957674 merged by Slyngshede:

[operations/dns@master] IDM Switchover

https://gerrit.wikimedia.org/r/957674

gerritbot added a comment.Sep 19 2023, 1:14 PM

Change 957676 merged by Slyngshede:

[operations/puppet@production] P:IDM: Failover Redis

https://gerrit.wikimedia.org/r/957676

Maintenance_bot removed a project: Patch-For-Review.Sep 19 2023, 1:31 PM
ops-monitoring-bot added a comment.Sep 20 2023, 7:09 AM

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1001 for host idm1001.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Sep 20 2023, 7:42 AM

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1001 for host idm1001.wikimedia.org with OS bookworm completed:

  • idm1001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309200724_slyngshede_4193245_idm1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 20 2023, 1:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1001 for host idm-test1001.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Sep 20 2023, 1:49 PM

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1001 for host idm-test1001.wikimedia.org with OS bookworm completed:

  • idm-test1001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309201331_slyngshede_78977_idm-test1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
SLyngshede-WMF closed this task as Resolved.Sep 21 2023, 8:07 AM
SLyngshede-WMF claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL