Productionising the IDM:
- Create Ganeti instances and apply the Puppet roles
- Setup monitoring for basic availability of the end point
Description
Description
Details
Details
Related Changes in Gerrit:
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T189531 All Wikimedia developer services should use single sign-on | |||
| Resolved | None | T161859 Make Wikitech an SUL wiki | |||
| Duplicate | None | T367287 Update Wikitech's LDAP credentials to be read-only | |||
| Resolved | PRODUCTION ERROR | Tgr | T195253 Special:Notifications gives a consistent PHP exception on load ("The trash icon is not registered") for users with OpenStackManager notifications | ||
| Open | None | T106123 Extensions needing to be removed from Wikimedia wikis | |||
| Resolved | Request | None | T367220 Archive the OpenStackManager extension | ||
| Resolved | taavi | T161553 Remove OpenStackManager from Wikitech | |||
| Resolved | taavi | T196171 Developer account creation without OpenStackManager | |||
| Declined | None | T179463 Create a single application to provision and manage developer (LDAP) accounts | |||
| Resolved | None | T319405 Create an IDM for Wikimedia developer accounts | |||
| Resolved | SLyngshede-WMF | T320603 IDM milestone 2 "Initial limited deployment" | |||
| Resolved | SLyngshede-WMF | T320797 Initial production deployment of the IDM |
Event Timeline
Comment Actions
Cookbook cookbooks.sre.ganeti.reimage was started by slyngshede@cumin1001 for host idm1001.wikimedia.org with OS bullseye
Comment Actions
Cookbook cookbooks.sre.ganeti.reimage started by slyngshede@cumin1001 for host idm1001.wikimedia.org with OS bullseye completed:
- idm1001 (PASS)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Set boot to disk
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/ganeti/reimage/202302161020_slyngshede_2708067_idm1001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
Comment Actions
Cookbook cookbooks.sre.ganeti.reimage was started by slyngshede@cumin1001 for host idm2001.wikimedia.org with OS bullseye
Comment Actions
Cookbook cookbooks.sre.ganeti.reimage started by slyngshede@cumin1001 for host idm2001.wikimedia.org with OS bullseye completed:
- idm2001 (PASS)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Set boot to disk
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/ganeti/reimage/202302171000_slyngshede_2978437_idm2001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
Comment Actions
Change 890801 had a related patch set uploaded (by Slyngshede; author: Slyngshede):
[operations/puppet@production] C:idm::deployment use Redis password
Comment Actions
Change 890801 merged by Slyngshede:
[operations/puppet@production] C:idm::deployment use Redis password
Comment Actions
Change 890815 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Set ferm access for redis
Comment Actions
Change 890815 merged by Slyngshede:
[operations/puppet@production] Set ferm access for redis
Comment Actions
Change 891318 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] idm::jobs: Adapt auto restart to only run of idm-rq is active/present
Comment Actions
Change 891318 merged by Muehlenhoff:
[operations/puppet@production] idm::jobs: Adapt auto restart to only run of idm-rq is active/present
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2023-02-27T14:33:02Z] <jbond@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on idm2001.wikimedia.org with reason: host still been configuered - T320797
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2023-02-27T14:33:06Z] <jbond@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on idm2001.wikimedia.org with reason: host still been configuered - T320797
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2023-02-27T14:33:13Z] <jbond@cumin2002> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on idm2001.wikimedia.org with reason: host still been configuered - T320797
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2023-02-27T14:33:19Z] <jbond@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on idm2001.wikimedia.org with reason: host still been configuered - T320797
Comment Actions
Change 896112 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):
[operations/puppet@production] Add Cumin aliases for IDM
Comment Actions
Change 896112 merged by Muehlenhoff:
[operations/puppet@production] Add Cumin aliases for IDM