Page MenuHomePhabricator
Create Task
Maniphest T320797

Initial production deployment of the IDM
Closed, ResolvedPublic
Actions

Description

Productionising the IDM:

  • Create Ganeti instances and apply the Puppet roles
  • Setup monitoring for basic availability of the end point

Details

SubjectRepoBranchLines +/-
Add Cumin aliases for IDMoperations/puppetproduction+2 -0
idm::jobs: Adapt auto restart to only run of idm-rq is active/presentoperations/puppetproduction+3 -1
Set ferm access for redisoperations/puppetproduction+1 -1
C:idm::deployment use Redis passwordoperations/puppetproduction+7 -4
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT189531 All Wikimedia developer services should use single sign-on
 OpenNoneT363125 sustainability of wikitech.wikimedia.org
 OpenNoneT161859 Make Wikitech an SUL wiki
 OpenSLyngshede-WMFT163478 Allow viewing/searching LDAP account creations including date
 ResolvedNoneT153821 wikitech.wikimedia.org missing from pageviews API
 OpenNoneT237773 Move Wikitech onto the production MW cluster
 ResolvedMarosteguiT167973 Move database for wikitech (labswiki) to a main cluster section
 ResolvedAndrewT188029 Move labswiki database to m5
ResolvedMarosteguiT282074 Audit labswiki grants
 ResolvedAndrewT282209 Enable connection from labweb1001/labweb1002 to s6
 DeclinedAndrewT282573 Move labweb/cloudweb hosts to private IPs
 ResolvedAndrewT284106 Replicate & sanitize wikitech data
 ResolvedAndrewT284108 Bring labswiki database tables up to date
 ResolvedLadsgroupT269348 wikitech database has almost all of its varbinary fields wrong
 DuplicateNoneT284736 Create views on labswiki on clouddb* hosts
 Resolved• BstormT287442 Create views for labswiki (wikitech)
 DeclinedNoneT237889 Install php-ldap on all MW appservers
 Resolved• taaviT237890 Remove and empty useless user groups
 ResolvedJdforrester-WMFT196466 Remove 'shell user' right on wikitech
 DeclinedNoneT246405 On wikitech, make sysops unable to edit site JS, and then retire contentadmin
 OpenjijikiT292707 Migrate Wikitech to Kubernetes
 OpenNoneT359551 Replace wikitech as source of two-factor auth protection for developer accounts
 OpenFeatureNoneT359554 Use IDP for authentication in Striker
 StalledFeatureNoneT359590 Use IDP for authentication in Horizon
 OpenFeatureSLyngshede-WMFT359552 Enable self-service IDP two-factor authentication management
 OpenNoneT359820 Add developer account (un)blocking support to Bitu
 OpenAndrewT360795 Set up a bitu instance for codfw1dev
 OpenSLyngshede-WMFT362128 Site: 1 VM for codfw1dev bitu deployment
 ResolvedABran-WMFT362619 Database request for Bitu Cloud DEV installation
 Resolved• taaviT360883 Disable email address changes in Wikitech
 OpenNoneT364605 Move Striker to Bitu username validation API
 In ProgressSLyngshede-WMFT362318 Add Bitu container to Striker development environment
 ResolvedPRODUCTION ERRORTgrT195253 Special:Notifications gives a consistent PHP exception on load ("The trash icon is not registered") for users with OpenStackManager notifications
 OpenNoneT106123 Extensions needing to be removed from Wikimedia wikis
 Resolvedbd808T53642 Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org
 ResolvedyuvipandaT103995 Build a simple tool to query which instances have which roles / puppet variables
 Resolvedbd808T161662 Replace SMW data on project instances with a tool using the OpenStack APIs
 Resolvedbd808T162508 Implement Tool Labs membership application and processing in Striker
 Resolvedbd808T164787 Keystone permissions exception when trying to approve Tool Labs membership request via Striker
 Open• taaviT161553 Remove OpenStackManager from Wikitech
 Resolved• taaviT196171 Developer account creation without OpenStackManager
 DeclinedNoneT179463 Create a single application to provision and manage developer (LDAP) accounts
 OpenNoneT319405 Create an IDM for Wikimedia developer accounts
 ResolvedSLyngshede-WMFT320603 IDM milestone 2 "Initial limited deployment"
 ResolvedSLyngshede-WMFT320797 Initial production deployment of the IDM
 OpenNoneT148048 Store Wikimedia unified account name (SUL) in LDAP directory
 Open• taaviT359428 Striker should use ID instead of username to identify SUL accounts

Event Timeline

MoritzMuehlenhoff created this task.Oct 14 2022, 12:19 PM
Dzahn renamed this task from Initial production deployment to Initial production deployment of the IDM.Nov 4 2022, 7:41 PM
ops-monitoring-bot added a comment.Feb 16 2023, 10:20 AM

Cookbook cookbooks.sre.ganeti.reimage was started by slyngshede@cumin1001 for host idm1001.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Feb 16 2023, 10:45 AM

Cookbook cookbooks.sre.ganeti.reimage started by slyngshede@cumin1001 for host idm1001.wikimedia.org with OS bullseye completed:

  • idm1001 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/ganeti/reimage/202302161020_slyngshede_2708067_idm1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
ops-monitoring-bot added a comment.Feb 17 2023, 10:00 AM

Cookbook cookbooks.sre.ganeti.reimage was started by slyngshede@cumin1001 for host idm2001.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Feb 17 2023, 10:28 AM

Cookbook cookbooks.sre.ganeti.reimage started by slyngshede@cumin1001 for host idm2001.wikimedia.org with OS bullseye completed:

  • idm2001 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/ganeti/reimage/202302171000_slyngshede_2978437_idm2001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
gerritbot added a comment.Feb 21 2023, 11:51 AM

Change 890801 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] C:idm::deployment use Redis password

https://gerrit.wikimedia.org/r/890801

gerritbot added a project: Patch-For-Review.Feb 21 2023, 11:51 AM
gerritbot added a comment.Feb 21 2023, 12:03 PM

Change 890801 merged by Slyngshede:

[operations/puppet@production] C:idm::deployment use Redis password

https://gerrit.wikimedia.org/r/890801

Maintenance_bot removed a project: Patch-For-Review.Feb 21 2023, 12:11 PM
gerritbot added a comment.Feb 21 2023, 1:27 PM

Change 890815 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Set ferm access for redis

https://gerrit.wikimedia.org/r/890815

gerritbot added a project: Patch-For-Review.Feb 21 2023, 1:27 PM
gerritbot added a comment.Feb 21 2023, 1:31 PM

Change 890815 merged by Slyngshede:

[operations/puppet@production] Set ferm access for redis

https://gerrit.wikimedia.org/r/890815

Maintenance_bot removed a project: Patch-For-Review.Feb 21 2023, 2:11 PM
gerritbot added a comment.Feb 22 2023, 3:54 PM

Change 891318 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] idm::jobs: Adapt auto restart to only run of idm-rq is active/present

https://gerrit.wikimedia.org/r/891318

gerritbot added a project: Patch-For-Review.Feb 22 2023, 3:54 PM
gerritbot added a comment.Feb 23 2023, 8:05 AM

Change 891318 merged by Muehlenhoff:

[operations/puppet@production] idm::jobs: Adapt auto restart to only run of idm-rq is active/present

https://gerrit.wikimedia.org/r/891318

Maintenance_bot removed a project: Patch-For-Review.Feb 23 2023, 8:11 AM
Stashbot added a comment.Feb 27 2023, 2:33 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-27T14:33:02Z] <jbond@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on idm2001.wikimedia.org with reason: host still been configuered - T320797

Stashbot added a comment.Feb 27 2023, 2:33 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-27T14:33:06Z] <jbond@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on idm2001.wikimedia.org with reason: host still been configuered - T320797

Stashbot added a comment.Feb 27 2023, 2:33 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-27T14:33:13Z] <jbond@cumin2002> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on idm2001.wikimedia.org with reason: host still been configuered - T320797

Stashbot added a comment.Feb 27 2023, 2:33 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-27T14:33:19Z] <jbond@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on idm2001.wikimedia.org with reason: host still been configuered - T320797

gerritbot added a comment.Mar 9 2023, 3:17 PM

Change 896112 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add Cumin aliases for IDM

https://gerrit.wikimedia.org/r/896112

gerritbot added a project: Patch-For-Review.Mar 9 2023, 3:17 PM
gerritbot added a comment.Mar 13 2023, 3:03 PM

Change 896112 merged by Muehlenhoff:

[operations/puppet@production] Add Cumin aliases for IDM

https://gerrit.wikimedia.org/r/896112

Maintenance_bot removed a project: Patch-For-Review.Mar 13 2023, 3:10 PM
Aklapper added a project: Bitu.May 10 2023, 11:45 AM
SLyngshede-WMF closed this task as Resolved.Jun 28 2023, 12:48 PM
SLyngshede-WMF claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL