Page MenuHomePhabricator
Create Task
Maniphest T260441

(Need By: ASAP) rack/setup/install clouddb10[13-20]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of clouddb10[13-20]

Need By: A date of ASAP due to Q1 OKRs was stated on parent task, but has not been reviewed/approved/agreed to by the ops-eqiad engineers at the time of this task creation.

Hostname / Racking / Installation Details

Hostnames: clouddb1013, clouddb1014, clouddb1015
Racking Proposal: These should be spread out away from other systems as much as is practical along with the other 5 systems from T258088. There may be limitations to how much they can be spread out because there are 8 of them.
Networking/Subnet/VLAN/IP: 10GbE on the private vlan (with the same access as the existing labsdb1009-11 hosts)
Partitioning/Raid: RAID10 + 256k stripe size. Normal db raid recipe @Marostegui will take care of this puppet part (or @Bstorm will copy his work).
OS Distro: Buster

Hostnames: clouddb1016, clouddb1017, clouddb1018,clouddb1019, clouddb1020
Racking Proposal: These should be spread out away from other systems as much as is practical along with the other 3 systems from T257987. There may be limitations to how much they can be spread out because there are 8 of these, total.
Networking/Subnet/VLAN/IP: 10GbE on the private vlan (with the same access as the existing labsdb1009-11 hosts)
Partitioning/Raid: RAID10 + 256k stripe size. Normal db raid recipe @Marostegui will take care of this puppet part (or @Bstorm will copy his work).
OS Distro: Buster

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

clouddb1013:

  • - receive in system on procurement task T257987 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620529
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

clouddb1014:

  • - receive in system on procurement task T257987 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620529
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

clouddb1015:

  • - receive in system on procurement task T257987 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620529
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

clouddb1016:

  • - receive in system on procurement task T257987 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620529
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

clouddb1017:

  • - receive in system on procurement task T257987 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620529
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

clouddb1018:

  • - receive in system on procurement task T257987 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620529
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

clouddb1019:

  • - receive in system on procurement task T257987 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620529
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

clouddb1020:

  • - receive in system on procurement task T257987 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible) https://gerrit.wikimedia.org/r/c/operations/puppet/+/620529
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Details

SubjectRepoBranchLines +/-
clouddb1020 mac address updateoperations/puppetproduction+1 -1
Add mac addresses to dhcp for clouddb1013-1020operations/puppetproduction+40 -0
mariadb: Allow the installation of clouddb hostsoperations/puppetproduction+23 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedRobHT260441 (Need By: ASAP) rack/setup/install clouddb10[13-20]
 Resolved• BstormT260389 Redesign and rebuild the wikireplicas service using a multi-instance architecture
 Resolved• BstormT260843 Set up roles for new wiki replicas layout
 ResolvedMarosteguiT265135 wikireplicas: Define MW sections per host
 ResolvedMarosteguiT267090 Productionize clouddb10[13-20]
 ResolvedMarosteguiT268312 Deploy labsdbuser and views to new clouddb hosts
 Resolved• BstormT269200 Create end-user accounts on the new clouddb hosts
 Resolved• BstormT269620 maintain-dbusers doesn't close connections right on harvest-replicas
 ResolvedMoritzMuehlenhoffT268725 Include mail on standard_packages.pp
 ResolvedMarosteguiT268742 Test upgrading sanitarium hosts to Buster + 10.4
 ResolvedMarosteguiT272008 Move wikireplicas under the new sanitarium hosts (db1154, db1155)
 Resolved• CmjohnsonT272125 Memory errors on clouddb1019
 ResolveddcaroT272127 2021-01-15: ** PROBLEM alert - labstore1004/Ensure mysql credential creation for tools users is running is CRITICAL **
 ResolvedMarosteguiT280492 Upgrade all sanitarium masters to 10.4 and Buster
 ResolvedRequestwiki_willyT281794 decommission db1082.eqiad.wmnet
 ResolvedRequest• CmjohnsonT281959 decommission db1074.eqiad.wmnet
 ResolvedRequest• CmjohnsonT282079 decommission db1079.eqiad.wmnet
 ResolvedRequest• CmjohnsonT282093 decommission db1087.eqiad.wmnet
 ResolvedRequest• CmjohnsonT282096 decommission db1085.eqiad.wmnet
 DeclinedNoneT267376 Set up IP addresses for the new wiki replicas setup
 Resolved• KormatT268098 Make innodb_change_buffering configurable
 ResolvedArielGlennT261145 Enable access for wmcs-admins to run wmcs-prefixed cookbooks on cumin hosts
 Resolved• BstormT264254 Prepare Quarry for multiinstance wiki replicas
 Declined• BstormT267989 Do some checks of how many Quarry queries will break in a multiinstance environment
 Resolved• BstormT272723 Create a way to sample wikireplicas usage data
 Resolved• BstormT266266 Improve dry-run behavior of wmcs.wikireplicas.add_wiki cookbook
 Resolved• JhernandezT268498 Feedback from Quarry and PAWS users, and other wiki editors affected by the new architecture
 Resolved• razziT269211 Convert labsdb1012 from multi-source to multi-instance
 ResolvedMarosteguiT270473 Ensure InnoDB is compressed on the new clouddb hosts
 ResolvedMilimetricT274690 Update sqoop to work with multi-instance clouddb1021 mariadb host
 Resolved• BstormT281287 Now that labsdb1012 is clouddb1021, connect it to maintain-dbusers
 Resolved• BstormT269399 Set up a way to persist non-default number of wikireplicas connections across all instances
 Resolved• BstormT260511 Parametrize wmf-pt-kill so it can connect to different sockets
 Resolved• BstormT274044 Fix systemd and possibly logrotate around the wmf-pt-kill service for multi-instance wikireplicas
 Resolved• BstormT271476 Iron out issues in the proxy structure for multi-instance wikireplicas
 ResolvedAndrewT272553 Enable the creation of the svc.wikimedia.cloud wikireplicas addresses in Designate
 Resolved• BstormT272720 Allocate service IPs for new wikireplicas setup
 ResolvedayounsiT273248 wikireplicas last-minute infra work to discuss / resolve
 Resolved• JhernandezT272523 Early testing of the new Wiki Replicas multi-instance architecture
 ResolvedMarosteguiT273593 Clean up heartbeat table on clouddb hosts
 ResolvedRagesossT278983 New Replica returns different data than old Replica for query using `change_tag` table
 Resolved• BstormT276284 Establish a working setup for PAWS with multi-instance wikireplicas
 Resolved• BstormT278252 Make alias for tools.db.svc.wikimedia.cloud
 Resolved• BstormT281732 Check into the configuration, cause and usefulness of memory alerts for multiinstance replicas
 ResolvedMarosteguiT266912 Degraded RAID on clouddb1019

Event Timeline

RobH created this task.Aug 14 2020, 3:45 PM
Restricted Application added a project: SRE. · View Herald TranscriptAug 14 2020, 3:45 PM
RobH added a parent task: Unknown Object (Task).Aug 14 2020, 3:45 PM
RobH unsubscribed.
Marostegui updated the task description. (Show Details)Aug 17 2020, 5:53 AM
gerritbot added a comment.Aug 17 2020, 6:03 AM

Change 620529 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Allow the installation of clouddb hosts

https://gerrit.wikimedia.org/r/620529

gerritbot added a project: Patch-For-Review.Aug 17 2020, 6:03 AM
gerritbot added a comment.Aug 18 2020, 7:50 AM

Change 620529 merged by Marostegui:
[operations/puppet@production] mariadb: Allow the installation of clouddb hosts

https://gerrit.wikimedia.org/r/620529

Marostegui added a comment.Aug 18 2020, 8:00 AM

I have merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/620529, so the new hosts will get installed with notifications disabled, RAID10 and spare role.
From  DC-Ops the pending commits are the usual DNS and DHCP entries

@Bstorm once the hosts are installed we can create a different task to talk about the final puppet roles, which I guess it should the role(mariadb::dbstore_multiinstance) which allows multi-instance mysql deployment and take it from there to see what things we need to migrate from the role(labs::db::wikireplica_web) and role(labs::db::wikireplica_analytics) to it.
This can be discussed in a different task.

Marostegui updated the task description. (Show Details)Aug 18 2020, 8:01 AM
Maintenance_bot removed a project: Patch-For-Review.Aug 18 2020, 8:10 AM
• Bstorm added a comment.Aug 19 2020, 4:30 PM
In T260441#6392164, @Marostegui wrote:

@Bstorm once the hosts are installed we can create a different task to talk about the final puppet roles, which I guess it should the role(mariadb::dbstore_multiinstance) which allows multi-instance mysql deployment and take it from there to see what things we need to migrate from the role(labs::db::wikireplica_web) and role(labs::db::wikireplica_analytics) to it.
This can be discussed in a different task.

Sounds good. I'll make the task! That role will probably get us pretty far for the initial install, I imagine.

wiki_willy moved this task from Backlog to Racking Tasks on the ops-eqiad board.Aug 19 2020, 6:11 PM
bd808 moved this task from Backlog to Racking / Decom on the cloud-services-team (Hardware) board.Sep 1 2020, 8:09 PM
• Cmjohnson updated the task description. (Show Details)Oct 1 2020, 6:09 PM

@Marostegui and @Bstorm I will be racking these in the next few days. Can you please review your racking plan and confirm that this it is correct. Thanks!

• Bstorm added a comment.Oct 1 2020, 9:16 PM

Things look right to me, at least. @Marostegui?

Marostegui added a subscriber: LSobanski.Oct 2 2020, 5:59 AM

Great news, thank you @Cmjohnson!
The racking plan looks good to me, we don't have much requirements other than trying to spread them across racks and rows as much as possible and to get them with Buster.
Just as a reminder, I left all the puppet recipes and initial roles (spare) ready on puppet, so from DC-Ops side the only puppet changes needed are the dhcp entries one.

• Cmjohnson added a comment.Oct 23 2020, 6:44 PM

Sorry for the delay, these are in progress. I expect to have them
completed by the end of next week.

• Cmjohnson updated the task description. (Show Details)Oct 26 2020, 2:47 PM
gerritbot added a comment.Oct 28 2020, 4:04 PM

Change 636971 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Add mac addresses to dhcp for clouddb1013-1020

https://gerrit.wikimedia.org/r/636971

gerritbot added a project: Patch-For-Review.Oct 28 2020, 4:04 PM
• Cmjohnson updated the task description. (Show Details)Oct 28 2020, 4:05 PM
gerritbot added a comment.Oct 28 2020, 4:06 PM

Change 636971 merged by Cmjohnson:
[operations/puppet@production] Add mac addresses to dhcp for clouddb1013-1020

https://gerrit.wikimedia.org/r/636971

Maintenance_bot removed a project: Patch-For-Review.Oct 28 2020, 4:10 PM
• Cmjohnson reassigned this task from • Cmjohnson to RobH.Oct 28 2020, 4:14 PM
• Cmjohnson added subscribers: RobH, • Cmjohnson.

@RobH these are ready for install, the raid configuration has been completed. Just need to do the final OS install.

• Bstorm added a subtask: T260389: Redesign and rebuild the wikireplicas service using a multi-instance architecture.Oct 29 2020, 7:19 PM
ops-monitoring-bot added a comment.Oct 30 2020, 10:48 PM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

['clouddb1013.eqiad.wmnet', 'clouddb1014.eqiad.wmnet', 'clouddb1015.eqiad.wmnet', 'clouddb1016.eqiad.wmnet', 'clouddb1017.eqiad.wmnet', 'clouddb1018.eqiad.wmnet', 'clouddb1019.eqiad.wmnet', 'clouddb1020.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202010302248_robh_26788.log.

ops-monitoring-bot added a comment.Oct 30 2020, 11:48 PM

Completed auto-reimage of hosts:

['clouddb1020.eqiad.wmnet']

Of which those FAILED:

['clouddb1020.eqiad.wmnet']
RobH updated the task description. (Show Details)Oct 30 2020, 11:58 PM

All but clouddb1020 are set to staged in netbox, and calling into puppet. I'll investigate whats up with clouddb1020.

Marostegui added a subtask: T266912: Degraded RAID on clouddb1019.Oct 31 2020, 7:22 AM
gerritbot added a comment.Nov 3 2020, 12:19 AM

Change 638205 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] clouddb1020 mac address update

https://gerrit.wikimedia.org/r/638205

gerritbot added a project: Patch-For-Review.Nov 3 2020, 12:19 AM

Change 638205 merged by RobH:
[operations/puppet@production] clouddb1020 mac address update

https://gerrit.wikimedia.org/r/638205

ops-monitoring-bot added a comment.Nov 3 2020, 12:23 AM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

clouddb1020.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202011030023_robh_25838_clouddb1020_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 3 2020, 12:23 AM

Completed auto-reimage of hosts:

['clouddb1020.eqiad.wmnet']

Of which those FAILED:

['clouddb1020.eqiad.wmnet']
ops-monitoring-bot added a comment.Nov 3 2020, 12:26 AM

Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts:

clouddb1020.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202011030026_robh_26206_clouddb1020_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Nov 3 2020, 12:49 AM

Completed auto-reimage of hosts:

['clouddb1020.eqiad.wmnet']

and were ALL successful.

RobH closed this task as Resolved.Nov 3 2020, 12:51 AM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)

all hosts installed, calling into puppet, staged in netbox.

Marostegui added a comment.Nov 3 2020, 6:25 AM

Thanks.
RAID (level and stripe size), memory and CPU looks good.

Marostegui closed subtask T260843: Set up roles for new wiki replicas layout as Resolved.Nov 17 2020, 3:39 PM
elukey mentioned this in T269211: Convert labsdb1012 from multi-source to multi-instance.Mar 4 2021, 6:59 PM
bd808 closed subtask T260389: Redesign and rebuild the wikireplicas service using a multi-instance architecture as Resolved.Oct 29 2021, 10:50 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL