Unlike most other edge sites, eqsin and ulsfo have 6 machines per cluster, not 8. ulsfo sees much less traffic, so this is fine there; however, eqsin has become a sizable fraction of our overall traffic at its daily peak: approx as loaded as eqiad's peak; approx 1/3rd of esams peak; and approx 1/3rd of global rps at eqsin daily peak.
We should probably increase from 6->8 machines for upload and text in eqsin.
| Project | Branch | Lines +/- | Subject | |
|---|---|---|---|---|
| operations/puppet | production | +4 -0 | Add missing cache::nodes for cp501[3456] | |
| operations/puppet | production | +28 -8 | Puppetize cp501[3456] |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T274888 cp_upload @ eqsin cascading failures, February 2021 | |||
| Resolved | BBlack | T275046 provision more machines for eqsin caches | |||
| Unknown Object (Task) | |||||
| Unknown Object (Task) | |||||
| Resolved | RobH | T278182 (Need By: TBD) rack/setup/install cp501[3-6] |
Change 683026 had a related patch set uploaded (by BBlack; author: BBlack):
[operations/puppet@production] Puppetize cp501[3456]
Script wmf-auto-reimage was launched by bblack on cumin1001.eqiad.wmnet for hosts:
['cp5013.eqsin.wmnet', 'cp5014.eqsin.wmnet', 'cp5015.eqsin.wmnet', 'cp5016.eqsin.wmnet']
The log can be found in /var/log/wmf-auto-reimage/202105042216_bblack_1599.log.
Completed auto-reimage of hosts:
['cp5015.eqsin.wmnet', 'cp5013.eqsin.wmnet', 'cp5014.eqsin.wmnet', 'cp5016.eqsin.wmnet']
and were ALL successful.
These are just about ready and running correct puppetization, but don't pool these yet. I think they may have some bad BIOS settings or something, at least related to power mgmt. cpufreq keeps attempting to reset the governor on every puppet run. Will check tomorrow.
I checked the BIOS/iDRAC settings on cp5013 against https://wikitech.wikimedia.org/wiki/Platform-specific_documentation/Dell_Documentation#Initial_System_Setup (+ the one custom setting we use on these modern cps, which is to disable the unused onboard NICs), and ended up making these 3 changes to bring it into conformance:
| BIOS Section | Setting | Old Value | New Value |
|---|---|---|---|
| Integrated Devices | Embedded NIC1 and NIC2 | Enabled | Disabled (OS) |
| System Profile Settings | System Profile | Performance Per Watt (DAPC) | Performance Per Watt (OS) |
| Miscellaneous | Assert Tag | (Blank) | (Value from netbox) |
The cpufreq issue is gone and the extra eno[12] interfaces vanished as well, so I think we're good. Will check the other three shortly, assuming they'll all need the same changes.
The others were in the same state. All are fixed and rebooted now, icinga downtimes are removed, netbox status is set to Active, and confctl weights are set correctly, but the pooled attribute is still set to inactive.
Will begin pooling these into service today while eqsin is in its daily load valley.
These are all pooled now and slowly filling their caches. Optimistically closing this task for now!
Change 691170 had a related patch set uploaded (by BBlack; author: BBlack):
[operations/puppet@production] Add missing cache::nodes for cp501[3456]
Change 691170 merged by BBlack:
[operations/puppet@production] Add missing cache::nodes for cp501[3456]