Let's add the misc section in orchestrator.
This requires first:
- Adding the grants to all the hosts.
- Cleaning up the heartbeat table T268336
- Enable report_host on all the hosts T266483
Progress:
- m1
- m2
- m3
- m5
- es4
- es5
Let's add the misc section in orchestrator.
This requires first:
Progress:
Subject | Repo | Branch | Lines +/- | |
---|---|---|---|---|
db-eqiad.php: Depool es4 from writes | operations/mediawiki-config | master | +1 -1 |
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Restricted Task | |||||
Resolved | Legoktm | T281515 June 2021 Datacenter switchover | |||
Resolved | • Marostegui | T285139 DB maintenance work to do while eqiad is passive (June 2021) | |||
Resolved | • Marostegui | T266483 Enable report_host for mariadb | |||
Resolved | • Marostegui | T272568 Add m* and es4/es5 sections to Orchestrator | |||
Resolved | • Marostegui | T271540 Upgrade and restart m1 master (db1080) | |||
Resolved | • Marostegui | T272596 Restart m3 (phabricator) database master db1132 | |||
Resolved | • Marostegui | T272388 Restart m5 master (db1128) | |||
Resolved | • Marostegui | T272964 Restart m2 database master (db1107) |
We need to decide what to do with hosts on es1, es2 and es3. They do not have replication enabled so we need to decide if we want them in orchestrator as standalone or not.
At the moment they are on tendril but not on the tree view (cause there is no tree to represent).
My personal opinion is to include them on orchestrator even if they are not replicating from each other - @Kormat thoughts?
Sounds good, I will clean up the heartbeat table there. It needs re-creation with the newer one schema, as the "shard" section doesn't exist on that one. Once that is done, hopefully orchestrator will group them by the same alias, but just not draw their replication relationship.
So looks like that even if the "alias" query works for all the three hosts, just the first one that gets discovered is the one that gets placed on es1 cluster, the rest are just their own cluster, which is not ideal as we'd have 18 hosts (3 hosts per dc on es1, es2 and es3) as their own clusters which can be a bit messy on the UI.
Going to forget those instances until we decide a work around for this.
There is also the fact that they will always show as lagged as pt-heartbeat entry doesn't run there...if they are downtimed forerever then the lag won't show up, but still not ideal either.
Standalone hosts aren't really thought to be on orchestrator.
Change 660772 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool es4 from writes
Change 660772 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool es4 from writes