terbium is a maintenance script server and backend for dbtree
wasat is supposed to be the equivalent of terbium in codfw but not yet a working backend for dbtree
it was removed as one in T162976#3186233 https://gerrit.wikimedia.org/r/#/c/348456/1/hieradata/role/common/cache/misc.yaml
after it was added when dbtree was made active/active in https://gerrit.wikimedia.org/r/#/c/346572/
make wasat a working backend and add it back above once we can.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Joe | T154658 Prepare and improve the datacenter switchover procedure | |||
| Resolved | LSobanski | T156937 Provide cross-dc redundancy (active-active or active-passive) to all important misc services | |||
| Resolved | BBlack | T162976 dbtree broken (for some users?) | |||
| Declined | None | T163141 dbtree: make wasat a working backend and become active-active | |||
| Resolved | jcrespo | T149557 Site: 2 VM request for tendril (switch tendril from einsteinium to dbmonitor*) | |||
| Resolved | Dzahn | T162183 tendril cert expiry alerts on dbmonitor hosts |
not sure if this should be tagged as traffic or not. please feel free to remove it. it just got auto-added because it copies tags when you create something as a subtask
Yeah, leave the traffic tag as we'll want to basically revert https://gerrit.wikimedia.org/r/#/c/348456/ once dbtree is ready for it.
status update:
nowadays terbium and wasat use the identical role and profile in site.pp, as in:
2600 # mediawiki maintenance servers (https://wikitech.wikimedia.org/wiki/Terbium)
2601 node 'terbium.eqiad.wmnet', 'wasat.codfw.wmnet' {
2602 role(mediawiki_maintenance)So both are getting the dbtree Apache site installed. Both have /etc/apache2/sites-enabled/50-dbtree-wikimedia-org.conf.
Yet, only terbium actually works, as in:
[bast1001:~] $ curl -v --silent -H "Host:dbtree.wikimedia.org" terbium.eqiad.wmnet 2>&1| grep "<title>"
<title>Core Databases</title>
[bast1001:~] $ curl -v --silent -H "Host:dbtree.wikimedia.org" wasat.codfw.wmnet 2>&1| grep "<title>"
[bast1001:~] $This is what is to debug next.
reason: database connection to tendril on tendril-backend.eqiad.wmnet failed
^ It should probably not try something in .eqiad., needs "tendril-backend.codfw.wmnet", right @jynus
Mentioned in SAL (#wikimedia-operations) [2017-05-18T20:47:03Z] <mutante> wasat - git pull - bring to latest, the last changed had never been deployed here like on terbium, but it's also not a backend for dbtree yet (T163141)
We should not enable active-active on dbtree (or enable it failing, as it is the current case). Dbtree database backend is db1011, which is only on eqiad. This year the plan is to setup a second node on codfw, which would allow local queries only.
Meanwhile, enabling it would mean cross-dc traffic which we should not allow for privacy reasons.
Once a backend exist locally on both db, we can put both dbmonitor2 and wasat as active-active (where no privacy or performance concerns happen). Alternatively, if we are in a hurry (I don't think we are), we can enable TLS (which we should probably do anyway).
This is stalled because tendril cannot work with multiple db backends. We would need to setup a different backend to support it- which we wanted to do anyway (even if we reuse many of the existing schema), but it is not trivial.
Once we have a working replication, we could make it active-active as it is a read-only functionality.
Closing this as we won't be really working on this anymore, but on deprecating tendril in favour of something else.
Replication won't be working anyways, tendril has too many writes per second for the slave in codfw to keep up with.
We do have a host in codfw, as a backup in case the one in eqiad breaks, so we'd need to manually switch dbmonitor to point to the codfw DB.