I noticed today that racktables has only been installed on miscweb1002. it appears that the main racktables install still has some manual steps which got missed.
As this installations was missing i had to add a work around which should be removed once racktables is installed on all machines
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| racktables: remove work around for missing install in codfw | operations/puppet | production | +6 -9 |
Mentioned in SAL (#wikimedia-operations) [2021-08-27T12:49:56Z] <mutante> rsynced /srv/org/wikimedia/racktables from miscweb1002 to miscweb2002 (T269746)
Change 715233 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] racktables: remove work around for missing install in codfw
I rsynced /srv/org/wikimedia/racktables over from eqiad to codfw
and then ran puppet.
This means the formerly missing file is there now but puppet adjusted
the mysql host to the codfw master.
I confirmed I could connect to the codfw DB master, racktables DB
and it has content. I assume this is readonly anyways and can't go wrong
to connect to the db from both DCs.
@Kormat @Marostegui Per Jaime's comment on https://gerrit.wikimedia.org/r/c/operations/puppet/+/715233 I am pinging you guys on the ticket to let you know about changed "misc db usage/expectations".
racktables used to be only in eqiad but now it exists in both eqiad and codfw. (miscweb1002, miscweb2002)
Puppet is configured to use "m1-master.eqiad.wmnet" in eqiad and "m1-master.codfw.wmnet" in codfw. User/pass/db name are the same.
I confirmed manually with mysql that I could connect to m1-master.codfw from miscweb2002 to db "racktables" and I could see content.
I assumed nothing can go wrong this way because racktables should be readonly either way.
But I also noticed that I could use the same password to connect to m1-master.eqiad also from the codfw backend. Maybe this should not work?
Change 715233 merged by Dzahn:
[operations/puppet@production] racktables: remove work around for missing install in codfw
Thanks for the ping - yes, codfw is read-only. We have no plans to have multi-master on misc any time in the future, so if there's something trying to write to codfw, it will fail (and might generate log traces).
Keep in mind that we've never switched misc services when doing DC switches, but in the future I do want to include that as part as a datacenter switchover, in that case eqiad will become read-only and codfw will become writable. Will something be needed from racktables side?
I am going to hold the +1 on the gerrit until we are sure about the above question just in case.
But I also noticed that I could use the same password to connect to m1-master.eqiad also from the codfw backend. Maybe this should not work?
They do share the same password, however there should not be any cross-dc queries and if they cannot be avoided by any means, they do need to be done via SSL.
N.B. This implies you can't use the the m1-master CNAME, or the dbproxy it points to. (Thanks, mysql protocol :( ). You'd have to know the hostname of the actual master, and connect directly to it.
Thanks for the replies, DBAs!
racktables should be readonly and never change again. regardless of data center. it just needs to stay around as-is for reading. nothing should try to write to it, and if it would try then it would be good if that fails.
So if I understand right we don't need to do anything else and can close this.