Create es4 and es5 sections in dbctl
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Marostegui
	Feb 21 2020, 7:23 AM

Description

We are in process of setting up es4 and es5 as new external store sections. Once those are online, es2 and es3 will become read-only like es1.
The servers are almost ready and this is the layout:

es4:

eqiad
es1020 - master
es1021 - slave
es1022 - slave

codfw (which replicates from eqiad)
es2020 - master
es2021 - slave
es2022 - slave

es5:

eqiad
es1023 - master
es1024 - slave
es1025 - slave

codfw (which replicates from eqiad)
es2023 - master
es2024 - slave
es2025 - slave

The idea is to get the sections in place in dbctl and later, and once we are ready, do the changes in mediawiki config so they can started being used, which I think it can be done in separate steps.
I believe setting up new sections in dbctl is independent from making them usable in mediawiki, right?
This is the first time we set up a new section in dbctl, can you help us here @CDanis?

Details

Project	Branch	Lines +/-	Subject
operations/software/conftool	master	+2 -2	dbctl: schema: allow es4 and es5 sections
operations/mediawiki-config	master	+2 -0	etcd.php: Add es4 and es5
operations/mediawiki-config	master	+2 -0	etcd.php: support es4/es5 sections
operations/puppet	production	+2 -0	dbctl: create sections es4/es5 in eqiad
operations/puppet	production	+2 -0	dbctl: create sections es4/es5 in codfw
operations/puppet	production	+2 -2	dbctl: schema: allow es4 and es5 sections

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Marostegui	T226704 Setup es4 and es5 replica sets for new read-write external store service
Resolved	Marostegui	T243052 Productionize es1020-es1025, es2020-es2025
Resolved	CDanis	T245806 Create es4 and es5 sections in dbctl

Event Timeline

Marostegui triaged this task as Medium priority.Feb 21 2020, 7:23 AM

Marostegui created this task.

Marostegui moved this task from Triage to Pending comment on the DBA board.

CDanis claimed this task.Feb 24 2020, 5:41 AM

Change 574268 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] dbctl: schema: allow es4 and es5 sections

https://gerrit.wikimedia.org/r/574268

Change 574269 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] dbctl: create sections s4/s5 in codfw

https://gerrit.wikimedia.org/r/574269

Change 574270 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] dbctl: create sections s4/s5 in eqiad

https://gerrit.wikimedia.org/r/574270

Change 574272 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/software/conftool@master] dbctl: schema: allow es4 and es5 sections

https://gerrit.wikimedia.org/r/574272

Change 574268 merged by CDanis:
[operations/puppet@production] dbctl: schema: allow es4 and es5 sections

https://gerrit.wikimedia.org/r/574268

Change 574269 merged by CDanis:
[operations/puppet@production] dbctl: create sections es4/es5 in codfw

https://gerrit.wikimedia.org/r/574269

Change 574270 merged by CDanis:
[operations/puppet@production] dbctl: create sections es4/es5 in eqiad

https://gerrit.wikimedia.org/r/574270

Mentioned in SAL (#wikimedia-operations) [2020-02-24T07:29:14Z] <cdanis> dbctl: edit es4/es5 sections in codfw (flavor & master fields) T245806

Mentioned in SAL (#wikimedia-operations) [2020-02-24T07:30:13Z] <cdanis> dbctl: (and min_replicas field) T245806

Mentioned in SAL (#wikimedia-operations) [2020-02-24T07:31:03Z] <cdanis> dbctl: edit es4/es5 sections in eqiad (flavor & master & min_replicas fields) T245806

This should be ready to go:

✔️ cdanis@cumin2001.codfw.wmnet ~ 🕞🍵 dbctl section all get | grep 'es[45]'
{"es4": {"readonly": false, "min_replicas": 1, "flavor": "external", "master": "es2020", "ro_reason": "PLACEHOLDER"}, "tags": "datacenter=codfw"}
{"es5": {"readonly": false, "min_replicas": 1, "flavor": "external", "master": "es2023", "ro_reason": "PLACEHOLDER"}, "tags": "datacenter=codfw"}
{"es4": {"readonly": false, "min_replicas": 1, "flavor": "external", "master": "es1020", "ro_reason": "PLACEHOLDER"}, "tags": "datacenter=eqiad"}
{"es5": {"readonly": false, "min_replicas": 1, "flavor": "external", "master": "es1023", "ro_reason": "PLACEHOLDER"}, "tags": "datacenter=eqiad"}

I see you created the instance objects already, so all that's left is filling in their details/associating them with the section as appropriate.

And then we need some mediawiki-config changes in etcd.php (see $externalStoreNameMap there), and also I think in db-{eqiad,codfw}.php (I think 'templateOverridesByCluster' and possibly also $wgDefaultExternalStore? Very likely you know better than I, or at least I hope ;)

Thank you @CDanis - I was planning on editing all the instances now, I guess it is safe to do so and also create them as "pooled", as nothing really uses, them

And then we need some mediawiki-config changes in etcd.php (see $externalStoreNameMap there), and also I think in db-{eqiad,codfw}.php (I think 'templateOverridesByCluster' and possibly also $wgDefaultExternalStore? Very likely you know better than I, or at least I hope ;)

This is probably for CPT to do/advise, as from our side most likely all we know is about db-eqiad.php/db-codfw.php flags are templateOverridesByCluster and $wgDefaultExternalStore indeed

Once we are fully done from dbctl side, I will create a more specific task for CPT, which should be the last step of this process.

In T245806#5911112, @Marostegui wrote:

Thank you @CDanis - I was planning on editing all the instances now, I guess it is safe to do so and also create them as "pooled", as nothing really uses, them

Yeah, AIUI, that should be fine.

Mentioned in SAL (#wikimedia-operations) [2020-02-24T08:01:29Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Add instances to es4 codfw - T245806', diff saved to https://phabricator.wikimedia.org/P10489 and previous config saved to /var/cache/conftool/dbconfig/20200224-080128-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-02-24T08:07:09Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Add instances to es4 eqiad - T245806', diff saved to https://phabricator.wikimedia.org/P10490 and previous config saved to /var/cache/conftool/dbconfig/20200224-080708-marostegui.json

Change 574377 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/mediawiki-config@master] etcd.php: support es4/es5 sections

https://gerrit.wikimedia.org/r/574377

Change 574378 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] etcd.php: Add es4 and es5

https://gerrit.wikimedia.org/r/574378

Mentioned in SAL (#wikimedia-operations) [2020-02-24T08:28:48Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Add instances to es5 codfw - T245806', diff saved to https://phabricator.wikimedia.org/P10491 and previous config saved to /var/cache/conftool/dbconfig/20200224-082848-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-02-24T08:29:22Z] <marostegui> Temporary put es1020 (es4) and es1023 (es5) on RO on a mysql level - T245806

Change 574377 abandoned by CDanis:
etcd.php: support es4/es5 sections

Reason:
superceded by I7215d1977

https://gerrit.wikimedia.org/r/574377

Change 574378 merged by jenkins-bot:
[operations/mediawiki-config@master] etcd.php: Add es4 and es5

https://gerrit.wikimedia.org/r/574378

Mentioned in SAL (#wikimedia-operations) [2020-02-24T08:33:58Z] <marostegui@deploy1001> Synchronized wmf-config/etcd.php: Add es4 and es5 (unused new external store sections to etcd - T245806 (duration: 00m 58s)

Mentioned in SAL (#wikimedia-operations) [2020-02-24T08:40:28Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Add instances to es5 eqiad - T245806', diff saved to https://phabricator.wikimedia.org/P10492 and previous config saved to /var/cache/conftool/dbconfig/20200224-084027-marostegui.json

All the hosts have been added to es4 and es5, both, eqiad and codfw:
P10492
P10491
P10490
P10489

I have set es1020 (es4 master) and es1023 (es5 master) as read only on MySQL for a few hours, to make sure nothing tries to write there, if there is something, an error will be thrown and we can investigate.
In a few hours, if all looks good, I will set them back to RW and close this.

Thank you @CDanis for your help.

Nothing on fatals or logstash in general about trying to insert (and failing) on es4 and es5, so I am going to remove RO from es4 and es5 master.

Mentioned in SAL (#wikimedia-operations) [2020-02-24T14:59:07Z] <marostegui> read_only=0 on es1020 (es4) and es1023 (es5) - unused new external store masters - T245806

for i in es1020 es1023; do echo $i; mysql.py -h$i -e "select @@read_only"; done
es1020
+-------------+
| @@read_only |
+-------------+
|           0 |
+-------------+
es1023
+-------------+
| @@read_only |
+-------------+
|           0 |
+-------------+

Marostegui mentioned this in T246072: Enable es4 and es5 as writable new external store sections and set es2 and es3 as read only.Feb 25 2020, 8:50 AM

Change 574272 merged by jenkins-bot:
[operations/software/conftool@master] dbctl: schema: allow es4 and es5 sections

https://gerrit.wikimedia.org/r/574272

CDanis mentioned this in rOSCT22e501a92829: dbctl: schema: allow es4 and es5 sections.Mar 24 2020, 1:57 PM

Maintenance_bot removed a project: Patch-For-Review.Mar 24 2020, 2:11 PM