Page MenuHomePhabricator
Create Task
Maniphest T190704

Convert all sanitarium hosts to multi-instance and increase its reliability/redundancy
Closed, ResolvedPublic
Actions

Description

db1095 is still in multi-source format, hosting s1, s3, s5 and s8. We may purchase soon an extra host to handle the growing capacity of the different sections. This is the main standalone goal for #DBAs for 2017-2018 Q4 (there could be other more important tasks and goals, but this should have no hard external dependencies, other than hardware provisioning.

  • Design a strategy (do we setup a db1095-equivalent to prevent downtime? Distribution of sections?)
  • Define the purchases needed for the new host(s) and request them (T189590) [Note the decision seems firm, but the purchases have not yet gone through)
  • Setup new hardware needed. If it doesn't arrive on time, maybe use one of the new 8 core host as a temporary measure
    • eqiad T194780
    • codfw T194781
    • Copied to a temporary new host until the new definitive hardware arrives (db1116, db1120, db2075, db2092)
  • Fix sanitarium_multiinstance puppetization https://gerrit.wikimedia.org/r/#/c/425087/
  • Copy data needed for the service
    • eqiad
      • db1124
      • db1125
    • codfw
      • db2094
      • db2095
  • Change codfw sanitariums (db2094 and db2095) to replicate from codfw hosts instead from eqiad ones.
    • db2094
      • s1
      • s3
      • s5
      • s8
    • db2095
      • s2
      • s4
      • s6
      • s7
  • Switchover replicas (labsdb hosts) to use the new hosts
    • labsdb1009
    • labsdb1010
    • labsdb1011

Follow up / clean up tasks:
Clean up sanitarium_multisource related code T196376
Productionize old/temporary eqiad sanitariums T196527
Implement a script to facilitate sanitarium failovers between DCs T196367

Details

SubjectRepoBranchLines +/-
db-eqiad,db.codfw.php: Unify and update sanitarium commentsoperations/mediawiki-configmaster+16 -16
events_sanitarium: Update sanitarium hostsoperations/softwaremaster+1 -1
db-eqiad.php: Depool all sanitarium mastersoperations/mediawiki-configmaster+31 -31
dbproxy1010: Depool labsdb1011operations/puppetproduction+2 -2
dbproxy1010: Depool labsdb1010operations/puppetproduction+2 -2
dbproxy1010: Depool labsdb1010operations/puppetproduction+2 -2
db-eqiad.php: Depool all sanitarium mastersoperations/mediawiki-configmaster+30 -30
redact_sanitarium.sh: Add db1124,db1125operations/puppetproduction+1 -1
db-codfw.php: Repool db2059 and db2075operations/mediawiki-configmaster+3 -3
db-eqiad.php: Depool db1121operations/mediawiki-configmaster+5 -5
mariadb: Move db2075 back to s5operations/puppetproduction+5 -16
db-codfw.php: Repool db2092 and db2062operations/mediawiki-configmaster+4 -4
db-eqiad.php: Depool sanitarium mastersoperations/mediawiki-configmaster+29 -29
install_server: Allow reimage db2092operations/puppetproduction+1 -2
sX.hosts: db2092 is now multiinstanceoperations/softwaremaster+4 -1
mariadb: Move db2092 back to s1operations/puppetproduction+4 -15
mariadb: Convert db2092 to sanitarium multi-instanceoperations/puppetproduction+35 -5
db2094,db2095: Enable notificationsoperations/puppetproduction+0 -2
db-eqiad.php: Depool sanitarium masters for s1,s3,s5,s8operations/mediawiki-configmaster+18 -18
db-eqiad.php: Depool s2,s4,s6,s7 sanitarium mastersoperations/mediawiki-configmaster+11 -11
s*.hosts: Add db2094 to s1,s3,s5,s8operations/softwaremaster+4 -0
check_private_data: Adapt it for the new sanitariumsoperations/puppetproduction+41 -10
db1124,db1125: Enable notificationsoperations/puppetproduction+0 -2
s*.hosts: Add db2095 to s2,s4,s6,s7operations/softwaremaster+4 -0
*.hosts: Add db1125 and db2078operations/softwaremaster+9 -2
s1,3,5,6.hosts: Add db1124operations/softwaremaster+4 -0
mariadb: Make explicit they use ROW based replicationoperations/puppetproduction+8 -0
db-codfw.php: Specifiy sanitarium mastersoperations/mediawiki-configmaster+8 -8
mariadb: Convert db2075 to temp sanitariumoperations/puppetproduction+33 -4
db-codfw.php: Depool db2075operations/mediawiki-configmaster+1 -1
db2092.yaml: Enable notificationsoperations/puppetproduction+0 -1
db-codfw.php: Depool db2092operations/mediawiki-configmaster+1 -1
redact_sanitarium.sh: Remove db1069operations/puppetproduction+2 -2
s8.hosts: Add db1116:3318 to s8operations/softwaremaster+1 -0
s5.hosts: Add db1116:3315 to s5operations/softwaremaster+1 -0
db-eqiad,db-codfw.php: Include db1116:3311 in configoperations/mediawiki-configmaster+3 -0
s1.hosts: Add db1116:3311operations/softwaremaster+1 -0
db-eqiad.php: Depool db1106operations/mediawiki-configmaster+4 -4
mariadb: Convert db1116 as sanitarium multi-instanceoperations/puppetproduction+31 -0
mariadb: migrate sanitarium to role/profile and abstract instancesoperations/puppetproduction+116 -67
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.May 31 2018, 8:31 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

['db2092.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805310831_marostegui_27000.log.

ops-monitoring-bot added a comment.May 31 2018, 8:51 AM

Completed auto-reimage of hosts:

['db2092.codfw.wmnet']

and were ALL successful.

gerritbot added a comment.May 31 2018, 11:09 AM

Change 436506 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool sanitarium masters

https://gerrit.wikimedia.org/r/436506

gerritbot added a comment.May 31 2018, 11:12 AM

Change 436506 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool sanitarium masters

https://gerrit.wikimedia.org/r/436506

Stashbot added a comment.May 31 2018, 11:14 AM

Mentioned in SAL (#wikimedia-operations) [2018-05-31T11:14:22Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool sanitarium masters - T190704 (duration: 01m 22s)

Stashbot added a comment.May 31 2018, 12:30 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-31T12:30:21Z] <marostegui> Stop replication on all sanitarium masters - T190704

Marostegui added a comment.May 31 2018, 1:12 PM

labsdb1009 has been moved under the new sanitarium hosts. We will leave it replicating till Monday before proceeding with labsdb1010

Marostegui updated the task description. (Show Details)May 31 2018, 1:12 PM
Stashbot added a comment.May 31 2018, 1:41 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-31T13:41:44Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool sanitarium masters - T190704 (duration: 01m 21s)

gerritbot added a comment.Jun 1 2018, 5:31 AM

Change 436724 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Repool db2092 and db2062

https://gerrit.wikimedia.org/r/436724

gerritbot added a comment.Jun 1 2018, 5:34 AM

Change 436724 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Repool db2092 and db2062

https://gerrit.wikimedia.org/r/436724

gerritbot added a comment.Jun 1 2018, 5:42 AM

Change 436725 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db2075 back to s5

https://gerrit.wikimedia.org/r/436725

gerritbot added a comment.Jun 1 2018, 5:44 AM

Change 436725 merged by Marostegui:
[operations/puppet@production] mariadb: Move db2075 back to s5

https://gerrit.wikimedia.org/r/436725

ops-monitoring-bot added a comment.Jun 1 2018, 5:45 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

['db2075.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201806010545_marostegui_5586.log.

ops-monitoring-bot added a comment.Jun 1 2018, 6:06 AM

Completed auto-reimage of hosts:

['db2075.codfw.wmnet']

and were ALL successful.

Stashbot added a comment.Jun 1 2018, 6:15 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-01T06:15:38Z] <marostegui> Stop MySQL on db2059 to clone db2075 - T190704

ops-monitoring-bot added a comment.Jun 1 2018, 7:23 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

['db2059.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201806010723_marostegui_26798.log.

ops-monitoring-bot added a comment.Jun 1 2018, 7:46 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

['db2059.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201806010746_marostegui_8297.log.

ops-monitoring-bot added a comment.Jun 1 2018, 8:13 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

['db2059.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201806010813_marostegui_14194.log.

ops-monitoring-bot added a comment.Jun 1 2018, 9:21 AM

Completed auto-reimage of hosts:

['db2059.codfw.wmnet']

Of which those FAILED:

['db2059.codfw.wmnet']
gerritbot added a comment.Jun 4 2018, 5:46 AM

Change 437165 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1121

https://gerrit.wikimedia.org/r/437165

gerritbot added a comment.Jun 4 2018, 5:49 AM

Change 437165 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1121

https://gerrit.wikimedia.org/r/437165

Stashbot added a comment.Jun 4 2018, 5:50 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-04T05:50:34Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1121 - T190704 (duration: 00m 49s)

Stashbot added a comment.Jun 4 2018, 5:52 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-04T05:52:12Z] <marostegui> Stop replication in sync on db1121 and db2051 - T190704

Marostegui added a comment.Jun 4 2018, 5:58 AM

db2095:s4 has been finally moved under db2073 as db2051 (codfw master) already caught up with eqiad and so did db2073 (the sanitarium master for that section)

Marostegui updated the task description. (Show Details)Jun 4 2018, 5:58 AM
Stashbot added a comment.Jun 4 2018, 6:05 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-04T06:05:26Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1121 - T190704 (duration: 00m 49s)

gerritbot added a comment.Jun 4 2018, 6:08 AM

Change 437172 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Repool db2059 and db2075

https://gerrit.wikimedia.org/r/437172

gerritbot added a comment.Jun 4 2018, 6:16 AM

Change 437172 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Repool db2059 and db2075

https://gerrit.wikimedia.org/r/437172

Stashbot added a comment.Jun 4 2018, 6:18 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-04T06:18:41Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Repool db2059, db2075 - T190704 (duration: 00m 49s)

gerritbot added a comment.Jun 4 2018, 7:24 AM

Change 437179 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] redact_sanitarium.sh: Add db1124,db1125

https://gerrit.wikimedia.org/r/437179

gerritbot added a comment.Jun 4 2018, 7:24 AM

Change 437179 merged by Marostegui:
[operations/puppet@production] redact_sanitarium.sh: Add db1124,db1125

https://gerrit.wikimedia.org/r/437179

gerritbot added a comment.Jun 4 2018, 9:36 AM

Change 437204 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1010: Depool labsdb1010

https://gerrit.wikimedia.org/r/437204

gerritbot added a comment.Jun 4 2018, 9:39 AM

Change 437204 merged by Marostegui:
[operations/puppet@production] dbproxy1010: Depool labsdb1010

https://gerrit.wikimedia.org/r/437204

Stashbot added a comment.Jun 4 2018, 9:39 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-04T09:39:34Z] <marostegui> Reload haproxy on dbproxy1010 to depool labsdb1010 - https://phabricator.wikimedia.org/T190704

gerritbot added a comment.Jun 4 2018, 1:00 PM

Change 437235 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool all sanitarium masters

https://gerrit.wikimedia.org/r/437235

Marostegui mentioned this in T196367: Implement a script to facilitate sanitarium failovers between DCs.Jun 4 2018, 1:15 PM
gerritbot added a comment.Jun 4 2018, 2:04 PM

Change 437235 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool all sanitarium masters

https://gerrit.wikimedia.org/r/437235

Stashbot added a comment.Jun 4 2018, 2:05 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-04T14:05:45Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool all sanitariums masters - T190704 (duration: 00m 49s)

Stashbot added a comment.Jun 4 2018, 2:10 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-04T14:09:59Z] <marostegui> Stop replication on all sanitarium masters to move labsdb1010 to another sanitarium host - T190704

Marostegui updated the task description. (Show Details)Jun 4 2018, 2:30 PM

labsdb1010 was switched over to the new sanitarium hosts.

Stashbot added a comment.Jun 4 2018, 2:34 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-04T14:34:05Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool all sanitariums masters - T190704 (duration: 00m 49s)

Stashbot added a comment.Jun 4 2018, 2:34 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-04T14:34:16Z] <marostegui> Reload haproxy on dbproxy1010 to repool labsdb1010 - T190704

Marostegui created subtask T196376: Productionize old/temporary eqiad sanitariums.Jun 4 2018, 3:13 PM
Marostegui updated the task description. (Show Details)
ops-monitoring-bot added a comment.Jun 5 2018, 5:44 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

['db2059.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201806050544_marostegui_25872.log.

ops-monitoring-bot added a comment.Jun 5 2018, 5:57 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

['db2059.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201806050556_marostegui_28049.log.

ops-monitoring-bot added a comment.Jun 5 2018, 6:26 AM

Completed auto-reimage of hosts:

['db2059.codfw.wmnet']

and were ALL successful.

Marostegui changed the status of subtask T196376: Productionize old/temporary eqiad sanitariums from Stalled to Open.Jun 5 2018, 9:35 AM
gerritbot added a comment.Jun 6 2018, 5:21 AM

Change 437670 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1010: Depool labsdb1010

https://gerrit.wikimedia.org/r/437670

gerritbot added a comment.Jun 6 2018, 5:22 AM

Change 437670 merged by Marostegui:
[operations/puppet@production] dbproxy1010: Depool labsdb1010

https://gerrit.wikimedia.org/r/437670

Stashbot added a comment.Jun 6 2018, 5:24 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T05:24:07Z] <marostegui> Reload haproxy on dbproxy1010 to depool labsdb1010 - T190704

Stashbot added a comment.Jun 6 2018, 5:31 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T05:31:56Z] <marostegui> Reload haproxy on dbproxy1010 to repool labsdb1010 - T190704

gerritbot added a comment.Jun 6 2018, 5:44 AM

Change 437674 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1010: Depool labsdb1011

https://gerrit.wikimedia.org/r/437674

gerritbot added a comment.Jun 6 2018, 5:45 AM

Change 437674 merged by Marostegui:
[operations/puppet@production] dbproxy1010: Depool labsdb1011

https://gerrit.wikimedia.org/r/437674

Stashbot added a comment.Jun 6 2018, 5:46 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T05:46:29Z] <marostegui> Reload haproxy on dbproxy1010 to depool labsdb1011 - T190704

gerritbot added a comment.Jun 6 2018, 5:55 AM

Change 437676 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool all sanitarium masters

https://gerrit.wikimedia.org/r/437676

gerritbot added a comment.Jun 6 2018, 6:03 AM

Change 437676 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool all sanitarium masters

https://gerrit.wikimedia.org/r/437676

Stashbot added a comment.Jun 6 2018, 6:04 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T06:04:46Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool all sanitariums masters - T190704 (duration: 01m 09s)

Stashbot added a comment.Jun 6 2018, 7:48 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T07:48:54Z] <marostegui> Stop replication on all sanitarium masters to move labsdb1011 - T190704

Stashbot added a comment.Jun 6 2018, 8:14 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T08:14:57Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool all sanitariums masters - T190704 (duration: 00m 57s)

Stashbot added a comment.Jun 6 2018, 8:25 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T08:25:17Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool all sanitariums masters - T190704 (duration: 00m 56s)

Marostegui added a comment.Jun 6 2018, 8:26 AM

labsdb1011 has been moved over the new sanitarium.
This was the last host to be moved.
Let's wait to make sure it goes fine.

Marostegui updated the task description. (Show Details)Jun 6 2018, 8:26 AM
Stashbot added a comment.Jun 6 2018, 8:29 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T08:29:05Z] <marostegui> Reload haproxy on dbproxy1010 to repool labsdb1011 - T190704

Marostegui mentioned this in T196527: Clean up sanitarium_multisource related code.Jun 6 2018, 8:46 AM
Marostegui changed the status of subtask T196527: Clean up sanitarium_multisource related code from Open to Stalled.
• jcrespo added a comment.Jun 6 2018, 5:39 PM

There is a script operations/software/dbtools/events_sanitarium.sql that should be checked, updated and deployed (?) to all sanitarium hosts.

Marostegui added a comment.Jun 6 2018, 5:57 PM

I do see it is deployed on db1095 and on db1102 on the ops database
It needs some checking, but I guess there is lots of historical reasons involved on those triggers, so I am not sure why they were created and if they are really needed anymore.

Marostegui added a comment.Jun 6 2018, 5:59 PM

It is from 3 years ago...: https://gerrit.wikimedia.org/r/#/q/events_sanitarium.sql

• jcrespo added a comment.Jun 6 2018, 6:03 PM

if they are really needed anymore

They are needed, a different thing is how much changes they need, but cloud users relay on information_schema_p (and so we do so they don't bring down production trying to query information_schema).

• jcrespo added a comment.Jun 6 2018, 6:06 PM

See: T196570

Marostegui added a comment.Jun 6 2018, 6:09 PM

I definitely think we do not need the ops database one on sanitarium hosts, those are probably entries from where this particular instance in db1102 was build and ops database was copied over, pretty much the same for db1095 one:

mysql:root@localhost [ops]> select * from event_log;
+-----------+---------------------+-------------------------------+------------------------------------------------------------------------------------------------------------------------+
| server_id | stamp               | event                         | content                                                                                                                |
+-----------+---------------------+-------------------------------+------------------------------------------------------------------------------------------------------------------------+
|    104822 | 2015-01-12 12:22:33 | wmf_slave_wikiuser_slow       | kill 12225880154; SELECT /* SpecialWhatLinksHere::showIndirectLinks 178.255.215.84 */  page_id,page_namespace,page_tit |
| 171974686 | 2017-07-03 15:00:05 | wmf_slave_wikiuser_sleep      | kill 3804646727                                                                                                        |
| 171974686 | 2017-07-03 16:21:05 | wmf_slave_wikiuser_sleep      | kill 3806982004                                                                                                        |
| 171974686 | 2017-07-04 00:16:33 | wmf_slave_wikiuser_slow (>60) | kill 3819361168; SELECT /* ApiQueryRecentChanges::run  */  rc_id,rc_timestamp,rc_namespace,rc_title,rc_cur_id,rc_type  |
| 171974686 | 2017-07-04 00:16:33 | wmf_slave_wikiuser_slow (>60) | kill 3819361168; SELECT /* ApiQueryRecentChanges::run  */  rc_id,rc_timestamp,rc_namespace,rc_title,rc_cur_id,rc_type  |
| 171974686 | 2017-07-04 00:18:03 | wmf_slave_wikiuser_slow (>60) | kill 3819392154; SELECT /* ApiQueryRecentChanges::run  */  rc_id,rc_timestamp,rc_namespace,rc_title,rc_cur_id,rc_type  |
| 171974686 | 2017-07-04 00:18:03 | wmf_slave_wikiuser_slow (>60) | kill 3819392154; SELECT /* ApiQueryRecentChanges::run  */  rc_id,rc_timestamp,rc_namespace,rc_title,rc_cur_id,rc_type  |
+-----------+---------------------+-------------------------------+------------------------------------------------------------------------------------------------------------------------+
7 rows in set (0.00 sec)

mysql:root@localhost [ops]> show events;
+-----+--------------------------+----------------+-----------+-----------+------------+----------------+----------------+---------------------+------+---------+------------+----------------------+----------------------+--------------------+
| Db  | Name                     | Definer        | Time zone | Type      | Execute at | Interval value | Interval field | Starts              | Ends | Status  | Originator | character_set_client | collation_connection | Database Collation |
+-----+--------------------------+----------------+-----------+-----------+------------+----------------+----------------+---------------------+------+---------+------------+----------------------+----------------------+--------------------+
| ops | wmf_slave_overload       | root@localhost | SYSTEM    | RECURRING | NULL       | 10             | SECOND         | 2017-04-28 00:00:01 | NULL | ENABLED |  171974686 | utf8                 | utf8_general_ci      | binary             |
| ops | wmf_slave_purge          | root@localhost | SYSTEM    | RECURRING | NULL       | 15             | MINUTE         | 2017-04-28 00:00:00 | NULL | ENABLED |  171974686 | utf8                 | utf8_general_ci      | binary             |
| ops | wmf_slave_wikiuser_sleep | root@localhost | SYSTEM    | RECURRING | NULL       | 30             | SECOND         | 2017-04-28 00:00:05 | NULL | ENABLED |  171974686 | utf8                 | utf8_general_ci      | binary             |
| ops | wmf_slave_wikiuser_slow  | root@localhost | SYSTEM    | RECURRING | NULL       | 30             | SECOND         | 2017-04-28 00:00:03 | NULL | ENABLED |  171974686 | utf8                 | utf8_general_ci      | binary             |
+-----+--------------------------+----------------+-----------+-----------+------------+----------------+----------------+---------------------+------+---------+------------+----------------------+----------------------+--------------------+
4 rows in set (0.00 sec)
mysql:root@localhost [ops]> select @@hostname;
+------------+
| @@hostname |
+------------+
| db1095     |
+------------+
1 row in set (0.00 sec)

mysql:root@localhost [ops]> select * from event_log;
+-----------+---------------------+--------------------------+------------------------------------------------------------------------------------------------------------------------+
| server_id | stamp               | event                    | content                                                                                                                |
+-----------+---------------------+--------------------------+------------------------------------------------------------------------------------------------------------------------+
|    101616 | 2014-08-31 10:03:35 | wmf_slave_wikiuser_sleep | kill 6537719837                                                                                                        |
|    101616 | 2014-08-31 10:04:05 | wmf_slave_wikiuser_sleep | kill 6537740876                                                                                                        |
|    101616 | 2014-08-31 10:04:35 | wmf_slave_wikiuser_sleep | kill 6537748124                                                                                                        |
|    101616 | 2014-08-31 18:05:35 | wmf_slave_wikiuser_sleep | kill 6553774149                                                                                                        |
|    101616 | 2014-08-31 20:32:35 | wmf_slave_wikiuser_sleep | kill 6558617326                                                                                                        |
|    101616 | 2014-08-31 20:38:05 | wmf_slave_wikiuser_sleep | kill 6558790024                                                                                                        |
|    101616 | 2014-08-31 20:39:05 | wmf_slave_wikiuser_sleep | kill 6558806160                                                                                                        |
|    101616 | 2014-08-31 23:29:05 | wmf_slave_wikiuser_sleep | kill 6564200174                                                                                                        |
|    101633 | 2015-03-29 09:23:03 | wmf_slave_wikiuser_slow  | kill 10384582902; SELECT /* SpecialWhatLinksHere::showIndirectLinks redacted */  page_id,page_namespace,page_titl |
|    101633 | 2015-03-29 09:23:03 | wmf_slave_wikiuser_slow  | kill 10384582902; SELECT /* SpecialWhatLinksHere::showIndirectLinks redacted */  page_id,page_namespace,page_titl |
|    101633 | 2015-03-29 09:24:03 | wmf_slave_wikiuser_slow  | kill 10384607000; SELECT /* SpecialWhatLinksHere::showIndirectLinks redacted */  page_id,page_namespace,page_titl |
|    101633 | 2015-03-29 09:24:03 | wmf_slave_wikiuser_slow  | kill 10384607000; SELECT /* SpecialWhatLinksHere::showIndirectLinks redacted */  page_id,page_namespace,page_titl |
|    101633 | 2015-03-30 00:49:33 | wmf_slave_wikiuser_slow  | kill 10409920681; SELECT /* SpecialWhatLinksHere::showIndirectLinks redacted ... */  page_id,page_namespace,page |
|    101633 | 2015-03-30 00:49:33 | wmf_slave_wikiuser_slow  | kill 10409920681; SELECT /* SpecialWhatLinksHere::showIndirectLinks redacted ... */  page_id,page_namespace,page |
+-----------+---------------------+--------------------------+------------------------------------------------------------------------------------------------------------------------+
14 rows in set (0.02

We can probably do a clean up on the ones for the ops database and deploy the information_schema ones

Marostegui added a comment.Jun 6 2018, 6:19 PM

Nevermind my comments above. They have nothing to do with the sanitarium events.
The ones on the file are indeed needed. I will deploy them tomorrow.

gerritbot added a comment.Jun 6 2018, 6:26 PM

Change 437802 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] events_sanitarium: Update sanitarium hosts

https://gerrit.wikimedia.org/r/437802

gerritbot added a comment.Jun 7 2018, 5:07 AM

Change 437802 merged by jenkins-bot:
[operations/software@master] events_sanitarium: Update sanitarium hosts

https://gerrit.wikimedia.org/r/437802

Stashbot added a comment.Jun 7 2018, 5:15 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-07T05:15:06Z] <marostegui> Deploy event_sanitarium on codfw sanitariums - T190704

Stashbot added a comment.Jun 7 2018, 6:26 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-07T06:26:26Z] <marostegui> Deploy sanitarium events on db1125 - T190704

Stashbot added a comment.Jun 8 2018, 5:34 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-08T05:34:28Z] <marostegui> Deploy sanitarium events on db1124 - T190704

gerritbot added a comment.Jun 8 2018, 9:11 AM

Change 438215 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db.codfw.php: Unify and update sanitarium comments

https://gerrit.wikimedia.org/r/438215

gerritbot added a comment.Jun 8 2018, 9:16 AM

Change 438215 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db.codfw.php: Unify and update sanitarium comments

https://gerrit.wikimedia.org/r/438215

Stashbot added a comment.Jun 8 2018, 9:18 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-08T09:18:21Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Unify and update sanitarium comments - T190704 (duration: 00m 50s)

Stashbot added a comment.Jun 8 2018, 9:19 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-08T09:19:30Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Unify and update sanitarium comments - T190704 (duration: 00m 50s)

Marostegui closed this task as Resolved.Jun 11 2018, 5:53 AM
Marostegui claimed this task.

Everything has been fine for more than a week now (including the events)
Today I even restarted MySQL on all the sanitariums to pick up a new filter. I think we can consider the scope of the goal now completed.
There are some follow up/clean up tasks which already have their own task to be followed up at (T196527 T196376 T196367)

Marostegui updated the task description. (Show Details)Jun 11 2018, 5:55 AM
Marostegui updated the task description. (Show Details)
Marostegui closed subtask T196527: Clean up sanitarium_multisource related code as Resolved.Jun 26 2018, 5:43 AM
• Vvjjkkii reopened subtask T196527: Clean up sanitarium_multisource related code as Open.Jul 1 2018, 1:05 AM
• Vvjjkkii reopened subtask T194781: rack/setup/install db209[45].codfw.wmnet (sanitarium expansion) as Open.Jul 1 2018, 1:09 AM
• Vvjjkkii reopened subtask T194780: rack/setup/install db112[45].eqiad.wmnet (sanitarium expansion) as Open.
Marostegui closed subtask T194780: rack/setup/install db112[45].eqiad.wmnet (sanitarium expansion) as Resolved.Jul 1 2018, 6:37 PM
Marostegui closed subtask T194781: rack/setup/install db209[45].codfw.wmnet (sanitarium expansion) as Resolved.Jul 1 2018, 6:46 PM
Marostegui closed subtask T196527: Clean up sanitarium_multisource related code as Resolved.Jul 1 2018, 8:13 PM
• jcrespo closed subtask T196376: Productionize old/temporary eqiad sanitariums as Resolved.Sep 28 2018, 2:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits