Page MenuHomePhabricator
Create Task
Maniphest T178359

Support multi-instance on core hosts
Closed, ResolvedPublic
Actions

Description

We currently have dbstore servers supporting multi-instance with systemd running on Debian Stretch

We need support for multi-instance on the core servers now.
To do that we have some paths that we could follow:

Jessie + 10.0 + support for init.d with mysqld_multi
Jessie + 10.1
Stretch + 10.1 (only on those hosts).

Some work has already been done and db2084 is being used as a testing host: https://gerrit.wikimedia.org/r/#/c/384452/

The idea is to use 8 hosts to serve all the recentchanges services across all the core shards.
Splitting them across them so we can have redundancy:

codfw:

[] `db2092 (B8): will not be done, because we don't need special replicas on s3 https://gerrit.wikimedia.org/r/394573

eqiad

Pending things:

  • Fully pool db1098:3316 and db1098:3317 (to be finished today 7th)
  • Finish compressing tables on db1099:3311 after reimporting a few of them
  • Pool db1099:3311
  • Compress db1099:3318
  • Compress db1096:3316
    • To be done once db1098:3316 is fully pooled

Details

ProjectBranchLines +/-Subject
operations/mediawiki-configmaster+6 -6db-eqiad.php: Depool db1096:3316
operations/mediawiki-configmaster+6 -6db-eqiad.php: Depool db1055
operations/mediawiki-configmaster+1 -1db-eqiad.php: Fully pool db1099:3311
operations/mediawiki-configmaster+3 -8db-eqiad.php: Repool db1093 as main traffic
operations/mediawiki-configmaster+1 -1db-eqiad.php: Increase traffic for db1099:3311
operations/mediawiki-configmaster+1 -1db-eqiad.php: Increase traffic for db1099:3311
operations/mediawiki-configmaster+6 -0db-eqiad.php: Pool db1099:3311
operations/mediawiki-configmaster+2 -2db-eqiad.php: Fully pool db1098:331{6,7}
operations/mediawiki-configmaster+16 -3db-eqiad,db-codfw.php: Pool db1098:331{6,7}
operations/puppetproduction+0 -1db1098.yaml: Enable notifications
operations/softwaremaster+1 -0s7.hosts: Add db1098:3317
operations/mediawiki-configmaster+6 -6db-eqiad.php: Depool db1034
operations/softwaremaster+1 -1s6.hosts: db1098 will replicate on 3316
operations/mediawiki-configmaster+6 -6mariadb: Pool db2085:3311 (s1) after being moved from db2092
operations/puppetproduction+12 -4mariadb: Convert db1098 to multiinstance
operations/mediawiki-configmaster+2 -2db-eqiad.php: Fully pool db1096:3316
operations/mediawiki-configmaster+0 -6db-eqiad.php: Remove db1099 from s5
operations/puppetproduction+0 -1mariadb: Reenable notifications on db2085 after s1 reimport
operations/mediawiki-configmaster+4 -4db-eqiad.php: Increase traffic for db1096:331{5,6}
operations/puppetproduction+1 -1install_server: Allow reimage of db1098
operations/mediawiki-configmaster+12 -4db-eqiad,db-codfw.php: Pool db1096:3315
operations/softwaremaster+1 -0s6.hosts: Add db1096:3316
operations/mediawiki-configmaster+7 -7db-eqiad.php: Depool db1098
operations/softwaremaster+1 -3Remove db2092; setup db1085 with s1 and s8
operations/puppetproduction+4 -9Set db2085 with s1 and s8; make db2092 pending to be deployed
operations/mediawiki-configmaster+11 -45mariadb: Undeploy db2092, use db2085 for s1 (remove s3 special slaves)
operations/puppetproduction+9 -6mariadb: Convert db1096 to multi-instance: s5,s6
operations/mediawiki-configmaster+6 -6db-eqiad.php: Depool db1096
operations/mediawiki-configmaster+3 -3db-eqiad.php: Pool db1099 with low traffic
operations/puppetproduction+4 -2mariadb: Notifications for db1099,reimage db1096
operations/mediawiki-configmaster+10 -8db-eqiad,db-codfw.php: Add db1099:3311 and 3318
operations/mediawiki-configmaster+3 -3db-eqiad.php: Repool db1055
operations/softwaremaster+1 -0s1.hosts: Add db1099:3311
operations/mediawiki-configmaster+6 -6db-eqiad.php: Depool db1055
operations/softwaremaster+1 -1s5.hosts: db1099 now replicates 3318
operations/puppetproduction+9 -5mariadb: Convert db1099 to multiinstance
operations/puppetproduction+4 -1mariadb: Reimage db1099
operations/mediawiki-configmaster+1 -1db-eqiad.php: Fully pool db1097:3314
operations/mediawiki-configmaster+2 -2db-eqiad.php: Increase weight for db1097:3314
operations/mediawiki-configmaster+2 -2db-eqiad.php: Pool db1097 in s4
operations/mediawiki-configmaster+3 -3db-eqiad.php: Increase traffic for db1101:3318
operations/mediawiki-configmaster+16 -4db-eqiad,db-codfw.php: Pool db1097 in s5
operations/mediawiki-configmaster+9 -9db-eqiad.php: Repool db1101:3318 with low weight
operations/puppetproduction+0 -1db1097.yaml: Enable notifications
operations/softwaremaster+1 -0s5.hosts: Add db1097:3315
operations/softwaremaster+1 -1s5.hosts: Update db1101 port
operations/softwaremaster+1 -1s4.hosts: db1097 is now a rc slave in s4 and s5
operations/mediawiki-configmaster+6 -6db-eqiad.php: Depool db1053
operations/puppetproduction+11 -4mariadb: Convert db1097 to multi-instance
operations/mediawiki-configmaster+2 -2db-eqiad.php: Start building db1097 as rc in s4,s5
operations/mediawiki-configmaster+1 -1db-eqiad.php: Fully pool db1101 in s7
operations/mediawiki-configmaster+2 -2db-eqiad.php: Increase weight on db1101.s7
operations/puppetproduction+0 -1db1101.yaml: Enable notifications
operations/mediawiki-configmaster+16 -8db-eqiad,db-codfw.php: Add db1101 as rc to s5,s7
operations/mediawiki-configmaster+2 -2db-eqiad.php: Repool db1096 with low weight
operations/mediawiki-configmaster+6 -6db-eqiad.php: Depool db1096
operations/mediawiki-configmaster+6 -5db-eqiad.php: Depool db1097
operations/mediawiki-configmaster+7 -7db-codfw.php: Repool db2038
operations/softwaremaster+2 -1db1101: Moved it from s2 to s5 and s7
operations/mediawiki-configmaster+9 -9db-eqiad.php: Depool db1034
operations/puppetproduction+8 -4mariadb: Convert db1101 to multi-instance
operations/puppetproduction+1 -0db1101.yaml: Disable notifications
operations/puppetproduction+2 -0install_server: Reimage db1101 as stretch
operations/puppetproduction+0 -4mariadb: Enable notifications in a few hosts
operations/mediawiki-configmaster+7 -7db-eqiad.php: Depool db1101
operations/mediawiki-configmaster+2 -2db-eqiad.php: Fully pool db1105 in s1 and s2
operations/mediawiki-configmaster+2 -2db-eqiad.php: Increase traffic for db1105
operations/mediawiki-configmaster+4 -4db-eqiad.php: Increase traffic for db1105
operations/mediawiki-configmaster+2 -2db-eqiad.php: Give db1105 more traffic
operations/mediawiki-configmaster+20 -7db-eqiad,db-codfw.php: Pool db1105 as rc for s1,s2
operations/puppetproduction+0 -1db1105.yaml: Enable notifications
operations/mediawiki-configmaster+4 -4db-eqiad.php: Increase weight for db1103
operations/mediawiki-configmaster+5 -5db-eqiad.php: Increase db1103 traffic
operations/mediawiki-configmaster+3 -3db-eqiad.php: Increase traffic for db1103
operations/mediawiki-configmaster+19 -6db-eqiad,db-codfw.php: Pool db1103 as rc for s2,s4
operations/puppetproduction+0 -1db1103.yaml: Enable notifications
operations/mediawiki-configmaster+2 -2db-eqiad.php: Restore db1055 original weight
operations/mediawiki-configmaster+2 -2db-eqiad.php: Repool db1055 with low weight
operations/mediawiki-configmaster+6 -6db-eqiad.php: Depool db1055
operations/softwaremaster+2 -1s1,s2,s5.hosts: Move db1105 to s1 and s2
operations/mediawiki-configmaster+6 -6db-eqiad.php: Depool db1051
operations/puppetproduction+8 -4mariadb: db1105 to become multiinstance for s1,s2
operations/mediawiki-configmaster+16 -0db-eqiad,db-codfw.php: Pool db2092 in s1 and s3
operations/mediawiki-configmaster+16 -0db-eqiad,db-codfw.php: Pool db2091 in s2 and s4
operations/mediawiki-configmaster+16 -4db-eqiad,db-codfw.php: Pool db2086 in s5 and s7
operations/mediawiki-configmaster+16 -0db-eqiad,db-codfw.php: Pool db2089 in s6 and s5
operations/softwaremaster+1 -0s2.hosts: Add db1103
operations/mediawiki-configmaster+16 -0db-eqiad,db-codfw.php: Add db2087 to s6 and s7
operations/puppetproduction+0 -1db2087.yaml: Enable notifications
operations/mediawiki-configmaster+2 -2db-eqiad.php: Restore db1101 original weight
operations/mediawiki-configmaster+16 -2db-eqiad,db-codfw.php: Pool db2088 in s1 and s2
operations/puppetproduction+1 -3mariadb: Enable notifications on db2085 and db2088
operations/mediawiki-configmaster+16 -0db-eqiad,db-codfw.php: Pool db2085 in s3 and s5
operations/mediawiki-configmaster+2 -1db-eqiad.php: Add db2084:3314 and db2084:3315
operations/mediawiki-configmaster+7 -7db-eqiad.php: Depool db1101
operations/softwaremaster+1 -1s4.hosts: db1103 s4 instance listen on 3314
operations/puppetproduction+11 -3mariadb: Convert db1103 into multinstance: s2 s4
operations/softwaremaster+2 -0s3,s5.hosts: Add db2085 to s3 and s5
operations/puppetproduction+2 -0install_server: Reimage db1103 as stretch
operations/softwaremaster+2 -0s5,s6.hosts: Add db2089
operations/puppetproduction+8 -1mariadb: Add db2085 to s3 and s5
operations/puppetproduction+2 -0install_server: Reimage db2085 with stretch
operations/puppetproduction+0 -1db2084.yaml: Enable notifications
operations/mediawiki-configmaster+14 -3db-codfw.php: Pool db2084 as multi-instance host
operations/puppetproduction+8 -5mariadb: Add db2089 to s6 and s5(s8)
operations/softwaremaster+2 -0s6,s7.hosts: Add db2087
operations/mediawiki-configmaster+7 -7db-codfw.php: Depool db2040
operations/puppetproduction+8 -1mariadb: Add db2087 to s6 and s7
operations/puppetproduction+2 -0install_server: Reimage db2087 as stretch
operations/puppetproduction+8 -4mariadb: Convert db2086 to multi-instance
operations/softwaremaster+2 -1s5,s7.hosts: Add db2086 to s5 and s7
operations/mediawiki-configmaster+6 -2db-codfw.php: Add db2088 and db2084 to the config
operations/puppetproduction+2 -0install_server: Reimage db2086 with stretch
operations/mediawiki-configmaster+7 -7db-codfw.php: Repool db2038
operations/softwaremaster+2 -0s2,s4.hosts: Add db2091 to s2 and s4
operations/puppetproduction+11 -0mariadb: Add db2091 to s2 and s4
operations/mediawiki-configmaster+7 -7db-codfw.php: Depool db2035
operations/softwaremaster+5 -3s1,s2.hosts: Add db2088 to s1 and s2
operations/puppetproduction+11 -0mariadb: Add db2088 to s1 and s2
operations/mediawiki-configmaster+13 -13db-codfw.php: Depool db2034 and db2036
operations/softwaremaster+2 -0s1,s3.hosts: Add db2092 to s1 and s3
operations/puppetproduction+14 -1mariadb: Add db2092 to s1 and s3 as rc
operations/softwaremaster+1 -0s4.hosts: Add db2084 to s4
operations/puppetproduction+2 -1mysql-core_codfw.yaml: Add db2084 to s4 and s5
operations/softwaremaster+1 -1s5.hosts: Change db2084 port to 3315
operations/puppetproduction+2 -0install_server: Reinstall db2084 with stretch
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
jcrespo added a comment.Dec 1 2017, 6:14 PM

So the summary of changes: db2085 (s3) and db2092 (s3) have disappeared (data has not been pysically deleted, but mysql is no longer running). db2092(s1) has been moved to db2085 (again, data has not been deleted, only server is shutdown. That means no special s3 multi-instance hosts, and db2078(m1), db2090 (empty) and db2092 (with garbage) can be dedicated to misc hosts?

gerritbot added a comment.Dec 1 2017, 6:18 PM

Change 394622 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Reenable notifications on db2085 after s1 reimport

https://gerrit.wikimedia.org/r/394622

Stashbot added a comment.Dec 2 2017, 5:55 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-02T17:55:52Z] <marostegui> Reboot db1096.s5 to pick up the correct innodb_buffer_pool size after finishing compressing s5 - T178359

gerritbot added a comment.Dec 4 2017, 6:27 AM

Change 394916 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1098

https://gerrit.wikimedia.org/r/394916

gerritbot added a comment.Dec 4 2017, 6:32 AM

Change 394916 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1098

https://gerrit.wikimedia.org/r/394916

Stashbot added a comment.Dec 4 2017, 6:34 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-04T06:34:03Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1098 - T178359 (duration: 00m 46s)

Stashbot added a comment.Dec 4 2017, 6:40 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-04T06:40:24Z] <marostegui> Stop MySQL on db1098 to clone db1096.s6 - T178359

Stashbot added a comment.Dec 4 2017, 7:17 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-04T07:17:24Z] <marostegui> Compress s1 on db1099 - T178359

Marostegui added a comment.Dec 4 2017, 7:44 AM

db1096.s6 is now replicating

Marostegui updated the task description. (Show Details)Dec 4 2017, 7:45 AM
gerritbot added a comment.Dec 4 2017, 7:47 AM

Change 394925 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s6.hosts: Add db1096:3316

https://gerrit.wikimedia.org/r/394925

gerritbot added a comment.Dec 4 2017, 7:49 AM

Change 394925 merged by jenkins-bot:
[operations/software@master] s6.hosts: Add db1096:3316

https://gerrit.wikimedia.org/r/394925

gerritbot added a comment.Dec 4 2017, 7:52 AM

Change 394926 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db1096:3315

https://gerrit.wikimedia.org/r/394926

gerritbot added a comment.Dec 4 2017, 8:09 AM

Change 394926 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db1096:3315

https://gerrit.wikimedia.org/r/394926

Stashbot added a comment.Dec 4 2017, 8:11 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-04T08:11:31Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Pool db1096:3315 - T178359 (duration: 00m 45s)

gerritbot added a comment.Dec 4 2017, 8:12 AM

Change 394928 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Allow reimage of db1098

https://gerrit.wikimedia.org/r/394928

Stashbot added a comment.Dec 4 2017, 8:12 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-04T08:12:22Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Pool db1096:3315 - T178359 (duration: 00m 44s)

gerritbot added a comment.Dec 4 2017, 8:15 AM

Change 394928 merged by Marostegui:
[operations/puppet@production] install_server: Allow reimage of db1098

https://gerrit.wikimedia.org/r/394928

gerritbot added a comment.Dec 4 2017, 8:21 AM

Change 394929 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Increase traffic for db1096:331{5,6}

https://gerrit.wikimedia.org/r/394929

Marostegui updated the task description. (Show Details)Dec 4 2017, 8:21 AM
gerritbot added a comment.Dec 4 2017, 8:29 AM

Change 394929 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Increase traffic for db1096:331{5,6}

https://gerrit.wikimedia.org/r/394929

Stashbot added a comment.Dec 4 2017, 8:30 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-04T08:30:31Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3315 and pool db1096:3316 - T178359 (duration: 00m 45s)

Stashbot added a comment.Dec 4 2017, 8:45 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-04T08:45:54Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3316 - T178359 (duration: 00m 45s)

Stashbot added a comment.Dec 4 2017, 8:57 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-04T08:57:50Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1096:3315 and 3316 - T178359 (duration: 00m 45s)

gerritbot added a comment.Dec 4 2017, 9:02 AM

Change 394944 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Remove db1099 from s5

https://gerrit.wikimedia.org/r/394944

gerritbot added a comment.Dec 4 2017, 9:02 AM

Change 394622 merged by Jcrespo:
[operations/puppet@production] mariadb: Reenable notifications on db2085 after s1 reimport

https://gerrit.wikimedia.org/r/394622

gerritbot added a comment.Dec 4 2017, 9:54 AM

Change 394944 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Remove db1099 from s5

https://gerrit.wikimedia.org/r/394944

gerritbot added a comment.Dec 4 2017, 10:00 AM

Change 394954 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Fully pool db1096:3316

https://gerrit.wikimedia.org/r/394954

gerritbot added a comment.Dec 4 2017, 10:03 AM

Change 394955 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Conver db1098 to multiinstance

https://gerrit.wikimedia.org/r/394955

gerritbot added a comment.Dec 4 2017, 10:05 AM

Change 394954 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Fully pool db1096:3316

https://gerrit.wikimedia.org/r/394954

gerritbot added a comment.Dec 4 2017, 10:19 AM

Change 394955 merged by Marostegui:
[operations/puppet@production] mariadb: Convert db1098 to multiinstance

https://gerrit.wikimedia.org/r/394955

ops-monitoring-bot added a comment.Dec 4 2017, 10:22 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

db1098.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201712041022_marostegui_22618_db1098_eqiad_wmnet.log.

gerritbot added a comment.Dec 4 2017, 10:35 AM

Change 394615 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Pool db2085:3311 (s1) after being moved from db2092

https://gerrit.wikimedia.org/r/394615

ops-monitoring-bot added a comment.Dec 4 2017, 10:40 AM

Completed auto-reimage of hosts:

['db1098.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Dec 4 2017, 10:53 AM

Change 394965 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s6.hosts: db1098 will replicate on 3316

https://gerrit.wikimedia.org/r/394965

gerritbot added a comment.Dec 4 2017, 10:56 AM

Change 394965 merged by jenkins-bot:
[operations/software@master] s6.hosts: db1098 will replicate on 3316

https://gerrit.wikimedia.org/r/394965

Marostegui added a comment.Dec 4 2017, 1:05 PM

db1098.s6 is now replicating.

Stashbot added a comment.Dec 4 2017, 1:05 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-04T13:05:52Z] <marostegui> Compress s6 on db1098 - T178359

gerritbot added a comment.Dec 5 2017, 6:50 AM

Change 395183 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1034

https://gerrit.wikimedia.org/r/395183

gerritbot added a comment.Dec 5 2017, 6:54 AM

Change 395183 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1034

https://gerrit.wikimedia.org/r/395183

Stashbot added a comment.Dec 5 2017, 6:55 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-05T06:55:35Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1034 - T178359! (duration: 00m 43s)

Stashbot added a comment.Dec 5 2017, 6:58 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-05T06:58:47Z] <marostegui> Stop MySQL on db1034 to clone db1098:3317 - T178359

Marostegui added a comment.Dec 5 2017, 9:16 AM

db1098:3317 is now replicating

Stashbot added a comment.Dec 5 2017, 9:20 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-05T09:20:59Z] <marostegui> Optimize s7 on db1098 - T178359

Marostegui added a comment.Dec 5 2017, 9:21 AM
In T178359#3811722, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2017-12-05T09:20:59Z] <marostegui> Optimize s7 on db1098 - T178359

s/optimize/compress

Stashbot added a comment.Dec 5 2017, 9:49 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-05T09:49:02Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1034 with low weight - T178359! (duration: 00m 43s)

Stashbot added a comment.Dec 5 2017, 10:05 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-05T10:05:55Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1034 - T178359! (duration: 00m 43s)

Stashbot added a comment.Dec 5 2017, 11:53 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-05T11:53:52Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1034 - T178359! (duration: 00m 43s)

gerritbot added a comment.Dec 6 2017, 7:00 AM

Change 395693 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s7.hosts: Add db1098:3317

https://gerrit.wikimedia.org/r/395693

gerritbot added a comment.Dec 6 2017, 7:04 AM

Change 395693 merged by jenkins-bot:
[operations/software@master] s7.hosts: Add db1098:3317

https://gerrit.wikimedia.org/r/395693

gerritbot added a comment.Dec 7 2017, 6:43 AM

Change 395928 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Pool db1099:3311

https://gerrit.wikimedia.org/r/395928

Stashbot added a comment.Dec 7 2017, 6:45 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-07T06:45:48Z] <marostegui> Stop replication on db1099:3311 to reimport: change_tag, tag_summary, user and watchlist tables and recompress again - T178359

gerritbot added a comment.Dec 7 2017, 6:54 AM

Change 395929 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db1098:331{6,7}

https://gerrit.wikimedia.org/r/395929

gerritbot added a comment.Dec 7 2017, 7:06 AM

Change 395930 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1098.yaml: Enable notifications

https://gerrit.wikimedia.org/r/395930

gerritbot added a comment.Dec 7 2017, 7:07 AM

Change 395930 merged by Marostegui:
[operations/puppet@production] db1098.yaml: Enable notifications

https://gerrit.wikimedia.org/r/395930

Marostegui updated the task description. (Show Details)Dec 7 2017, 7:16 AM
gerritbot added a comment.Dec 7 2017, 7:17 AM

Change 395929 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Pool db1098:331{6,7}

https://gerrit.wikimedia.org/r/395929

Stashbot added a comment.Dec 7 2017, 7:18 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-07T07:18:55Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Slowly pool db1098:3316 db1098:3317 - T178359 (duration: 00m 48s)

Stashbot added a comment.Dec 7 2017, 7:19 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-07T07:19:49Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Slowly pool db1098:3316 db1098:3317 - T178359 (duration: 00m 47s)

Marostegui added a comment.Dec 7 2017, 7:21 AM

db1098:3316 and 3317 starting to get pooled.

Pending things:

  • Fully pool db1098:3316 and db1098:3317 (to be finished today 7th)
    • Finished compressing tables on db1099:3311 after reimporting a few of them
      • pool db1099:3311
    • Compress db1096:3316
      • To be done once db1098:3316 is fully pooled
Stashbot added a comment.Dec 7 2017, 8:32 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-07T08:32:26Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098:3316 db1098:3317 - T178359 (duration: 00m 48s)

Marostegui updated the task description. (Show Details)Dec 7 2017, 8:44 AM
Stashbot added a comment.Dec 7 2017, 8:51 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-07T08:51:25Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098:3316 db1098:3317 - T178359 (duration: 00m 48s)

Stashbot added a comment.Dec 7 2017, 9:36 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-07T09:36:14Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1098:3316 db1098:3317 - T178359 (duration: 00m 52s)

gerritbot added a comment.Dec 7 2017, 10:10 AM

Change 395963 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Fully pool db1098:331{6,7}

https://gerrit.wikimedia.org/r/395963

gerritbot added a comment.Dec 7 2017, 10:13 AM

Change 395963 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Fully pool db1098:331{6,7}

https://gerrit.wikimedia.org/r/395963

Stashbot added a comment.Dec 7 2017, 10:14 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-07T10:14:34Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Fully pool db1098:3316 db1098:3317 - T178359 (duration: 00m 51s)

Marostegui updated the task description. (Show Details)Dec 7 2017, 10:15 AM
Marostegui updated the task description. (Show Details)Dec 7 2017, 11:32 AM
Stashbot added a comment.Dec 7 2017, 11:35 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-07T11:35:12Z] <marostegui> Compress s8 on db1099 - T178359

gerritbot added a comment.Dec 7 2017, 2:52 PM

Change 395928 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Pool db1099:3311

https://gerrit.wikimedia.org/r/395928

Stashbot added a comment.Dec 7 2017, 2:55 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-07T14:55:19Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Pool db1099:331 with low weight - T178359 (duration: 00m 47s)

gerritbot added a comment.Dec 7 2017, 3:18 PM

Change 396022 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Increase traffic for db1099:3311

https://gerrit.wikimedia.org/r/396022

gerritbot added a comment.Dec 7 2017, 3:23 PM

Change 396022 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Increase traffic for db1099:3311

https://gerrit.wikimedia.org/r/396022

Stashbot added a comment.Dec 7 2017, 3:24 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-07T15:24:39Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase db1099:3311 weight - T178359 (duration: 00m 48s)

gerritbot added a comment.Dec 7 2017, 3:39 PM

Change 396026 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Increase traffic for db1099:3311

https://gerrit.wikimedia.org/r/396026

gerritbot added a comment.Dec 7 2017, 3:43 PM

Change 396026 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Increase traffic for db1099:3311

https://gerrit.wikimedia.org/r/396026

Stashbot added a comment.Dec 7 2017, 3:44 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-07T15:44:39Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase db1099:3311 weight - T178359 (duration: 00m 48s)

gerritbot added a comment.Dec 7 2017, 3:54 PM

Change 396030 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1093 as main traffic

https://gerrit.wikimedia.org/r/396030

gerritbot added a comment.Dec 7 2017, 4:00 PM

Change 396030 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1093 as main traffic

https://gerrit.wikimedia.org/r/396030

Stashbot added a comment.Dec 7 2017, 4:01 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-07T16:01:51Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1093 back as main traffic in s6 - T178359 (duration: 00m 48s)

Stashbot added a comment.Dec 7 2017, 4:14 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-07T16:14:00Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase db1099:3311 weight - T178359 (duration: 00m 48s)

Marostegui updated the task description. (Show Details)Dec 7 2017, 4:23 PM
gerritbot added a comment.Dec 8 2017, 6:36 AM

Change 396304 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Fully pool db1099:3311

https://gerrit.wikimedia.org/r/396304

gerritbot added a comment.Dec 8 2017, 6:41 AM

Change 396304 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Fully pool db1099:3311

https://gerrit.wikimedia.org/r/396304

Stashbot added a comment.Dec 8 2017, 6:43 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-08T06:43:33Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Fully pool db1099:3311 - T178359 (duration: 00m 55s)

Marostegui updated the task description. (Show Details)Dec 11 2017, 6:13 AM
gerritbot added a comment.Dec 11 2017, 6:15 AM

Change 397232 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1096:3316

https://gerrit.wikimedia.org/r/397232

gerritbot added a comment.Dec 11 2017, 6:19 AM

Change 397232 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1096:3316

https://gerrit.wikimedia.org/r/397232

Stashbot added a comment.Dec 11 2017, 6:21 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-11T06:21:08Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 to compress InnoDB there - T178359 (duration: 00m 45s)

Stashbot added a comment.Dec 11 2017, 6:22 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-11T06:22:12Z] <marostegui> Compress s6 on db1096 - T178359

gerritbot added a comment.Dec 12 2017, 6:41 AM

Change 397739 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1055

https://gerrit.wikimedia.org/r/397739

gerritbot added a comment.Dec 12 2017, 6:47 AM

Change 397739 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1055

https://gerrit.wikimedia.org/r/397739

Stashbot added a comment.Dec 12 2017, 6:48 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-12T06:48:56Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1055 - T178359 T182653 (duration: 00m 56s)

Stashbot mentioned this in T182653: Power supply error on db1055.Dec 12 2017, 6:48 AM
Stashbot added a comment.Dec 12 2017, 9:23 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-12T09:23:10Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1096:3316 after InnoDB there - T178359 (duration: 00m 56s)

Marostegui closed this task as Resolved.Dec 12 2017, 9:23 AM
Marostegui updated the task description. (Show Details)
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:42 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL