Page MenuHomePhabricator

Install, configure and provision recently arrived db core machines
Closed, ResolvedPublic

Description

These machines: T131368

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 22 2016, 3:38 PM
jcrespo updated the task description. (Show Details)
jcrespo updated the task description. (Show Details)
jcrespo added a project: Operations.
jcrespo moved this task from Triage to Next on the DBA board.May 12 2016, 2:18 PM
jcrespo added subscribers: Xqt, valhallasw, Andrew and 8 others.

Prioritize db1058's replacement, which its lack is causing replication issues on T135100.

This is still block by DC work.

Although looking at the graphs, not only the current capacity is lower, there was also an unusualy high amount of INSERT and UPDATES.

@jcrespo If it's not a secret, maybe you can list the machines here? The task from the description is not visible ;)

@Luke081515 don't worry, you will be able to see the list when we start working on this ticket. That other ticket is hidden only because it is procurement (contains private communications with 3rd parties).

I just need to plan things myself- it is (hopefuly) 16 new servers, for a (total) of 3 new per shard.

jcrespo claimed this task.May 31 2016, 6:04 PM
jcrespo changed the status of subtask T135253: Rack and set up 16 db's db1079-1094 from Open to Stalled.
jcrespo moved this task from Next to In progress on the DBA board.

I've started installation, reporting issues on T135253.

jcrespo changed the status of subtask T135253: Rack and set up 16 db's db1079-1094 from Stalled to Open.Jun 1 2016, 11:35 AM

OS installed and initial puppetization/salt of:

db1079
db1080
db1081
db1083
db1084
db1087
db1088
db1090
db1091
db1092
db1093

Aside from the other hosts, they still require proper shard puppetization and provisioning.

Change 292197 had a related patch set uploaded (by Volans):
MariaDB: Add new coredb servers

https://gerrit.wikimedia.org/r/292197

Volans added a comment.Jun 1 2016, 6:38 PM

@jcrespo the above CR is according to the plan I've put in T111992, of course review the plan for possible mistakes and feel free to discard and/or split in smaller pieces as needed. I just had it already ready locally ;)

@Volans, while I appreciate the patch, I cannot use it as-is, as many machines have not yet been properly installed due to network/config issues.

I will of course use your plan and the patch iself as the basis of several, smaller patches, though. Thank you!

This is the last updated list:

db1079
db1080
db1081
db1083
db1084
db1087
db1088
db1089
db1090
db1091
db1092
db1093
db1094

Change 292380 had a related patch set uploaded (by Jcrespo):
Puppetize new shard s1 db servers

https://gerrit.wikimedia.org/r/292380

Change 292380 merged by Jcrespo:
Puppetize new shard s1 db servers

https://gerrit.wikimedia.org/r/292380

Change 292545 had a related patch set uploaded (by Jcrespo):
Add extra new core database server to s2 shard

https://gerrit.wikimedia.org/r/292545

Change 292545 merged by Jcrespo:
Add extra new core database server to s2 shard

https://gerrit.wikimedia.org/r/292545

Change 292546 had a related patch set uploaded (by Jcrespo):
Add 3 new s4 shard core db servers

https://gerrit.wikimedia.org/r/292546

Change 292546 merged by Jcrespo:
Add 3 new s4 shard core db servers

https://gerrit.wikimedia.org/r/292546

Change 292562 had a related patch set uploaded (by Jcrespo):
Add new s5 database core servers

https://gerrit.wikimedia.org/r/292562

Change 292562 merged by Jcrespo:
Add new s5 database core servers

https://gerrit.wikimedia.org/r/292562

Change 292564 had a related patch set uploaded (by Jcrespo):
Add new s6 core database servers

https://gerrit.wikimedia.org/r/292564

Change 292564 merged by Jcrespo:
Add new s6 core database servers

https://gerrit.wikimedia.org/r/292564

Change 292197 merged by Jcrespo:
MariaDB: Add new coredb servers

https://gerrit.wikimedia.org/r/292197

Change 293069 had a related patch set uploaded (by Jcrespo):
Depool db1070 for cloning

https://gerrit.wikimedia.org/r/293069

Change 293069 merged by Jcrespo:
Depool db1070 for cloning

https://gerrit.wikimedia.org/r/293069

Change 293082 had a related patch set uploaded (by Jcrespo):
Add new coredb servers to alias configuration

https://gerrit.wikimedia.org/r/293082

Change 293085 had a related patch set uploaded (by Jcrespo):
Pool new db hosts: db1082, db1087, db1092 with low weight

https://gerrit.wikimedia.org/r/293085

Change 293082 merged by Jcrespo:
Add new coredb servers to alias configuration

https://gerrit.wikimedia.org/r/293082

Change 293085 merged by Jcrespo:
Pool new db hosts: db1082, db1087, db1092 with low weight

https://gerrit.wikimedia.org/r/293085

Mentioned in SAL [2016-06-07T13:49:58Z] <jynus> about to pool new dewiki/wikidata servers T133398

Change 294029 had a related patch set uploaded (by Jcrespo):
Increase weight of new s5 database servers

https://gerrit.wikimedia.org/r/294029

Change 294029 merged by Jcrespo:
Increase weight of new s5 database servers

https://gerrit.wikimedia.org/r/294029

Change 294035 had a related patch set uploaded (by Jcrespo):
Depool db1052 for cloning

https://gerrit.wikimedia.org/r/294035

Change 294035 merged by Jcrespo:
Depool db1052 for cloning

https://gerrit.wikimedia.org/r/294035

Change 294253 had a related patch set uploaded (by Jcrespo):
Pool new s1 servers db1080, db1083, db1089

https://gerrit.wikimedia.org/r/294253

Change 294253 merged by Jcrespo:
Pool new s1 servers db1080, db1083, db1089

https://gerrit.wikimedia.org/r/294253

Change 294265 had a related patch set uploaded (by Jcrespo):
Depool db1040 for cloning

https://gerrit.wikimedia.org/r/294265

Change 294265 merged by Jcrespo:
Depool db1040 for cloning

https://gerrit.wikimedia.org/r/294265

Change 294295 had a related patch set uploaded (by Jcrespo):
Repool db1024, pool for the first time db1090 with low weight

https://gerrit.wikimedia.org/r/294295

Change 294295 merged by Jcrespo:
Repool db1024, pool for the first time db1090 with low weight

https://gerrit.wikimedia.org/r/294295

Change 294329 had a related patch set uploaded (by Jcrespo):
Repool db1040, pool for the first time db1081, db1084, db1091

https://gerrit.wikimedia.org/r/294329

Change 294329 merged by Jcrespo:
Repool db1040, pool for the first time db1081, db1084, db1091

https://gerrit.wikimedia.org/r/294329

Change 294439 had a related patch set uploaded (by Jcrespo):
Depool db1033 for cloning

https://gerrit.wikimedia.org/r/294439

Change 294439 merged by Jcrespo:
Depool db1033 for cloning

https://gerrit.wikimedia.org/r/294439

Change 294459 had a related patch set uploaded (by Jcrespo):
Repool db1033, first pool of db1079, db1086, db1094 with low weight

https://gerrit.wikimedia.org/r/294459

Change 294459 merged by Jcrespo:
Repool db1033, first pool of db1079, db1086, db1094 with low weight

https://gerrit.wikimedia.org/r/294459

Change 294470 had a related patch set uploaded (by Jcrespo):
Increase new enwiki dbs weight, depool db1023 for cloning

https://gerrit.wikimedia.org/r/294470

Change 294470 merged by Jcrespo:
Increase new enwiki dbs weight, depool db1023 for cloning

https://gerrit.wikimedia.org/r/294470

Change 294502 had a related patch set uploaded (by Jcrespo):
Repool db1023; pool db1085 (disabled), db1088, db1092 w/low weight

https://gerrit.wikimedia.org/r/294502

Change 294502 merged by Jcrespo:
Repool db1023; pool db1085 (disabled), db1088, db1092 w/low weight

https://gerrit.wikimedia.org/r/294502

Change 294672 had a related patch set uploaded (by Jcrespo):
Pool db1085, increase weight of all new db servers

https://gerrit.wikimedia.org/r/294672

Change 294672 merged by Jcrespo:
Pool db1085, increase weight of all new db servers

https://gerrit.wikimedia.org/r/294672

jcrespo closed this task as Resolved.Jun 16 2016, 8:46 AM

All 16 new servers (21 in total, 3 per shard) are pooled into production- we will do some adjustments over the following weeks of the weights, but that is something in constant change anyway.