Page MenuHomePhabricator

set min_replicas on database sections in dbctl
Closed, ResolvedPublic

Description

Part of dbctl's functionality is a min_replicas field for each database section, that ensures a minimum number of non-master replicas are always pooled for it -- a config commit command will exit with error if there aren't enough replicas pooled.

Assuming that this is a useful thing to enforce, DBAs should pick values for sections and configure them with e.g. dbctl -s eqiad section s3 edit.

Progress:

Event Timeline

CDanis created this task.Aug 22 2019, 3:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 22 2019, 3:24 PM
Marostegui moved this task from Triage to Next on the DBA board.Aug 26 2019, 8:36 AM
Marostegui added a subscriber: Marostegui.

 @CDanis I understand this is related to replicas for main traffic? as in not on specific groups? (ie: api).

That's right -- this just enforces total number of replicas pooled at the top level of a section.

We could add a min_pooled value for groups if it would be helpful.

That's right -- this just enforces total number of replicas pooled at the top level of a section.
We could add a min_pooled value for groups if it would be helpful.

As far as I know MW, will pick any host if an specific group has no slaves pooled, so that should be fine for now.

Marostegui triaged this task as Medium priority.
Marostegui updated the task description. (Show Details)

I will start taking care of this once eqiad and codfw sections are equal in HW, which is basically once T230106 is fully done.
Some sections have already been done, there so those can be also done here

Mentioned in SAL (#wikimedia-operations) [2019-08-26T14:59:18Z] <marostegui> Change min_replicas to 3 on s5 for eqiad and codfw T231019

s5 done - set it to 3:

root@cumin1001:~# dbctl -s codfw section s5 get
{
    "s5": {
        "master": "db2123",
        "min_replicas": 3,
        "readonly": false,
        "ro_reason": "PLACEHOLDER"
    },
    "tags": "datacenter=codfw"
}
root@cumin1001:~# dbctl -s eqiad section s5 get
{
    "s5": {
        "master": "db1070",
        "min_replicas": 3,
        "readonly": false,
        "ro_reason": "PLACEHOLDER"
    },
    "tags": "datacenter=eqiad"
}
Marostegui updated the task description. (Show Details)Aug 26 2019, 3:00 PM
Marostegui moved this task from Next to In progress on the DBA board.Aug 26 2019, 4:45 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-28T13:18:14Z] <marostegui> Change min_replicas to 3 on s6 for eqiad and codfw T231019

s6 done - set it to 3:

root@cumin1001:~# dbctl -s eqiad section s6 get
{
    "s6": {
        "master": "db1061",
        "min_replicas": 3,
        "readonly": false,
        "ro_reason": "PLACEHOLDER"
    },
    "tags": "datacenter=eqiad"
}
root@cumin1001:~# dbctl -s codfw section s6 get
{
    "s6": {
        "master": "db2129",
        "min_replicas": 3,
        "readonly": false,
        "ro_reason": "PLACEHOLDER"
    },
    "tags": "datacenter=codfw"
}
Marostegui updated the task description. (Show Details)Aug 28 2019, 1:19 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-28T13:20:47Z] <marostegui> Change min_replicas to 4 on s8 for eqiad and codfw T231019

s8 done - set it to 4:

root@cumin1001:~# dbctl -s codfw section s8 get
{
    "s8": {
        "master": "db2079",
        "min_replicas": 4,
        "readonly": false,
        "ro_reason": "PLACEHOLDER"
    },
    "tags": "datacenter=codfw"
}
root@cumin1001:~# dbctl -s eqiad section s8 get
{
    "s8": {
        "master": "db1104",
        "min_replicas": 4,
        "readonly": false,
        "ro_reason": "PLACEHOLDER"
    },
    "tags": "datacenter=eqiad"
}
Marostegui updated the task description. (Show Details)Aug 28 2019, 1:21 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-29T08:30:28Z] <marostegui> Change min_replicas to 2 on s3 for eqiad and codfw T231019

s3 done - set it to 2:

# dbctl -s eqiad section s3 get ; dbctl -s codfw section s3 get
{
    "s3": {
        "master": "db1075",
        "min_replicas": 2,
        "readonly": false,
        "ro_reason": "PLACEHOLDER"
    },
    "tags": "datacenter=eqiad"
}
{
    "s3": {
        "master": "db2105",
        "min_replicas": 2,
        "readonly": false,
        "ro_reason": "PLACEHOLDER"
    },
    "tags": "datacenter=codfw"
}
Marostegui updated the task description. (Show Details)Aug 29 2019, 8:31 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-29T08:36:00Z] <marostegui> Change min_replicas to 4 on s4 for eqiad and codfw T231019

s4 done - set it to 4:

# dbctl -s eqiad section s4 get ; dbctl -s codfw section s4 get
{
    "s4": {
        "master": "db1081",
        "min_replicas": 4,
        "readonly": false,
        "ro_reason": "PLACEHOLDER"
    },
    "tags": "datacenter=eqiad"
}
{
    "s4": {
        "master": "db2090",
        "min_replicas": 4,
        "readonly": false,
        "ro_reason": "PLACEHOLDER"
    },
    "tags": "datacenter=codfw"
}
Marostegui updated the task description. (Show Details)Aug 29 2019, 8:36 AM
Marostegui updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2019-08-29T08:43:45Z] <marostegui> Change min_replicas to 4 on s2 for eqiad and codfw T231019

s2 done - set it to 4:

# dbctl -s eqiad section s2 get ; dbctl -s codfw section s2 get
{
    "s2": {
        "master": "db1066",
        "min_replicas": 4,
        "readonly": false,
        "ro_reason": "PLACEHOLDER"
    },
    "tags": "datacenter=eqiad"
}
{
    "s2": {
        "master": "db2107",
        "min_replicas": 4,
        "readonly": false,
        "ro_reason": "PLACEHOLDER"
    },
    "tags": "datacenter=codfw"
}
Marostegui updated the task description. (Show Details)Aug 29 2019, 8:44 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-03T06:04:42Z] <marostegui> Change min_replicas to 4 on s7 for eqiad and codfw T231019

s7 done - set it to 4:

# dbctl -s eqiad section s7 get ; dbctl -s codfw section s7 get
{
    "s7": {
        "master": "db1062",
        "min_replicas": 4,
        "readonly": false,
        "ro_reason": "PLACEHOLDER"
    },
    "tags": "datacenter=eqiad"
}
{
    "s7": {
        "master": "db2118",
        "min_replicas": 4,
        "readonly": false,
        "ro_reason": "PLACEHOLDER"
    },
    "tags": "datacenter=codfw"
}
Marostegui updated the task description. (Show Details)Sep 3 2019, 6:06 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-03T07:16:48Z] <marostegui> Change min_replicas to 6 on s1 for eqiad and codfw T231019

s1 done - set it to 6

# dbctl -s eqiad section s1 get ; dbctl -s codfw section s1 get
{
    "s1": {
        "master": "db1067",
        "min_replicas": 6,
        "readonly": false,
        "ro_reason": "PLACEHOLDER"
    },
    "tags": "datacenter=eqiad"
}
{
    "s1": {
        "master": "db2048",
        "min_replicas": 6,
        "readonly": false,
        "ro_reason": "PLACEHOLDER"
    },
    "tags": "datacenter=codfw"
}
Marostegui closed this task as Resolved.Sep 3 2019, 7:17 AM
Marostegui updated the task description. (Show Details)

Everything is now done.