Page MenuHomePhabricator
Create Task
Maniphest T355424

Productionize es[2035-2040]
Closed, ResolvedPublic
Actions

Description

These will be the next two RW external store.
es4 and es5 will go RO once this ticket is done.

es6

  • es2035 master
  • es2036
  • es2037
  • Pending dbctl configuration
  • etcd.php changes
  • Set candidate masters

es7

  • es2038 master
  • es2039
  • es2040
  • Pending dbctl configuration
  • etcd.php changes
  • Set candidate masters

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedVolansT355343 Q3:rack/setup/install es[2035-2040]
 ResolvedMarosteguiT364446 Enable writes on es6 and es7
 ResolvedMarosteguiT355424 Productionize es[2035-2040]

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 19 2024, 1:39 PM
Marostegui changed the task status from Open to Stalled.Jan 19 2024, 1:39 PM
Marostegui triaged this task as Medium priority.
Marostegui moved this task from Triage to Blocked on the DBA board.
Marostegui changed the task status from Stalled to Open.Mar 5 2024, 4:55 PM
Marostegui moved this task from Blocked to Ready on the DBA board.
[16:55:08] marostegui@cumin1002:~$ sudo cumin es20[35-40].codfw.wmnet 'lvextend -L+1T /dev/mapper/tank-data ; xfs_growfs /srv/ ; df -hT /srv'
6 hosts will be targeted:
es[2035-2040].codfw.wmnet
OK to proceed on 6 hosts? Enter the number of affected hosts to confirm or "q" to quit: 6
===== NODE GROUP =====
(6) es[2035-2040].codfw.wmnet
----- OUTPUT of 'lvextend -L+1T /...v/ ; df -hT /srv' -----
  Size of logical volume tank/data changed from 9.09 TiB (2384188 extents) to 10.09 TiB (2646332 extents).
  Logical volume tank/data successfully resized.
meta-data=/dev/mapper/tank-data  isize=512    agcount=32, agsize=76294016 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=0
data     =                       bsize=4096   blocks=2441408512, imaxpct=5
         =                       sunit=64     swidth=256 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 2441408512 to 2709843968
Filesystem            Type  Size  Used Avail Use% Mounted on
/dev/mapper/tank-data xfs    11T   73G   11T   1% /srv
Marostegui moved this task from Ready to In progress on the DBA board.Mon, Apr 29, 10:08 AM
gerritbot added a comment.Mon, Apr 29, 10:17 AM

Change #1025289 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize es6

https://gerrit.wikimedia.org/r/1025289

gerritbot added a project: Patch-For-Review.Mon, Apr 29, 10:17 AM

Change #1025289 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize es6

https://gerrit.wikimedia.org/r/1025289

gerritbot added a comment.Mon, Apr 29, 10:22 AM

Change #1025291 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] valid_section.pp: Add es6 and es7

https://gerrit.wikimedia.org/r/1025291

gerritbot added a comment.Mon, Apr 29, 10:26 AM

Change #1025291 merged by Marostegui:

[operations/puppet@production] valid_section.pp: Add es6 and es7

https://gerrit.wikimedia.org/r/1025291

Maintenance_bot removed a project: Patch-For-Review.Mon, Apr 29, 10:31 AM
Marostegui added a comment.Mon, Apr 29, 10:42 AM

es6 hosts added to zarcillo.
Also added es6 and es7 as valid sections

Marostegui updated the task description. (Show Details)Mon, Apr 29, 2:04 PM
gerritbot added a comment.Mon, Apr 29, 2:09 PM

Change #1025370 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es2035: Make it es6 codfw master

https://gerrit.wikimedia.org/r/1025370

gerritbot added a project: Patch-For-Review.Mon, Apr 29, 2:09 PM
gerritbot added a comment.Mon, Apr 29, 2:12 PM

Change #1025370 merged by Marostegui:

[operations/puppet@production] es2035: Make it es6 codfw master

https://gerrit.wikimedia.org/r/1025370

Marostegui added a comment.Mon, Apr 29, 2:17 PM

es6 codfw is now showing up in orchestrator.

Maintenance_bot removed a project: Patch-For-Review.Mon, Apr 29, 2:31 PM
gerritbot added a comment.Tue, Apr 30, 7:06 AM

Change #1025603 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] sections.yaml: Add es6 as valid dbctl section

https://gerrit.wikimedia.org/r/1025603

gerritbot added a project: Patch-For-Review.Tue, Apr 30, 7:06 AM

Change #1025603 merged by Marostegui:

[operations/puppet@production] sections.yaml: Add es6 as valid dbctl section

https://gerrit.wikimedia.org/r/1025603

gerritbot added a comment.Tue, Apr 30, 7:26 AM

Change #1025668 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add es6 hosts

https://gerrit.wikimedia.org/r/1025668

gerritbot added a comment.Tue, Apr 30, 7:27 AM

Change #1025668 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add es6 hosts

https://gerrit.wikimedia.org/r/1025668

Maintenance_bot removed a project: Patch-For-Review.Tue, Apr 30, 7:31 AM
gerritbot added a comment.Tue, Apr 30, 7:37 AM

Change #1025670 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/mediawiki-config@master] etcd.php: Add es6

https://gerrit.wikimedia.org/r/1025670

gerritbot added a project: Patch-For-Review.Tue, Apr 30, 7:37 AM
Marostegui added a comment.Tue, Apr 30, 7:53 AM

es7 hosts added to zarcillo

gerritbot added a comment.Tue, Apr 30, 7:55 AM

Change #1025672 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Remove comments from es2035, es2036

https://gerrit.wikimedia.org/r/1025672

gerritbot added a comment.Tue, Apr 30, 7:56 AM

Change #1025672 merged by Marostegui:

[operations/puppet@production] mariadb: Remove comments from es2035, es2036

https://gerrit.wikimedia.org/r/1025672

Marostegui updated the task description. (Show Details)Tue, Apr 30, 8:34 AM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Tue, Apr 30, 9:14 AM

Change #1025670 merged by jenkins-bot:

[operations/mediawiki-config@master] etcd.php: Add es6

https://gerrit.wikimedia.org/r/1025670

Stashbot added a comment.Tue, Apr 30, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2024-04-30T09:14:59Z] <marostegui@deploy1002> Started scap: Backport for [[gerrit:1025670|etcd.php: Add es6 (T355285 T355424)]]

Stashbot mentioned this in T355285: Productionize es10[35-40].Tue, Apr 30, 9:15 AM
Stashbot added a comment.Tue, Apr 30, 9:17 AM

Mentioned in SAL (#wikimedia-operations) [2024-04-30T09:17:48Z] <marostegui@deploy1002> marostegui: Backport for [[gerrit:1025670|etcd.php: Add es6 (T355285 T355424)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Tue, Apr 30, 9:30 AM

Mentioned in SAL (#wikimedia-operations) [2024-04-30T09:30:01Z] <marostegui@deploy1002> Finished scap: Backport for [[gerrit:1025670|etcd.php: Add es6 (T355285 T355424)]] (duration: 15m 01s)

Maintenance_bot removed a project: Patch-For-Review.Tue, Apr 30, 9:30 AM
gerritbot added a comment.Tue, Apr 30, 9:36 AM

Change #1025699 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] profile: Add es6 to the regex of valid sections

https://gerrit.wikimedia.org/r/1025699

gerritbot added a project: Patch-For-Review.Tue, Apr 30, 9:36 AM
gerritbot added a comment.Tue, Apr 30, 9:39 AM

Change #1025699 merged by Marostegui:

[operations/puppet@production] profile: Add es6 to the regex of valid sections

https://gerrit.wikimedia.org/r/1025699

Stashbot added a comment.Tue, Apr 30, 9:51 AM

Mentioned in SAL (#wikimedia-operations) [2024-04-30T09:51:19Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Push es6 codfw config T355424', diff saved to https://phabricator.wikimedia.org/P61487 and previous config saved to /var/cache/conftool/dbconfig/20240430-095119-marostegui.json

Marostegui updated the task description. (Show Details)Tue, Apr 30, 9:54 AM
Maintenance_bot removed a project: Patch-For-Review.Tue, Apr 30, 10:31 AM
gerritbot added a comment.Tue, Apr 30, 10:34 AM

Change #1025713 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es203[57]: Enable notifications

https://gerrit.wikimedia.org/r/1025713

gerritbot added a project: Patch-For-Review.Tue, Apr 30, 10:34 AM

Change #1025713 merged by Marostegui:

[operations/puppet@production] es203[57]: Enable notifications

https://gerrit.wikimedia.org/r/1025713

Maintenance_bot removed a project: Patch-For-Review.Tue, Apr 30, 11:31 AM
Stashbot added a comment.Wed, May 1, 5:33 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-01T05:33:34Z] <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on 6 hosts with reason: Setting up T355285 T355424

Stashbot added a comment.Wed, May 1, 5:33 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-01T05:33:51Z] <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on 6 hosts with reason: Setting up T355285 T355424

Stashbot added a comment.Wed, May 1, 5:33 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-01T05:33:56Z] <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on es[1035,1039-1040].eqiad.wmnet with reason: Setting up T355285 T355424

Stashbot added a comment.Wed, May 1, 5:34 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-01T05:34:11Z] <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on es[1035,1039-1040].eqiad.wmnet with reason: Setting up T355285 T355424

gerritbot added a comment.Wed, May 1, 5:38 AM

Change #1025899 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize codfw es7 servers

https://gerrit.wikimedia.org/r/1025899

gerritbot added a project: Patch-For-Review.Wed, May 1, 5:38 AM

Change #1025899 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize codfw es7 servers

https://gerrit.wikimedia.org/r/1025899

gerritbot added a comment.Wed, May 1, 5:51 AM

Change #1025900 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es2038: Make it es7 master

https://gerrit.wikimedia.org/r/1025900

gerritbot added a comment.Wed, May 1, 5:51 AM

Change #1025900 merged by Marostegui:

[operations/puppet@production] es2038: Make it es7 master

https://gerrit.wikimedia.org/r/1025900

gerritbot added a comment.Wed, May 1, 6:01 AM

Change #1025902 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] conftool: Add es7 as valid section

https://gerrit.wikimedia.org/r/1025902

gerritbot added a comment.Wed, May 1, 6:06 AM

Change #1025902 merged by Marostegui:

[operations/puppet@production] conftool: Add es7 as valid section

https://gerrit.wikimedia.org/r/1025902

Maintenance_bot removed a project: Patch-For-Review.Wed, May 1, 6:30 AM
Marostegui added a comment.Wed, May 1, 7:21 AM

es7 codfw now in orchestrator

Marostegui added a comment.Wed, May 1, 7:24 AM

dbctl section (not config per host) created for es7

Marostegui updated the task description. (Show Details)Wed, May 1, 8:00 AM
Marostegui updated the task description. (Show Details)Wed, May 1, 8:39 AM
gerritbot added a comment.Wed, May 1, 9:09 AM

Change #1026092 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add es7 codfw hosts

https://gerrit.wikimedia.org/r/1026092

gerritbot added a project: Patch-For-Review.Wed, May 1, 9:09 AM

Change #1026092 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add es7 codfw hosts

https://gerrit.wikimedia.org/r/1026092

Stashbot added a comment.Wed, May 1, 9:13 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-01T09:13:52Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Push es7 codfw config T355424', diff saved to https://phabricator.wikimedia.org/P61561 and previous config saved to /var/cache/conftool/dbconfig/20240501-091352-marostegui.json

Marostegui added a comment.Wed, May 1, 9:14 AM

Pushed es7 dbctl config - tables aren't created

P61561 dbctl commit (dc=all): 'Push es7 codfw config T355424'
1--- codfw/externalLoads/es7 live
2+++ codfw/externalLoads/es7 generated
3@@ -1 +1,9 @@
4-{}
5+[
6+ {
7+ "es2038": 10
8+ },
9+ {
10+ "es2039": 100,
11+ "es2040": 100
12+ }
13+]
14--- codfw/hostsByName live
15+++ codfw/hostsByName generated
16@@ -100,5 +100,8 @@
17 "es2034": "10.192.48.164",
18 "es2035": "10.192.14.7",
19 "es2036": "10.192.32.22",
20- "es2037": "10.192.32.25"
21+ "es2037": "10.192.32.25",
22+ "es2038": "10.192.32.42",
23+ "es2039": "10.192.48.126",
24+ "es2040": "10.192.48.132"
25 }

Marostegui updated the task description. (Show Details)Wed, May 1, 9:14 AM
gerritbot added a comment.Wed, May 1, 9:24 AM

Change #1026096 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/mediawiki-config@master] etcd.php: Add es7

https://gerrit.wikimedia.org/r/1026096

gerritbot added a comment.Wed, May 1, 9:25 AM

Change #1026096 merged by jenkins-bot:

[operations/mediawiki-config@master] etcd.php: Add es7

https://gerrit.wikimedia.org/r/1026096

Stashbot added a comment.Wed, May 1, 9:27 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-01T09:27:32Z] <marostegui@deploy1002> Started scap: Backport for [[gerrit:1026096|etcd.php: Add es7 (T355285 T355424)]]

Stashbot added a comment.Wed, May 1, 9:30 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-01T09:30:18Z] <marostegui@deploy1002> marostegui: Backport for [[gerrit:1026096|etcd.php: Add es7 (T355285 T355424)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Maintenance_bot removed a project: Patch-For-Review.Wed, May 1, 9:30 AM
Marostegui updated the task description. (Show Details)Wed, May 1, 9:37 AM
Stashbot added a comment.Wed, May 1, 9:42 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-01T09:42:26Z] <marostegui@deploy1002> Finished scap: Backport for [[gerrit:1026096|etcd.php: Add es7 (T355285 T355424)]] (duration: 14m 53s)

Marostegui updated the task description. (Show Details)Thu, May 2, 1:37 PM
Marostegui added a comment.Fri, May 3, 5:26 AM

es2037 and es2039 set as candidate masters.

Marostegui updated the task description. (Show Details)Fri, May 3, 5:27 AM
Marostegui added a parent task: T364446: Enable writes on es6 and es7.Wed, May 8, 7:42 AM
gerritbot added a comment.Mon, May 13, 5:02 AM

Change #1030621 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es2039: Remove "to be setup"

https://gerrit.wikimedia.org/r/1030621

gerritbot added a project: Patch-For-Review.Mon, May 13, 5:02 AM
gerritbot added a comment.Mon, May 13, 5:08 AM

Change #1030621 merged by Marostegui:

[operations/puppet@production] es2039: Remove "to be setup"

https://gerrit.wikimedia.org/r/1030621

Marostegui closed this task as Resolved.Mon, May 13, 5:30 AM

This is completed

Maintenance_bot removed a project: Patch-For-Review.Mon, May 13, 5:31 AM
Maintenance_bot moved this task from In progress to Done on the DBA board.Mon, May 13, 6:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL