Page MenuHomePhabricator
Create Task
Maniphest T394260

Productionize pc8
Closed, ResolvedPublic
Actions

Description

  • pc1018
  • pc2018

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedMarosteguiT394260 Productionize pc8
 ResolvedJhancock.wmT393110 Q4:rack/setup/install pc2018
 ResolvedJclark-ctrT392492 Q4:rack/setup/install pc1018

Event Timeline

Marostegui created this task.May 14 2025, 5:47 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 14 2025, 5:47 AM
Marostegui triaged this task as Medium priority.May 14 2025, 5:48 AM
Marostegui moved this task from Triage to Blocked on the DBA board.
Marostegui added subtasks: T393110: Q4:rack/setup/install pc2018, T392492: Q4:rack/setup/install pc1018.

pc1018 has been installed - waiting for pc2018

Marostegui added a comment.May 14 2025, 12:53 PM
[12:53:19] marostegui@cumin1002:~$ sudo cumin 'pc1018*' 'lvextend -L+1000G /dev/mapper/tank-data ; xfs_growfs /srv ; df -hT /srv ; pvs'
1 hosts will be targeted:
pc1018.eqiad.wmnet
OK to proceed on 1 hosts? Enter the number of affected hosts to confirm or "q" to quit: 1
----- OUTPUT of 'lvextend -L+1000...f -hT /srv ; pvs' -----
  Size of logical volume tank/data changed from <7.56 TiB (1981022 extents) to 8.53 TiB (2237022 extents).
  Logical volume tank/data successfully resized.
meta-data=/dev/mapper/tank-data  isize=512    agcount=32, agsize=63392704 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=0
data     =                       bsize=4096   blocks=2028566528, imaxpct=5
         =                       sunit=64     swidth=256 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 2028566528 to 2290710528
Filesystem            Type  Size  Used Avail Use% Mounted on
/dev/mapper/tank-data xfs   8.6T   61G  8.5T   1% /srv
  PV         VG   Fmt  Attr PSize  PFree
  /dev/sda3  tank lvm2 a--  <8.69t 156.30g
================
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00,  1.32hosts/s]
FAIL |                                                                                                                                                                                                                                         |   0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'lvextend -L+1000...f -hT /srv ; pvs'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Stashbot added a comment.May 15 2025, 5:07 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-15T05:07:24Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool pc7 T394260', diff saved to https://phabricator.wikimedia.org/P76162 and previous config saved to /var/cache/conftool/dbconfig/20250515-050724-marostegui.json

gerritbot added a comment.May 15 2025, 5:12 AM

Change #1146175 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbconfig.schema: Add pc8

https://gerrit.wikimedia.org/r/1146175

gerritbot added a project: Patch-For-Review.May 15 2025, 5:12 AM
gerritbot added a comment.May 15 2025, 5:15 AM

Change #1146175 merged by Marostegui:

[operations/puppet@production] dbconfig.schema: Add pc8

https://gerrit.wikimedia.org/r/1146175

gerritbot added a comment.May 15 2025, 5:20 AM

Change #1146176 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] valid_section.pp: Add pc8

https://gerrit.wikimedia.org/r/1146176

gerritbot added a comment.May 15 2025, 5:25 AM

Change #1146176 merged by Marostegui:

[operations/puppet@production] valid_section.pp: Add pc8

https://gerrit.wikimedia.org/r/1146176

gerritbot added a comment.May 15 2025, 5:28 AM

Change #1146184 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] pc1018: Productionize

https://gerrit.wikimedia.org/r/1146184

gerritbot added a comment.May 15 2025, 5:30 AM

Change #1146184 merged by Marostegui:

[operations/puppet@production] pc1018: Productionize

https://gerrit.wikimedia.org/r/1146184

Maintenance_bot removed a project: Patch-For-Review.May 15 2025, 6:30 AM
Stashbot added a comment.May 15 2025, 8:52 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-15T08:52:57Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Repool pc7 T394260', diff saved to https://phabricator.wikimedia.org/P76208 and previous config saved to /var/cache/conftool/dbconfig/20250515-085256-marostegui.json

Marostegui added a comment.May 15 2025, 8:54 AM

pc1018 has been recloned and emptied. Host added to zarcillo, pc8 added as a section and pc1018 as eqiad's master.

Marostegui added a comment.May 16 2025, 5:23 AM

pc8 now showing up in orchestrator: https://orchestrator.wikimedia.org/web/cluster/alias/pc8
pending pc2018 which isn't yet racked T393110

gerritbot added a comment.May 16 2025, 6:41 AM

Change #1146871 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] sections.yaml: Add pc8

https://gerrit.wikimedia.org/r/1146871

gerritbot added a project: Patch-For-Review.May 16 2025, 6:41 AM
Ladsgroup awarded a token.May 16 2025, 10:47 AM
gerritbot added a comment.May 16 2025, 12:44 PM

Change #1146871 merged by Marostegui:

[operations/puppet@production] sections.yaml: Add pc8

https://gerrit.wikimedia.org/r/1146871

Maintenance_bot removed a project: Patch-For-Review.May 16 2025, 1:30 PM
gerritbot added a comment.May 21 2025, 7:26 AM

Change #1148786 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] installserver: Do not reimage pc1018

https://gerrit.wikimedia.org/r/1148786

gerritbot added a project: Patch-For-Review.May 21 2025, 7:26 AM
gerritbot added a comment.May 21 2025, 7:29 AM

Change #1148786 merged by Marostegui:

[operations/puppet@production] installserver: Do not reimage pc1018

https://gerrit.wikimedia.org/r/1148786

Maintenance_bot removed a project: Patch-For-Review.May 21 2025, 7:30 AM
Jhancock.wm closed subtask T393110: Q4:rack/setup/install pc2018 as Resolved.May 21 2025, 3:22 PM
gerritbot added a comment.May 22 2025, 4:46 AM

Change #1148998 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize pc2018

https://gerrit.wikimedia.org/r/1148998

gerritbot added a project: Patch-For-Review.May 22 2025, 4:46 AM
gerritbot added a comment.May 22 2025, 4:49 AM

Change #1148998 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize pc2018

https://gerrit.wikimedia.org/r/1148998

gerritbot added a comment.May 22 2025, 5:06 AM

Change #1148999 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add pc1018,pc2018

https://gerrit.wikimedia.org/r/1148999

gerritbot added a comment.May 22 2025, 5:08 AM

Change #1148999 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add pc1018,pc2018

https://gerrit.wikimedia.org/r/1148999

gerritbot added a comment.May 22 2025, 5:13 AM

Change #1149000 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instance.schema: Add pc8

https://gerrit.wikimedia.org/r/1149000

gerritbot added a comment.May 22 2025, 5:16 AM

Change #1149000 merged by Marostegui:

[operations/puppet@production] instance.schema: Add pc8

https://gerrit.wikimedia.org/r/1149000

Marostegui updated the task description. (Show Details)May 22 2025, 5:24 AM
Stashbot added a comment.May 22 2025, 5:26 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-22T05:26:49Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add pc1018 and pc2018 to dbctl depooled T394260', diff saved to https://phabricator.wikimedia.org/P76372 and previous config saved to /var/cache/conftool/dbconfig/20250522-052649-marostegui.json

Marostegui added a comment.May 22 2025, 5:27 AM

Both hosts added to dbctl.
Waiting for confirmation whether I can enable pc8 in dbctl.

Maintenance_bot removed a project: Patch-For-Review.May 22 2025, 5:30 AM
Marostegui mentioned this in T394371: Migrate to MariaDB 10.11.May 22 2025, 5:37 AM
Marostegui moved this task from Blocked to In progress on the DBA board.May 22 2025, 5:55 AM
Stashbot added a comment.May 22 2025, 9:50 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-22T09:50:17Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Add pc8 T394260', diff saved to https://phabricator.wikimedia.org/P76390 and previous config saved to /var/cache/conftool/dbconfig/20250522-095017-marostegui.json

Stashbot added a comment.May 22 2025, 9:50 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-22T09:50:35Z] <marostegui> dbmaint codfw eqiad Pool pc8 new section T394260

gerritbot added a comment.May 22 2025, 9:51 AM

Change #1149337 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] pc1018,pc2018: Enable notifications

https://gerrit.wikimedia.org/r/1149337

gerritbot added a project: Patch-For-Review.May 22 2025, 9:51 AM
gerritbot added a comment.May 22 2025, 9:54 AM

Change #1149337 merged by Marostegui:

[operations/puppet@production] pc1018,pc2018: Enable notifications

https://gerrit.wikimedia.org/r/1149337

Marostegui added a comment.May 22 2025, 9:57 AM

The section is receiving traffic now. Waiting to make sure grafana works well before resolving this.

Marostegui closed this task as Resolved.May 22 2025, 10:03 AM

All good https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&from=now-5m&to=now&timezone=utc&var-site=eqiad&var-group=parsercache&var-shard=$__all&var-role=$__all

gerritbot added a comment.May 22 2025, 10:11 AM

Change #1149344 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] parsercachepurging: Use for loop

https://gerrit.wikimedia.org/r/1149344

Maintenance_bot moved this task from In progress to Done on the DBA board.May 22 2025, 10:29 AM
gerritbot added a comment.May 22 2025, 10:46 AM

Change #1149344 merged by Ladsgroup:

[operations/puppet@production] parsercachepurging: Use for loop

https://gerrit.wikimedia.org/r/1149344

gerritbot added a comment.May 22 2025, 10:50 AM

Change #1149353 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] parsercachepurging: Add pc8

https://gerrit.wikimedia.org/r/1149353

gerritbot added a comment.May 22 2025, 11:19 AM

Change #1149353 merged by Ladsgroup:

[operations/puppet@production] parsercachepurging: Add pc8

https://gerrit.wikimedia.org/r/1149353

Maintenance_bot removed a project: Patch-For-Review.May 22 2025, 11:31 AM
Ladsgroup subscribed.May 22 2025, 11:35 AM

the purge script is now deployed.

gerritbot added a comment.May 23 2025, 6:17 AM

Change #1149539 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Add pc8-master CNAME

https://gerrit.wikimedia.org/r/1149539

gerritbot added a project: Patch-For-Review.May 23 2025, 6:17 AM

Change #1149539 merged by Marostegui:

[operations/dns@master] wmnet: Add pc8-master CNAME

https://gerrit.wikimedia.org/r/1149539

Maintenance_bot removed a project: Patch-For-Review.May 23 2025, 6:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits