Page MenuHomePhabricator

Marostegui (Manuel Aróstegui)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Sep 1 2016, 6:48 AM (154 w, 3 d)
Availability
Busy Busy until Aug 19.
IRC Nick
marostegui
LDAP User
Marostegui
MediaWiki User
MArostegui (WMF) [ Global Accounts ]

TZ: UTC +1/+2

Recent Activity

Wed, Aug 14

Marostegui claimed T210762: Prepare and check storage layer for nap.wikisource.

Thanks - I will take care of this next week.
Assigning this to me so it is known it is blocked on me before creating the views on the wikireplicas.

Wed, Aug 14, 2:31 PM · User-Banyek, cloud-services-team, Cloud-Services, DBA
Marostegui added a comment to T230459: Replace db2044 with db2063.

Forgot to mention that this host is not in use and it is downtimed, so this onsite maintenance can be done anytime without heads-up to the DBAs

Wed, Aug 14, 10:37 AM · DBA
Marostegui updated subscribers of T230459: Replace db2044 with db2063.

I have been trying to PXE boot this host but it has been impossible.
Even though I have manually set the PXE from the ipmitool locally it is still not working:

root@db2063:~# ipmitool  chassis bootparam get 5
Boot parameter version: 1
Boot parameter 5 is valid/unlocked
Boot parameter data: 0004000000
 Boot Flags :
   - Boot Flag Invalid
   - Options apply to only next boot
   - BIOS PC Compatible (legacy) boot
   - Boot Device Selector : Force PXE
   - Console Redirection control : System Default
   - BIOS verbosity : Console redirection occurs per BIOS configuration setting (default)
   - BIOS Mux Control Override : BIOS uses recommended setting of the mux at the end of POST
Wed, Aug 14, 8:49 AM · DBA
Marostegui closed T228969: Productionize db21[21-31}, a subtask of T227113: rack/setup/install db21[21-30].codfw.wmnet, as Resolved.
Wed, Aug 14, 7:32 AM · Goal, Operations, DBA, ops-codfw
Marostegui closed T228969: Productionize db21[21-31}, a subtask of T229251: (2019-08-31)rack/setup/install db2131.codfw.wmnet, as Resolved.
Wed, Aug 14, 7:32 AM · ops-codfw, Operations, DBA
Marostegui closed T228969: Productionize db21[21-31} as Resolved.

All these hosts have been populated and are provisioned

Wed, Aug 14, 7:32 AM · Goal, DBA
Marostegui removed a project from T228969: Productionize db21[21-31}: Patch-For-Review.
Wed, Aug 14, 7:32 AM · Goal, DBA
Marostegui triaged T230459: Replace db2044 with db2063 as Normal priority.
Wed, Aug 14, 7:03 AM · DBA
Marostegui created T230459: Replace db2044 with db2063.
Wed, Aug 14, 7:02 AM · DBA
Marostegui updated the task description for T202367: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4].
Wed, Aug 14, 6:02 AM · Patch-For-Review, DBA
Marostegui added a comment to T190572: Prepare a disaster recovery plan for failing over from phab1001 to phab2001 (or phab2001 to 1001).

The proxy at codfw is now provisioned.
It obviously points to the codfw databases, which are on read-only.
In case of disaster and if we had to switch everything to codfw, they'd need to be set as writable

root@cumin1001:~# mysql --skip-ssl -hm3-master.codfw.wmnet
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 319933
Server version: 10.1.39-MariaDB MariaDB Server
Wed, Aug 14, 5:47 AM · Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, Phabricator, Documentation, DBA, User-MModell
Marostegui moved T230458: hw troubleshooting: power supply for db1129 from Triage to In progress on the DBA board.
Wed, Aug 14, 5:05 AM · DBA, Operations, ops-eqiad, DC-Ops
Marostegui added a project to T230458: hw troubleshooting: power supply for db1129: DBA.
Wed, Aug 14, 5:04 AM · DBA, Operations, ops-eqiad, DC-Ops
Marostegui moved T230458: hw troubleshooting: power supply for db1129 from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Wed, Aug 14, 5:04 AM · DBA, Operations, ops-eqiad, DC-Ops
Marostegui created T230458: hw troubleshooting: power supply for db1129.
Wed, Aug 14, 5:04 AM · DBA, Operations, ops-eqiad, DC-Ops

Tue, Aug 13

Marostegui moved T51195: Drop filejournal table from WMF from Backlog to Next on the DBA board.
Tue, Aug 13, 11:11 AM · DBA, Performance-Team (Radar), MediaWiki-File-management
Marostegui added a comment to T33863: Fix use of DB schema so RenameUser is trivial.

After all the changes with the actor migration and so forth (per T224348#5226167), global renames are now pretty fast, almost instant, almost 8 years after this task was created. I was wondering, should we consider this resolved now?

Tue, Aug 13, 9:23 AM · DBA, MediaWiki-extensions-Renameuser
Marostegui updated the task description for T228258: Decommission db2043-db2069.
Tue, Aug 13, 8:51 AM · Operations, DBA
Marostegui added a project to T230394: Decommission db2057.codfw.wmnet: DC-Ops.

This host is ready for DC-Ops to decommission

Tue, Aug 13, 8:50 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui reassigned T230394: Decommission db2057.codfw.wmnet from Marostegui to RobH.
Tue, Aug 13, 8:50 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui updated the task description for T230394: Decommission db2057.codfw.wmnet.
Tue, Aug 13, 8:46 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui updated the task description for T230394: Decommission db2057.codfw.wmnet.
Tue, Aug 13, 7:54 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui triaged T230394: Decommission db2057.codfw.wmnet as Normal priority.
Tue, Aug 13, 7:35 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui moved T230394: Decommission db2057.codfw.wmnet from Triage to In progress on the DBA board.
Tue, Aug 13, 7:32 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui created T230394: Decommission db2057.codfw.wmnet.
Tue, Aug 13, 7:32 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui updated the task description for T228258: Decommission db2043-db2069.
Tue, Aug 13, 7:31 AM · Operations, DBA
Marostegui updated the task description for T228258: Decommission db2043-db2069.
Tue, Aug 13, 7:24 AM · Operations, DBA
Marostegui renamed T230107: Decommission db2069.codfw.wmnet from Decommission db2069 to Decommission db2069.codfw.wmnet.
Tue, Aug 13, 5:50 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui updated the task description for T228258: Decommission db2043-db2069.
Tue, Aug 13, 5:48 AM · Operations, DBA
Marostegui added a comment to T230391: Decommission db2050.codfw.wmnet.

This host is ready for DC-Ops to decommission

Tue, Aug 13, 5:48 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui reassigned T230391: Decommission db2050.codfw.wmnet from Marostegui to RobH.
Tue, Aug 13, 5:48 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui updated the task description for T230391: Decommission db2050.codfw.wmnet.
Tue, Aug 13, 5:39 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui updated the task description for T230391: Decommission db2050.codfw.wmnet.
Tue, Aug 13, 5:29 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui updated the task description for T228258: Decommission db2043-db2069.
Tue, Aug 13, 5:29 AM · Operations, DBA
Marostegui triaged T230391: Decommission db2050.codfw.wmnet as Normal priority.
Tue, Aug 13, 5:27 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui created T230391: Decommission db2050.codfw.wmnet.
Tue, Aug 13, 5:27 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui added a comment to T51195: Drop filejournal table from WMF.

@aaron are you sure this table isn't created by default? I just checked the most recent wiki we created yuewiktionary T205714: Prepare and check storage layer for yuewiktionary which was created the 14th Nov 2018 and filejournal table is there.

-rw-rw---- 1 mysql mysql 128K Nov 14  2018 filejournal.ibd
Tue, Aug 13, 5:17 AM · DBA, Performance-Team (Radar), MediaWiki-File-management
Marostegui updated the task description for T228969: Productionize db21[21-31}.
Tue, Aug 13, 5:06 AM · Goal, DBA

Mon, Aug 12

Marostegui claimed T51195: Drop filejournal table from WMF.

Ah! Ok :)
Didn't know
Thanls for the clarification

Mon, Aug 12, 6:46 PM · DBA, Performance-Team (Radar), MediaWiki-File-management
Marostegui moved T119154: Move echo tables from local wiki databases onto extension1 cluster for mediawikiwiki, metawiki, and officewiki from Backlog to Next on the DBA board.
Mon, Aug 12, 2:07 PM · Collaboration-Team-Triage, DBA, Operations
Marostegui moved T111929: Puppetize grants for mysql hosts that are the source of recovery (dbstore, passive misc) from Backlog to Next on the DBA board.
Mon, Aug 12, 2:07 PM · Operations, DBA
Marostegui removed a project from T51195: Drop filejournal table from WMF: DBA.

Removing the DBA tag per T51195#4769106ç
The table is still present on the archives/patch-filejournal.sql

cat archives/patch-filejournal.sql
-- File backend operation journal
CREATE TABLE /*_*/filejournal (
  -- Unique ID for each file operation
  fj_id bigint unsigned NOT NULL PRIMARY KEY auto_increment,
  -- UUID of the batch this operation belongs to
  fj_batch_uuid varbinary(32) NOT NULL,
  -- The registered file backend name
  fj_backend varchar(255) NOT NULL,
  -- The storage path that was affected (may be internal paths)
  fj_path blob NOT NULL,
  -- Primitive operation description (create/update/delete)
  fj_op varchar(16) NOT NULL default '',
  -- SHA-1 file content hash in base-36
  fj_new_sha1 varbinary(32) NOT NULL default '',
  -- Timestamp of the batch operation
  fj_timestamp varbinary(14) NOT NULL default ''
) /*$wgDBTableOptions*/;
Mon, Aug 12, 1:13 PM · DBA, Performance-Team (Radar), MediaWiki-File-management
Marostegui updated subscribers of T227717: Drop DB tables for now-deleted zerowiki from production.

Actually, db grants might also be a good first step from the other direction – instead of truncating, revoke both the read and write grant from the MW user. That step would be easy to reverse, but if without issue for weeks, could then lead to a proper drop without much risk.

Mon, Aug 12, 1:10 PM · Release-Engineering-Team-TODO, DBA, Reading-Infrastructure-Team-Backlog
Marostegui removed a project from T60674: Drop page.page_restrictions column from Wikimedia wikis: DBA.

This cannot proceed, this column is still showing up on tables.sql and the related files. It needs to be removed from there and merged so we can start dropping it in production.
A grep in the core repo reveals lots of references to this column.
Going to remove the DBA tag from here until there is an actionable for us (I will remain subscribed to the task though).
Once this is ready to go, please send this ticket back with the schema-change template: https://wikitech.wikimedia.org/wiki/Schema_changes#Workflow_of_a_schema_change

Mon, Aug 12, 1:05 PM · MediaWiki-Page-protection, Schema-change
Marostegui closed T230055: Remove globalblocks tables from wikis, a subtask of T54921: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking), as Resolved.
Mon, Aug 12, 12:03 PM · Epic, DBA, Tracking-Neverending
Marostegui closed T230055: Remove globalblocks tables from wikis as Resolved.

All done

Mon, Aug 12, 12:03 PM · DBA, Schema-change
Marostegui updated the task description for T230055: Remove globalblocks tables from wikis.
Mon, Aug 12, 12:03 PM · DBA, Schema-change
Marostegui updated the task description for T230055: Remove globalblocks tables from wikis.
Mon, Aug 12, 10:01 AM · DBA, Schema-change
Marostegui updated the task description for T230106: Switchover codfw primary database masters to new hosts.
Mon, Aug 12, 9:57 AM · Goal, DBA
Marostegui updated subscribers of T230317: Error: 1146 Table 'valid_tag' doesn't exist when upgrading from an ancient MediaWiki version.
Mon, Aug 12, 9:45 AM · Core Platform Team Workboards (Clinic Duty Team), MediaWiki-Maintenance-scripts
Marostegui updated the task description for T230055: Remove globalblocks tables from wikis.
Mon, Aug 12, 9:36 AM · DBA, Schema-change
Marostegui updated the task description for T208323: Predictive failures on disk S.M.A.R.T. status.
Mon, Aug 12, 9:34 AM · Operations, DBA
Marostegui updated the task description for T228258: Decommission db2043-db2069.
Mon, Aug 12, 9:34 AM · Operations, DBA
Marostegui reassigned T230311: Decommission db2043.codfw.wmnet from Marostegui to RobH.

This host is ready for DC-Ops to decommission

Mon, Aug 12, 9:34 AM · Patch-For-Review, ops-codfw, decommission, Operations
Marostegui updated the task description for T230311: Decommission db2043.codfw.wmnet.
Mon, Aug 12, 9:33 AM · Patch-For-Review, ops-codfw, decommission, Operations
Marostegui updated the task description for T230311: Decommission db2043.codfw.wmnet.
Mon, Aug 12, 9:31 AM · Patch-For-Review, ops-codfw, decommission, Operations
Marostegui moved T230055: Remove globalblocks tables from wikis from Next to In progress on the DBA board.
Mon, Aug 12, 9:25 AM · DBA, Schema-change
Marostegui updated the task description for T230055: Remove globalblocks tables from wikis.
Mon, Aug 12, 9:24 AM · DBA, Schema-change
Marostegui claimed T230055: Remove globalblocks tables from wikis.
Mon, Aug 12, 9:22 AM · DBA, Schema-change
Marostegui updated the task description for T230311: Decommission db2043.codfw.wmnet.
Mon, Aug 12, 9:21 AM · Patch-For-Review, ops-codfw, decommission, Operations
Marostegui triaged T230311: Decommission db2043.codfw.wmnet as Normal priority.
Mon, Aug 12, 8:30 AM · Patch-For-Review, ops-codfw, decommission, Operations
Marostegui created T230311: Decommission db2043.codfw.wmnet.
Mon, Aug 12, 8:30 AM · Patch-For-Review, ops-codfw, decommission, Operations
Marostegui updated the task description for T195847: Clean up artifacts from LaTeX based math rendering.
Mon, Aug 12, 5:49 AM · Operations, Math
Marostegui closed T196055: Remove table `math` from the database, a subtask of T54921: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking), as Resolved.
Mon, Aug 12, 5:48 AM · Epic, DBA, Tracking-Neverending
Marostegui closed T196055: Remove table `math` from the database, a subtask of T195847: Clean up artifacts from LaTeX based math rendering, as Resolved.
Mon, Aug 12, 5:48 AM · Operations, Math
Marostegui closed T196055: Remove table `math` from the database as Resolved.

All done!

Mon, Aug 12, 5:48 AM · Patch-For-Review, DBA, Math
Marostegui updated the task description for T196055: Remove table `math` from the database.
Mon, Aug 12, 5:48 AM · Patch-For-Review, DBA, Math
Marostegui updated the task description for T196055: Remove table `math` from the database.
Mon, Aug 12, 5:03 AM · Patch-For-Review, DBA, Math

Sun, Aug 11

Marostegui added a comment to T230290: Fatal DBQueryError exception: Lock wait timeout exceeded in MessageGroups::setPriority.

These are the actual details of the stack trace:

exception.file	       	/srv/mediawiki/php-1.34.0-wmf.17/includes/libs/rdbms/database/Database.php:1596
Sun, Aug 11, 6:08 AM · MediaWiki-extensions-Translate, Wikimedia-production-error

Fri, Aug 9

Marostegui added a comment to T229657: Switchover m5 primary master: db1073 to db1133.

@Marostegui to pool db1133 with weight 0 on wikitech section via dbctl instance db1133 edit so it can be later set as master.

You likely also want a dbctl config commit here.
Otherwise LGTM!

Fri, Aug 9, 1:18 PM · Patch-For-Review, cloud-services-team (Kanban), wikitech.wikimedia.org, Operations, DBA
Marostegui added a comment to T229657: Switchover m5 primary master: db1073 to db1133.
Fri, Aug 9, 1:15 PM · Patch-For-Review, cloud-services-team (Kanban), wikitech.wikimedia.org, Operations, DBA
Marostegui added a comment to T229677: #dbctl: add 'comment'/'description' metadata to instances.

The way to get notes and candidate info would be the dbctl instance all get command.
To find candidate masters then one could do e.g. dbctl instance all get | jq 'select(.. | .sections? | has("s1")) | select(.. | .candidate_master?)' to find all the candidate masters for s1.
Hosts with notes attached could be found with dbctl instance all get | jq 'select(.. | .note? | length > 0)'

Fri, Aug 9, 1:04 PM · DBA, conftool
Marostegui updated the task description for T196055: Remove table `math` from the database.
Fri, Aug 9, 10:49 AM · Patch-For-Review, DBA, Math
Marostegui updated subscribers of T229657: Switchover m5 primary master: db1073 to db1133.

I have submitted the patches for review, I would appreciate if the cloud-services-team folks can give them a look (specially @JHedden as he will be online supporting this). Also @CDanis for the dbctl part
The procedure for the failover, on a high level description, along these lines:

Fri, Aug 9, 10:23 AM · Patch-For-Review, cloud-services-team (Kanban), wikitech.wikimedia.org, Operations, DBA
Marostegui updated the task description for T196055: Remove table `math` from the database.
Fri, Aug 9, 9:57 AM · Patch-For-Review, DBA, Math
Marostegui updated the task description for T196055: Remove table `math` from the database.
Fri, Aug 9, 9:06 AM · Patch-For-Review, DBA, Math
Marostegui updated the task description for T196055: Remove table `math` from the database.
Fri, Aug 9, 8:58 AM · Patch-For-Review, DBA, Math
Marostegui updated the task description for T208323: Predictive failures on disk S.M.A.R.T. status.
Fri, Aug 9, 8:38 AM · Operations, DBA
Marostegui added a project to T230107: Decommission db2069.codfw.wmnet: DC-Ops.

This host is ready for DC-Ops to decommission

Fri, Aug 9, 8:36 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui reassigned T230107: Decommission db2069.codfw.wmnet from Marostegui to RobH.
Fri, Aug 9, 8:36 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui added a comment to T229677: #dbctl: add 'comment'/'description' metadata to instances.

OK, for now I'm planning on changing the instance schema in the following way:

  • Add a candidate_master field, type boolean, default false. No extra code or feature support, aside from it being settable in instance FOO edit and appearing in instance (FOO|all) get output (where you could use jq to find candidate masters for a given section).
Fri, Aug 9, 8:35 AM · DBA, conftool
Marostegui updated the task description for T230107: Decommission db2069.codfw.wmnet.
Fri, Aug 9, 8:29 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui updated the task description for T196055: Remove table `math` from the database.
Fri, Aug 9, 5:40 AM · Patch-For-Review, DBA, Math

Thu, Aug 8

Marostegui updated the task description for T229657: Switchover m5 primary master: db1073 to db1133.
Thu, Aug 8, 2:31 PM · Patch-For-Review, cloud-services-team (Kanban), wikitech.wikimedia.org, Operations, DBA
Marostegui updated the task description for T229657: Switchover m5 primary master: db1073 to db1133.
Thu, Aug 8, 2:30 PM · Patch-For-Review, cloud-services-team (Kanban), wikitech.wikimedia.org, Operations, DBA
Marostegui updated the task description for T228258: Decommission db2043-db2069.
Thu, Aug 8, 2:01 PM · Operations, DBA
Marostegui updated the task description for T196055: Remove table `math` from the database.
Thu, Aug 8, 1:33 PM · Patch-For-Review, DBA, Math
Marostegui updated the task description for T196055: Remove table `math` from the database.
Thu, Aug 8, 1:09 PM · Patch-For-Review, DBA, Math
Marostegui updated the task description for T196055: Remove table `math` from the database.
Thu, Aug 8, 9:27 AM · Patch-For-Review, DBA, Math
Marostegui updated the task description for T230107: Decommission db2069.codfw.wmnet.
Thu, Aug 8, 9:19 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui updated the task description for T230107: Decommission db2069.codfw.wmnet.
Thu, Aug 8, 9:05 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui triaged T230107: Decommission db2069.codfw.wmnet as Normal priority.
Thu, Aug 8, 8:59 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui created T230107: Decommission db2069.codfw.wmnet.
Thu, Aug 8, 8:59 AM · DC-Ops, ops-codfw, decommission, Operations
Marostegui updated the task description for T196055: Remove table `math` from the database.
Thu, Aug 8, 8:51 AM · Patch-For-Review, DBA, Math
Marostegui renamed T230106: Switchover codfw primary database masters to new hosts from Switchover codfw masters to new hosts to Switchover codfw primary database masters to new hosts.
Thu, Aug 8, 8:46 AM · Goal, DBA
Marostegui triaged T230106: Switchover codfw primary database masters to new hosts as Normal priority.
Thu, Aug 8, 8:46 AM · Goal, DBA
Marostegui moved T230106: Switchover codfw primary database masters to new hosts from Triage to In progress on the DBA board.
Thu, Aug 8, 8:46 AM · Goal, DBA
Marostegui updated the task description for T230106: Switchover codfw primary database masters to new hosts.
Thu, Aug 8, 8:46 AM · Goal, DBA
Marostegui created T230106: Switchover codfw primary database masters to new hosts.
Thu, Aug 8, 8:42 AM · Goal, DBA
Marostegui updated the task description for T228258: Decommission db2043-db2069.
Thu, Aug 8, 7:49 AM · Operations, DBA