Page MenuHomePhabricator

Kormat (Stevie Beth Mhaol)
SRE

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Apr 14 2020, 7:57 AM (128 w, 2 d)
Availability
Available
LDAP User
Kormat
MediaWiki User
SBMhaol (WMF) [ Global Accounts ]

Recent Activity

May 17 2022

Kormat committed rSCHCH069d02e4924d: fix_revision.rev_timestamp_type_T298560.py: New schema change. (authored by Kormat).
fix_revision.rev_timestamp_type_T298560.py: New schema change.
May 17 2022, 1:37 PM
Kormat committed rSCHCHf7eb02b078a0: fix_logging.log_timestamp_type_T298555.py: New schema change. (authored by Kormat).
fix_logging.log_timestamp_type_T298555.py: New schema change.
May 17 2022, 1:37 PM

May 2 2022

Kormat updated the task description for T303603: Add actor and comment columns to cu_changes.
May 2 2022, 1:29 PM · DBA, Schema-change-in-production
Kormat updated the task description for T303603: Add actor and comment columns to cu_changes.
May 2 2022, 1:28 PM · DBA, Schema-change-in-production
Kormat committed rSCHCH0b67e8d866a6: cu_changes_actor_comment_cols_T303603.py: New schema change. (authored by Kormat).
cu_changes_actor_comment_cols_T303603.py: New schema change.
May 2 2022, 9:01 AM
Kormat committed rSCHCH55502aa6cc1a: change_localuser.lu_attached_timestamp_T302659.py: New schema change (authored by Kormat).
change_localuser.lu_attached_timestamp_T302659.py: New schema change
May 2 2022, 9:00 AM
Kormat moved T300774: Drop fr_img_* columns from Ready to Blocked on the DBA board.
May 2 2022, 8:27 AM · DBA, Schema-change-in-production
Kormat changed the status of T303171: Upgrade s1 to Bullseye, a subtask of T298585: Upgrade WMF database-and-backup-related hosts to bullseye, from Open to Stalled.
May 2 2022, 7:24 AM · DBA
Kormat changed the status of T303171: Upgrade s1 to Bullseye from Open to Stalled.
May 2 2022, 7:24 AM · DBA
Kormat added a comment to T303171: Upgrade s1 to Bullseye.

Stalling until db1164 is back in service: T307198: db1164 fails to POST/boot/etc

May 2 2022, 7:23 AM · DBA

Apr 29 2022

Kormat added a comment to T307198: db1164 fails to POST/boot/etc.

I've set the host to 'failed' in netbox: https://netbox.wikimedia.org/dcim/devices/2999/

Apr 29 2022, 3:41 PM · SRE, ops-eqiad, DBA
Kormat added a comment to T307198: db1164 fails to POST/boot/etc.

On the web console, i can see it get as far as this before resetting:

image.png (1×1 px, 52 KB)

Apr 29 2022, 10:32 AM · SRE, ops-eqiad, DBA
Kormat created T307198: db1164 fails to POST/boot/etc.
Apr 29 2022, 10:27 AM · SRE, ops-eqiad, DBA
Kormat archived P27009 (An Untitled Masterwork).
Apr 29 2022, 10:15 AM
Kormat created P27009 (An Untitled Masterwork).
Apr 29 2022, 10:03 AM
Kormat updated the task description for T303171: Upgrade s1 to Bullseye.
Apr 29 2022, 9:36 AM · DBA
Kormat updated the task description for T303171: Upgrade s1 to Bullseye.
Apr 29 2022, 7:58 AM · DBA
Kormat updated the task description for T303171: Upgrade s1 to Bullseye.
Apr 29 2022, 7:09 AM · DBA

Apr 28 2022

Kormat updated the task description for T303171: Upgrade s1 to Bullseye.
Apr 28 2022, 3:47 PM · DBA
Kormat closed T307101: Reboot pc1013 as Resolved.
Apr 28 2022, 2:32 PM · DBA, SRE, Security
Kormat updated the task description for T307101: Reboot pc1013.
Apr 28 2022, 2:24 PM · DBA, SRE, Security
Kormat updated the task description for T303171: Upgrade s1 to Bullseye.
Apr 28 2022, 1:54 PM · DBA
Kormat updated the task description for T303171: Upgrade s1 to Bullseye.
Apr 28 2022, 1:48 PM · DBA
Kormat updated the task description for T307101: Reboot pc1013.
Apr 28 2022, 1:43 PM · DBA, SRE, Security
Kormat updated the task description for T303171: Upgrade s1 to Bullseye.
Apr 28 2022, 1:38 PM · DBA
Kormat updated the task description for T303171: Upgrade s1 to Bullseye.
Apr 28 2022, 1:36 PM · DBA
Kormat updated the task description for T307101: Reboot pc1013.
Apr 28 2022, 1:11 PM · DBA, SRE, Security
Kormat updated the task description for T307101: Reboot pc1013.
Apr 28 2022, 1:11 PM · DBA, SRE, Security
Kormat triaged T307101: Reboot pc1013 as Medium priority.
Apr 28 2022, 1:09 PM · DBA, SRE, Security
Kormat updated the task description for T307101: Reboot pc1013.
Apr 28 2022, 1:09 PM · DBA, SRE, Security
Kormat created T307101: Reboot pc1013.
Apr 28 2022, 1:03 PM · DBA, SRE, Security
Kormat closed T306983: Reboot pc1012 as Resolved.

Success.

Apr 28 2022, 11:04 AM · DBA, SRE, Security
Kormat updated the task description for T306983: Reboot pc1012.
Apr 28 2022, 10:57 AM · DBA, SRE, Security
Kormat triaged T306983: Reboot pc1012 as Medium priority.
Apr 28 2022, 10:46 AM · DBA, SRE, Security
Kormat updated the task description for T306983: Reboot pc1012.
Apr 28 2022, 10:46 AM · DBA, SRE, Security
Kormat updated the task description for T306983: Reboot pc1012.
Apr 28 2022, 10:45 AM · DBA, SRE, Security
Kormat updated the task description for T306983: Reboot pc1012.
Apr 28 2022, 10:43 AM · DBA, SRE, Security

Apr 27 2022

Kormat updated the task description for T306983: Reboot pc1012.
Apr 27 2022, 1:26 PM · DBA, SRE, Security
Kormat updated the task description for T306983: Reboot pc1012.
Apr 27 2022, 1:20 PM · DBA, SRE, Security
Kormat created T306983: Reboot pc1012.
Apr 27 2022, 1:19 PM · DBA, SRE, Security

Apr 26 2022

Kormat closed T306892: Reboot pc1011 as Resolved.
Apr 26 2022, 1:42 PM · DBA, SRE
Kormat updated the task description for T306892: Reboot pc1011.
Apr 26 2022, 1:24 PM · DBA, SRE
Kormat updated the task description for T306892: Reboot pc1011.
Apr 26 2022, 1:24 PM · DBA, SRE
Kormat updated the task description for T306892: Reboot pc1011.
Apr 26 2022, 1:21 PM · DBA, SRE
Kormat updated the task description for T306892: Reboot pc1011.
Apr 26 2022, 12:45 PM · DBA, SRE
Kormat updated the task description for T306892: Reboot pc1011.
Apr 26 2022, 12:34 PM · DBA, SRE
Kormat updated the task description for T306892: Reboot pc1011.
Apr 26 2022, 12:32 PM · DBA, SRE
Kormat triaged T306892: Reboot pc1011 as Medium priority.
Apr 26 2022, 12:32 PM · DBA, SRE
Kormat removed a project from T306892: Reboot pc1011: Security.
Apr 26 2022, 12:32 PM · DBA, SRE
Kormat created T306892: Reboot pc1011.
Apr 26 2022, 12:31 PM · DBA, SRE

Apr 25 2022

Kormat added a comment to T303603: Add actor and comment columns to cu_changes.

As @Zabe says, T233004 is stalled on this. Is there anything I can help with? This might be more of a "teach Dan how to do this" slowdown than a help, but gets you an extra person to help with these kinds of changes in the future? Anyway, I'm game.

Apr 25 2022, 1:12 PM · DBA, Schema-change-in-production

Apr 21 2022

Kormat added a comment to T299021: Reduce running time of refreshLinkRecommendations.php to a maximum of 60 minutes.

The script dropped the connection after 2h15m.

Apr 21 2022, 4:19 PM · Growth-Team, Add-Link
Kormat added a comment to T299021: Reduce running time of refreshLinkRecommendations.php to a maximum of 60 minutes.

As an example, i depooled a host (db1120) so i could reboot it. 1.5h later, i'm still waiting for refreshLinkRecommendations to drop it's connection so that i can actually do the reboot. I have no way of knowing when that will happen. I also don't know how bad/disruptive/etc it would be to just reboot it anyway.

Apr 21 2022, 3:31 PM · Growth-Team, Add-Link

Apr 20 2022

Kormat archived P25584 (An Untitled Masterwork).
Apr 20 2022, 9:40 AM
Kormat created P25584 (An Untitled Masterwork).
Apr 20 2022, 9:06 AM

Apr 11 2022

Kormat added a comment to T305469: codfw: Dedicate Rack B1 for cloudX-dev servers.

For what is worth, es2029 and es2030 are scheduled to be done 14th, which is a bank holiday for me, so someone else would need to bring mysql up once that is done.

Apr 11 2022, 9:57 AM · SRE, ops-codfw

Apr 5 2022

Kormat closed T301315: Move orchestrator from db2093 to db1115, a subtask of T297605: Shutdown Tendril and dbtree, as Resolved.
Apr 5 2022, 9:54 AM · serviceops-radar, Patch-For-Review, DBA
Kormat closed T301315: Move orchestrator from db2093 to db1115 as Resolved.

The only thing left is to check (by DBAs) that the backup available at (e.g.) : dbprov1002:/srv/backups/dumps/latest/dump.db_inventory.2022-04-05--03-13-31 looks as expected (content-wise).

Apr 5 2022, 9:54 AM · Patch-For-Review, database-backups, DBA

Mar 31 2022

Kormat added a comment to T301315: Move orchestrator from db2093 to db1115.

One last thing- does orchestrator + zarcillo server have a name? Backups (jobs) have a completely arbitrary identifier for monitoring purposes- any suggestion (even if it is orchestrator OR zarcillo containing both)? Alternatively, it is possible to setup separate backup jobs, even for the same server.

Mar 31 2022, 2:04 PM · Patch-For-Review, database-backups, DBA
Kormat added a comment to T301315: Move orchestrator from db2093 to db1115.

@Kormat I can help setting up backups, but someone has to make clear first what to backup (what databases?), from which servers/dcs, and how frequently they are needed.

Mar 31 2022, 1:51 PM · Patch-For-Review, database-backups, DBA
Kormat updated subscribers of T301315: Move orchestrator from db2093 to db1115.

@jcrespo: Can i hand this task over to you? I think the dumps grants probably need some cleanup, if you can have a look at that, too.

Mar 31 2022, 12:40 PM · Patch-For-Review, database-backups, DBA
Kormat updated the task description for T301315: Move orchestrator from db2093 to db1115.
Mar 31 2022, 12:35 PM · Patch-For-Review, database-backups, DBA

Mar 30 2022

Kormat archived P23832 (An Untitled Masterwork).
Mar 30 2022, 3:08 PM
Kormat created P23832 (An Untitled Masterwork).
Mar 30 2022, 2:56 PM
Kormat updated the task description for T301315: Move orchestrator from db2093 to db1115.
Mar 30 2022, 2:01 PM · Patch-For-Review, database-backups, DBA
Kormat updated the task description for T301315: Move orchestrator from db2093 to db1115.
Mar 30 2022, 1:58 PM · Patch-For-Review, database-backups, DBA
Kormat updated the task description for T301315: Move orchestrator from db2093 to db1115.
Mar 30 2022, 1:40 PM · Patch-For-Review, database-backups, DBA
Kormat added a comment to T301315: Move orchestrator from db2093 to db1115.

Regarding the plan I would include the following steps before dropping the database in db2093:

  • Check current grants on db2093 to make sure they are the same on db1115
Mar 30 2022, 1:28 PM · Patch-For-Review, database-backups, DBA
Kormat updated the task description for T301315: Move orchestrator from db2093 to db1115.
Mar 30 2022, 1:22 PM · Patch-For-Review, database-backups, DBA

Mar 29 2022

Kormat created T304915: Puppetise wmfdb.
Mar 29 2022, 8:55 AM · wmfdb

Mar 28 2022

Kormat updated the task description for T301315: Move orchestrator from db2093 to db1115.
Mar 28 2022, 2:50 PM · Patch-For-Review, database-backups, DBA
Kormat updated the task description for T301315: Move orchestrator from db2093 to db1115.
Mar 28 2022, 2:45 PM · Patch-For-Review, database-backups, DBA
Kormat updated the task description for T301315: Move orchestrator from db2093 to db1115.
Mar 28 2022, 2:36 PM · Patch-For-Review, database-backups, DBA

Mar 8 2022

Kormat created T303272: Move VTRS db passwords to a different hiera location.
Mar 8 2022, 2:35 PM · Znuny, serviceops, SRE
Kormat moved T303178: Export list of sections from puppet / add support to wmfdb from Triage to Ready on the DBA board.
Mar 8 2022, 11:31 AM · DBA, wmfdb

Mar 7 2022

Kormat added a comment to T276589: migrate services from cumin2001 to cumin2002.

Granted removed:

Mar 7 2022, 2:53 PM · Patch-For-Review, SRE
Kormat added a comment to T276589: migrate services from cumin2001 to cumin2002.

Granted removed:

  • es1
  • es2
  • es3
  • es4
  • es5
  • m1
  • m2
  • m3
  • m5
  • s1
  • s2
  • s3
  • s4
  • s5
  • s6
  • s7
  • s8
  • db_inventory
  • x1
  • x2
  • pc1
  • pc2
  • pc3
  • mediabackupstemp
  • A:db-core-test
Mar 7 2022, 2:10 PM · Patch-For-Review, SRE
jcrespo awarded T303178: Export list of sections from puppet / add support to wmfdb a Like token.
Mar 7 2022, 1:47 PM · DBA, wmfdb
Kormat updated the task description for T300471: Switchover s6 master (db1173 -> db1131).
Mar 7 2022, 1:44 PM · User-notice-archive, DBA
Kormat added a comment to T302283: Read-only window needed for s6.

I'm afraid we have to cancel this maintenance for now. Hopefully we'll be able to do in the next couple of weeks, but i'll open a new task for that when i have a firm date. Sorry for the hassle <3

Mar 7 2022, 1:44 PM · CommRel-Specialists-Support (Jan-Mar-2022), User-notice
Kormat updated the task description for T300471: Switchover s6 master (db1173 -> db1131).
Mar 7 2022, 1:41 PM · User-notice-archive, DBA
Kormat created T303178: Export list of sections from puppet / add support to wmfdb.
Mar 7 2022, 1:35 PM · DBA, wmfdb
Kormat added a comment to T276589: migrate services from cumin2001 to cumin2002.

Change 768657 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove cumin2001 from mysql root clients and related grants

https://gerrit.wikimedia.org/r/768657

Apart from merging this, we also need to remove the grants from production (@Kormat would you be able to take care of that?)

Mar 7 2022, 10:28 AM · Patch-For-Review, SRE

Mar 3 2022

Kormat triaged T302951: db2147 SMART error as High priority.
Mar 3 2022, 11:00 AM · SRE, DBA, ops-codfw

Mar 1 2022

Kormat added a comment to T276589: migrate services from cumin2001 to cumin2002.

I'm working on the wmfdb + wmfmariadbpy sides of this for data-persistence. wmfdb is good to go, i'm currently doing some testing with wmfmariadbpy in pontoon in preparation for doing a new release in the next day or so.

Mar 1 2022, 1:50 PM · Patch-For-Review, SRE
Kormat closed T302796: Deploy wmfmariadbpy 0.9, a subtask of T276589: migrate services from cumin2001 to cumin2002, as Resolved.
Mar 1 2022, 1:50 PM · Patch-For-Review, SRE
Kormat closed T302796: Deploy wmfmariadbpy 0.9 as Resolved.

Deployment complete.

Mar 1 2022, 1:49 PM · DBA
Kormat updated the task description for T302796: Deploy wmfmariadbpy 0.9.
Mar 1 2022, 1:47 PM · DBA
Kormat updated the task description for T302796: Deploy wmfmariadbpy 0.9.
Mar 1 2022, 1:45 PM · DBA
Kormat updated the task description for T302796: Deploy wmfmariadbpy 0.9.
Mar 1 2022, 1:43 PM · DBA
Kormat updated the task description for T302796: Deploy wmfmariadbpy 0.9.
Mar 1 2022, 1:41 PM · DBA
Kormat updated the task description for T302796: Deploy wmfmariadbpy 0.9.
Mar 1 2022, 1:40 PM · DBA
Kormat committed rOSMDdbec737ec27d: Prepare for 0.9 release. (authored by Kormat).
Prepare for 0.9 release.
Mar 1 2022, 1:07 PM
Kormat added a subtask for T276589: migrate services from cumin2001 to cumin2002: T302796: Deploy wmfmariadbpy 0.9.
Mar 1 2022, 12:59 PM · Patch-For-Review, SRE
Kormat added a parent task for T302796: Deploy wmfmariadbpy 0.9: T276589: migrate services from cumin2001 to cumin2002.
Mar 1 2022, 12:59 PM · DBA
Kormat changed the status of T302796: Deploy wmfmariadbpy 0.9 from Open to In Progress.
Mar 1 2022, 12:57 PM · DBA
Kormat updated the task description for T302796: Deploy wmfmariadbpy 0.9.
Mar 1 2022, 12:49 PM · DBA
Kormat created T302796: Deploy wmfmariadbpy 0.9.
Mar 1 2022, 12:49 PM · DBA

Feb 28 2022

Kormat updated subscribers of T276589: migrate services from cumin2001 to cumin2002.

Any update on this? This upgrade is blocking serviceops who needs bullseye for the kubernetes python libraries and cookbooks.

Feb 28 2022, 4:06 PM · Patch-For-Review, SRE
Kormat updated the task description for T300471: Switchover s6 master (db1173 -> db1131).
Feb 28 2022, 9:55 AM · User-notice-archive, DBA