Page MenuHomePhabricator

Kormat (Stevie Shirley (she/her))
SRE

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Apr 14 2020, 7:57 AM (31 w, 6 d)
Availability
Available
LDAP User
Kormat
MediaWiki User
SShirley (WMF) [ Global Accounts ]

Recent Activity

Yesterday

Kormat added a comment to T268336: Cleanup heartbeat.heartbeat on all production instances.

Cleaning up the heartbeat tables in prod is a bit tricky, as there's a lot of cruft, and a mix of STATEMENT vs ROW replication. My suggestion is that we update stale rows to set ts to 0, instead of trying to delete the rows. That makes it easy to filter them out from queries.

Mon, Nov 23, 3:51 PM · Patch-For-Review, DBA, Operations
Kormat updated the task description for T268469: Reboot es1024 (es5 master).
Mon, Nov 23, 3:47 PM · Patch-For-Review, User-Kormat, DBA, Operations, Security
Kormat created P13371 (An Untitled Masterwork).
Mon, Nov 23, 2:15 PM
Kormat updated the task description for T268469: Reboot es1024 (es5 master).
Mon, Nov 23, 1:21 PM · Patch-For-Review, User-Kormat, DBA, Operations, Security
Kormat updated the task description for T268469: Reboot es1024 (es5 master).
Mon, Nov 23, 1:20 PM · Patch-For-Review, User-Kormat, DBA, Operations, Security
Kormat updated the task description for T268469: Reboot es1024 (es5 master).
Mon, Nov 23, 1:05 PM · Patch-For-Review, User-Kormat, DBA, Operations, Security
Kormat updated the task description for T268469: Reboot es1024 (es5 master).
Mon, Nov 23, 1:03 PM · Patch-For-Review, User-Kormat, DBA, Operations, Security
Kormat created T268469: Reboot es1024 (es5 master).
Mon, Nov 23, 1:03 PM · Patch-For-Review, User-Kormat, DBA, Operations, Security
Kormat created P13360 gerrit keybinding ignore.js.
Mon, Nov 23, 9:26 AM
Kormat closed T268098: Make innodb_change_buffering configurable as Resolved.

Resolved by https://gerrit.wikimedia.org/r/641986

Mon, Nov 23, 8:27 AM · Data-Services, cloud-services-team (Kanban)
Kormat closed T268098: Make innodb_change_buffering configurable, a subtask of T260843: Set up roles for new wiki replicas layout, as Resolved.
Mon, Nov 23, 8:26 AM · Patch-For-Review, Data-Services, cloud-services-team (Kanban)

Fri, Nov 20

Kormat updated the task description for T268336: Cleanup heartbeat.heartbeat on all production instances.
Fri, Nov 20, 3:12 PM · Patch-For-Review, DBA, Operations
Kormat added a comment to T268316: Base replication lag detection on heartbeat.

Ways to achieve this:

  1. Clean up the heartbeat table so that it only contains entries that are supposed to be there (T268336: Cleanup heartbeat.heartbeat on all production instances)
    • With this, we can use the oldest entry in the heartbeat table to measure the current lag
    • This will work for both the current situation, circular replication, and active-active.
  2. Only run pt-heartbeat on primary masters
    • With this, we can use the newest entry in the heartbeat table to measure the current lag, because none of the other entries matter.
    • This will work for the current situation, but will not for circular replication/active-active, as we're back to having >1 entry that matters.
Fri, Nov 20, 2:54 PM · Patch-For-Review, Orchestrator, DBA, Operations
Kormat added a comment to T268316: Base replication lag detection on heartbeat.

Considerations:

  • A non-primary master instance may be running heartbeat (e.g. all parsercache nodes, section masters in backup DC)
    • Ignoring any heartbeat entry generated by the local instance causes all primary masters to show an arbitrary amount of lag, as they will only evaluate stale entries in the heartbeat table.
  • Solution must allow circular replication, like we do leading up to DC switchovers
  • Ideally an instance would display lag relative to it's DC-local master.
    • Otherwise if master in the secondary DC is lagging, _all_ instances in that DC will show lag, which just adds a lot of noise for no information gain.
Fri, Nov 20, 2:00 PM · Patch-For-Review, Orchestrator, DBA, Operations
Kormat added a comment to T268316: Base replication lag detection on heartbeat.

@jcrespo: Orchestrator only supports a single query for all instances. This means we can't supply per-DC/-section/-etc parameters. It also means we can't rely on the section's primary master to be in the $mw_primary DC, as that wasn't true for the misc sections while mw was running in codfw a couple of months ago. So, in short, there's no way to do this using an existing wheel.

Fri, Nov 20, 1:54 PM · Patch-For-Review, Orchestrator, DBA, Operations
Kormat updated the task description for T268336: Cleanup heartbeat.heartbeat on all production instances.
Fri, Nov 20, 1:43 PM · Patch-For-Review, DBA, Operations
Kormat added a comment to T268316: Base replication lag detection on heartbeat.

Trying to do this in a reasonable fashion doesn't seem possible without T268336: Cleanup heartbeat.heartbeat on all production instances being done first.

Fri, Nov 20, 1:40 PM · Patch-For-Review, Orchestrator, DBA, Operations
Kormat added a parent task for T268336: Cleanup heartbeat.heartbeat on all production instances: T268316: Base replication lag detection on heartbeat.
Fri, Nov 20, 1:39 PM · Patch-For-Review, DBA, Operations
Kormat added a subtask for T268316: Base replication lag detection on heartbeat: T268336: Cleanup heartbeat.heartbeat on all production instances.
Fri, Nov 20, 1:39 PM · Patch-For-Review, Orchestrator, DBA, Operations
Kormat created P13352 (An Untitled Masterwork).
Fri, Nov 20, 1:19 PM
Kormat created T268336: Cleanup heartbeat.heartbeat on all production instances.
Fri, Nov 20, 1:13 PM · Patch-For-Review, DBA, Operations
Kormat added a comment to P13346 (An Untitled Masterwork).
root@db2094.codfw.wmnet[(none)]> select * from heartbeat.heartbeat;
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+--------
| ts                         | server_id | file              | position   | relay_master_log_file | exec_master_log_pos | shard | datacen
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+--------
| 2018-04-25T13:12:01.000970 | 171966669 | db1075-bin.002735 |  145906585 | NULL                  |                NULL | s3    | eqiad  
| 2017-04-20T16:15:01.000960 | 171970589 | db1040-bin.002987 |  587360761 | db2019-bin.002532     |           773446073 | s4    | eqiad  
| 2018-07-18T06:01:44.001480 | 171970637 | db1052-bin.005945 |  479413784 | NULL                  |                NULL | s1    | eqiad  
| 2020-11-20T12:53:24.001090 | 171970661 | db1083-bin.005754 |  610841608 | NULL                  |                NULL | s1    | eqiad  
| 2017-01-26T08:16:47.000790 | 171974683 | db1057-bin.002885 |  576316184 | db1052-bin.004556     |           210334207 | s1    | eqiad  
| 2019-11-14T06:01:21.000750 | 171974720 | db1067-bin.003024 |  445994072 | NULL                  |                NULL | s1    | eqiad  
| 2017-11-16T17:15:23.000810 | 171974884 | db1063-bin.001382 |  234441239 | NULL                  |                NULL | s5    | eqiad  
| 2017-08-22T07:34:57.000440 | 171978775 | db1068-bin.000516 |  439889287 | NULL                  |                NULL | s4    | eqiad  
| 2018-04-25T13:12:01.000630 | 171978777 | db1070-bin.001894 |   46969145 | NULL                  |                NULL | s5    | eqiad  
| 2018-04-25T13:12:01.000630 | 171978778 | db1071-bin.006549 |  147776760 | NULL                  |                NULL | s8    | eqiad  
| 2019-09-11T05:33:39.001920 | 180355171 | db2048-bin.005384 |  980454332 | db1067-bin.002779     |           346367189 | s1    | codfw  
| 2017-05-10T13:29:31.001050 | 180359172 | db2016-bin.002977 |  503605243 | db1052-bin.004793     |           401461576 | s1    | codfw  
| 2017-05-16T06:34:43.000930 | 180359174 | db2018-bin.003211 | 1032717323 | db1075-bin.001439     |           632039344 | s3    | codfw  
| 2017-05-09T09:24:39.001160 | 180359175 | db2019-bin.002580 |  364384048 | db1068-bin.000049     |           373607821 | s4    | codfw  
|                            | 180359179 | NULL              |       NULL | NULL                  |                NULL | NULL  | NULL   
| 2020-11-20T12:53:24.000700 | 180363268 | db2112-bin.002509 |  488527459 | db1083-bin.005754     |           610841608 | s1    | codfw  
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+--------
Fri, Nov 20, 12:54 PM
Kormat added a comment to P13346 (An Untitled Masterwork).
root@pc2010.codfw.wmnet[(none)]> SELECT NOW(),server_id,ts,TIMEDIFF(NOW(),ts) FROM heartbeat.heartbeat WHERE server_id!=@@server_id ORDER BY ts DESC LIMIT 1;
+---------------------+-----------+----------------------------+--------------------+
| NOW()               | server_id | ts                         | TIMEDIFF(NOW(),ts) |
+---------------------+-----------+----------------------------+--------------------+
| 2020-11-20 10:50:20 | 171966644 | 2020-11-20T10:49:57.000820 | 00:00:22.999180    |
+---------------------+-----------+----------------------------+--------------------+
1 row in set (0.034 sec)
Fri, Nov 20, 10:50 AM
Kormat added a comment to P13346 (An Untitled Masterwork).
root@db1139.eqiad.wmnet[(none)]> SELECT NOW(),ts,TIMEDIFF(NOW(),ts) FROM heartbeat.heartbeat ORDER BY ts DESC LIMIT 1;
+---------------------+----------------------------+--------------------+
| NOW()               | ts                         | TIMEDIFF(NOW(),ts) |
+---------------------+----------------------------+--------------------+
| 2020-11-20 10:44:16 | 2020-11-19T02:54:04.001120 | 31:50:11.998880    |
+---------------------+----------------------------+--------------------+
Fri, Nov 20, 10:44 AM
Kormat added a comment to P13346 (An Untitled Masterwork).
root@db2093.codfw.wmnet[heartbeat]> SELECT TIMEDIFF(ts,NOW()) FROM heartbeat.heartbeat ORDER BY ts DESC LIMIT 1;
+--------------------+
| TIMEDIFF(ts,NOW()) |
+--------------------+
| 00:00:00.000880    |
+--------------------+
1 row in set (0.034 sec)
Fri, Nov 20, 10:39 AM
Kormat created P13346 (An Untitled Masterwork).
Fri, Nov 20, 10:31 AM
Kormat created T268320: Configure mariadb to notice/recover from replication issues quicker.
Fri, Nov 20, 9:42 AM · Orchestrator, DBA, Operations
Kormat created T268316: Base replication lag detection on heartbeat.
Fri, Nov 20, 9:31 AM · Patch-For-Review, Orchestrator, DBA, Operations

Thu, Nov 19

Kormat added a comment to T268233: thanos u/i gives errors if left idle for a few hours.

Just reproduced this in chrome, and got this message:

Access to XMLHttpRequest at 'https://idp.wikimedia.org/login?service=https%3a%2f%2fthanos.wikimedia.org%2fapi%2fv1%2fquery%3fquery%3dsum_over_time(mysql_exporter_last_scrape_error%255B5m%255D)%2520%253E%25201%26dedup%3dtrue%26partial_response%3dtrue%26time%3d1605799406.496%26_%3d1605795670420' (redirected from 'https://thanos.wikimedia.org/api/v1/query?query=sum_over_time(mysql_exporter_last_scrape_error%5B5m%5D)%20%3E%201&dedup=true&partial_response=true&time=1605799406.496&_=1605795670420') from origin 'https://thanos.wikimedia.org' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Thu, Nov 19, 3:24 PM · CAS-SSO, observability, Operations
Kormat created T268233: thanos u/i gives errors if left idle for a few hours.
Thu, Nov 19, 2:38 PM · CAS-SSO, observability, Operations
Kormat claimed T268098: Make innodb_change_buffering configurable.
Thu, Nov 19, 1:00 PM · Data-Services, cloud-services-team (Kanban)
Marostegui awarded T267389: Orchestrator doesn't use FQDN when manipulating replicas a Love token.
Thu, Nov 19, 12:46 PM · Orchestrator, DBA
Kormat closed T267389: Orchestrator doesn't use FQDN when manipulating replicas as Resolved.

Tested a simple case in pontoon, works correctly

Thu, Nov 19, 12:45 PM · Orchestrator, DBA
Kormat updated the task description for T267929: Investigate hostname/fqdn handling in orchestrator.
Thu, Nov 19, 11:18 AM · Orchestrator, DBA

Wed, Nov 18

Kormat added a comment to T266452: Explore orchestrator hooks to integrate them with dbctl, !log, irc alerts and emails.

One thing that's not currently clear how to handle is starting/stopping pt-heartbeat on masters.

Wed, Nov 18, 3:22 PM · Orchestrator, Operations, User-Kormat, DBA
Kormat added a comment to T264797: Update 2030.wikimedia.org redirect to new URI.

Hi. I have merged the gerrit change, but

  • i'm not sure how long it's going to take to take effect
  • the current redirect is a 301 (a 'permenant' redirect), so browsers that have visited the url before are likely to still go to the old location.
Wed, Nov 18, 2:11 PM · Wikimedia-Apache-configuration, Operations
Kormat closed T267401: orchestrator: Use ssl for talking to db servers as Resolved.

Fixed by https://gerrit.wikimedia.org/r/639765. From the commit description:

Wed, Nov 18, 12:57 PM · Orchestrator, Operations, DBA
Kormat added a comment to T267389: Orchestrator doesn't use FQDN when manipulating replicas.

This should now be fixed by https://gerrit.wikimedia.org/r/641402. Needs testing.

Wed, Nov 18, 12:35 PM · Orchestrator, DBA
Kormat claimed T267767: Drop default of revactor_timestamp.
Wed, Nov 18, 11:31 AM · Blocked-on-schema-change, DBA
Kormat closed T259831: Schema change to make change_tag.ct_rc_id unsigned, a subtask of T243318: FY2020-2021 Q1 codfw -> eqiad switchback, as Resolved.
Wed, Nov 18, 11:24 AM · Operations
Kormat closed T259831: Schema change to make change_tag.ct_rc_id unsigned as Resolved.

Check completed successfully, we're done \o/

Wed, Nov 18, 11:24 AM · DBA, Operations, User-Kormat, Blocked-on-schema-change
Kormat added a comment to P13329 (An Untitled Masterwork).
root@db2093.codfw.wmnet[orchestrator]> select hostname from database_instance;
+--------------------+
| hostname           |
+--------------------+
| db1077             |
| pc1010             |
| pc1007.eqiad.wmnet |
| pc1008.eqiad.wmnet |
| pc2007.codfw.wmnet |
| pc2008.codfw.wmnet |
| pc2010.codfw.wmnet |
| db1077.eqiad.wmnet |
| pc1010.eqiad.wmnet |
+--------------------+
Wed, Nov 18, 11:23 AM
Kormat added a comment to T267929: Investigate hostname/fqdn handling in orchestrator.

If there's an entry in database_resolve that maps to a bare hostname, e.g.:

Wed, Nov 18, 10:59 AM · Orchestrator, DBA
Kormat added a comment to T259831: Schema change to make change_tag.ct_rc_id unsigned.

Running one last check over all section instances to confirm the change has been made everywhere.

Wed, Nov 18, 9:55 AM · DBA, Operations, User-Kormat, Blocked-on-schema-change
Kormat updated the task description for T259831: Schema change to make change_tag.ct_rc_id unsigned.
Wed, Nov 18, 9:51 AM · DBA, Operations, User-Kormat, Blocked-on-schema-change

Tue, Nov 17

Kormat updated the task description for T259831: Schema change to make change_tag.ct_rc_id unsigned.
Tue, Nov 17, 10:21 AM · DBA, Operations, User-Kormat, Blocked-on-schema-change
Kormat closed T267929: Investigate hostname/fqdn handling in orchestrator, a subtask of T267389: Orchestrator doesn't use FQDN when manipulating replicas, as Resolved.
Tue, Nov 17, 10:04 AM · Orchestrator, DBA
Kormat closed T267929: Investigate hostname/fqdn handling in orchestrator as Resolved.

Conclusion - these settings are required:

  • "HostnameResolveMethod": "cname" - causes bare hostnames to be resolved to fqdn's correctly.
  • "MysqlHostnameResolveMethod": "@@report_host" - use the host's fqdn as its identity, instead of the bare hostname (which will make SSL verification fail).
  • "DiscoverByShowSlaveHosts": true - as we have skip_resolve_names set to ON in mariadb the replication clients in processlist show IPs, not hostnames; with @@report_host configured we can use 'show slave hosts' which will show the fqdn instead.
Tue, Nov 17, 10:04 AM · Orchestrator, DBA
Kormat updated the task description for T267929: Investigate hostname/fqdn handling in orchestrator.
Tue, Nov 17, 9:48 AM · Orchestrator, DBA
Kormat awarded T238966: Apply updates for MCR, actor migration, and content migration, to production wikis. a 100 token.
Tue, Nov 17, 9:35 AM · MW-1.35-notes (1.35.0-wmf.36; 2020-06-09), Cloud-Services, DBA, Schema-change, Platform Team Workboards (Clinic Duty Team), Platform Team Initiatives (MCR Schema Migration)

Mon, Nov 16

Kormat updated the task description for T267929: Investigate hostname/fqdn handling in orchestrator.
Mon, Nov 16, 2:58 PM · Orchestrator, DBA
Kormat updated the task description for T267929: Investigate hostname/fqdn handling in orchestrator.
Mon, Nov 16, 2:49 PM · Orchestrator, DBA
Kormat created T267929: Investigate hostname/fqdn handling in orchestrator.
Mon, Nov 16, 2:49 PM · Orchestrator, DBA
Kormat updated the task description for T259831: Schema change to make change_tag.ct_rc_id unsigned.
Mon, Nov 16, 1:09 PM · DBA, Operations, User-Kormat, Blocked-on-schema-change
Kormat updated the task description for T256972: Refactor mariadb puppet code.
Mon, Nov 16, 10:42 AM · Patch-For-Review, DBA, Operations, User-jbond, User-Kormat

Fri, Nov 6

Kormat added a comment to P13238 (An Untitled Masterwork).

with @@report_host, fqdn, and DiscoverByShowSlaveHosts:

Fri, Nov 6, 2:24 PM
Kormat added a comment to P13238 (An Untitled Masterwork).

with @@report_host and fqdn for discover:

Fri, Nov 6, 2:11 PM
Kormat added a comment to P13238 (An Untitled Masterwork).

with @@report_host and bare host for discover:

Fri, Nov 6, 2:08 PM
Kormat added a comment to P13238 (An Untitled Masterwork).

with @@hostname and fqdn for discover:

Fri, Nov 6, 2:04 PM
Kormat edited P13238 (An Untitled Masterwork).
Fri, Nov 6, 2:03 PM
Kormat edited P13238 (An Untitled Masterwork).
Fri, Nov 6, 2:02 PM
Kormat edited P13238 (An Untitled Masterwork).
Fri, Nov 6, 2:02 PM
Kormat created P13238 (An Untitled Masterwork).
Fri, Nov 6, 2:00 PM
Kormat added a comment to T267401: orchestrator: Use ssl for talking to db servers.

Looking at the code, it looks like this is what happens:

  • if MySQLTopologyUseMixedTLS is set, check if the host 'requires' ssl
  • if it can auth to the db host without ssl, then it doesn't 'require' ssl, so ssl won't be used.
Fri, Nov 6, 1:25 PM · Orchestrator, Operations, DBA
Kormat created T267401: orchestrator: Use ssl for talking to db servers.
Fri, Nov 6, 1:19 PM · Orchestrator, Operations, DBA
Kormat added a comment to T267389: Orchestrator doesn't use FQDN when manipulating replicas.
case registerCliCommand("register-hostname-unresolve", "Instance, meta", `Assigns the given instance a virtual (aka "unresolved") name`):
Fri, Nov 6, 10:41 AM · Orchestrator, DBA
Kormat added a comment to T267389: Orchestrator doesn't use FQDN when manipulating replicas.

From looking at the code, it seems there's a hostname_unresolve table in the orchestrator db that could map from the bare hostname to "something else" (not clear to me yet). In our case, the hostname_unresolve table is empty, so orchestrator will just use the bare hostname, causing the issue.

Fri, Nov 6, 10:38 AM · Orchestrator, DBA

Thu, Nov 5

Kormat updated the task description for T259831: Schema change to make change_tag.ct_rc_id unsigned.
Thu, Nov 5, 7:31 PM · DBA, Operations, User-Kormat, Blocked-on-schema-change
Kormat closed T267220: db2077 hung on reboot as Resolved.

Great, thanks :)

Thu, Nov 5, 3:24 PM · ops-codfw, DBA, Operations
Kormat closed T267220: db2077 hung on reboot, a subtask of T216240: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092, as Resolved.
Thu, Nov 5, 3:24 PM · Operations, DBA
Kormat added a comment to T267220: db2077 hung on reboot.

@Papaul: it's powering off now. Thanks!

Thu, Nov 5, 2:47 PM · ops-codfw, DBA, Operations
Kormat added a comment to T267220: db2077 hung on reboot.

Perfect. I'll bring it down now, and update here when done.

Thu, Nov 5, 2:44 PM · ops-codfw, DBA, Operations
Kormat updated the task description for T259831: Schema change to make change_tag.ct_rc_id unsigned.
Thu, Nov 5, 1:25 PM · DBA, Operations, User-Kormat, Blocked-on-schema-change
Kormat updated the task description for T259831: Schema change to make change_tag.ct_rc_id unsigned.
Thu, Nov 5, 1:06 PM · DBA, Operations, User-Kormat, Blocked-on-schema-change
Kormat updated the task description for T259831: Schema change to make change_tag.ct_rc_id unsigned.
Thu, Nov 5, 12:47 PM · DBA, Operations, User-Kormat, Blocked-on-schema-change
Kormat updated the task description for T259831: Schema change to make change_tag.ct_rc_id unsigned.
Thu, Nov 5, 11:40 AM · DBA, Operations, User-Kormat, Blocked-on-schema-change
Kormat updated the task description for T259831: Schema change to make change_tag.ct_rc_id unsigned.
Thu, Nov 5, 11:24 AM · DBA, Operations, User-Kormat, Blocked-on-schema-change
Kormat updated the task description for T266483: Enable report_host for mariadb.
Thu, Nov 5, 10:14 AM · Orchestrator, DBA, User-Kormat
Kormat moved T266483: Enable report_host for mariadb from Unsorted 💣 to Blocked 🚧 on the User-Kormat board.
Thu, Nov 5, 10:13 AM · Orchestrator, DBA, User-Kormat
Kormat moved T266483: Enable report_host for mariadb from In progress to Ready on the DBA board.
Thu, Nov 5, 10:13 AM · Orchestrator, DBA, User-Kormat
Kormat edited P13146 T266483 run.
Thu, Nov 5, 9:49 AM
Kormat updated the task description for T266483: Enable report_host for mariadb.
Thu, Nov 5, 9:39 AM · Orchestrator, DBA, User-Kormat
Kormat updated the task description for T266483: Enable report_host for mariadb.
Thu, Nov 5, 9:37 AM · Orchestrator, DBA, User-Kormat
Kormat added a comment to T266483: Enable report_host for mariadb.

es3 eqiad:

  • es1014
  • es1017
  • es1019
  • es1028
  • es1031
Thu, Nov 5, 9:36 AM · Orchestrator, DBA, User-Kormat
Kormat updated the task description for T266483: Enable report_host for mariadb.
Thu, Nov 5, 9:36 AM · Orchestrator, DBA, User-Kormat
Kormat added a comment to T266483: Enable report_host for mariadb.

es2 eqiad:

  • es1011
  • es1013
  • es1015
  • es1026
  • es1030
Thu, Nov 5, 9:35 AM · Orchestrator, DBA, User-Kormat
Kormat updated the task description for T266483: Enable report_host for mariadb.
Thu, Nov 5, 9:35 AM · Orchestrator, DBA, User-Kormat
Kormat added a comment to T266483: Enable report_host for mariadb.

es1 eqiad:

  • es1012
  • es1016
  • es1018
  • es1027
  • es1029
Thu, Nov 5, 9:35 AM · Orchestrator, DBA, User-Kormat
Kormat updated the task description for T266483: Enable report_host for mariadb.
Thu, Nov 5, 9:30 AM · Orchestrator, DBA, User-Kormat
Kormat updated the task description for T266483: Enable report_host for mariadb.
Thu, Nov 5, 9:29 AM · Orchestrator, DBA, User-Kormat
Kormat updated the task description for T266483: Enable report_host for mariadb.
Thu, Nov 5, 9:27 AM · Orchestrator, DBA, User-Kormat
Kormat edited P13146 T266483 run.
Thu, Nov 5, 9:27 AM
Kormat updated the task description for T266483: Enable report_host for mariadb.
Thu, Nov 5, 9:24 AM · Orchestrator, DBA, User-Kormat
Kormat edited P13146 T266483 run.
Thu, Nov 5, 9:23 AM
Kormat assigned T267220: db2077 hung on reboot to Papaul.

Can you run a firmware upgrade on this host, please? Let me know a day that works for you, and i can have the host powered down safely.

Thu, Nov 5, 8:49 AM · ops-codfw, DBA, Operations
Kormat triaged T267220: db2077 hung on reboot as Medium priority.
Thu, Nov 5, 8:48 AM · ops-codfw, DBA, Operations

Wed, Nov 4

Kormat changed the status of T238966: Apply updates for MCR, actor migration, and content migration, to production wikis., a subtask of T215466: Remove revision_comment_temp and revision_actor_temp, from Stalled to Open.
Wed, Nov 4, 3:44 PM · Platform Engineering Roadmap Decision Making, Analytics-Radar, MW-1.35-notes (1.35.0-wmf.32; 2020-05-12), Platform Team Initiatives (Revision Storage Schema Improvements), Technical-Debt, Epic
Kormat changed the status of T238966: Apply updates for MCR, actor migration, and content migration, to production wikis., a subtask of T243318: FY2020-2021 Q1 codfw -> eqiad switchback, from Stalled to Open.
Wed, Nov 4, 3:44 PM · Operations
Kormat changed the status of T238966: Apply updates for MCR, actor migration, and content migration, to production wikis. from Stalled to Open.

Unstalling, @Marostegui is planning to do this soon.

Wed, Nov 4, 3:44 PM · MW-1.35-notes (1.35.0-wmf.36; 2020-06-09), Cloud-Services, DBA, Schema-change, Platform Team Workboards (Clinic Duty Team), Platform Team Initiatives (MCR Schema Migration)
Kormat changed the status of T259831: Schema change to make change_tag.ct_rc_id unsigned, a subtask of T243318: FY2020-2021 Q1 codfw -> eqiad switchback, from Stalled to Open.
Wed, Nov 4, 3:42 PM · Operations