Page MenuHomePhabricator

Kormat (Stevie Shirley (she/her))
SRE

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Apr 14 2020, 7:57 AM (39 w, 6 d)
Availability
Available
LDAP User
Kormat
MediaWiki User
SShirley (WMF) [ Global Accounts ]

Recent Activity

Today

Kormat created P13800 (An Untitled Masterwork).
Mon, Jan 18, 2:31 PM
Kormat updated the task description for T266106: orchestrator: Support SSO.
Mon, Jan 18, 10:55 AM · Patch-For-Review, Orchestrator, CAS-SSO, SRE, User-Kormat, DBA
Kormat updated the task description for T266106: orchestrator: Support SSO.
Mon, Jan 18, 9:19 AM · Patch-For-Review, Orchestrator, CAS-SSO, SRE, User-Kormat, DBA
Kormat created P13789 (An Untitled Masterwork).
Mon, Jan 18, 8:48 AM

Fri, Jan 15

Kormat updated the task description for T266106: orchestrator: Support SSO.
Fri, Jan 15, 11:05 AM · Patch-For-Review, Orchestrator, CAS-SSO, SRE, User-Kormat, DBA
Kormat updated the task description for T266106: orchestrator: Support SSO.
Fri, Jan 15, 10:16 AM · Patch-For-Review, Orchestrator, CAS-SSO, SRE, User-Kormat, DBA

Thu, Jan 14

Kormat updated the task description for T267767: Drop default of revactor_timestamp.
Thu, Jan 14, 12:49 PM · Blocked-on-schema-change, DBA
Kormat added a comment to T267767: Drop default of revactor_timestamp.

s6 codfw progress:

Thu, Jan 14, 12:48 PM · Blocked-on-schema-change, DBA
Kormat updated the task description for T267767: Drop default of revactor_timestamp.
Thu, Jan 14, 12:38 PM · Blocked-on-schema-change, DBA
Kormat archived P13752 (An Untitled Masterwork).
Thu, Jan 14, 9:49 AM

Wed, Jan 13

Kormat created P13752 (An Untitled Masterwork).
Wed, Jan 13, 10:34 AM

Mon, Jan 11

Kormat added a comment to P13714 (An Untitled Masterwork).
root@db2079.codfw.wmnet[wikidatawiki]> select * from information_schema.columns where table_name='revision_actor_temp';
+---------------+--------------+---------------------+--------------------+------------------+----------------+-------------+
| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME          | COLUMN_NAME        | ORDINAL_POSITION | COLUMN_DEFAULT | IS_NULLABLE |
+---------------+--------------+---------------------+--------------------+------------------+----------------+-------------+
| def           | wikidatawiki | revision_actor_temp | revactor_rev       |                1 | NULL           | NO          |
| def           | wikidatawiki | revision_actor_temp | revactor_actor     |                2 | NULL           | NO          |
| def           | wikidatawiki | revision_actor_temp | revactor_timestamp |                3 |                | NO          |
| def           | wikidatawiki | revision_actor_temp | revactor_page      |                4 | NULL           | NO          |
+---------------+--------------+---------------------+--------------------+------------------+----------------+-------------+
Mon, Jan 11, 1:58 PM

Fri, Jan 8

Kormat archived P13661 (An Untitled Masterwork).
Fri, Jan 8, 9:00 AM

Thu, Jan 7

Kormat added a comment to P13661 (An Untitled Masterwork).
Date:   Thu Jan 7 14:46:40 2021 +0000
Thu, Jan 7, 2:47 PM
Kormat created P13661 (An Untitled Masterwork).
Thu, Jan 7, 2:24 PM

Tue, Jan 5

Kormat moved T221159: FY18/19 TEC1.6 Q4: Improve or replace the usage of GTID_WAIT with pt-heartbeat in MW from Unsorted 💣 to Watching 👀 on the User-Kormat board.
Tue, Jan 5, 9:04 AM · User-Kormat, MW-1.36-notes (1.36.0-wmf.20; 2020-12-01), Patch-For-Review, Performance-Team (Radar), User-mobrovac, Services (watching), Goal, Wikimedia-Rdbms, DBA
Kormat added a project to T221159: FY18/19 TEC1.6 Q4: Improve or replace the usage of GTID_WAIT with pt-heartbeat in MW: User-Kormat.
Tue, Jan 5, 9:03 AM · User-Kormat, MW-1.36-notes (1.36.0-wmf.20; 2020-12-01), Patch-For-Review, Performance-Team (Radar), User-mobrovac, Services (watching), Goal, Wikimedia-Rdbms, DBA

Dec 11 2020

Kormat committed rOSMD38660e943a91: integration: Complete framework for running basic tests (authored by Kormat).
integration: Complete framework for running basic tests
Dec 11 2020, 3:53 PM

Dec 9 2020

Kormat added a comment to P13535 (An Untitled Masterwork).

Startup failure:

$ pytest  wmfmariadbpy/test/integration/cli_admin/test_mysqlpy.py 
Image already up-to-date
FATAL: Already running
Exit: integration env setup failed: Command 'integration-env build && integration-env start' returned non-zero exit status 1.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! _pytest.outcomes.Exit: integration env setup failed: Command 'integration-env build && integration-env start' returned non-zero exit status 1. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Dec 9 2020, 1:05 PM
Kormat created P13535 (An Untitled Masterwork).
Dec 9 2020, 10:01 AM

Dec 7 2020

Kormat added a comment to T267376: Set up IP addresses for the new wiki replicas setup.

Nevermind; the mysql protocol is not compatible with SNI: https://stackoverflow.com/a/50301712/13706377

Dec 7 2020, 9:38 AM · Patch-For-Review, Data-Services, cloud-services-team (Kanban)
Kormat added a comment to T267376: Set up IP addresses for the new wiki replicas setup.

Would it be feasible to use SNI on haproxy to select the backend? It looks like it has SNI support.

Dec 7 2020, 8:58 AM · Patch-For-Review, Data-Services, cloud-services-team (Kanban)

Dec 3 2020

Kormat archived P13518 (An Untitled Masterwork).
Dec 3 2020, 8:31 AM

Dec 2 2020

Kormat created P13518 (An Untitled Masterwork).
Dec 2 2020, 3:29 PM

Dec 1 2020

Kormat added a comment to T268233: thanos u/i gives errors if left idle for a few hours.

Do you get this error on all expressions, a specific expression or spasmodically? have also tagged observability in case there is something other then CORS in play

Dec 1 2020, 12:55 PM · CAS-SSO, observability, SRE
Kormat added a comment to T268233: thanos u/i gives errors if left idle for a few hours.

I'm still getting failures, but it's not clear where the issue is.

Dec 1 2020, 10:37 AM · CAS-SSO, observability, SRE
jcrespo awarded T269097: Standardize extra-port for mariadb instances a Like token.
Dec 1 2020, 10:24 AM · DBA
Kormat created T269097: Standardize extra-port for mariadb instances.
Dec 1 2020, 10:21 AM · DBA
Kormat archived P13476 (An Untitled Masterwork).
Dec 1 2020, 9:12 AM

Nov 30 2020

Kormat edited projects for T269015: Requesting access to security@, added: Security-Team; removed Security.
Nov 30 2020, 4:37 PM · Security-Team
Kormat created T269015: Requesting access to security@.
Nov 30 2020, 4:36 PM · Security-Team
Kormat created P13476 (An Untitled Masterwork).
Nov 30 2020, 1:54 PM

Nov 26 2020

Kormat moved T265266: Create integration test env for wmfmariadbpy from Ready to In progress on the DBA board.
Nov 26 2020, 10:04 AM · Release-Engineering-Team, Continuous-Integration-Config, User-Kormat, DBA, SRE
Kormat updated the task description for T265266: Create integration test env for wmfmariadbpy.
Nov 26 2020, 10:04 AM · Release-Engineering-Team, Continuous-Integration-Config, User-Kormat, DBA, SRE
Kormat closed T257033: Standardize/centralize mapping from section to mariadb port/socket and prom-mysql-exporter port as Resolved.

I think it's good enough to resolve at this point. Thanks for the reminder :)

Nov 26 2020, 9:25 AM · Patch-For-Review, DBA, SRE, User-jbond, User-Kormat
Kormat closed T257033: Standardize/centralize mapping from section to mariadb port/socket and prom-mysql-exporter port, a subtask of T256972: Refactor mariadb puppet code, as Resolved.
Nov 26 2020, 9:25 AM · Patch-For-Review, DBA, SRE, User-jbond, User-Kormat

Nov 25 2020

Kormat moved T265266: Create integration test env for wmfmariadbpy from Back Burner 🏛️ to Active 🚁 on the User-Kormat board.
Nov 25 2020, 1:13 PM · Release-Engineering-Team, Continuous-Integration-Config, User-Kormat, DBA, SRE
Kormat moved T268735: debdeploy skipped hosts and assumed they're up to date(?) from Unsorted 💣 to Watching 👀 on the User-Kormat board.
Nov 25 2020, 11:27 AM · User-Kormat, SRE
Kormat triaged T268735: debdeploy skipped hosts and assumed they're up to date(?) as Medium priority.
Nov 25 2020, 11:27 AM · User-Kormat, SRE
Kormat created T268735: debdeploy skipped hosts and assumed they're up to date(?).
Nov 25 2020, 11:08 AM · User-Kormat, SRE
Kormat created P13406 (An Untitled Masterwork).
Nov 25 2020, 10:53 AM
Kormat added a comment to T268316: Base replication lag detection on heartbeat.

(and re-starting heartbeat makes the lag disappear ~instantly)

Nov 25 2020, 9:54 AM · Orchestrator, DBA, SRE
Kormat added a comment to T268316: Base replication lag detection on heartbeat.

I've tested it in pontoon — stopping heartbeat on the master causes immediate lag to show up for the entire tree.

Nov 25 2020, 9:53 AM · Orchestrator, DBA, SRE
Kormat closed T268316: Base replication lag detection on heartbeat as Resolved.

The orchestrator config change has been deployed, and the heartbeat tables for pc{1,2,3} have been cleaned up. Other sections will need similar cleanups before orchestrator can manage them properly.

Nov 25 2020, 9:46 AM · Orchestrator, DBA, SRE
Kormat closed T268469: Reboot es1024 (es5 master) as Resolved.

Completed.

Nov 25 2020, 9:06 AM · User-Kormat, DBA, SRE, Security
Kormat updated the task description for T268469: Reboot es1024 (es5 master).
Nov 25 2020, 9:05 AM · User-Kormat, DBA, SRE, Security
Kormat updated the task description for T268469: Reboot es1024 (es5 master).
Nov 25 2020, 8:43 AM · User-Kormat, DBA, SRE, Security
Kormat updated the task description for T268469: Reboot es1024 (es5 master).
Nov 25 2020, 8:40 AM · User-Kormat, DBA, SRE, Security
Kormat updated the task description for T268469: Reboot es1024 (es5 master).
Nov 25 2020, 8:33 AM · User-Kormat, DBA, SRE, Security
Kormat added a comment to T268233: thanos u/i gives errors if left idle for a few hours.

This is still occurring for me. From firefox:

Nov 25 2020, 8:26 AM · CAS-SSO, observability, SRE
Kormat updated the task description for T268469: Reboot es1024 (es5 master).
Nov 25 2020, 8:14 AM · User-Kormat, DBA, SRE, Security
Kormat updated the task description for T268469: Reboot es1024 (es5 master).
Nov 25 2020, 8:04 AM · User-Kormat, DBA, SRE, Security

Nov 24 2020

Kormat committed rOSMDadd8ed157392: Prepare for 0.6 release. (authored by Kormat).
Prepare for 0.6 release.
Nov 24 2020, 3:31 PM
Kormat added a comment to T268636: Remove 'kraken' user.

It looks like this is veery old. The only mentions in the private repo:

Nov 24 2020, 3:23 PM · Analytics, DBA, WMF-NDA

Nov 23 2020

Kormat added a comment to T268336: Cleanup heartbeat.heartbeat on all production instances.

Cleaning up the heartbeat tables in prod is a bit tricky, as there's a lot of cruft, and a mix of STATEMENT vs ROW replication. My suggestion is that we update stale rows to set ts to 0, instead of trying to delete the rows. That makes it easy to filter them out from queries.

Nov 23 2020, 3:51 PM · Orchestrator, DBA
Kormat updated the task description for T268469: Reboot es1024 (es5 master).
Nov 23 2020, 3:47 PM · User-Kormat, DBA, SRE, Security
Kormat created P13371 (An Untitled Masterwork).
Nov 23 2020, 2:15 PM
Kormat updated the task description for T268469: Reboot es1024 (es5 master).
Nov 23 2020, 1:21 PM · User-Kormat, DBA, SRE, Security
Kormat updated the task description for T268469: Reboot es1024 (es5 master).
Nov 23 2020, 1:20 PM · User-Kormat, DBA, SRE, Security
Kormat updated the task description for T268469: Reboot es1024 (es5 master).
Nov 23 2020, 1:05 PM · User-Kormat, DBA, SRE, Security
Kormat updated the task description for T268469: Reboot es1024 (es5 master).
Nov 23 2020, 1:03 PM · User-Kormat, DBA, SRE, Security
Kormat created T268469: Reboot es1024 (es5 master).
Nov 23 2020, 1:03 PM · User-Kormat, DBA, SRE, Security
Kormat created P13360 gerrit keybinding ignore.js.
Nov 23 2020, 9:26 AM
Kormat closed T268098: Make innodb_change_buffering configurable as Resolved.

Resolved by https://gerrit.wikimedia.org/r/641986

Nov 23 2020, 8:27 AM · Data-Services, cloud-services-team (Kanban)
Kormat closed T268098: Make innodb_change_buffering configurable, a subtask of T260843: Set up roles for new wiki replicas layout, as Resolved.
Nov 23 2020, 8:26 AM · Patch-For-Review, Data-Services, cloud-services-team (Kanban)

Nov 20 2020

Kormat updated the task description for T268336: Cleanup heartbeat.heartbeat on all production instances.
Nov 20 2020, 3:12 PM · Orchestrator, DBA
Kormat added a comment to T268316: Base replication lag detection on heartbeat.

Ways to achieve this:

  1. Clean up the heartbeat table so that it only contains entries that are supposed to be there (T268336: Cleanup heartbeat.heartbeat on all production instances)
    • With this, we can use the oldest entry in the heartbeat table to measure the current lag, because all entries are relevant.
    • This will work for both the current situation, circular replication, and active-active.
  2. Only run pt-heartbeat on primary masters
    • With this, we can use the newest entry in the heartbeat table to measure the current lag, because none of the other entries matter.
    • This will work for the current situation, but will not for circular replication/active-active, as we're back to having >1 entry that matters.
Nov 20 2020, 2:54 PM · Orchestrator, DBA, SRE
Kormat added a comment to T268316: Base replication lag detection on heartbeat.

Considerations:

  • A non-primary master instance may be running heartbeat (e.g. all parsercache nodes, section masters in backup DC)
    • Ignoring any heartbeat entry generated by the local instance causes all primary masters to show an arbitrary amount of lag, as they will only evaluate stale entries in the heartbeat table.
  • Solution must allow circular replication, like we do leading up to DC switchovers
  • Solution must not assume primary master is in the MW primary DC.
    • This would have broken for the misc sections when MW was in CODFW recently.
  • Ideally an instance would display lag relative to it's DC-local master.
    • Otherwise if master in the secondary DC is lagging, _all_ instances in that DC will show lag, which just adds a lot of noise for no information gain.
Nov 20 2020, 2:00 PM · Orchestrator, DBA, SRE
Kormat added a comment to T268316: Base replication lag detection on heartbeat.

@jcrespo: Orchestrator only supports a single query for all instances. This means we can't supply per-DC/-section/-etc parameters. It also means we can't rely on the section's primary master to be in the $mw_primary DC, as that wasn't true for the misc sections while mw was running in codfw a couple of months ago. So, in short, there's no way to do this using an existing wheel.

Nov 20 2020, 1:54 PM · Orchestrator, DBA, SRE
Kormat updated the task description for T268336: Cleanup heartbeat.heartbeat on all production instances.
Nov 20 2020, 1:43 PM · Orchestrator, DBA
Kormat added a comment to T268316: Base replication lag detection on heartbeat.

Trying to do this in a reasonable fashion doesn't seem possible without T268336: Cleanup heartbeat.heartbeat on all production instances being done first.

Nov 20 2020, 1:40 PM · Orchestrator, DBA, SRE
Kormat added a parent task for T268336: Cleanup heartbeat.heartbeat on all production instances: T268316: Base replication lag detection on heartbeat.
Nov 20 2020, 1:39 PM · Orchestrator, DBA
Kormat added a subtask for T268316: Base replication lag detection on heartbeat: T268336: Cleanup heartbeat.heartbeat on all production instances.
Nov 20 2020, 1:39 PM · Orchestrator, DBA, SRE
Kormat created P13352 (An Untitled Masterwork).
Nov 20 2020, 1:19 PM
Kormat created T268336: Cleanup heartbeat.heartbeat on all production instances.
Nov 20 2020, 1:13 PM · Orchestrator, DBA
Kormat added a comment to P13346 (An Untitled Masterwork).
root@db2094.codfw.wmnet[(none)]> select * from heartbeat.heartbeat;
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+--------
| ts                         | server_id | file              | position   | relay_master_log_file | exec_master_log_pos | shard | datacen
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+--------
| 2018-04-25T13:12:01.000970 | 171966669 | db1075-bin.002735 |  145906585 | NULL                  |                NULL | s3    | eqiad  
| 2017-04-20T16:15:01.000960 | 171970589 | db1040-bin.002987 |  587360761 | db2019-bin.002532     |           773446073 | s4    | eqiad  
| 2018-07-18T06:01:44.001480 | 171970637 | db1052-bin.005945 |  479413784 | NULL                  |                NULL | s1    | eqiad  
| 2020-11-20T12:53:24.001090 | 171970661 | db1083-bin.005754 |  610841608 | NULL                  |                NULL | s1    | eqiad  
| 2017-01-26T08:16:47.000790 | 171974683 | db1057-bin.002885 |  576316184 | db1052-bin.004556     |           210334207 | s1    | eqiad  
| 2019-11-14T06:01:21.000750 | 171974720 | db1067-bin.003024 |  445994072 | NULL                  |                NULL | s1    | eqiad  
| 2017-11-16T17:15:23.000810 | 171974884 | db1063-bin.001382 |  234441239 | NULL                  |                NULL | s5    | eqiad  
| 2017-08-22T07:34:57.000440 | 171978775 | db1068-bin.000516 |  439889287 | NULL                  |                NULL | s4    | eqiad  
| 2018-04-25T13:12:01.000630 | 171978777 | db1070-bin.001894 |   46969145 | NULL                  |                NULL | s5    | eqiad  
| 2018-04-25T13:12:01.000630 | 171978778 | db1071-bin.006549 |  147776760 | NULL                  |                NULL | s8    | eqiad  
| 2019-09-11T05:33:39.001920 | 180355171 | db2048-bin.005384 |  980454332 | db1067-bin.002779     |           346367189 | s1    | codfw  
| 2017-05-10T13:29:31.001050 | 180359172 | db2016-bin.002977 |  503605243 | db1052-bin.004793     |           401461576 | s1    | codfw  
| 2017-05-16T06:34:43.000930 | 180359174 | db2018-bin.003211 | 1032717323 | db1075-bin.001439     |           632039344 | s3    | codfw  
| 2017-05-09T09:24:39.001160 | 180359175 | db2019-bin.002580 |  364384048 | db1068-bin.000049     |           373607821 | s4    | codfw  
|                            | 180359179 | NULL              |       NULL | NULL                  |                NULL | NULL  | NULL   
| 2020-11-20T12:53:24.000700 | 180363268 | db2112-bin.002509 |  488527459 | db1083-bin.005754     |           610841608 | s1    | codfw  
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+--------
Nov 20 2020, 12:54 PM
Kormat added a comment to P13346 (An Untitled Masterwork).
root@pc2010.codfw.wmnet[(none)]> SELECT NOW(),server_id,ts,TIMEDIFF(NOW(),ts) FROM heartbeat.heartbeat WHERE server_id!=@@server_id ORDER BY ts DESC LIMIT 1;
+---------------------+-----------+----------------------------+--------------------+
| NOW()               | server_id | ts                         | TIMEDIFF(NOW(),ts) |
+---------------------+-----------+----------------------------+--------------------+
| 2020-11-20 10:50:20 | 171966644 | 2020-11-20T10:49:57.000820 | 00:00:22.999180    |
+---------------------+-----------+----------------------------+--------------------+
1 row in set (0.034 sec)
Nov 20 2020, 10:50 AM
Kormat added a comment to P13346 (An Untitled Masterwork).
root@db1139.eqiad.wmnet[(none)]> SELECT NOW(),ts,TIMEDIFF(NOW(),ts) FROM heartbeat.heartbeat ORDER BY ts DESC LIMIT 1;
+---------------------+----------------------------+--------------------+
| NOW()               | ts                         | TIMEDIFF(NOW(),ts) |
+---------------------+----------------------------+--------------------+
| 2020-11-20 10:44:16 | 2020-11-19T02:54:04.001120 | 31:50:11.998880    |
+---------------------+----------------------------+--------------------+
Nov 20 2020, 10:44 AM
Kormat added a comment to P13346 (An Untitled Masterwork).
root@db2093.codfw.wmnet[heartbeat]> SELECT TIMEDIFF(ts,NOW()) FROM heartbeat.heartbeat ORDER BY ts DESC LIMIT 1;
+--------------------+
| TIMEDIFF(ts,NOW()) |
+--------------------+
| 00:00:00.000880    |
+--------------------+
1 row in set (0.034 sec)
Nov 20 2020, 10:39 AM
Kormat created P13346 (An Untitled Masterwork).
Nov 20 2020, 10:31 AM
Kormat created T268320: Configure mariadb to notice/recover from replication issues quicker.
Nov 20 2020, 9:42 AM · Orchestrator, DBA
Kormat created T268316: Base replication lag detection on heartbeat.
Nov 20 2020, 9:31 AM · Orchestrator, DBA, SRE

Nov 19 2020

Kormat added a comment to T268233: thanos u/i gives errors if left idle for a few hours.

Just reproduced this in chrome, and got this message:

Access to XMLHttpRequest at 'https://idp.wikimedia.org/login?service=https%3a%2f%2fthanos.wikimedia.org%2fapi%2fv1%2fquery%3fquery%3dsum_over_time(mysql_exporter_last_scrape_error%255B5m%255D)%2520%253E%25201%26dedup%3dtrue%26partial_response%3dtrue%26time%3d1605799406.496%26_%3d1605795670420' (redirected from 'https://thanos.wikimedia.org/api/v1/query?query=sum_over_time(mysql_exporter_last_scrape_error%5B5m%5D)%20%3E%201&dedup=true&partial_response=true&time=1605799406.496&_=1605795670420') from origin 'https://thanos.wikimedia.org' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Nov 19 2020, 3:24 PM · CAS-SSO, observability, SRE
Kormat created T268233: thanos u/i gives errors if left idle for a few hours.
Nov 19 2020, 2:38 PM · CAS-SSO, observability, SRE
Kormat claimed T268098: Make innodb_change_buffering configurable.
Nov 19 2020, 1:00 PM · Data-Services, cloud-services-team (Kanban)
Marostegui awarded T267389: Orchestrator doesn't use FQDN when manipulating replicas a Love token.
Nov 19 2020, 12:46 PM · Orchestrator, DBA
Kormat closed T267389: Orchestrator doesn't use FQDN when manipulating replicas as Resolved.

Tested a simple case in pontoon, works correctly

Nov 19 2020, 12:45 PM · Orchestrator, DBA
Kormat updated the task description for T267929: Investigate hostname/fqdn handling in orchestrator.
Nov 19 2020, 11:18 AM · Orchestrator, DBA

Nov 18 2020

Kormat added a comment to T266452: Explore orchestrator hooks to integrate them with dbctl, !log, irc alerts and emails.

One thing that's not currently clear how to handle is starting/stopping pt-heartbeat on masters.

Nov 18 2020, 3:22 PM · Orchestrator, User-Kormat, DBA
Kormat added a comment to T264797: Update 2030.wikimedia.org redirect to new URI.

Hi. I have merged the gerrit change, but

  • i'm not sure how long it's going to take to take effect
  • the current redirect is a 301 (a 'permenant' redirect), so browsers that have visited the url before are likely to still go to the old location.
Nov 18 2020, 2:11 PM · Wikimedia-Apache-configuration, SRE
Kormat closed T267401: orchestrator: Use ssl for talking to db servers as Resolved.

Fixed by https://gerrit.wikimedia.org/r/639765. From the commit description:

Nov 18 2020, 12:57 PM · Orchestrator, SRE, DBA
Kormat added a comment to T267389: Orchestrator doesn't use FQDN when manipulating replicas.

This should now be fixed by https://gerrit.wikimedia.org/r/641402. Needs testing.

Nov 18 2020, 12:35 PM · Orchestrator, DBA
Kormat claimed T267767: Drop default of revactor_timestamp.
Nov 18 2020, 11:31 AM · Blocked-on-schema-change, DBA
Kormat closed T259831: Schema change to make change_tag.ct_rc_id unsigned, a subtask of T243318: FY2020-2021 Q1 codfw -> eqiad switchback, as Resolved.
Nov 18 2020, 11:24 AM · SRE
Kormat closed T259831: Schema change to make change_tag.ct_rc_id unsigned as Resolved.

Check completed successfully, we're done \o/

Nov 18 2020, 11:24 AM · DBA, SRE, User-Kormat, Blocked-on-schema-change
Kormat added a comment to P13329 (An Untitled Masterwork).
root@db2093.codfw.wmnet[orchestrator]> select hostname from database_instance;
+--------------------+
| hostname           |
+--------------------+
| db1077             |
| pc1010             |
| pc1007.eqiad.wmnet |
| pc1008.eqiad.wmnet |
| pc2007.codfw.wmnet |
| pc2008.codfw.wmnet |
| pc2010.codfw.wmnet |
| db1077.eqiad.wmnet |
| pc1010.eqiad.wmnet |
+--------------------+
Nov 18 2020, 11:23 AM
Kormat added a comment to T267929: Investigate hostname/fqdn handling in orchestrator.

If there's an entry in database_resolve that maps to a bare hostname, e.g.:

Nov 18 2020, 10:59 AM · Orchestrator, DBA
Kormat added a comment to T259831: Schema change to make change_tag.ct_rc_id unsigned.

Running one last check over all section instances to confirm the change has been made everywhere.

Nov 18 2020, 9:55 AM · DBA, SRE, User-Kormat, Blocked-on-schema-change
Kormat updated the task description for T259831: Schema change to make change_tag.ct_rc_id unsigned.
Nov 18 2020, 9:51 AM · DBA, SRE, User-Kormat, Blocked-on-schema-change

Nov 17 2020

Kormat updated the task description for T259831: Schema change to make change_tag.ct_rc_id unsigned.
Nov 17 2020, 10:21 AM · DBA, SRE, User-Kormat, Blocked-on-schema-change