Page MenuHomePhabricator
Create Task
Maniphest T318126

Migrate deployment-prep db hosts to bullseye
Closed, ResolvedPublic
Actions

Description

Production has been migrated to Bullseye, so deployment-prep should do so as well.

Details

SubjectRepoBranchLines +/-
beta: Promote deployment-db09 as master, decom deployment-db07operations/mediawiki-configmaster+10 -21
beta: Add deployment-db10operations/mediawiki-configmaster+19 -8
beta: Pool deployment-db10operations/mediawiki-configmaster+17 -9
Remove deployment-db08operations/mediawiki-configmaster+0 -3
Pool deployment-db09, depool deployment-db08operations/mediawiki-configmaster+10 -10
Add back deployment-db08operations/mediawiki-configmaster+12 -9
Replace deployment-db08 with deployment-db09operations/mediawiki-configmaster+11 -11
Customize query in gerrit

Related Objects
Search...

Event Timeline

Zabe created this task.Sep 19 2022, 10:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 19 2022, 10:55 PM
Stashbot added a comment.Sep 19 2022, 11:57 PM

Mentioned in SAL (#wikimedia-releng) [2022-09-19T23:57:10Z] <zabe> create deployment-db09 as g3.cores8.ram16.disk20 # T318126

Stashbot added a comment.Sep 19 2022, 11:57 PM

Mentioned in SAL (#wikimedia-releng) [2022-09-19T23:57:28Z] <zabe> create volume db09 and attach to deployment-db09 # T318126

Stashbot added a comment.Sep 19 2022, 11:58 PM

Mentioned in SAL (#wikimedia-releng) [2022-09-19T23:58:19Z] <zabe> install mariadb 10.6 via role::mariadb::beta on deployment-db09 # T318126

Stashbot added a comment.Sep 20 2022, 2:06 AM

Mentioned in SAL (#wikimedia-releng) [2022-09-20T02:06:55Z] <zabe> created backup of all databases on deployment-db08 # T318126

TheresNoTime subscribed.Sep 20 2022, 10:03 AM
Stashbot added a comment.Sep 20 2022, 8:04 PM

Mentioned in SAL (#wikimedia-releng) [2022-09-20T20:04:49Z] <zabe> rsynced dump from deployment-db08 to deployment-db09 # T318126

Stashbot added a comment.Sep 20 2022, 8:06 PM

Mentioned in SAL (#wikimedia-releng) [2022-09-20T20:06:58Z] <zabe> deployment-db09: import dump into mariadb # T318126

Stashbot added a comment.Sep 20 2022, 10:00 PM

Mentioned in SAL (#wikimedia-releng) [2022-09-20T22:00:06Z] <zabe> deployment-db09: start replication # T318126

gerritbot added a comment.Sep 20 2022, 10:04 PM

Change 833461 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Replace deployment-db08 with deployment-db09

https://gerrit.wikimedia.org/r/833461

gerritbot added a project: Patch-For-Review.Sep 20 2022, 10:04 PM
gerritbot added a comment.Sep 21 2022, 1:27 PM

Change 833461 merged by jenkins-bot:

[operations/mediawiki-config@master] Replace deployment-db08 with deployment-db09

https://gerrit.wikimedia.org/r/833461

Maintenance_bot removed a project: Patch-For-Review.Sep 21 2022, 1:30 PM
Stashbot added a comment.Sep 21 2022, 1:32 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-21T13:32:01Z] <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/db-labs.php: Config: [[gerrit:833461|Replace deployment-db08 with deployment-db09 (T318126)]] (Beta-only, replace one replica with another) (duration: 03m 56s)

gerritbot added a comment.Sep 21 2022, 1:49 PM

Change 833776 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Add back deployment-db08

https://gerrit.wikimedia.org/r/833776

gerritbot added a project: Patch-For-Review.Sep 21 2022, 1:49 PM
gerritbot added a comment.Sep 21 2022, 1:52 PM

Change 833776 merged by jenkins-bot:

[operations/mediawiki-config@master] Add back deployment-db08

https://gerrit.wikimedia.org/r/833776

Stashbot added a comment.Sep 21 2022, 1:57 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-21T13:57:09Z] <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/db-labs.php: Config: [[gerrit:833776|Add back deployment-db08 (T318126)]] (Beta-only, restore old replica) (duration: 03m 48s)

Lucas_Werkmeister_WMDE subscribed.Sep 21 2022, 1:58 PM

db09 seems to be missing some grants; can be tested on the command line:

lucaswerkmeister-wmde@deployment-mediawiki12:~$ sudo -u www-data php /srv/mediawiki/multiversion/MWScript.php sql.php enwiki --replicadb deployment-db09
Wikimedia\Rdbms\DBConnectionError from line 1487 of /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php: Cannot access the database: Host '172.16.2.65' is not allowed to connect to this MariaDB server (deployment-db09)
#0 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(982): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#1 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(954): Wikimedia\Rdbms\LoadBalancer->getServerConnection(2, 'enwiki', 0)
#2 /srv/mediawiki/php-master/includes/libs/rdbms/database/DBConnRef.php(95): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(2, Array, 'enwiki', 0)
#3 /srv/mediawiki/php-master/includes/libs/rdbms/database/DBConnRef.php(101): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#4 /srv/mediawiki/php-master/includes/libs/rdbms/database/DBConnRef.php(175): Wikimedia\Rdbms\DBConnRef->__call('getLBInfo', Array)
#5 /srv/mediawiki/php-master/maintenance/sql.php(88): Wikimedia\Rdbms\DBConnRef->getLBInfo('master')
#6 /srv/mediawiki/php-master/maintenance/includes/MaintenanceRunner.php(309): MwSql->execute()
#7 /srv/mediawiki/php-master/maintenance/doMaintenance.php(85): MediaWiki\Maintenance\MaintenanceRunner->run()
#8 /srv/mediawiki/php-master/maintenance/sql.php(236): require_once('/srv/mediawiki/...')
#9 /srv/mediawiki/multiversion/MWScript.php(120): require_once('/srv/mediawiki/...')
#10 {main}
Lucas_Werkmeister_WMDE unsubscribed.Sep 21 2022, 1:58 PM
Stashbot added a comment.Sep 21 2022, 2:21 PM

Mentioned in SAL (#wikimedia-releng) [2022-09-21T14:21:16Z] <zabe> deployment-db09: restart mariadb # T318126

Zabe added a comment.EditedSep 21 2022, 2:23 PM

I restarted mariadb on deployment-db09 (after gracefully stopping replication), which fixed the issue.

zabe@deployment-mwmaint02:~$ sql --wiki=enwiki --host=deployment-db09
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 689
Server version: 10.6.8-MariaDB-log MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

wikiadmin@172.16.0.238(enwiki)>
Maintenance_bot removed a project: Patch-For-Review.Sep 21 2022, 2:30 PM
gerritbot added a comment.Sep 21 2022, 2:31 PM

Change 833783 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Pool deployment-db09, depool deployment-db08

https://gerrit.wikimedia.org/r/833783

gerritbot added a project: Patch-For-Review.Sep 21 2022, 2:31 PM
gerritbot added a comment.Sep 21 2022, 2:39 PM

Change 833783 merged by jenkins-bot:

[operations/mediawiki-config@master] Pool deployment-db09, depool deployment-db08

https://gerrit.wikimedia.org/r/833783

Stashbot added a comment.Sep 21 2022, 2:44 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-21T14:44:08Z] <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/db-labs.php: Config: [[gerrit:833783|Pool deployment-db09, depool deployment-db08 (T318126)]] (Beta-only, exchange one replica for another) (duration: 03m 48s)

Stashbot added a comment.Sep 21 2022, 2:50 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-21T14:50:05Z] <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/db-labs.php: Config: [[gerrit:833783|Pool deployment-db09, depool deployment-db08 (T318126)]] (Beta-only, exchange one replica for another) [*actually* sync it this time since I forgot to git rebase before the last sync 🤦] (duration: 03m 41s)

Maintenance_bot removed a project: Patch-For-Review.Sep 21 2022, 3:30 PM
gerritbot added a comment.Sep 21 2022, 5:19 PM

Change 833830 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] Remove deployment-db08

https://gerrit.wikimedia.org/r/833830

gerritbot added a project: Patch-For-Review.Sep 21 2022, 5:19 PM
Stashbot added a comment.Sep 21 2022, 5:54 PM

Mentioned in SAL (#wikimedia-releng) [2022-09-21T17:54:49Z] <zabe> create deployment-db10 as g3.cores8.ram16.disk20 # T318126

Stashbot added a comment.Sep 21 2022, 5:55 PM

Mentioned in SAL (#wikimedia-releng) [2022-09-21T17:55:14Z] <zabe> create volume db10 and attach to deployment-db10 # T318126

Stashbot added a comment.Sep 21 2022, 6:19 PM

Mentioned in SAL (#wikimedia-releng) [2022-09-21T18:19:52Z] <zabe> install mariadb 10.6 via role::mariadb::beta on deployment-db10 # T318126

gerritbot added a comment.Sep 21 2022, 8:03 PM

Change 833830 merged by jenkins-bot:

[operations/mediawiki-config@master] Remove deployment-db08

https://gerrit.wikimedia.org/r/833830

Stashbot added a comment.Sep 21 2022, 8:04 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-21T20:04:30Z] <samtar@deploy1002> Started scap: Backport for [[gerrit:833830|Remove deployment-db08 (T318126)]]

Stashbot added a comment.Sep 21 2022, 8:04 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-21T20:04:55Z] <samtar@deploy1002> samtar and zabe: Backport for [[gerrit:833830|Remove deployment-db08 (T318126)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet

Stashbot added a comment.Sep 21 2022, 8:09 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-21T20:09:46Z] <samtar@deploy1002> Finished scap: Backport for [[gerrit:833830|Remove deployment-db08 (T318126)]] (duration: 05m 16s)

Maintenance_bot removed a project: Patch-For-Review.Sep 21 2022, 8:30 PM
Stashbot added a comment.Sep 21 2022, 8:46 PM

Mentioned in SAL (#wikimedia-releng) [2022-09-21T20:46:02Z] <zabe> clone deployment-db10 from deployment-db08 # T318126

Stashbot added a comment.Sep 21 2022, 11:34 PM

Mentioned in SAL (#wikimedia-releng) [2022-09-21T23:34:49Z] <zabe> shutoff deployment-db08 # T318126

gerritbot added a comment.Sep 21 2022, 11:44 PM

Change 833865 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] beta: Add deployment-db10

https://gerrit.wikimedia.org/r/833865

gerritbot added a project: Patch-For-Review.Sep 21 2022, 11:44 PM

Change 833866 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] beta: Pool deployment-db10

https://gerrit.wikimedia.org/r/833866

Stashbot added a comment.Sep 22 2022, 10:38 AM

Mentioned in SAL (#wikimedia-releng) [2022-09-22T10:38:13Z] <zabe> deployment-db10: start replication # T318126

gerritbot added a comment.Sep 22 2022, 1:12 PM

Change 833866 abandoned by Zabe:

[operations/mediawiki-config@master] beta: Pool deployment-db10

Reason:

https://gerrit.wikimedia.org/r/833866

gerritbot added a comment.Sep 22 2022, 1:17 PM

Change 833865 merged by jenkins-bot:

[operations/mediawiki-config@master] beta: Add deployment-db10

https://gerrit.wikimedia.org/r/833865

gerritbot added a comment.Sep 22 2022, 1:45 PM

Change 834058 had a related patch set uploaded (by Zabe; author: Zabe):

[operations/mediawiki-config@master] beta: Promote deployment-db09 as master, decom deployment-db06

https://gerrit.wikimedia.org/r/834058

Stashbot added a comment.Sep 22 2022, 8:25 PM

Mentioned in SAL (#wikimedia-releng) [2022-09-22T20:25:04Z] <zabe> set deployment-prep as read-only # T318126

gerritbot added a comment.Sep 22 2022, 8:27 PM

Change 834058 merged by jenkins-bot:

[operations/mediawiki-config@master] beta: Promote deployment-db09 as master, decom deployment-db07

https://gerrit.wikimedia.org/r/834058

Stashbot added a comment.Sep 22 2022, 8:32 PM

Mentioned in SAL (#wikimedia-releng) [2022-09-22T20:32:38Z] <zabe> failover deployment-prep master from deployment-db07 to deployment-db09 # T318126

Stashbot added a comment.Sep 22 2022, 8:36 PM

Mentioned in SAL (#wikimedia-releng) [2022-09-22T20:36:34Z] <zabe> take deployment-prep out of read-only # T318126

Stashbot added a comment.Sep 22 2022, 8:40 PM

Mentioned in SAL (#wikimedia-releng) [2022-09-22T20:40:27Z] <zabe> shutoff deployment-db07 # T318126

Zabe closed this task as Resolved.Sep 22 2022, 9:22 PM
Zabe claimed this task.
Zabe removed a project: Patch-For-Review.

Both hosts are now running MariaDB 10.6 on Bullseye.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits