@JAllemandou can you clarify if this has any impact on Data Engineering? Thanks.

The impact on DE seems small IMO. I'm not an expert but my uinderstanding is that some databases/tables are to be moved to the x1 section, containing databases used across multiple projects. Those tables, if configured so, should be replicated to the redacted database, one of them being the an-redacteddb1001 we use to sqoop redacted data.
Two things to keep in mind here:

• If we're talking about the database/tables from T395072 and T387419, we currently don't sqoop them
• Longer term we wish to stop sqooping from an-redacteddb and only sqoop from analytics-replicas

Change #1247549 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add x1 section to an-redacteddb1001

https://gerrit.wikimedia.org/r/1247549

Change #1247549 merged by Btullis:

[operations/puppet@production] Add x1 section to an-redacteddb1001

https://gerrit.wikimedia.org/r/1247549

I've added the puppet definition of the x1 section to an-redacteddb1001 now.
However, the service didn't start because the mysql.db database is missing.

I have created a default database with this command:

btullis@an-redacteddb1001:~$ sudo /opt/wmf-mariadb1011/scripts/mariadb-install-db --defaults-file=/etc/mysql/mysqld.conf.d/x1.cnf --basedir /opt/wmf-mariadb1011 --datadir=/srv/sqldata.x1 --user=mysql
Installing MariaDB/MySQL system tables in '/srv/sqldata.x1' ...
OK

This will be deleted when we copy the data files and configure replication, which is dependent on T415219 being completed.

I have also fixed up the Icinga checks by manually creating the grants required to carry out the checks.

root@an-redacteddb1001:x1[(none)]> GRANT BINLOG MONITOR, PROCESS, SLAVE MONITOR ON *.* TO `nagios`@`localhost` IDENTIFIED VIA unix_socket;

root@an-redacteddb1001:x1[(none)]> GRANT BINLOG MONITOR, PROCESS, SLAVE MONITOR ON *.* TO `prometheus`@`localhost` IDENTIFIED VIA unix_socket WITH MAX_USER_CONNECTIONS 5;

root@an-redacteddb1001:x1[(none)]> flush privileges;

I'll now move it out of our current milestons, since we are awaiting the completion of T415219 before proceeding to replicate the data from an upstream host.

In T407485#11667548, @BTullis wrote:

I've added the puppet definition of the x1 section to an-redacteddb1001 now.
However, the service didn't start because the mysql.db database is missing.

I have created a default database with this command:

btullis@an-redacteddb1001:~$ sudo /opt/wmf-mariadb1011/scripts/mariadb-install-db --defaults-file=/etc/mysql/mysqld.conf.d/x1.cnf --basedir /opt/wmf-mariadb1011 --datadir=/srv/sqldata.x1 --user=mysql
Installing MariaDB/MySQL system tables in '/srv/sqldata.x1' ...
OK

This will be deleted when we copy the data files and configure replication, which is dependent on T415219 being completed.

This is not necessary and neither are the grants as those will be copied once we import the database.
You can shutdown the process and even remove that new directory, as it will be wiped anyway for the cloning.

In T407485#11667849, @Marostegui wrote:

In T407485#11667548, @BTullis wrote:

This will be deleted when we copy the data files and configure replication, which is dependent on T415219 being completed.

This is not necessary and neither are the grants as those will be copied once we import the database.
You can shutdown the process and even remove that new directory, as it will be wiped anyway for the cloning.

Thanks, but I only added it as a way to keep Icinga checks happy, since we don't know how long T415219 is going to take.
An alternative approach would be to set downtime for the three failing checks (mysqld processes, replication I/O, replication SQL) - but I thought that it might be some time until we populate the data directory,

If you have strong feelings about it, then I can definitely do as you suggest and delete /srv/sqldata.x1/ instead, then fix up the monitoring.

No, I really don't have strong feelings about it. As this is host is owned by your team, I am happy with whatever you guys decide to make your day-to-day easier.

Change #1249379 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/homer/public@master] definitions: Add port for x1 on the wiki replicas

https://gerrit.wikimedia.org/r/1249379

This is causing some issues with the wiki replicas account maintenance tooling given that the x1 port is not yet in the firewall rules. The patch above fixes that, assuming the temporary instance will stay in place.

Change #1249379 merged by jenkins-bot:

[operations/homer/public@master] definitions: Add port for x1 on the wiki replicas

https://gerrit.wikimedia.org/r/1249379

New problem:

Mar 10 10:08:41 cloudcontrol1007 maintain-dbusers[3910393]: WARNING [root._create_accounts_on_host:1080] Could not connect to an-redacteddb1001.eqiad.wmnet:3320 due to (1130, "Host '10.64.148.21' is not allowed to connect to this MariaDB server").  Skipping.

Another problem relating to the wikireplica views maintenance, as discovered by @fnegri in T414240.
The x1 section has been added to /etc/maintain-views.yaml and /etc/index-conf.yaml and this causes errors like this:

pymysql.err.OperationalError: (1045, "Access denied for user 'maintainindexes'@'localhost' (using password: YES)")

...from the maintenance commands:

• /usr/bin/maintain-replica-indexes
• /usr/bin/maintain-views
• /usr/bin/maintain-meta_p

I think that the easiest thing might be to remove this section until we are ready to populate it, but I'll check that there isn't another workaround first.

Change #1251494 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Disable the x1 section on an-redacteddb1001 until we can populate it

https://gerrit.wikimedia.org/r/1251494

Change #1251494 merged by Btullis:

[operations/puppet@production] Disable the x1 section on an-redacteddb1001 until we can populate it

https://gerrit.wikimedia.org/r/1251494