In T202764#4587022, @Krinkle wrote:

@Marostegui So to confirm, recentchanges db hosts are the same within and between eqiad/codfw. But the api db hosts are different, right? Only api dbhosts in codfw have the two extra indexes. Is that right?

Removed the DBA tag here as there is nothing for us to do. I will remain subscribed here just in case!
Thanks!

The creation of the tables shouldn't be an issue and I'm fine with that.
However, I would prefer if we do not enable the extension until we are back on eqiad (so that means 11th Oct)

So taking a quick look at the schema differences between rc groups in different DCs, they look the same:

Confirmed from my end too!

/admin1-> help
[Usage]
    show   [<options>] [<target>] [<properties>]
           [<propertyname>== <propertyvalue>]
    set    [<options>] [<target>] <propertyname>=<value>
    cd     [<options>] [<target>]

All done

The biggest table is on metawiki which is 22M, so probably good to go directly on the masters someday early in the morning.

Table imported on testwikidatawiki on eqiad hosts:

root@db1075.eqiad.wmnet[testwikidatawiki]> show create table wbqc_constraints;
+------------------+-------------------------------------------------------------------------------------------------------------------
| Table            | Create Table
+------------------+-------------------------------------------------------------------------------------------------------------------
| wbqc_constraints | CREATE TABLE `wbqc_constraints` (
  `constraint_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `constraint_guid` varbinary(255) NOT NULL,
  `pid` int(11) NOT NULL,
  `constraint_type_qid` varbinary(255) NOT NULL,
  `constraint_parameters` blob,
  PRIMARY KEY (`constraint_id`),
  UNIQUE KEY `wbqc_constraints_guid_uniq` (`constraint_guid`),
  KEY `wbqc_constraints_pid_index` (`pid`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=binary |

In T204127#4581347, @jcrespo wrote:

db2068 has been recloned, but needs time to catch up replication and then be slowly repooled with the above patch.

The table has been imported in eqiad hosts for wikidatawiki

root@neodymium:/home/marostegui# mysql.py -hdb1071 wikidatawiki -e "show create table wbqc_constraints\G"
*************************** 1. row ***************************
       Table: wbqc_constraints
Create Table: CREATE TABLE `wbqc_constraints` (
  `constraint_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `constraint_guid` varbinary(63) NOT NULL,
  `pid` int(11) NOT NULL,
  `constraint_type_qid` varbinary(25) NOT NULL,
  `constraint_parameters` blob,
  PRIMARY KEY (`constraint_id`),
  UNIQUE KEY `wbqc_constraints_guid_uniq` (`constraint_guid`),
  KEY `wbqc_constraints_pid_index` (`pid`)
) ENGINE=InnoDB AUTO_INCREMENT=22320 DEFAULT CHARSET=binary ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8

db1071 GTID enabled

db2054 has been recloned, it is catching up. Once it has sync'ed with its master, I will remove its downtime and repool it into s7

db1068 GTID enabled

GTID enabled on all eqiad masters but db1071 (s8) and db1068 (s4) as they are currently running a big alter.

All done

Replication has been disconnected from eqiad to codfw:

root@neodymium:/home/marostegui# for i in db2048 db2035 db2043 db2051 db2052 db2039 db2040 db2045 db2034 es2016 es2017; do mysql.py -h$i -e "show slave status\G" ; done
root@neodymium:/home/marostegui#

Thanks

We, DBAs, don't use it.

In T204127#4578456, @jcrespo wrote:

In T204127#4578185, @Marostegui wrote:

We could probably reclone one of these hosts (for example db2054) from an eqiad slave, and then move it under codfw master. That way we don't have to depool an active codfw s7 slave, as that might be too much, 3 hosts out.
Once db2054 is up to date and under codfw master, we can reclone db2068 from it.

If only there was some kind of "provisioning server"- you know, a place were to take backups and recover them to production! ;-D Spoiler, there is one, we call it dbstore200X! No need for a slow process of cloning cross-dc. This is exactly the reason why we setup dbstore. We use it right now for logical backups, but we can use it for a cold copy too :-)

We could probably reclone one of these hosts (for example db2054) from an eqiad slave, and then move it under codfw master. That way we don't have to depool an active codfw s7 slave, as that might be too much, 3 hosts out.
Once db2054 is up to date and under codfw master, we can reclone db2068 from it.

With course_token you mean that column on the ep_courses table?
If that is the case, there is no need to play around with the views for mysqldump, as that table has course_token already filtered on sanitariums with the triggers:

| ep_courses_insert       | INSERT | ep_courses       | SET NEW.course_token = ''
| ep_courses_update       | UPDATE | ep_courses       | SET NEW.course_token = ''

I am going to close this as resolved with the creation of information_schema_p and the current content (tables and schemata)
The only part pending apparently was:

I have renamed the file from research-grants.sql.erb to analytics-grants.sql.erb so we can have all the users that are actually active (T200801#4570548) on dbstore1002 puppetized.
Also added some new users that were not on that file yet.
We are still auditing and we will probably drop all the ones that are not really used as per T200801#4570548 //cc @elukey - for now. All the ones that do have had connections are now on that sql file. If we end up discovering that some others aren't active but should be there, I will edit that file.
I am going to close this for now as resolved and we can modify the file depending on the outcome of T200801

Security any comments about ^

As there are no more actionables for the DBA Team here I will untag it.
I will remain subscribed here though in case you have further questions.
Once this is good to go just send the schema change ticket as described in the link in my previous comment above and we will take it from there.

In T204006#4574614, @aezell wrote:

1. We'd like the change on all production databases but I know "all databases" isn't a valid answer for your work. However, I'm new and don't know all of the possible databases. I'm consulting with my team to get the right answer. In the short term, testwiki and betawiki would be a good start, I suppose.

Thanks for amending the task. I still have some questions.

In T204006#4574533, @aezell wrote:

@Marostegui No problem. I'll get it fixed up. Sorry for the confusion.