To start some discussion. The questions are easy, the details are not:

Do we trust 10.0 enough?

Yes, once we have done proper testing (profiling thatis already ongoing, and more will be done).

Which shard should we do first?

All shards are production, doesn't matter. The important thing is having a plan to roll back version and changes (separately each one).

Should we attempt to trial 10.0 under real(ish) master load (thread pool, concurrency, not just serial replicated transactions, etc)?

Yes.

My main issue with 10 is that we need GTID-like features, but we cannot close our doors by chaining ourselves to diverging implementations and exclusive features. But that is not what this ticket is about.

(Please associate a project to this task if possible so it can be found on project searches / workboards. Is this SRE ?)

All s1-7 slaves are now 10.0.

In T105135#1437745, @jcrespo wrote:

Which shard should we do first?

All shards are production, doesn't matter. The important thing is having a plan to roll back version and changes (separately each one).

Agree in theory (and technically), but in terms of potential fallout after a mistake, they are not equal:

• s1 and s4 are huge and would be painful to get wrong (in terms of upset users/public)
• s7 has centralauth, so problems would affect all logged-in users on all shards
• s5 is wikidata, which has non-standard write load and replag
• s3 has many more tables/file-handles (maybe not big deal)

Thinking that doing s2 or s6 first would be safest.

My main issue with 10 is that we need GTID-like features, but we cannot close our doors by chaining ourselves to diverging implementations and exclusive features. But that is not what this ticket is about.

I think this ticket should be about that, because if we do not discuss it now, before migration is complete... then when :-) ? So, please feel free to throw up road blocks now!

Things I think they are a big issues:

• Multisource replication (it does not work well, bugs with parallel?)
• TokuDB (the versions we use have very important bugs, but I do not know if all of them are fixed in the latest versions)
• Filtering (works well, but makes some tools more complex to use and error-prone)

None I think are related to masters:

I think we can avoid filtering, at least on production and maybe on labs, with some work.

TokuDB is not used except on dbstores/labs. We should update those to the latest version. Maybe for some of them (research) evaluate alternatives- for OLAP.
For storage saving I think InnoDB compressed was used for a while (what was your experience like?)
It would be lovely to also have MASTER_DELAY on those, which is not available on 10, only on 5.6.

Multisource is the big issue, there is an alternative on 5.7, but not a "short term" thing.

Main issue for me is GTID and ROW, but I do not have and answer to those yet, plus it has a dependency on sanitarium.

Now, my question is: why replacing 5.5 now? It works well and should work in jessie. Is there any specific issue we want to solve that requires update? (this is a rethorical question, but I want to know what are for you the main pain issues- queing, online alters?) HA is needed, but it is independent of the master version.

I am trying to get a list of blockers (we can edit the task description):

• Prepare a deployment and rollback plan
• Perform a table checksum on all affected servers
• Find the best node on each shard to failover to (may need reinstall)
• Create/review the role for mariadb production master
• Test write load (for example, with pt-upgrade or pt-table-checksum)
• Fix pending issues with jessie installer [soft block]
• Setup proxies and put them into production [soft block]
• Add extra monitoring only available on the mysql::coredb role [soft block]
  • Ganglia mysql parameters
  • Extra nagios checks (evaluate things like pt-heartbeart, read only mode or query strange states)

Some of these can be done in small increments (only one shard or server at a time)

Adding replication error as a blocking task, let's not rush and discard any replication issues with 10 masters first.

Doing s2 next week due to T122048

All s* servers now have mariadb10 masters. We keep the old 5.5 masters in case a rollback is needed.

There are still some core servers in 5.5, mainly x1.

Change 284455 had a related patch set uploaded (by Volans):
Change eqiad masters for s1,s3-s7

https://gerrit.wikimedia.org/r/284455

Change 284455 merged by jenkins-bot:
Change eqiad masters for s1,s3-s7

https://gerrit.wikimedia.org/r/284455

Mentioned in SAL [2016-04-20T14:56:36Z] <volans@tin> Synchronized wmf-config/db-eqiad.php: Change eqiad masters for s1,s3-s7 - T105135 (duration: 00m 28s)