There was some cleanup about where different connections were going that seem to have revealed an issue where some databases are only replicated to some replicas. I'm going to fiddle things around to hide the problem until the underlying issue has been found.

(Specifically labsdb1003 has part, but not all, of the databases replicated)

metatron wrote:

See also completely missing databases:
https://bugzilla.wikimedia.org/show_bug.cgi?id=71043

labsdb1003 hasn't upgraded to a single MariaDB 10 instance yet.

metatron wrote:

(In reply to Sean Pringle from comment #4)

labsdb1003 hasn't upgraded to a single MariaDB 10 instance yet.

What does that mean? Upgrade hasn't started yet? Or is currently in progress?
If none of the former, what caused the current outages? When will it be fixed and - first and foremost - when will the lost databases be back?

Though a major blocker, I couldn't find anything about such things in labs-l or elsewhere.

It means only that labsd1003 is in the same state it has been for the last year[1]: three MariaDB 5.5 instances replicating S3, S6, and S7 on ports 3306, 3307, and 3308. Unlike labsdb1001 and labsdb1002, which have been upgraded to a single MariaDB 10 instance on port 3306, labsdb1003 does not yet have all wikis.

@Marc, what was the "cleanup" you mentioned? DNAT or [fr|es|ru]wiki.labsdb changes?

[1] https://lists.wikimedia.org/pipermail/labs-l/2014-August/002797.html

metatron wrote:

@Sean: Thanks for info & clarification.

All s6 and s7 wikis I checked are affected.

This should be fixed right now; with Sean having merged all three of the labsdb1003 slices and the hosts file adjusted accordingly.

metatron wrote:

Some user databases still missing. See #3.

Is there anything left to fix here?

This seems to be resolved.