Let me know if I can help with this at all.

In T344090#9088031, @Matthewrbowker wrote:

Let me know if I can help with this at all.

Bring it on! As stated in the task description, I can handle the replica querying part since that's getting moved to the ToolforgeBundle. Beyond that, we probably just need more $this->isWMF checks to make XTools compatible with 3rd party wikis. Please feel free to contribute to that effort. Note however these days we always inject dependencies such as parameters. Thanks to autowiring this usually as simple as adding bool $isWMF to the constructor signature (example). The property $isWMF is already available in all Repository classes.

Sounds great!! Will do whatever I can on my end.

Great to hear about the third-party support for the XTools.
I suppose the same approach will be needed for any specific tool such as the Copyvios tool. https://github.com/earwig/copyvios https://copyvios.toolforge.org/
Can you please confirm if it should work after the site URL and project name changes?
Is the configuration needed limited to this or anything else?

Our approach:

• Setup the Copyvios code on one of our servers
• Update the Code for the third-party project Site URL

Is that correct?

Thank you in advance.
@MusikAnimal

In T344090#9228418, @Abhishek.lal wrote:

Great to hear about the third-party support for the XTools.
I suppose the same approach will be needed for any specific tool such as the Copyvios tool. https://github.com/earwig/copyvios https://copyvios.toolforge.org/
Can you please confirm if it should work after the site URL and project name changes?
Is the configuration needed limited to this or anything else?

Our approach:

• Setup the Copyvios code on one of our servers
• Update the Code for the third-party project Site URL

Is that correct?

Thank you in advance.
@MusikAnimal

Sorry, I do not maintain Copyvios (although I do have maintainer access to the tool, which I only ever used to simple restarts as needed). You'll need to contact the author, Earwig, regarding implementation details and feasibility of 3rd party support.

Change #1277281 had a related patch set uploaded (by Alien4444; author: Alien4444):

[labs/xtools@main] [WIP] Readd third-party suppport

https://gerrit.wikimedia.org/r/1277281

Change #1309801 had a related patch set uploaded (by Jsn.sherman; author: Jsn.sherman):

[labs/xtools@main] Repository: config-driven dblist assembly

https://gerrit.wikimedia.org/r/1309801

Change #1309830 had a related patch set uploaded (by Jsn.sherman; author: Jsn.sherman):

[labs/xtools@main] Repository: remove dead code, add test coverage

https://gerrit.wikimedia.org/r/1309830

Change #1277281 merged by jenkins-bot:

[labs/xtools@main] Readd third-party suppport

https://gerrit.wikimedia.org/r/1277281

Change #1309801 merged by jenkins-bot:

[labs/xtools@main] Repository: config-driven dblist assembly

https://gerrit.wikimedia.org/r/1309801

Change #1309830 merged by jenkins-bot:

[labs/xtools@main] Repository: cover isWMF branches

https://gerrit.wikimedia.org/r/1309830

One issue that we're having since r1277281 is that information_schema.schemata apparently is not necessarily accurate (!?). testwiki appears in both s3 and s5… it should be just s3. It is also missing entirely from https://noc.wikimedia.org/db.php?format=json yet I do see it at https://noc.wikimedia.org/conf/dblists/s3.dblist and not at https://noc.wikimedia.org/conf/dblists/s5.dblist .

So for the WMF cluster, it seems the way we were fetching the DB list before (scraping the .dblist pages) may be the most accurate. However per the comments, the whole point was to get a more updated DB list to prevent incidents when new wikis are created but not yet replicated, like T322466 and T420632.

Sounds like we need a combination of the two solutions?

@MusikAnimal
tl;dr: just going back to using information_schema.tables as that patch previously did should fix it.

On inacurracy: here is the complete list of schema names for wikis appearing in multiple sections' schemata:

amwikimedia_p
cnwikimedia_p
donatewiki_p
gewikimedia_p
grwikimedia_p
hiwikimedia_p
idwikimedia_p
maiwikimedia_p
mediawikiwiki_p
ngwikimedia_p
nostalgiawiki_p
punjabiwikimedia_p
romdwikimedia_p
rswikimedia_p
testwiki_p
vewikimedia_p
votewiki_p
wbwikimedia_p

Haven't tried all but for all of these it seems that they're wikis which live on s3 but have an empty schema (as in no tables with that schema) in s5.
To be honest, if it weren't for mediawikiwiki in there I wouldn't even bother fixing this, because like wikimedia affiliate wikis are not content wikis, there's literally nothing on donatewiki or votewiki, and testwiki is testwiki.
information_schema being mysql metadata means it's never really inaccurate.
As it happens, it says that schemas under those names exist in s5, which is fair enough.
However, what we really care about is the existence of tables we can query.
That information is directly accessible with information_schema.tables.
(The third-party patch previously used tables to get the schema names, until it was pointed out to me during review that there's a table for schemas, which we switched to not knowing that empty schemas were around.)

Change #1320944 had a related patch set uploaded (by Alien4444; author: Alien4444):

[labs/xtools@main] Repository: use tables not schemata in getDbList

https://gerrit.wikimedia.org/r/1320944

Change #1320944 merged by jenkins-bot:

[labs/xtools@main] Repository: use tables not schemata in getDbList

https://gerrit.wikimedia.org/r/1320944