Page MenuHomePhabricator

Make populateSitesTable.php more robust
Closed, DeclinedPublic

Description

As part of T158730, it'd be good to make populateSitesTable.php more robust

https://wikitech.wikimedia.org/w/index.php?title=Add_a_wiki&oldid=1560435#Wikidata doesn't inspire much confidence!

That script is known to be troublesome, you might want to ask Marius (hoo) or Katie (aude) run it for you or just create a ticket (that may be done anytime after the wiki was created).

Beware: The script sometimes fails with a duplicate key conflict. In that case, go to the wiki's master database and empty the sites and site_identifiers tables, then run the script again. It's probably also wise to backup these tables from Wikidata and at least one Wikipedia before running the script across the whole fleet. Breaking the sites, site_identifiers tables will break page rendering of many wikis!

Event Timeline

Addshore moved this task from Legacy Inbox to Research on the wdwb-tech board.

Looks like this text is still on the page:

In order to be able to link the new wiki from Wikidata, and to allow interwiki links from Wikidata to the new wiki, run extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https on at least all Wikidata clients (including wikidatawiki itself and testwikidata).

foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https

That script is known to be troublesome, you might want to ask Marius (hoo) or Amir Sarabadani (Amir1) run it for you or just create a ticket (that may be done anytime after the wiki was created).

Beware: The script sometimes fails with a duplicate key conflict. In that case, go to the wiki's master database and empty the sites and site_identifiers tables, then run the script again. It's probably also wise to backup these tables from Wikidata and at least one Wikipedia before running the script across the whole fleet. Breaking the sites, site_identifiers tables will break page rendering of many wikis!

@hoo @Ladsgroup would you be able to specify some changes that we might be able to make & clarify the problem exactly?

I've been running it for years now, I have seen it break I think last month too while it's rather rare. The problem is that you need to run this for +1k wikis and well, some would not work for any number of reasons and it would break the whole wiki. The proper fix for this is to have a centralized sitestore (Something along the lines of T113034: RFC: Overhaul Interwiki map, unify with Sites and WikiMap) so only one run would be needed after each wiki creation.

I'm not aware of a problem of any concrete issues we could tackle, other than T122520 which seems fine now. Mid-term we should really try to get rid of this, though, as Amir pointed out.

In that case I'm going to decline this in favour of T113034