Page MenuHomePhabricator

Add Wikidata support to new wikis
Closed, ResolvedPublic3 Story Points

Description

Several wikis were created (see parent tasks). Add wikidata support to them, please.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 19 2018, 11:22 AM
Addshore moved this task from Incoming to Ready to go on the Wikidata-Campsite board.

I guess this can be picked up @Lydia_Pintscher ? :)

Addshore triaged this task as Normal priority.Nov 19 2018, 1:39 PM
Addshore set the point value for this task to 3.Nov 20 2018, 2:08 PM
Ladsgroup updated the task description. (Show Details)Nov 20 2018, 7:24 PM
Ladsgroup updated the task description. (Show Details)

This needs to be done before rebuilding the sites table: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/473510

I can merge and deploy it tomorrow I guess.

Hello903hello added a comment.EditedNov 21 2018, 6:15 AM

Does "yue" need to be added to InterwikiSortOrders.php? Seems like there's only zh-yue now.

Does "yue" need to be added to InterwikiSortOrders.php? Seems like there's only zh-yue now.

There is no need for yue language in the interwiki sort order.

And there's no SWAT until Monday. This can't go forward until then.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptNov 21 2018, 12:01 PM

@Ladsgroup

There is no need for yue language in the interwiki sort order.

Why no need? Have you ever read T202602?

Why no need? Have you ever read T202602?

Yes, The thing is language code is important here and the database name doesn't matter and language of zh-yue wiktionary is "yue" and it's already there.

...and language of zh-yue wiktionary is "yue" and it's already there.

I'm confused. There seems to be no "yue" throughout the whole InterwikiSortOrders.php.

Now that's interesting.

@Amire80 Hey, @Lydia_Pintscher and I are having problems with the new wikis:

  • Is this patch sane to you? All the orders are correct?
  • We have two different wikis with the same language but they havedifferent database names. zh_yuewiki and yuewiktionary. They both have "yue" as their language now but InterwikiSortOrder actually doesn't contain the new name. Should we just replace zh-yue with yue or keep both for a while?

@Amire80 Hey, @Lydia_Pintscher and I are having problems with the new wikis:

  • Is this patch sane to you? All the orders are correct?

Yes.

  • We have two different wikis with the same language but they havedifferent database names. zh_yuewiki and yuewiktionary. They both have "yue" as their language now but InterwikiSortOrder actually doesn't contain the new name. Should we just replace zh-yue with yue or keep both for a while?

Ugh. Not sure. Perhaps @Nikerabbit will have a better idea?

Are these interwiki codes or language codes? I see simple on the list but simplewiki uses en as the content language. https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&meta=siteinfo&siprop=interwikimap lists boths zh-yue and yue so I suppose you want to include both codes as long as both can be used.

hoo added a comment.Dec 5 2018, 8:06 AM

@Ladsgroup What's missing here to be able to update the sites tables? Just https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/473510?

In case that was not noticed/included: li.wikinews also is unrecognized via wikidata.

In case that was not noticed/included: li.wikinews also is unrecognized via wikidata.

As you can see, this task is not closed yet - so this is not considered as done. Please wait until this task is closed, after that, you can check and reopen in case it won't work.

@Ladsgroup What's missing here to be able to update the sites tables? Just https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/473510?

It needs 'yue' too. Let me make the patch.

Change 477771 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Add yue to InterwikiSortOrders.php

https://gerrit.wikimedia.org/r/477771

Change 477771 merged by jenkins-bot:
[operations/mediawiki-config@master] Add yue to InterwikiSortOrders.php

https://gerrit.wikimedia.org/r/477771

Mentioned in SAL (#wikimedia-operations) [2018-12-17T12:50:24Z] <ladsgroup@deploy1001> Synchronized wmf-config/InterwikiSortOrders.php: Update InterwikiSortOrders.php (T209820) (duration: 00m 45s)

Mentioned in SAL (#wikimedia-operations) [2018-12-17T13:04:57Z] <Amir1> ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T209820)

you need to wait for forty minutes until the cache for sitelinks expire and then you can add links to these wikis. I will run my bot on those three wikis.

Okay, it seems there is a huge bug when making yuewiktionary, the wiki is marked as a "special" wiki (wikis that doesn't belong to any group) but it should have been marked as a "wiktionary" wiki. you can find the faulty result from https://meta.wikimedia.org/w/api.php?action=sitematrix I'm investigating why it's happening.

Reedy added a comment.EditedDec 18 2018, 3:19 PM

It's because 'yue' is not in langlist: https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/langlist
@Reedy was it intentional?

I didn't make the config patch.

I'm guessing it's a confusion with this being the first yue project, but the others being zh-yue - honestly, I'm not sure exactly how this should be handled, so it's not really surprising. Do we consider yue and zh-yue to be the same? Different?

Okay, it seems there is a huge bug when making yuewiktionary, the wiki is marked as a "special" wiki (wikis that doesn't belong to any group) but it should have been marked as a "wiktionary" wiki. you can find the faulty result from https://meta.wikimedia.org/w/api.php?action=sitematrix I'm investigating why it's happening.

It's definitely in the right dblists though

reedy@deploy1001:/srv/mediawiki-staging/dblists$ grep ^yue *.dblist
all.dblist:yuewiktionary
commonsuploads.dblist:yuewiktionary
nowikidatadescriptiontaglines.dblist:yuewiktionary
s3.dblist:yuewiktionary
small.dblist:yuewiktionary
wikidataclient.dblist:yuewiktionary
wiktionary.dblist:yuewiktionary

It's definitely in the right dblists though

Yes but according to https://wikitech.wikimedia.org/wiki/Add_a_wiki, it needs to have its language in the langlist otherwise sitematrix wouldn't work:

For a new language project, add the language code (ISO 639 code: usually provided in the task) to langlist. This will be used at Special:SiteMatrix and for interwiki linking etc.

One simple solution to avoid all sorts of issues is to change language of yuewiktionary from 'yue' to 'zh-yue'

Reedy added a comment.Dec 18 2018, 3:41 PM

Sure, I don't disagree that it probably should be there. But like I'm saying, it's a fairly obvious a confusion between zh-yue and yue, and the presumption that that lang code existed in the right places

I'm not the right person to advise on the pros/cons of yue vs zh-yue :)

One simple solution to avoid all sorts of issues is to change language of yuewiktionary from 'yue' to 'zh-yue'

You mean $wgLanguageCode ? This seems the easiest option in the short term rather than adding yue all over the place

Ladsgroup added a comment.EditedDec 18 2018, 3:45 PM

I'm not the right person to advise on the pros/cons of yue vs zh-yue :)

Yeah, let's ask @Amire80 :)

You mean $wgLanguageCode ? This seems the easiest option in the short term rather than adding yue all over the place

Yes, I'm afraid having two language codes in langlist that actually point to one language would cause lots of trouble. Fun thing: language of zh-yue.wikipedia.org is set to yue and not 'zh-yue'. How so it hasn't exploded so far? https://zh-yue.wikipedia.org/w/api.php?action=query&meta=siteinfo&siinlanguagecode=1

One simple solution to avoid all sorts of issues is to change language of yuewiktionary from 'yue' to 'zh-yue'

The documentation for $wgLanguageCode clearly states:

* Warning: Don't use any of MediaWiki's deprecated language codes listed in
* LanguageCode::getDeprecatedCodeMapping or $wgDummyLanguageCodes, like "no"
* for Norwegian (use "nb" instead). If you do, things will break unexpectedly.

I assume we don't want things breaking unexpectedly. Whatever SiteMatrix does, it is not based on $wgLanguageCode. If it would, then it would confuse Simple English Wikipedia with English Wikipedia because they have the same language code.

Liuxinyu970226 added a comment.EditedDec 26 2018, 4:11 PM

Should I say, that I can accept adding a "yue" line that only Wiktionary link is blue? Let's just add yue to langlist file, why not?

Whatever SiteMatrix does, it is not based on $wgLanguageCode.

You're wrong:
From wmf-config/CommonSettings.php:

// Config for sitematrix
$wgSiteMatrixFile = ( $wmfRealm === 'labs' ) ? "$IP/../langlist-labs" : "$IP/../langlist";

Then it's loaded in SiteMatrix.php

If it would, then it would confuse Simple English Wikipedia with English Wikipedia because they have the same language code.

It's a different story. Both simple and enwiki have their language codes in the langlist but in case of Cantonese wiktionary, its language code doesn't exist at all so it thinks it's a special wiki.

I'm more inclined to include both yue and zh-yue until we find a proper fix (T113034: RFC: Overhaul Interwiki map, unify with Sites and WikiMap). Connecting zh-yue Wikipedia with yue Wiktionary will be another fun topic.

Yes. We need to add yue to langlist (https://github.com/wikimedia/mediawiki-extensions-SiteMatrix/blob/master/includes/SiteMatrix.php#L117):

foreach ( $this->sites as $site ) {
	$m = [];
	if ( preg_match( "/(.*)$site\$/", $db, $m ) ) {
		$lang = $m[1];
		$langhost = str_replace( '_', '-', $lang );
		if ( isset( $xLanglist[$langhost] ) ) {
			$this->matrix[$site][$langhost] = 1;
			$this->countPerSite[$site]++;
		} else {
			$this->specials[] = [ $lang, $site ];
		}
		$found = true;
		break;
	}
}
Lydia_Pintscher closed this task as Resolved.Jan 26 2019, 1:11 PM