Implement a SiteLookup based on a nested array structure.
Open, HighPublic
Actions

Assigned To

None

Authored By

	daniel
	May 12 2016, 4:57 PM

Description

As per T113034: RFC: Overhaul Interwiki map, unify with Sites and WikiMap, we want to move towards maintaining meta-information about other sites (aka interwiki info) in files, using the structure outlined in P3044. This structure would be stored in JSON or PHP files, and will be represented internally by nested arrays. The SiteLookup should:

provide access to the Site objects represented by nested arrays, see T135147: Make the domain model implemented by Site/SiteLookup/SiteStore more flexible
be able to load such data from JSON or PHP files (this could be in a separate class, if we want)
combine multiple such data-structures (deep-merge)
build indexes for efficient access by id or group. If such an index was already included in the data provided, it should be used.

NOTE: FileBasedSiteLookup exists. Perhaps it can be adopted. We would probably need compatibility code so it can continue to support the current/legacy JSON structure (see docs/sitescache.txt). However, the SiteLookup should perhaps not know about files at all, only about nested arrays.

The proposed structure, for reference:

P3044 interwiki.json

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125

1	{ "enwiki": { "type": "mediawiki", "ids": { "global": [ "enwiki", "some-old-alias" ], "interwiki": "en", "domain": "en.wikipedia.org" }, "groups": { "language": "en", "size": "big", "family": "wikipedia", "db-cluster": "s1" }, "paths": { "article": "//en.wikipedia.org/wiki/$1", "api": "https://en.wikipedia.org/w/api.php" }, "props": { "database": "enwiki", "language": "en" } }, "enwiktionary": { "type": "mediawiki", "ids": { "global": "enwiktionary", "interwiki": [ "wiktionary", "wikt" ], "domain": "en.wiktionary.org" }, "groups": { "language": "en", "family": "wiktionary", "db-cluster": "s2" }, "paths": { "article": "//en.wiktionary.org/wiki/$1", "api": "https://en.wiktionary.org/w/api.php" }, "props": { "database": "enwiktionary", "language": "en", "capital-links": false } }, "commonswiki": { "type": "mediawiki", "ids": { "global": [ "commonswiki", "commons" ], "interwiki": [ "commons", "c" ], "domain": "commons.wikimedia.org" }, "groups": { "language": "en", "family": "commons", "db-cluster": "s2" }, "paths": { "article": "//commons.wikimedia.org/wiki/$1", "api": "https://commons.wikimedia.org/w/api.php" }, "props": { "database": "commonswiki", "multilingual": true, "transcludable": true } }, "bb": { "type": "unknown", "ids": { "global": ["bb", "boingboing" ], "interwiki": "bb", "domain": "boingboing.net" }, "groups": { "language": "en" }, "paths": { "article": "https://boingboing.net/$1.html" }, "props": { "language": "en" } }, "_by_ids": { "global": { "some-old-alias": "enwiki", "enwiki": "enwiki", "enwiktionary": "enwiktionary", "commonswiki": "commonswiki", "commons": "commonswiki", "bb": "bb", "boingboing": "bb" }, "interwiki": { "en": "enwiki", "wiktionary": "enwiktionary", "wikt": "enwiktionary", "c": "commonswiki", "commons": "commonswiki", "bb": "bb" }, "domain": { "en.wikipedia.org": "enwiki", "en.wiktionary.org": "enwiktionary", "commons.wikimedia.org": "commonswiki", "boingboing.net": "bb" } }, "_by_groups": { "language": { "en": [ "enwiki", "enwiktionary", "bb" ], "mul": [ "commonswiki" ] }, "family": { "wikipedia": [ "enwiki" ], "wiktionary": [ "enwiktionary" ], "commons": [ "commonswiki" ] }, "db-cluster": { "s1": [ "enwiki" ], "s2": [ "enwiktionary", "commonswiki" ] } } class="o">}

{ "enwiki": { "type": "mediawiki", "ids": { "global": [ "enwiki", "some-old-alias" ], "interwiki": "en", "domain": "en.wikipedia.org" }, "groups": { "language": "en", "size": "big", "family": "wikipedia", "db-cluster": "s1" }, "paths": { "article": "//en.wikipedia.org/wiki/$1", "api": "https://en.wikipedia.org/w/api.php" }, "props": { "database": "enwiki", "language": "en" } }, "enwiktionary": { "type": "mediawiki", "ids": { "global": "enwiktionary", "interwiki": [ "wiktionary", "wikt" ], "domain": "en.wiktionary.org" }, "groups": { "language": "en", "family": "wiktionary", "db-cluster": "s2" }, "paths": { "article": "//en.wiktionary.org/wiki/$1", "api": "https://en.wiktionary.org/w/api.php" }, "props": { "database": "enwiktionary", "language": "en", "capital-links": false } }, "commonswiki": { "type": "mediawiki", "ids": { "global": [ "commonswiki", "commons" ], "interwiki": [ "commons", "c" ], "domain": "commons.wikimedia.org" }, "groups": { "language": "en", "family": "commons", "db-cluster": "s2" }, "paths": { "article": "//commons.wikimedia.org/wiki/$1", "api": "https://commons.wikimedia.org/w/api.php" }, "props": { "database": "commonswiki", "multilingual": true, "transcludable": true } }, "bb": { "type": "unknown", "ids": { "global": ["bb", "boingboing" ], "interwiki": "bb", "domain": "boingboing.net" }, "groups": { "language": "en" }, "paths": { "article": "https://boingboing.net/$1.html" }, "props": { "language": "en" } }, "_by_ids": { "global": { "some-old-alias": "enwiki", "enwiki": "enwiki", "enwiktionary": "enwiktionary", "commonswiki": "commonswiki", "commons": "commonswiki", "bb": "bb", "boingboing": "bb" }, "interwiki": { "en": "enwiki", "wiktionary": "enwiktionary", "wikt": "enwiktionary", "c": "commonswiki", "commons": "commonswiki", "bb": "bb" }, "domain": { "en.wikipedia.org": "enwiki", "en.wiktionary.org": "enwiktionary", "commons.wikimedia.org": "commonswiki", "boingboing.net": "bb" } }, "_by_groups": { "language": { "en": [ "enwiki", "enwiktionary", "bb" ], "mul": [ "commonswiki" ] }, "family": { "wikipedia": [ "enwiki" ], "wiktionary": [ "enwiktionary" ], "commons": [ "commonswiki" ] }, "db-cluster": { "s1": [ "enwiki" ], "s2": [ "enwiktionary", "commonswiki" ] } } class="o">}

Related Objects
Search...

Status	Assigned	Task
Open	None	T124286 [Epic] Wikidata language support
Open	None	T39459 [Task] Don't try to add labels in non-existing languages: restrict to Language::isKnownLanguageTag
Open	None	T145535 language code disparity (Wikidata tracking)
Open	None	T66649 'Add link' function in Wikipedia creates items with wrong language code
Open	None	T137536 Use correct term language when creating items from the client
Open	None	T137534 Map dummy language codes in sites
Open	None	T137538 Use interwiki code from Sites facility to create language links.
Open	None	T137537 Ensure correct information about Wikimedia sites in the Sites facility on the Wikimedia cluster.
Open	None	T191405 Mark interwiki links as external links when they fall outside the WMF family
Open	None	T113034 RFC: Overhaul Interwiki map, unify with Sites and WikiMap
Open	None	T135153 Allow Sites and Interwiki facilities to be configured consistently
Open	None	T135149 Implement a SiteLookup based on a nested array structure.

Event Timeline

daniel created this task.May 12 2016, 4:57 PM

daniel removed a parent task: T135146: Create an adapter that implements InterwikiLookup on top of SiteLookup.

daniel added a parent task: T113034: RFC: Overhaul Interwiki map, unify with Sites and WikiMap.

We should probably have some way to indicate that a mediawiki site is local or not

Or maybe that's just based on the presence of groups db-cluster or props database?

Why is language in both groups and props?

Also this would have to be per-wiki with the current format - en: is the interwiki prefix for enwiki on wikipedias, but on (for example) wikibooks it should point to enwikibooks

daniel updated the task description. (Show Details)May 12 2016, 5:08 PM

daniel mentioned this in T135153: Allow Sites and Interwiki facilities to be configured consistently.May 12 2016, 5:27 PM

daniel added a parent task: T135153: Allow Sites and Interwiki facilities to be configured consistently.

daniel created subtask T135154: Maintenance script to convert between different representations of interwiki / sites info.May 12 2016, 5:32 PM

daniel removed a subtask: T135154: Maintenance script to convert between different representations of interwiki / sites info.

daniel mentioned this in T135156: Create a SiteStore that can write JSON and PHP files.May 12 2016, 5:39 PM

daniel added a project: Wikidata-Sprint-2016-05-24.May 12 2016, 6:06 PM

• Quiddity unsubscribed.May 12 2016, 10:03 PM

Lydia_Pintscher moved this task from Proposed to Backlog on the Wikidata-Sprint-2016-05-24 board.May 24 2016, 1:03 PM

daniel added a parent task: T137537: Ensure correct information about Wikimedia sites in the Sites facility on the Wikimedia cluster..Jun 10 2016, 10:42 AM

Bumping prio to high. The Wikidata team has this on their road map for 2016/Q3, because it ultimately blocks T39459: [Task] Don't try to add labels in non-existing languages: restrict to Language::isKnownLanguageTag and T137536: Use correct term language when creating items from the client as well as T137538: Use interwiki code from Sites facility to create language links..

daniel added a project: WMDE-TLA-Team.Sep 19 2016, 1:58 PM

daniel removed daniel as the assignee of this task.Sep 19 2016, 2:00 PM

daniel claimed this task.Oct 31 2016, 4:22 PM

daniel added a project: Wikidata.

Ladsgroup edited projects, added Wikidata-Former-Sprint-Board; removed WMDE-TLA-Team.Nov 18 2016, 10:57 AM

Liuxinyu970226 subscribed.Apr 9 2017, 2:43 PM

Lydia_Pintscher moved this task from incoming to ready to go on the Wikidata board.May 5 2017, 12:56 PM

Lydia_Pintscher removed a project: Wikidata-Former-Sprint-Board.Nov 21 2017, 8:57 AM

Lucas_Werkmeister_WMDE subscribed.Mar 7 2018, 2:47 PM

This task has been assigned to the same task owner for more than two years. Resetting task assignee due to inactivity, to decrease task cookie-licking and to get a slightly more realistic overview of plans. Please feel free to assign this task to yourself again if you still realistically work or plan to work on this task - it would be welcome!

For tips how to manage individual work in Phabricator (noisy notifications, lists of task, etc.), see https://phabricator.wikimedia.org/T228575#6237124 for available options.
(For the records, two emails were sent to assignee addresses before resetting assignees. See T228575 for more info and for potential feedback. Thanks!)

ItamarWMDE subscribed.Sep 6 2023, 8:20 AM

Winston_Sung moved this task from Backlog to General on the MediaWiki-Site-system board.Feb 2 2024, 4:33 PM

Implement a SiteLookup based on a nested array structure.Open, HighPublicActions

Description

Related ObjectsSearch...

Event Timeline

Implement a SiteLookup based on a nested array structure.
Open, HighPublic
Actions

Related Objects
Search...