This is rather lots of work, SpecialWikibasePage is not the right place to keep the list, Making it per special page also is complex, like Special:SetLabel can be used on property and item and if the person is blocked on item namespace, there's no way to make sure it's applied until the user actually tries it (in which EntityStore should handle it properly using PermissionManager), so maybe we should just drop it? I would prefer if the special page still checks for site-wide bans.

It would have been so much easier if EntitySchemas were entities, we just could have re-used the entity datatype boilerplate

https://www.wikidata.org/w/index.php?title=Q4115189&diff=963952400&oldid=963896842

I just put this here:

In T225496#5255602, @Jdforrester-WMF wrote:

This seems to have had a ~20% speed improvement – picking two patches merged into MW-core either side of this change (but without any change in the number or nature of tests), the before durations are 135 and 156 seconds, and the after durations are 162 and 187. Of course, the durations bump around based on CI server load, but 30 seconds saved is 30 seconds saved.

In T160690#5251867, @matmarex wrote:

The first user is probably going to be this MobileFrontend patch: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontend/+/510067/10..11/extension.json

Another note and observation: Running tests in HHVM is extremely slower than php7. phpunit excluding database takes 2 minutes in hhvm and 40 seconds in php7. Selenium takes 4 minutes on hhvm and 1 minute on php7. These jobs hopefully will be dropped soon.

In T225524#5249945, @Jdforrester-WMF wrote:

Yes, the fixes to make phan not slow down CI mean that it's now non-recursive in dependency fetching. See T225112.

I guess the real dependency tree for WikibaseLexeme needs to be updated.

Number of DBQueries drop by 1000:

amsa@amsa-Latitude-7480:~/Downloads$ grep DBQuery mw-debug-www.log | wc -l
24814
amsa@amsa-Latitude-7480:~/Downloads$ grep DBQuery mw-debug-www.log.1 | wc -l
23775

^ This patch reduces the time for selenium tests from 1:18 to 0:50 ([https://integration.wikimedia.org/ci/job/mediawiki-quibble-composer-mysql-php70-docker/20462/consoleFull| 1]] 2). Number of calls to SqlBagOStuff reduces drastically:

amsa@amsa-Latitude-7480:~/Downloads$ grep SqlBag mw-debug-www.log | wc -l
1623
amsa@amsa-Latitude-7480:~/Downloads$ grep SqlBag mw-debug-www.log.1 | wc -l
113

It works in beta as expected:

• https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Special:UserLogout&returnto=Main+Page
• https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Special:UserLogout

In T225420#5246721, @Nikerabbit wrote:

I don't have any idea how to do deterministic random sorts (for paging) efficiently.

I started the patch, It's way far from being ready.

In T223602#5245075, @Ladsgroup wrote:

2- indentation is tricky and problematic sometimes. You know, like this.

Let me drill down on this a bit. The most annoying part is when you make a mistake in json, it explodes, it gives you error "Unable to parse json" but when you make a mistake in yaml, it works but the output is changed in a sneaky way and it explodes when you deploy it :)

In T191231#5245169, @HappyDog wrote:

That's an argument against code commenting in general.
I strongly disagree with the suggestion that we should limit the amount of comments a developer is allowed to provide, either explicitly or implicitly (by choice of format). I can guarantee that any recommendation that developers should add their comments somewhere other than alongside the code they are writing will simply result in developers not writing comments at all.

Well, Developers should avoid writing comments as much as possible too. I know it seems counter-intuitive but there's a whole chapter of "Clean Code" by Uncle Bob dedicated to this. There are lots of reasons for that:

• The code should be self-explanatory through variables/classes/methods names and structure (readability of the code is an important factor in maintainability of it) then comments become redundant and a cognitive load without giving out anything new.
• When you have code and comments, they might get out of sync, they become misleading, they decay and now you need to main two things instead of one. (If you're maintaining documentation, like what we do for schema structures at mediawiki.org, you have three things to maintain, the DDL, the comments, the mediawiki.org documentation).

\o/

In T223602#5235411, @ArielGlenn wrote:

@Ladsgroup what sort of issues have you had with yaml in python? How complicated are the structures you needed to represent?

In T191231#5236862, @Skizzerz wrote:

Abstract schemas is an argument for keeping MSSQL, Oracle, and other lightly-supported DBMSes in Core, not splitting them out.

You are completely right but that's true when there's an existing abstraction in place. With current system and specially in case of oracle, it's not possible to use Doctrine DBAL to build the abstraction or requires a great amount of developer work to write a DBAL from scratch. (See this)

(Warning. Long comment ahead)
This week, I have been so frustrated with slow jenkins that I spent the last two days trying to make it faster. I learned a lot about mediawiki's caching actually which is surprising to me. While I share the same concern, I think making browser tests non-voting defies the point of having them in the first place (we did a similar thing with Wikibase, we moved most of browser tests to a daily jenkins job. Guess what happened. The tests have been red for around two years now.
So what should we do?

• Find and fix slow phpunit tests. We have report of tests run time which is pretty useful in finding weird cases.
  • The slowest one needs to be better: T225184: CirrusSearch\SearcherTest::testSearchText PHPUnit tests take a while and runs for everyone I already merged the patch that makes it 30 seconds faster.
  • One of wikibase unit tests takes 8.6 seconds because it sleeps more than 8 seconds to make sure TTL is expired. That's fixed in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/515023
  • There are lots of other ones that I don't know if they are possible to fix. People should take a look at their codebase and fix those.
• Move browser tests to another parallel job.
• Separate unit tests from integration tests (T87781: Split mediawiki tests into unit and integration tests) It doesn't just make the tests faster, it has another fun benefit too. quibble is really smart, at first it runs phpunit excluding database group (we should start honoring "database" tag in our codebase, I don't think we do). Then it runs the browser tests and then at last runs database phpunit tests. This is a smart way to stop early when the build is going to fail anyway, run fast ones first and abort when they fail. By separating unittests, we can put them in front of quibble queue, making it abort jobs that fail in unittests before running lots of other tests.
• Improve caching in our tests. I thought phpunit uses the default caching for database but it actually sets all of caches to 'hash' (opcache) in all types of tests. If we do this for browser tests too, it makes them blazing fast. Number of database queries if you set caching to database is unbelievable (Try it). I set browsertests caching to APCu but it hasn't been merged yet: https://gerrit.wikimedia.org/r/515702 That will improve things drastically. But still we might want to consider using persistent cache in our unittests because for example making Language object is pretty expensive but they are cached.
• Speaking of Language object, can this be more mockable or split? Most of slow phpunit tests are slow because they try to make the object. In my (rather slow) computer it takes around 50ms to just construct and do a basic use of Language object (you can see the difference by changing the language so it reuses the cache)

I made T225351: Regression in diffing unreviewed changes in edit mode in FlaggedRevs for the regression. The point of the task is done (Renewing icons). It should be reopened if the regression causes it to be reverted which happened yet (it's more likely that this regression gets fixed instead)

So this is the backtrace when you set $wgWBRepoSettings['useEntitySourceBasedFederation'] = true;:

/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityMetaDataLookup.php:106
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/Sql/TypeDispatchingWikiPageEntityMetaDataAccessor.php:88
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php:264
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php:186
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php:125
/var/lib/mediawiki2/extensions/Wikibase/data-access/src/ByTypeDispatchingEntityRevisionLookup.php:54
/var/lib/mediawiki2/extensions/WikibaseLexeme/src/DataAccess/Store/SenseRevisionLookup.php:55
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/TypeDispatchingEntityRevisionLookup.php:54
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php:104
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php:87
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php:104
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php:87
/var/lib/mediawiki2/extensions/Wikibase/repo/includes/Api/GetEntities.php:275
/var/lib/mediawiki2/extensions/Wikibase/repo/includes/Api/GetEntities.php:257
/var/lib/mediawiki2/extensions/Wikibase/repo/includes/Api/GetEntities.php:136
/var/lib/mediawiki2/includes/api/ApiMain.php:1595
/var/lib/mediawiki2/includes/api/ApiMain.php:500
/var/lib/mediawiki2/tests/phpunit/includes/api/ApiTestCase.php:106
/var/lib/mediawiki2/extensions/Wikibase/repo/tests/phpunit/includes/Api/WikibaseApiTestCase.php:156
/var/lib/mediawiki2/extensions/WikibaseLexeme/tests/phpunit/mediawiki/Api/LexemeGetEntitiesTest.php:105
/var/lib/mediawiki2/tests/phpunit/MediaWikiTestCase.php:427
/var/lib/mediawiki2/maintenance/doMaintenance.php:99

I just added @WMDE-Fisch to wmf-deployment LDAP group which seems was not done with this

If you set $wgWBRepoSettings['useEntitySourceBasedFederation'] = true; in localhost, it works just fine:

{
    "entities": {
        "L16-F1": {
            "pageid": 513,
            "ns": 146,
            "title": "Lexeme:L16",
            "lastrevid": 515,
            "modified": "2019-05-22T15:06:48Z",
            "id": "L16-F1",
            "representations": {
                "en": {
                    "language": "en",
                    "value": "ff"
                }
            },
            "grammaticalFeatures": [],
            "claims": {}
        }
    },
    "success": 1
}

This is all of wikibase-related configs that differ between production and beta clsuter:

1 wgWBQualityConstraintsWikibaseFormId:"Q505134"
1 wgWBQualityConstraintsWikibaseFormId:"Q54285143"
1 wgWBQualityConstraintsWikibaseItemId:"Q29934200"
1 wgWBQualityConstraintsWikibaseItemId:"Q505131"
1 wgWBQualityConstraintsWikibaseLexemeId:"Q505133"
1 wgWBQualityConstraintsWikibaseLexemeId:"Q51885771"
1 wgWBQualityConstraintsWikibasePropertyId:"Q29934218"
1 wgWBQualityConstraintsWikibasePropertyId:"Q505132"
1 wgWBQualityConstraintsWikibaseSenseId:"Q505135"
1 wgWBQualityConstraintsWikibaseSenseId:"Q54285715"
1 wgWikibaseMultiRepositoryServiceWiringFiles:["\/srv\/mediawiki\/php-1.34.0-wmf.8\/extensions\/Wikibase\/client\/..\/data-access\/src\/MultiRepositoryServiceWiring.php"]
1 wgWikibaseMultiRepositoryServiceWiringFiles:["\/srv\/mediawiki-staging\/php-master\/extensions\/Wikibase\/client\/..\/data-access\/src\/MultiRepositoryServiceWiring.php"]
1 wgWikibasePerRepositoryServiceWiringFiles:["\/srv\/mediawiki\/php-1.34.0-wmf.8\/extensions\/Wikibase\/client\/..\/data-access\/src\/PerRepositoryServiceWiring.php"]
1 wgWikibasePerRepositoryServiceWiringFiles:["\/srv\/mediawiki-staging\/php-master\/extensions\/Wikibase\/client\/..\/data-access\/src\/PerRepositoryServiceWiring.php"]
1 wmgWikibaseClientBadgeClassNames:{"Q17437796":"badge-featuredarticle","Q17437798":"badge-goodarticle","Q17559452":"badge-recommendedarticle","Q17506997":"badge-featuredlist","Q17580674":"badge-featuredportal","Q20748091":"badge-notproofread","Q20748094":"badge-problematic","Q20748092":"badge-proofread","Q20748093":"badge-validated","Q28064618":"badge-digitaldocument","Q51759403":"badge-goodlist"}
1 wmgWikibaseClientBadgeClassNames:{"Q49444":"badge-goodarticle","Q49447":"badge-featuredarticle","Q49448":"badge-recommendedarticle","Q49449":"badge-featuredlist","Q49450":"badge-featuredportal","Q98649":"badge-notproofread","Q98650":"badge-problematic","Q98658":"badge-proofread","Q98651":"badge-validated"}
1 wmgWikibaseClientEchoIcon:{"path":"\/static\/images\/wikibase\/echoIcon.svg"}
1 wmgWikibaseClientEchoIcon:{"url":"\/static\/images\/wikibase\/echoIcon.svg"}
1 wmgWikibaseClientPropertyOrderUrl:"https:\/\/www.wikidata.org\/w\/index.php?title=MediaWiki:Wikibase-SortedProperties&action=raw&sp_ver=1"
1 wmgWikibaseClientRepoConceptBaseUri:"http:\/\/www.wikidata.beta.wmflabs.org\/entity\/"
1 wmgWikibaseClientRepoConceptBaseUri:"http:\/\/www.wikidata.org\/entity\/"
1 wmgWikibaseClientRepositories:{"":{"repoDatabase":"wikidatawiki","baseUri":"http:\/\/www.wikidata.org\/entity\/","entityNamespaces":{"item":0,"property":120,"lexeme":146},"prefixMapping":{"":""}}}
1 wmgWikibaseClientRepositories:{"":{"repoDatabase":"wikidatawiki","entityNamespaces":{"item":0,"property":120,"lexeme":146},"baseUri":"https:\/\/wikidata.beta.wmflabs.org\/entity\/","prefixMapping":{"":""}},"commons":{"repoDatabase":"commonswiki","entityNamespaces":{"mediainfo":"6\/mediainfo"},"baseUri":"https:\/\/commons.wikimedia.beta.wmflabs.org\/entity\/","prefixMapping":[]}}
1 wmgWikibaseClientRepoUrl:"https:\/\/wikidata.beta.wmflabs.org"
1 wmgWikibaseClientRepoUrl:"https:\/\/www.wikidata.org"
1 wmgWikibaseClientUseTermsTableSearchFields:false
1 wmgWikibaseClientUseTermsTableSearchFields:true
1 wmgWikibaseFeatureFlagWbeditentitySetEmptyAliases:false
1 wmgWikibaseFeatureFlagWbeditentitySetEmptyAliases:true
1 wmgWikibaseRepoBadgeItems:{"Q17437798":"wb-badge-goodarticle","Q17437796":"wb-badge-featuredarticle","Q17559452":"wb-badge-recommendedarticle","Q17506997":"wb-badge-featuredlist","Q17580674":"wb-badge-featuredportal","Q20748091":"wb-badge-notproofread","Q20748094":"wb-badge-problematic","Q20748092":"wb-badge-proofread","Q20748093":"wb-badge-validated","Q28064618":"wb-badge-digitaldocument","Q51759403":"wb-badge-goodlist"}
1 wmgWikibaseRepoBadgeItems:{"Q49444":"wb-badge-goodarticle","Q49447":"wb-badge-featuredarticle","Q49448":"wb-badge-recommendedarticle","Q49449":"wb-badge-featuredlist","Q49450":"wb-badge-featuredportal","Q98649":"wb-badge-notproofread","Q98650":"wb-badge-problematic","Q98658":"wb-badge-proofread","Q98651":"wb-badge-validated"}
1 wmgWikibaseSearchIndexProperties:[]
1 wmgWikibaseSearchIndexPropertiesExclude:[]
1 wmgWikibaseSearchIndexPropertiesExclude:["P304","P433","P478","P558","P3903","P3921","P4316","P1433","P2860"]
1 wmgWikibaseSearchIndexProperties:["P31","P279"]
1 wmgWikibaseSearchStatementBoosts:[]
1 wmgWikibaseSearchStatementBoosts:{"P31=Q4167410":-10,"P31=Q13442814":-5,"P31=Q18918145":-5}
1 wmgWikibaseSSRTermboxServerUrl:"https:\/\/ssr-termbox.wmflabs.org\/termbox"
1 wmgWikibaseUseSSRTermbox:true

Beta cluster lexeme-related configs:

wgLexemePrefixRescoreProfile:"lexeme_prefix"
wgLexemeEnableRepo:true
wgLexemeNamespace:146
wgLexemePrefixSearchProfiles:[]
wgWBQualityConstraintsWikibaseSenseId:"Q505135"
wgLexemeRescoreFunctions:[]
wmgUseWikibaseLexemeCirrusSearch:true
wgLexemeEnableSenses:true
wgLexemeLanguageCodePropertyId:"P218"
wmgUseWikibaseLexeme:true
wgLexemeDisableCirrus:true
wgLexemeUseCirrus:true
wgLexemeTalkNamespace:147
wmgNewWikibaseLexemeCirrusSearch:true
wgLexemeFulltextRescoreProfile:"lexeme_fulltext"
wgLexemePrefixSearchProfile:"lexeme_prefix"
wgWBQualityConstraintsWikibaseLexemeId:"Q505133"

It fails on test wikidata https://www.wikidata.org/w/api.php?action=wbgetentities&format=json&origin=*&ids=L1-S1

Oops!... I licked It Again

Unbelievably, it works in beta cluster but not in production and my localhost: https://wikidata.beta.wmflabs.org/w/api.php?action=wbgetentities&ids=L53-S1

This can't be done at the given time since most SRE people will be at SRE summit and hence no deployment (no SWAT, no train, no service, literally nothing of course except when production is on fire)

In T225092#5236838, @Reedy wrote:

$wgMessagesDirs['FlaggedRevs'] = __DIR__ . '/i18n/flaggedrevs';
$wgMessagesDirs['RevisionReview'] = __DIR__ . '/i18n/revisionreview';
$wgMessagesDirs['Stabilization'] = __DIR__ . '/i18n/stabilization';
$wgMessagesDirs['ReviewedVersions'] = __DIR__ . '/i18n/reviewedversions';
$wgMessagesDirs['UnreviewedPages'] = __DIR__ . '/i18n/unreviewedpages';
$wgMessagesDirs['PendingChanges'] = __DIR__ . '/i18n/pendingchanges';
$wgMessagesDirs['ProblemChanges'] = __DIR__ . '/i18n/problemchanges';
$wgMessagesDirs['ReviewedPages'] = __DIR__ . '/i18n/reviewedpages';
$wgMessagesDirs['StablePages'] = __DIR__ . '/i18n/stablepages';
$wgMessagesDirs['ConfiguredPages'] = __DIR__ . '/i18n/configuredpages';
$wgMessagesDirs['QualityOversight'] = __DIR__ . '/i18n/qualityoversight';
$wgMessagesDirs['ValidationStatistics'] = __DIR__ . '/i18n/validationstatistics';
$wgMessagesDirs['FlaggedRevsApi'] = __DIR__ . '/i18n/api';
$wgExtensionMessagesFiles['FlaggedRevsMagic'] = "$langDir/FlaggedRevs.i18n.magic.php";
$wgExtensionMessagesFiles['FlaggedRevsAliases'] = "$langDir/FlaggedRevs.alias.php";

In T225092#5236804, @zeljkofilipin wrote:

@Ladsgroup will this be fixed by train window tomorrow?

Definitely a train blocker.

In T220733#5236718, @zeljkofilipin wrote:

@Ladsgroup it's deployed, right?

https://tools.wmflabs.org/sal/log/AWsnSUSLOwpQ-3PkVwpT

Unlicking the cookie

I will do it ASAP.

Given that wb_terms is also fully replicated I don't see any reason to change it now. We remove terms from the term store when the page gets deleted/suppressed etc.

Deployment blocker: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/FlaggedRevs/+/514446 this needs to backported to wmf.8 before going to group1

Here's my two cents:

• I have done a similar thing with ores, in couple of months we made a huge Makefile that contains instructions for extracting and building the models for ores into per-wiki config files that turned into a generated Makefile at the end. I have some lessons learned from that action.
• We need to have a migration path. There's always lots of edge cases that can't be migrated in one go. Migrating model instructions took us months to finish and it was way simpler than InitialiseSettings.php. The way that we did it was we introduced the generated Makefile as "Makefile.automated" and started to include that empty file into the main one and per-wiki we moved the manual instructions to the new one (we also did an audit of each wiki and we found one error per wiki on average, because automation helped us check for inconsistencies). And after we were almost done, we renamed the Makefile to Makefile.manual and Makefile.automated to Makefile. The Makefile.manual still exists for weird edge cases (like wikidata)
• A similar course of action can be done here. Before getting things fully moved, we can define "InitialiseSettings-generated.php" and load it before InitialiseSettings.php (so values in InitialiseSettings.php override this one) and then in rather large batches move content and horrors of InitialiseSettings.php to per-wiki config files.
• The "InitialiseSettings-generated.php" can be generated before each deployment, we can put to .gitignore but the script that generates the file needs to show the diff and asks the user if the changes made make sense.
• Once everything is moved to InitialiseSettings-generated.php, then we can decide on how to properly do config management in mediawiki in production. The script can get all of those config files and put them somewhere else (even etcd directly) if needed. My idea basically narrows down the focus of this solution and make it very local to add space for breathing for future changes.
• I'm personally in favor of using pretty json for two reasons: 1- There's no native/built-in support for yaml in python or php. I've been dealing with yaml and it's a big headache even with python. 2- json is sorta standard in mediawiki these days, we have extension.json (and similar things like composer.json), etc. 3- (It's not a real reason): We can add support for comments the way we added support for config variables in extension.json. We can ignore any key value that starts with _ or @ in generating the result. (Which also brings the question of should we output them as comment in the result or we should omit comments).

I think having one or several (HA) VPS nodes is way better than having them in toolforge. We have more control over the node. I might be able to setup something quickly. Is there a help on how to setup a standalone service?

In T191231#5230413, @Fheredia wrote:

a) It looks like colliding with https://phabricator.wikimedia.org/T113831

Thanks for the comment :)

To recap. I suggest these two:

• Drop official support for Oracle and MSSQL
• Use Doctorine DBAL to handle SQL generation of schema and schem changes (practically turning .sql files to .json files)

In T218155#5228955, @Ankry wrote:

@Ladsgroup Are thre any tickets describing what are the script problems that this task should depend on?

In T205045#5225805, @jcrespo wrote:

and DBAs to babysit the deployment for a couple of days

I can do that if you tell me anyting specific -metrics- to check for (other than typical error count and general stability. Specially if initial deployment happens at the beginning of next week for example, near EU timezone.

I agreed to work on it but I would like to keep these tickets unassigned to me in order to avoid "licking the cookie". i.e. if someone beats me to do it, I don't want to block them. I might not be able to get to it any time soon.

In T205045#5223716, @Lydia_Pintscher wrote:

@jcrespo thank you! That's useful information.

@Ladsgroup @Addshore What's needed to get this ready for pickup in camp?

Reza is @Yamaha5 :)

One quick update. I worked a little bit on this in the hackathon. A POC is almost ready, I'm trying to finish it soon.
Also, by checking the code, it seems "templatelinks" table is the best option to start for two reasons: 1- the target column is only referenced around 20 times in the code, making the migration at code level easier than other tables (categorylinks is referenced around 80 times) 2- it's the biggest links table. (It's basically lowest and heaviest hanging fruit.)

In the discussion with @daniel and @Anomie it become apparent that it's better to drop support for Oracle and MSSQL before trying to abstract it and moving support to extensions and if someone wants to maintain the support, they should do it in the extension.

Adding Performance because handshakes on TLS 1.3 are 100ms faster and also it caches handshakes (https://kinsta.com/blog/tls-1-3/). Hope that's fine for you.

Okay, now the first batch is ran