Still happening (very occasionally):

Maybe EntityUsageTable::INSERT_BATCH_SIZE_FACTOR should be changed from 5 to 3/2.5/2?

I merged the patch and will deploy it probably in the next days… but without a configuration change, this is a no-op for now.

Done with https://gerrit.wikimedia.org/r/c/wikidata/query/gui/+/483780/1 and https://gerrit.wikimedia.org/r/c/wikidata/query/gui/+/483781/1, I suppose.

In T189506#4858317, @Lydia_Pintscher wrote:

@hoo are you still working on this? Should it move to camp?

So this is just about registering a dummy special page (which is not yet able to actually create a schema)?!

In T211935#4824420, @ArielGlenn wrote:

In T211935#4824159, @hoo wrote:

I tested this with testwikidatawiki on mwdebug1002, php7.2 should be fine as far as the Wikibase entity dumps are concerned (output is exactly equal to php7.0/hhvm).

Awesome! What types was this for?

I tested this with testwikidatawiki on mwdebug1002, php7.2 should be fine as far as the Wikibase entity dumps are concerned (output is exactly equal to php7.0/hhvm).

In T184933#4819706, @Lea_Lacroix_WMDE wrote:

@hoo Another question was also raised: is it possible to adapt the scale of the map (=zoom level) depending on what it is about?
(see discussion here)

In T204531#4813322, @hoo wrote:

I just revisited the original logs… and I found that the way way way more common error back then was from LoadBalancer::getRandomNonLagged:

2018-09-19 07:13:27 [1d74bb1b0e64917ec3cd47e7] snapshot1008 wikidatawiki 1.32.0-wmf.20 DBReplication ERROR: Wikimedia\Rdbms\LoadBalancer::getRandomNonLagged: server db1109 is not replicating? {"host":"db1109"}

I'll try to have a look at this soon.

This is scheduled to be enabled on Dec 20 after the train (probably around 19:00–20:00 UTC).

Still needs to be enabled via mediawiki-configuration.

I suggest to enable this tracking per group for all groups (wikipedia, wikiquote, …) and for the following wikis specifically:

1. commonswiki
2. ruwiki
3. ukwiki
4. arwiki
5. zhwiki
6. cawiki
7. frwiki
8. enwiki
9. svwiki
10. itwiki

This looks very good. I just checked and even the latest PHP still uses the "zlib compatibility functions" which had troubles with chunked bzip2s before (see T118379) (it uses them for both the compress.bzip2 stream wrapper and for bzopen).

In T184933#4812900, @TheDJ wrote:

Fancy...
Just a followup/enhancement idea: since we have precision.. wouldn't this be a good place to draw a precision circle ?

I just revisited the original logs… and I found that the way way way more common error back then was from LoadBalancer::getRandomNonLagged:

Given the lock timeouts happen on SELECT, I don't really see how using a single query instead of SELECT+UPDATE/INSERT would greatly help us here. While that's definitely nice to have performance wise (less queries, less roundtrips), the main problem seems to me that something is getting a new id and then keeping the transaction (=> the lock) open to do other things before commit/rollback.

Maybe duplicate/ overlapping with T210044: Data corruption when loading RDF data into WDQS?

Anything that depends on a particular db server makes me uneasy

Or we could alter LoadMonitor::getServerStates to not log for a certain db-server more than every N seconds (where N could be derived from LAG_WARN_THRESHOLD, which currently is 10s).

I just looked into this some more… We could also create a new LB settings key which would be passed as lagWarnThreshold to LoadMonitor (currently this options is never set, as their is no way to do so, thus this always defaults to LoadMonitor::LAG_WARN_THRESHOLD (10s)).

The issue at hands has been solved, and we also halved the time the dumper instances are running.

I just scheduled a deployment to enable this on Wikidata for December 12 12:00–13:00 UTC.

This can now be seen on testwikidata: https://test.wikidata.org/wiki/Q22#P10 (the newest version of the authority control gadget is also already in place, so that's how it's going to look on Wikidata)

Resolved with T175273#4800204.

Deployment of this has been scheduled for today 14:00-15:00 UTC.

Actually enabling arbitrary access on wiktionaries is being tracked in T175273, this is just a (now resolved) prerequisite for this.

This was closed due to misunderstanding.

3 years later and this is still happening :/

@Ladsgroup What's missing here to be able to update the sites tables? Just https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/473510?

In T208317#4798101, @Lydia_Pintscher wrote:

Uhmmm this is exactly what I've been told this is doing. We even mentioned it in the weekly summary. What's going on?

In T208317#4797677, @Lea_Lacroix_WMDE wrote:

So if I understand correctly, the result of this ticket is not changing anything for users? ie it's not enabling arbitrary access on all Wiktionaries?

In T184933#4791041, @Lucas_Werkmeister_WMDE wrote:

I don’t think so, why? The already merged patches are mostly for displaying coordinates on the repo, whereas the linked chain is for the client, if I understand correctly.

@Lucas_Werkmeister_WMDE This is done for Wikibase repo… but there's still a patch (by @Tpt) pending for the wikitext formatter (for clients). Do we want to create a new bug for that?

Adapted the AuthorityControl gadget for this: https://www.wikidata.org/w/index.php?title=MediaWiki%3AGadget-AuthorityControl.js&type=revision&diff=800479818&oldid=744294484

In T144932#4751268, @Smalyshev wrote:

Is this still relevant? We do have regular deploys now at 10am PDT Monday.

In T147997#4734766, @Lydia_Pintscher wrote:

Do we know how many items we currently have that are completely empty?

Using Kartographer on Wikibase Repo is now ready to be reviewed IMO :)

Just for the record, the errors were:
2018-09-19 07:13:15 [580058d7b2babd5f85e725ec] snapshot1008 wikidatawiki 1.32.0-wmf.20 DBReplication ERROR: Server db1109 has 29.26508808136 seconds of lag (>= 10) {"host":"db1109","lag":29.26508808136,"maxlag":10}
2018-09-19 07:13:31 [c36272088a424bcc438a31e9] snapshot1008 wikidatawiki 1.32.0-wmf.20 DBReplication ERROR: Wikimedia\Rdbms\LoadBalancer::getRandomNonLagged: server db1109 is not replicating? {"host":"db1109"}
Was there anything else?

In T176282#4604145, @Krinkle wrote:

Is this obsolete per T176277 being declined?

In T147797#4717951, @Addshore wrote:

@hoo is this still an issue?

Related https://github.com/wmde/WikibaseDataModelServices/pull/200

Given that this seems to only happen rarely, I think this can wait for the train next week.

@aaron Suggested to use a separate master DB connection for this (if were in a transaction and don't immediately autocommit), so that it never ends up in the main transaction. That might mean slightly more wasted IDs, but should significantly reduce contention.

In T204066#4707450, @Ladsgroup wrote:

I looked at this today and I think we should revisit the estimation for this task. The reason it doesn't work is that ImageLinksDataUpdater gets an entity and only adds image usages for the entity and not its subentities. The only possible way that I can think of is to add a "getSubEntities()" to EntityDocument interface and implement it with empty value everywhere, then use it in EntityStatementDataUpdaterAdapter + praying that in future we don't end up having subentities that would have subentities of their own.

AFAICT this would only make one practical difference: It would also load Lexeme on the wiktionaries (which we totally want).