In T144932#4751268, @Smalyshev wrote:

Is this still relevant? We do have regular deploys now at 10am PDT Monday.

In T147997#4734766, @Lydia_Pintscher wrote:

Do we know how many items we currently have that are completely empty?

Using Kartographer on Wikibase Repo is now ready to be reviewed IMO :)

Just for the record, the errors were:
2018-09-19 07:13:15 [580058d7b2babd5f85e725ec] snapshot1008 wikidatawiki 1.32.0-wmf.20 DBReplication ERROR: Server db1109 has 29.26508808136 seconds of lag (>= 10) {"host":"db1109","lag":29.26508808136,"maxlag":10}

In T176282#4604145, @Krinkle wrote:

Is this obsolete per T176277 being declined?

In T147797#4717951, @Addshore wrote:

@hoo is this still an issue?

Related https://github.com/wmde/WikibaseDataModelServices/pull/200

Given that this seems to only happen rarely, I think this can wait for the train next week.

@aaron Suggested to use a separate master DB connection for this (if were in a transaction and don't immediately autocommit), so that it never ends up in the main transaction. That might mean slightly more wasted IDs, but should significantly reduce contention.

In T204066#4707450, @Ladsgroup wrote:

I looked at this today and I think we should revisit the estimation for this task. The reason it doesn't work is that ImageLinksDataUpdater gets an entity and only adds image usages for the entity and not its subentities. The only possible way that I can think of is to add a "getSubEntities()" to EntityDocument interface and implement it with empty value everywhere, then use it in EntityStatementDataUpdaterAdapter + praying that in future we don't end up having subentities that would have subentities of their own.

AFAICT this would only make one practical difference: It would also load Lexeme on the wiktionaries (which we totally want).

This seems to be fairly common now (at least on some days):

All changes deployed, this should be fully functional now.

The change mentioned above got deployed, but I just noticed that we also need to add trwiktionary to dblists/wikidataclient.dblist. I'll take care of this and make sure it gets deployed today or tomorrow.

Patch: https://github.com/DataValues/Number/pull/121

Scheduled deployment of this on Monday (Oct 29) 18:00-19:00 UTC.

One thing we could possibly do next: Add a hook in getConnection (or somewhere close) that let's us kill the connection attempt (or the entire script) in case an unwanted replica is selected.

In T138208#4679458, @jcrespo wrote:

From another thread, from TimS:

It sounds like the snapshot hosts were in fact needing and using the host. It's not LoadBalancer's fault if some maintenance script calls wfGetDB() without specifying a query group. Throwing an exception is the correct thing to do if the MW configuration is incorrect, since it allows the maintenance script to terminate and be restarted with different configuration.

Change deployed and I did the corresponding change on meta: 12:48, 20 October 2018 Hoo man (talk | contribs | block) changed global group permissions for Special:GlobalUsers/steward: added (none); removed globalgrouppermissions, globalgroupmembership (Now handled by meta-local rights T207531) .

In T207278#4677853, @ArielGlenn wrote:

That time would be ok for me (my evening but it's not too late). @hoo?

Well, last week Sunday afternoon was the only possible time for this :( … mid-term we will (hopefully) solve T206535, but for now we should either disable the cron(s) or I can carefully stop the scripts on demand and continue them after the maintenance.

In T206535#4675204, @Smalyshev wrote:

I just thought about this a bit and we might want to split the dumping process up into two steps:

I thought that's what is happening now? Or I miss something?

I just thought about this a bit and we might want to split the dumping process up into two steps:

1. Generating the dump via the maintenance script (sharded) and concatenating the shards
2. Re-Compress/ format conversion (ttl <> nt)/ …

In T206535#4673798, @ArielGlenn wrote:

I've enabled the use of lbzip2 for the xml/sql dumps starting with the Oct 20th run; we could consider using this for the wikidata weeklies recompression into bz2 files, at, say, four threads (half the number of shards). As far as I can tell it puts out binary-format compatible dumps to those produced by bzip2, though not byte-identical output. What do folks think?

In T147169#4669703, @Marostegui wrote:

In T147169#4669702, @hoo wrote:

We currently run the dumper instances for about 1h... would it be helpful to shorten this to maybe 30m?

It will definitely help, but it may still get into race conditions, but obviously less than if it is every 1h :-)

In T147169#4666218, @jcrespo wrote:

@jcrespo Do you have any more information for what to look out here?

The problem is most likely the loadbalancer- working with old data, even if the server is not needed, generates lag warnings for some complex dependency of how lag check works (generating 40K logs per minute). Abandon ship- this is not going to work, and nothing that wikibase developers or DBAs do can fix it.

Although you could do a new database requests every 10 minutes or so to mitigate that. Kill the process and restart it so it reloads the new config.

In T147169#4506139, @jcrespo wrote:

I still have no idea how this could happen (also these errors don't seem to be on mwlog1001?!).

Does dumpJson.php call any function that could force any check on all replicas to fail (and because it has old configuration, it fails on a depooled non-slow host?

What's the state of https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PropertySuggester/+/458187? @Jonas can it be merged?

Thanks @Marostegui but this sadly didn't help. Do you have any other ideas what could cause these discrepencies?

In T205865#4655143, @Addshore wrote:

Reversing this experiment now that we have switched back I see:

mwmaint 1001: 1.4454369544983
mwmaint 2001: 73.811069011688

In T44325#4650472, @Addshore wrote:

@hoo

related to https://www.wikidata.org/wiki/Wikidata:Project_chat/Archive/2014/05#Items_with_same_sitelink and https://www.wikidata.org/wiki/Wikidata:Contact_the_development_team#True_duplicates_clean_up?

What script did you run to generate the list?

In T179008#4597143, @Addshore wrote:

@hoo are you working on this one?

It indeed is the lockmanager. I ran the following on both mwmaint2001 and mwmaint1001:

I poked at potential DB related changes a bit and found none. The tables are consistent across the cluster AFAICT and the queries are being executed in the same way.

I have first shot at this up at https://github.com/mariushoch/revscoring/blob/bug/T158430/T158430.py, but this doesn't yet fully work as the required API change will only be deployed tomorrow.

@Halfak The patch is currently waiting to be reviewed.

Hm, it seems to me the way forward here would be to include python-logstash and then add it as logging handler via the deployed logging_config.yaml.

I think this grew from some discussion back then, but given it had no clear goals and I don't think this is as relevant anymore, let's decline it.

In T147169#4498457, @jcrespo wrote:

No need to be sorry, could this be a load balancer bug?

In T196297#4278095, @Addshore wrote:

This means https://www.wikidata.org/w/api.php?action=wbgetentities&ids=L4-S1 for example should work

Ok, so I just looked into this again and removing the redundant languages after the summary is not easily possible. These are needed as it's possible to add/remove/change several glosses at once and still be able to denote their respective languages.

After discussing this with Lydia, we decided to change the message content and no longer append the language key. The message should look like this: "Changed [de] gloss of Sense L1-F1: foobar".

This will need some more love.

Hm, the message key for this is wikibase-lexeme-summary-set-sense-glosses (which is in line with wikibase-lexeme-summary-set-form-representations for form representations). Form representation change edit summaries currently render as: Changed representation of Form L1-F1: de-at: foo.

Keeping this open as https://gerrit.wikimedia.org/r/450968 still needs to be resolved.

One question I just came across… what language shall we show the available languages in? Shall we use autonyms (native names)? What about languages that are not in CLDR?

The respective patch has been merged into master.

Talking to @Addshore, we probably want to go for something along the lines of action=query&meta=wikibasecontentlanguages&context=gloss.

It seems that we do this differently for Monolingual texts atm: There this is handled via wikibase.WikibaseContentLanguages which is basically an access wrapper for mw.config.get( 'wgULSLanguages' ) (some languages supported by the backed are missing, though). This is being discussed in T124758.

I talked about this with @Lydia_Pintscher today and we agreed that the most viable option for this is to simply track the number of calls to each Lua function when they happen. Tracking the number of function calls per page cannot easily be supported by our current infrastructure (attaching this information to pages/ parser cache entries in an easily queryable way is hard).

In T201418#4485482, @Lucas_Werkmeister_WMDE wrote:

Thanks, should I rebase the patch against master and upload it on Gerrit as a regular change? (I assume we don’t need to bother with a pre-release announcement or anything for this extension.)

Deployed the above patch and result looks good: https://test.wikidata.org/w/api.php?action=wbformatvalue&format=json&datavalue={%22value%22%3A{%22id%22%3A%22L99-F2%22}%2C%22type%22%3A%22wikibase-entityid%22}&options={%22lang%22%3A%22en%22}&generate=text%2Fhtml%3B%20disposition%3Dverbose&property=P75109 (the < in asldflkd<sdfsfjsdlfkjsdf is correctly escaped).

If you prepare the patch, I can look into deploying this (unless @Bawolff wants to).