Currently these link fixes are being done in the Gadget-CentralAuthInterlinkFixer.js, but only for Special:Log and Special:Recentchanges. That's probable the place to look at for this cosmetic fix.

I looked into this a bit and found https://grafana.wikimedia.org/dashboard/snapshot/08uo4Z113gB4aJJet3qVUq1Ock5sfkv6?panelId=1&fullscreen&orgId=1 (filter for SqlEntityInfoBuilder_collectTermsForEntities) very interesting.

Please have a look at jQuery.wikibase.entityselector (can be loaded via the jquery.wikibase.entityselector RL module). I didn't test it, but that should just work.

In T229564#5404225, @DannyS712 wrote:

In T229564#5404223, @hoo wrote:

In T229564#5403493, @Jony wrote:

I added @hoo, maybe he know documentation for this.

I'm not sure we have documentation for this anywhere (I doubt it)… but these are automated edits done by Wikibase whenever a linked page is deleted (or moved). What would you like to know/ have documented about these?

xtools just links tools to their documentation page, so having a related target is ideal

In T229564#5403493, @Jony wrote:

I added @hoo, maybe he know documentation for this.

@Seb35 What you describe here sounds like T142691: [Bug] wb_terms table truncates labels exceeding 255 bytes, possibly leaving invalid UTF-8. This task is about the columns storing entity ids as string (like Q123 or P456).

Yeah, this is on purpose… this is rather complex (unfortunately). The thing is that we make sure that usage are "correct" by: Immediately purging no-longer accurate usages on page touches/ re-parses on edit (it might be that we also add new usages here, but that's not technically needed), then we later on add all usage to the table that come up when parsing the page in different ways (different language, mostly, maybe exclusively).

We backported this to wmf14 now to check for fallout. If there's none, we will also target wmf13 later on.

I assume this is fixed now (didn't test, though).

Alternative patch: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/523412

This is again related to dbgroupdefault (caused by ab09cf2fa59).

In T227063#5332149, @Johan wrote:

@hoo Shouldn't happen, as in "won't happen, we're worrying unnecessarily"?

In T227063#5328194, @Lucas_Werkmeister_WMDE wrote:

but strangely UpdateRepoOnMoveJob (more precisely UpdateRepoJob) doesn't fail if it can't edit, it just sends a debug message and act like nothing happened in saveChanges

More specifically, saveChanges() does return true/false to indicate if everything’s okay, but run() ignores the return value and unconditionally returns true itself. I don’t know if this was intentional at the time, but I also think it would probably be better to retry such cases (i. e. return $this->saveChanges( $item, $user );).

Please try to follow these steps: T226499#5326479

In T226499#5316346, @Rosalie_WMDE wrote:

I followed this documentation https://www.mediawiki.org/wiki/Wikibase/Installation/Advanced_configuration#Define_links_for_external_identifiers localy and the Formatted URL did not work. It happened just like on Factgrid. I am thinking it is a bug on our side.

Apparently this is related to the --dbgroupdefault (doesn't seem to depend on the value):

This is all done.

GenericServices::getCompactBaseDataModelSerializerFactory, which is set to omit hashes for references and mainsnaks, is used for everything, but references are correctly included. So either they're injected later on (but I don't see that happening) or the requested omission is not working correctly, it seems.

Only thing left to do: Merge https://gerrit.wikimedia.org/r/520501 (makes the integration test cover the actual serialization code).

This was caused by a1c5b7ed3eccef653a7041f42d65a09ff83bf6da

In T226008#5303836, @alaa_wmde wrote:

I think we also need to pass domain in the call to ILoadBalancer::getConnection in PrefetchingPropertyTermLookup::getDbr (remember, when a specific domain is requested, you also need to mark the connection with LoadBalancer::reuseConnection later on).

@hoo what are domains actually?

I think we also need to pass domain in the call to ILoadBalancer::getConnection in PrefetchingPropertyTermLookup::getDbr (remember, when a specific domain is requested, you also need to mark the connection with LoadBalancer::reuseConnection later on).

This broke beta commons: https://commons.wikimedia.beta.wmflabs.org/wiki/File:CLD_test.webm (Error: 1146 Table 'commonswiki.wbt_property_terms' doesn't exist (172.16.4.147:3306))

This should(tm) should never happen again.

I assume this is done now… feel free to move this back if anything is missing.

In T221774#5296984, @alaa_wmde wrote:

@maho re feedback on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/512393

Do we even want to do this via Prometheus (WikimediaPrometheusSparqlEndpointReplicationStatus)

Sounds like the right thing to do so to me too, as per https://phabricator.wikimedia.org/T221774#5165165. Warning though, I have very limited knowledge so far on what Prometheus is setup to monitor in our stack. Links are welcome :)
If I understand correctly, it's main and probably only downside is when Prometheus has outdated data on the lag for some reason (if WDQS instance is down/depooled? or the refresh job fails often?) .. which is the same scenario that raises your last questions, I believe:

Shall we disregard lag data if it's really old

I think here I'd rather keep counting that lag. It is at the end the latest known lag to the user querying the lag (last refresh, refresh rate and interval are implementation details). And to this implementation detail, if the latest lag that is known is much farther in the past than I expected it to be, then I should for safety sake go with the assumption that the instance is actually lagging at least that much rather than assuming than risking ignoring that lag, which might in edge cases result in processing the edit instead of rejecting it due to lag.
Does that sound reasonable? or did I misunderstood too much in there?

I looked into why these incomplete and broken dumps were even published, found and fixed the cause(s) (see above).

In T226639#5287258, @alaa_wmde wrote:

I wonder if waiting for replication might be another option here to avoid duplicate inserts caused by lag altogether?

I just discovered a very similar case when locking AureliaSperry. Nothing of interest in the logs at a glance (no obviously related exceptions, fatals, …).

I was asked to quickly chime in on this, AFAIS the best way for you would be to hook into AbuseFilter-filterAction and then figure out your context based on the other variables present (yeah this is not exactly nice :/).

I just realized that limiting the type and language in TermIdsResolver wont do the trick, as we then still need to get all wbpt_term_in_lang_ids first.

In T198676#5246717, @ArielGlenn wrote:

Oh, congrats belatedly then :-) I don't know if dcat has special format requirements; if not, then even a simple text file looking kind of like http://get-simple.info/start/changelog/ would be fine as far as I'm concerned. @hoo begging you for assistance once again!

@alaa_wmde @Lucas_Werkmeister_WMDE Does that make sense to ^?

We could change resolveTermIds( array $termIds ): array (from TermIdsResolver) to resolveTermIds( array $termIds, array $types = null, array $languages = null ): array where we restrict the result set to only contain thing of $types and/ or only things in language(s) $languages. I would change resolveGroupedTermIds accordingly.

Moving this to review, so that we can decide if we want to abandon this (probably, yes).

I dimly remember @hoo saying that this would cause debug log warnings and we should expliclitly specify the correct value in mediawiki-config instead.

I'm not sure what level exactly it logs at, but it at very least debug logs… and given we resolve this on almost every Wikidata related request, we should try to avoid this also for performance reasons (even if no one listens to that log).

I also don't see any reason to filter this, the current design is mostly equivalent to wb_terms which is also fully available (only public data is present).

In T224269#5225831, @alaa_wmde wrote:

@hoo for wrapping DatabasePropertyTermStore/DatabaseItemTermStore logic (storeTerms() and deleteTerms() methods) inside transactions, I'd probably create a separate task for that (and merge the changes that introduce the classes for now). Make sense to you?

As also briefly noted in Gerrit, relying on application logic only to ensure that rows are (still) in place seems to be a fairly weak consistency pledge. Exceptions, DB failures, sudden HW failures, …, can prevent these (re-)checks from happening, leaving the tables in an inconsistent state… unlike in transactions, this wont result in a rollback of all changes, but leave us in that inconsistent state.
That shouldn't block us from doing this, but it's something we need to be very well aware of… doing this means we can never be sure all tables are consistent (although I guess we can't ever be really sure about that, thus shouldn't make that assumption).

This might be fixed along with T223995 which both might have the same root cause.

This used to work, see T196297 (but we didn't write tests back then :S )

Also results in error: https://www.wikidata.org/w/api.php?action=wbgetentities&format=json&origin=*&ids=L11533-F7&languages=de&languagefallback=1

@fgiunchedi @Smalyshev How about I do something like this:

$result = json_decode(file_get_contents('http://prometheus.svc.eqiad.wmnet/ops/api/v1/query?query=blazegraph_lastupdated'), true)['data']['result'];
foreach ( $result as $resultByInstance ) {
	echo $resultByInstance['metric']['instance'] . ' (' .$resultByInstance['metric']['cluster'] . ') has been updated at ' . $resultByInstance['value'][1] . PHP_EOL;
}

wdqs1004:9193 (wdqs) has been updated at 1557755605
wdqs1005:9193 (wdqs) has been updated at 1557755946
wdqs1006:9193 (wdqs) has been updated at 1557758395
wdqs1003:9193 (wdqs-internal) has been updated at 1557758386
wdqs1007:9193 (wdqs-internal) has been updated at 1557758374
wdqs1008:9193 (wdqs-internal) has been updated at 1557758353
wdqs1009:9193 (wdqs-test) has been updated at 1557758403
wdqs1010:9193 (wdqs-test) has been updated at 1557758409