We backported this to wmf14 now to check for fallout. If there's none, we will also target wmf13 later on.

I assume this is fixed now (didn't test, though).

Alternative patch: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/523412

This is again related to dbgroupdefault (caused by ab09cf2fa59).

In T227063#5332149, @Johan wrote:

@hoo Shouldn't happen, as in "won't happen, we're worrying unnecessarily"?

In T227063#5328194, @Lucas_Werkmeister_WMDE wrote:

but strangely UpdateRepoOnMoveJob (more precisely UpdateRepoJob) doesn't fail if it can't edit, it just sends a debug message and act like nothing happened in saveChanges

More specifically, saveChanges() does return true/false to indicate if everything’s okay, but run() ignores the return value and unconditionally returns true itself. I don’t know if this was intentional at the time, but I also think it would probably be better to retry such cases (i. e. return $this->saveChanges( $item, $user );).

Please try to follow these steps: T226499#5326479

In T226499#5316346, @Rosalie_WMDE wrote:

I followed this documentation https://www.mediawiki.org/wiki/Wikibase/Installation/Advanced_configuration#Define_links_for_external_identifiers localy and the Formatted URL did not work. It happened just like on Factgrid. I am thinking it is a bug on our side.

Apparently this is related to the --dbgroupdefault (doesn't seem to depend on the value):

This is all done.

GenericServices::getCompactBaseDataModelSerializerFactory, which is set to omit hashes for references and mainsnaks, is used for everything, but references are correctly included. So either they're injected later on (but I don't see that happening) or the requested omission is not working correctly, it seems.

Only thing left to do: Merge https://gerrit.wikimedia.org/r/520501 (makes the integration test cover the actual serialization code).

This was caused by a1c5b7ed3eccef653a7041f42d65a09ff83bf6da

In T226008#5303836, @alaa_wmde wrote:

I think we also need to pass domain in the call to ILoadBalancer::getConnection in PrefetchingPropertyTermLookup::getDbr (remember, when a specific domain is requested, you also need to mark the connection with LoadBalancer::reuseConnection later on).

@hoo what are domains actually?

I think we also need to pass domain in the call to ILoadBalancer::getConnection in PrefetchingPropertyTermLookup::getDbr (remember, when a specific domain is requested, you also need to mark the connection with LoadBalancer::reuseConnection later on).

This broke beta commons: https://commons.wikimedia.beta.wmflabs.org/wiki/File:CLD_test.webm (Error: 1146 Table 'commonswiki.wbt_property_terms' doesn't exist (172.16.4.147:3306))

This should(tm) should never happen again.

I assume this is done now… feel free to move this back if anything is missing.

In T221774#5296984, @alaa_wmde wrote:

@maho re feedback on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/512393

Do we even want to do this via Prometheus (WikimediaPrometheusSparqlEndpointReplicationStatus)

Sounds like the right thing to do so to me too, as per https://phabricator.wikimedia.org/T221774#5165165. Warning though, I have very limited knowledge so far on what Prometheus is setup to monitor in our stack. Links are welcome :)
If I understand correctly, it's main and probably only downside is when Prometheus has outdated data on the lag for some reason (if WDQS instance is down/depooled? or the refresh job fails often?) .. which is the same scenario that raises your last questions, I believe:

Shall we disregard lag data if it's really old

I think here I'd rather keep counting that lag. It is at the end the latest known lag to the user querying the lag (last refresh, refresh rate and interval are implementation details). And to this implementation detail, if the latest lag that is known is much farther in the past than I expected it to be, then I should for safety sake go with the assumption that the instance is actually lagging at least that much rather than assuming than risking ignoring that lag, which might in edge cases result in processing the edit instead of rejecting it due to lag.
Does that sound reasonable? or did I misunderstood too much in there?

I looked into why these incomplete and broken dumps were even published, found and fixed the cause(s) (see above).

In T226639#5287258, @alaa_wmde wrote:

I wonder if waiting for replication might be another option here to avoid duplicate inserts caused by lag altogether?

I just discovered a very similar case when locking AureliaSperry. Nothing of interest in the logs at a glance (no obviously related exceptions, fatals, …).

I was asked to quickly chime in on this, AFAIS the best way for you would be to hook into AbuseFilter-filterAction and then figure out your context based on the other variables present (yeah this is not exactly nice :/).

I just realized that limiting the type and language in TermIdsResolver wont do the trick, as we then still need to get all wbpt_term_in_lang_ids first.

In T198676#5246717, @ArielGlenn wrote:

Oh, congrats belatedly then :-) I don't know if dcat has special format requirements; if not, then even a simple text file looking kind of like http://get-simple.info/start/changelog/ would be fine as far as I'm concerned. @hoo begging you for assistance once again!

@alaa_wmde @Lucas_Werkmeister_WMDE Does that make sense to ^?

We could change resolveTermIds( array $termIds ): array (from TermIdsResolver) to resolveTermIds( array $termIds, array $types = null, array $languages = null ): array where we restrict the result set to only contain thing of $types and/ or only things in language(s) $languages. I would change resolveGroupedTermIds accordingly.

Moving this to review, so that we can decide if we want to abandon this (probably, yes).

I dimly remember @hoo saying that this would cause debug log warnings and we should expliclitly specify the correct value in mediawiki-config instead.

I'm not sure what level exactly it logs at, but it at very least debug logs… and given we resolve this on almost every Wikidata related request, we should try to avoid this also for performance reasons (even if no one listens to that log).

I also don't see any reason to filter this, the current design is mostly equivalent to wb_terms which is also fully available (only public data is present).

In T224269#5225831, @alaa_wmde wrote:

@hoo for wrapping DatabasePropertyTermStore/DatabaseItemTermStore logic (storeTerms() and deleteTerms() methods) inside transactions, I'd probably create a separate task for that (and merge the changes that introduce the classes for now). Make sense to you?

As also briefly noted in Gerrit, relying on application logic only to ensure that rows are (still) in place seems to be a fairly weak consistency pledge. Exceptions, DB failures, sudden HW failures, …, can prevent these (re-)checks from happening, leaving the tables in an inconsistent state… unlike in transactions, this wont result in a rollback of all changes, but leave us in that inconsistent state.
That shouldn't block us from doing this, but it's something we need to be very well aware of… doing this means we can never be sure all tables are consistent (although I guess we can't ever be really sure about that, thus shouldn't make that assumption).

This might be fixed along with T223995 which both might have the same root cause.

This used to work, see T196297 (but we didn't write tests back then :S )

Also results in error: https://www.wikidata.org/w/api.php?action=wbgetentities&format=json&origin=*&ids=L11533-F7&languages=de&languagefallback=1

@fgiunchedi @Smalyshev How about I do something like this:

$result = json_decode(file_get_contents('http://prometheus.svc.eqiad.wmnet/ops/api/v1/query?query=blazegraph_lastupdated'), true)['data']['result'];
foreach ( $result as $resultByInstance ) {
	echo $resultByInstance['metric']['instance'] . ' (' .$resultByInstance['metric']['cluster'] . ') has been updated at ' . $resultByInstance['value'][1] . PHP_EOL;
}

wdqs1004:9193 (wdqs) has been updated at 1557755605
wdqs1005:9193 (wdqs) has been updated at 1557755946
wdqs1006:9193 (wdqs) has been updated at 1557758395
wdqs1003:9193 (wdqs-internal) has been updated at 1557758386
wdqs1007:9193 (wdqs-internal) has been updated at 1557758374
wdqs1008:9193 (wdqs-internal) has been updated at 1557758353
wdqs1009:9193 (wdqs-test) has been updated at 1557758403
wdqs1010:9193 (wdqs-test) has been updated at 1557758409

This has always been the case: Ever since that was introduced with 9f25055cd98293aa0b49fe16acd9490b231b29c1 (the function was later renamed, but this is about the implementation).

Per https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/509437#message-f7b925c6397cdb5a95b2561998b51e3d7c798d51: "can be merged after Tuesday next week." (so after May 14)

@Michael In https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/509100#message-105c66f111a3e15944690f1bef71889e210a61b5 you said that https://gerrit.wikimedia.org/r/509356 will probably fix all of that? So… is this done, can we abandon the other patches?

@WMDE-leszek Why was this reopened?

@Smalyshev Do you think it would be enough to look at http://prometheus.svc.eqiad.wmnet/ops/api/v1/query?query=blazegraph_lastupdated and http://prometheus.svc.codfw.wmnet/ops/api/v1/query?query=blazegraph_lastupdated (no matter which DC MW is running in) and just take all servers into account? Or do we need a whitelist/blacklist (or both) or some other mechanism to make sure we don't eg. take servers into account that are being maintained.

In T222259#5159066, @Lea_Lacroix_WMDE wrote:

I still see the "edit" text on https://wikidata-shex.wmflabs.org/w/index.php?title=EntitySchema:E44&action=edit does it mean it's not yet merged?

Alternative approach: