In T252091#6171258, @tstarling wrote:

Load is average queue size, if you take the currently running batch as being part of the queue. WDQS currently does not monitor the queue size. I gather (after an hour or so of research, I'm new to all this) that with some effort, KafkaPoller could obtain an estimate of the queue size by subtracting the current partition offsets from KafkaConsumer.endOffsets().

I think that schema:ImageObject should be kept since we may have AudioObject and VideoObject

@JAllemandou I think that is an option as well, the thing is that is it is transitional to help to bootstrap a test of the full pipeline. In the end we won't be using jumbo and thus won't be able to rely on a 30days retention on main so hopefully we'll be able to reset the retention back to 7days once we're done with the test.
To circumvent this particular problem (time to make the dumps available > retention) we could either:

• send back the events that matter back to kafka and have higher retention like you suggest
• create a dedicated job running on the analytics network to read the events stored in HDFS and figure out a way to make the resulting data available in kafka main

Wikibase has now a way to override the default namespaces, for commons it should happen thanks to https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/569260 .

@WMDE-leszek oops, sorry I replied before reading you comment and was reading an old code base... if this is just a config change it can hopefully be merged soon. Thanks!

Looks like it was decided not to use wikidata specific prefixes for MediaInfo exports but uses a more specific sdc for these (see: T222995).
The code does still look to be hardcoded with wikidata specific prefixes.
It does not look to me like that we could make this happen quickly.
I created T253798 to track this work. Since it seems that some refactoring will have to happen (initially we thought it might just be a config change) I wonder if making the dumps available should be blocked by T253798 or go ahead and make them available with a short notice explaining that prefixes might change in the future with a link to that same ticket.
@CBogen I'll leave that decision to you.

Just a note on the current problem:
the prefixes defined in ttl dumps are identical to the ones used by wikidata e.g.:

@ArielGlenn we plan to make a subtle change to the dump (prefixes), this won't be technically a breaking change but could cause some confusion if users start to assume the presence of some prefixes. Would it be possible to pause the publication of the dumps while we change this? Sorry for the late notice.

The munger should exclude rdf:type statement by default:

SELECT ?o {
  wd:M19705716 a ?o .
}

returns :

schema:ImageObject
schema:MediaObject
wikibase:Mediainfo

In T244341#6129321, @Pfps wrote:

I thus view it misleading to state in this Phabricator ticket that "performance issues [of the WDQS] cause edits on wikidata to be throttled", which gives the impression that the WDQS forms a part of the Wikidata editing process or some other essential part of Wikidata itself.

In T244341#6124894, @Pfps wrote:

I was completely unaware that WDQS is so integrated into the inner workings of Wikidata. Where is this described? Was this mentioned in the announcement of the proposed change?

In T244341#6124638, @Pfps wrote:

If 'unskolemizing' is a trivial step then that should be implemented by WDQS, instead of pushing it to every consumer (including indirect consumers) of Wikidata information, if this change is simply a change to make WDQS work faster.

In T244341#6113236, @Lucas_Werkmeister_WMDE wrote:

Is anyone proposing a change to Wikibase (or Wikidata)?

Yes – the goal is that the RDF in the query service, the RDF dumps, and the output of Special:EntityData all change.

Closing this, @JAllemandou has done plenty of work on this already.

In T249260#6112298, @DD063520 wrote:

Is there a way to reindex?

If the index already exists perhaps forcing a reindex might help.
For this you need to run:
php updateSearchIndexConfig.php --reindexAndRemoveOk --indexIdentifier now

checkpointing works as expected now

happened a couple of times on a test run:

FailedOp(FullImport(Q93246620,2020-05-04T12:57:49Z,1173447691),org.wikidata.query.rdf.tool.exception.ContainedException: Didn't get a revision id for [])
FailedOp(FullImport(Q12439094,2020-05-04T13:23:11Z,1173459859),org.wikidata.query.rdf.tool.exception.ContainedException: Didn't get a revision id for [])
FailedOp(Diff(Q93114550,2020-05-04T14:37:37Z,1173500364,1173485266),org.wikidata.query.rdf.tool.exception.ContainedException: Didn't get a revision id for [])
FailedOp(Diff(Q93265456,2020-05-04T15:13:36Z,1173518285,1173505327),org.wikidata.query.rdf.tool.exception.ContainedException: Didn't get a revision id for [])
FailedOp(Diff(Q93265456,2020-05-04T15:16:04Z,1173520772,1173518285),org.wikidata.query.rdf.tool.exception.ContainedException: Didn't get a revision id for [])
FailedOp(Diff(Q93265456,2020-05-04T15:36:20Z,1173531303,1173520772),org.wikidata.query.rdf.tool.exception.ContainedException: Didn't get a revision id for [])
FailedOp(Diff(Q93335412,2020-05-05T09:49:16Z,1174201293,1174194685),org.wikidata.query.rdf.tool.exception.ContainedException: Didn't get a revision id for [])
FailedOp(Diff(Q93335412,2020-05-05T09:49:16Z,1174201299,1174201293),org.wikidata.query.rdf.tool.exception.ContainedException: Didn't get a revision id for [])
FailedOp(Diff(Q93248652,2020-05-05T09:48:09Z,1174200244,1173531244),org.wikidata.query.rdf.tool.exception.ContainedException: Didn't get a revision id for [])

In T244341#6097277, @Pfps wrote:

I don't understand why it was considered necessary to make a breaking change the RDF dump to improve WDQS performance when there is a solution that does not make a breaking change to the dump.

@Multichill the discussion seems to have stalled. Thanks to Peter the pros and cons have been well summarized now. I also understand that part of the misunderstanding of this change was the lack of clarity on the motivations as to why we require a breaking change like that. I hope it had been addressed in the linked discussion.
Do you have additional comments to make here? Thanks!

I think the best approach here is to wait for the cleanup in T249613 and its report then make sure that true duplicates are removed and then schedule a new full reload of all the servers.
In the meantime items can manually be fixed by doing a null edit, this is far from ideal but I don't think we have a better option at the moment.

@Peter_James thanks! The current update strategy assumes that entity <> sitelink pairs are unique and thus when a sitelink is removed it blindly assumes that it's not used elsewhere. Not doing so would require a much more costly update process that would have to verify if it's being used by other entities.
T44325 (perhaps exacerbated by T249565) is probably the root cause.

running:

curl -d 'query=SELECT * WHERE { wd:Q163320 p:P1106 / psv:P1106 ?a .  ?a wikibase:quantityUnit ?unit .  }&format=json' http://localhost/bigdata/namespace/wdq/sparql

on wdqs1010.eqiad.wmnet.

Raising to high, this issue might be hard to solve as it sounds related to the blazegraph design flaw of running with unbounded thread pools.
We might perhaps at least try to add some debugging code to isolate the request that's causing the deadlock and sees if we find a pattern.

@ArielGlenn no not yet, this is still blocked on T243292 which requires some investigation to determine which component (dump or the wdqs transformation process) is wrong.

merged in T246568 which is where we'll announce that the full reload has been done.

In T242453#6071767, @Addshore wrote:

This just happened again, so depooled and restarted 1006, and switched traffic over to codfw.
Seems to always be 1006?

In T244341#6064237, @Dipsacus_fullonum wrote:

Many queries use the optimizer hint hint:Prior hint:rangeSafe true. when e.g. comparing date or number values with constants in a filter as suggested at https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/query_optimization#Fixed_values_and_ranges. Is there a risc that such queries will fail or give wrong results when somevalue become IRI's, and thus the values will be of different types?

In T244341#6062795, @Dipsacus_fullonum wrote:

Yes, isLiteral should still work for properties where the real values are literals. Without knowing the internal workings of Blazegraph I would guess that it is more efficient than STRSTARTS( STR(?o), 'http://www.wikidata.org/prop/somevalue/' ) . Maybe that could be used in some way?

We deployed a fix to blazegraph so the wikidata/query/deploy patch should no longer be needed. Last failure occurrence is around 2020-03-21 and the fix was deployed last week on 2020-04-09, it's probably too early to claim victory.
Next step is to do a full reload of the category graph and monitor carefully that the daily dumps are properly applied.

Aliases were put in the labels field for performance reasons, we need to investigated whether it's feasible or not to add an alias field (note that due to the multilingual nature of wikidata this is something we have to ponder carefully because the current strategy is to add one field per language).