@Multichill eventually yes, but since they are not being used anywhere yet it's too early to document them. Once RDF export is properly set up to use these prefixes then we can document them officially.
Fri, Aug 23
All should be updated now.
Thu, Aug 22
Immediate RDF breakage fixed, now I'll have to update lexemes that were affected.
Ok I am getting multiple builds taking 50+ minutes again for Wikibase, e.g.:
RDF generated is:
wdq6: 17:16:23.890 [update 0] WARN org.wikidata.query.rdf.tool.Updater - Contained error syncing. Giving up on L60296 org.wikidata.query.rdf.tool.exception.ContainedException: RDF parsing error for https://www.wikidata.org/wiki/Special:EntityData/L60296.ttl?flavor=dump&nocache=1566494183408 at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.collectStatementsFromUrl(WikibaseRepository.java:401) at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRdfForEntity(WikibaseRepository.java:457) at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRdfForEntity(WikibaseRepository.java:433) at org.wikidata.query.rdf.tool.Updater.handleChange(Updater.java:362) at org.wikidata.query.rdf.tool.Updater.lambda$handleChanges$0(Updater.java:236) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.openrdf.rio.RDFParseException: Default namespace used but not defined [line 42] at org.openrdf.rio.helpers.RDFParserHelper.reportFatalError(RDFParserHelper.java:440) at org.openrdf.rio.helpers.RDFParserBase.reportFatalError(RDFParserBase.java:685) at org.openrdf.rio.turtle.TurtleParser.reportFatalError(TurtleParser.java:1405) at org.openrdf.rio.helpers.RDFParserBase.getNamespace(RDFParserBase.java:342) at org.openrdf.rio.turtle.TurtleParser.parseQNameOrBoolean(TurtleParser.java:1032) at org.openrdf.rio.turtle.TurtleParser.parseValue(TurtleParser.java:643) at org.openrdf.rio.turtle.TurtleParser.parseSubject(TurtleParser.java:474) at org.openrdf.rio.turtle.TurtleParser.parseTriples(TurtleParser.java:407) at org.openrdf.rio.turtle.TurtleParser.parseStatement(TurtleParser.java:259) at org.openrdf.rio.turtle.TurtleParser.parse(TurtleParser.java:214) at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.collectStatementsFromUrl(WikibaseRepository.java:392) ... 8 common frames omitted
So that's why it's not updated. I'll check why this happens.
Wed, Aug 21
Tags should work, at least for now, I think, if I can filter by tag efficiently. There's not a lot of data edits so far, compared to overall Commons edit volume.
Probably not a lot. Search for English labels returns 188 results, unfortunately search for statements and every label doesn't seem to work (probably needs a reindex?) so I don't know how many but probably also not a lot. I'll check tomorrow if I can get more specific figures.
Tue, Aug 20
@Addshore I was planning to do 0.3.2 pretty soon, if that is easier for you you could use that.
Mon, Aug 19
Though maybe creating empty wikidata.jnl under blazegraph user is ok... Not sure.
I think this is done since the export now exists and is enabled, we can tweak the specifics in follow-up tasks.
https://federated-commons.wmflabs.org/wiki/Special:EntityData/M10694.ttl?flavor=dump looks good to me now. Anything else missing?
Hmm looks like I updated all Qs but forgot about some Ps... These should be fine now.
Should be back to normal now.
Thu, Aug 15
Furthermore, https://query.wikidata.org/css/embed.style.min.fa3ff6a142279256ede4.css gives 404
Looks like Belgium is back to Q31 and cats are cats again. I will investigate what happened to the URIs (most likely the order in the dictionary switched somehow because of changing of underlying storage class but I have missed it since the content of the dictionary is still the same).
Probably my fault, I've deployed new WDQS with some URL refactoring, but looks like something went wrong with URI scheme. I will be rolling it back.
This should not be hard to do.
http://dcatap.wmflabs.org/ is not up and can be queried.
@WMDE-leszek you patch is still WIP, are you still working on it?
Wed, Aug 14
Anther point: while WDQS does fetch data from both clusters (at least is supposed to), it only tracks its timestamp by one topic: reportingTopic, which is currently eqiad.mediawiki.revision-create. Otherwise we get weird jumps in the timestamps, since different topics can get different events in different sequence. So if eqiad does not get any events, the updater seems to lag even though it is processing events from codfw. I am not sure whether "nothing useful" message is related anyhow.
Tue, Aug 13
We'll also probably need to update items edited in that timeframe manually, just in case. I'll do that a bit later (and also will add docs for doing this).
This is pretty weird, the updater should be able to consume from both eqiad and codfw, maybe something between the brokers did not work, or we're not connecting to the right endpoint? The messages and the situation definitely looks like it stopped getting events - in general Did not find anything useful in this batch, returning existing data is normal if it happens occasionally (it means no new events) but if it happens all the time that means there's trouble since we're not getting events. So we need to check maybe we're missing something in our kafka setup.
Mon, Aug 12
I've just remembered we'd need VPS for testing SPARQL service for T141602, so we might allocate some quote for those as well.
I've added fix for one of the issues in T222497 already but it doesn't fix everything. I think it's still would be interesting to test what happens in production - maybe not full dump but just partial, to estimate what we're dealing with and how bad is it? Maybe due to the fact we don't have yet too many mediainfo records and they're small we could be still fine?
Sat, Aug 10
Fri, Aug 9
This may be moved to a separate endpoint, probably in Toolforge.
Setting it to High priority since it will start generating problems in production as soon as the train gets there, I imagine.