Page MenuHomePhabricator

What to do if the WDQS does not synchronise with Wikibase?
Open, NormalPublic

Description

I am running a wikibase-container at http://185.54.113.154:8181/wiki/Main_Page. I have populated this wikibase instance with 176,113 items using a bot written with the Wikidata integrator.
However, after completion that WDQS remains empty.

During the bot bot import the WDQS crashed and I had to restart the docker image. Can the WDQS be synchronized afterwards?

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 31 2018, 9:38 PM

Hmm, if WDQS just stops, you should be able to restart it and it should pick up from the last point in recent changes that it took data from.
What error did the query service / updater crash out with?

What happens when you restart the container?
Remember the data is stored in the wdqs container / service and the updater is run within the updater container, so if the updater crashes the wdqs itself should still be running.
Also when restarting these things, you need to restart both containers, if you just restart wdqs no updating will happen, you also need to start the updater.

In any case, as long as you don't have more than 30 days of edits / all of your edits are accessible via recent changes you should be able to ditch both the wdqs updater and container (and data from them) and start new ones and all of the data will be picked up from recent changes.

See T182394 which allows the number of days that recent changes can be used for to be configured, as well as forcing a initial run time to be added to the query service which could also help out here or in the future (although these are not in the images yet)

I actually don't know when things went wrong. I launched all containers using docker compose. Upon initiation, seems to work. I then launched a bot to populate the wiki base instance. Which ran overnight to at around 180lk items. When I then tried to run some SPARQL queries in the WDQS the WDQS gave a nginx error, which I solved by relaunching all containers.

I have restarted all with the following commands:

sudo /usr/local/bin/docker-compose down
ubuntu@wikibase:~/wikibase-docker$ sudo /usr/local/bin/docker-compose down
Stopping wikibasedocker_wdqs-updater_1  ... done
Stopping wikibasedocker_wdqs-frontend_1 ... done
Stopping wikibasedocker_wikibase_1      ... done
Stopping wikibasedocker_wdqs-proxy_1    ... done
Stopping wikibasedocker_mysql_1         ... done
Stopping wikibasedocker_wdqs_1          ... done
Removing wikibasedocker_wdqs-frontend_1 ... done
Removing wikibasedocker_wikibase_1      ... done
Removing wikibasedocker_wdqs-proxy_1    ... done
Removing wikibasedocker_mysql_1         ... done
Removing wikibasedocker_wdqs_1          ... done

The be sure, a reboot, followed by:

ubuntu@wikibase:~/wikibase-docker$ sudo /usr/local/bin/docker-compose up --no-build -d
wikibasedocker_wdqs_1 is up-to-date
wikibasedocker_mysql_1 is up-to-date
wikibasedocker_wdqs-proxy_1 is up-to-date
wikibasedocker_wikibase_1 is up-to-date
wikibasedocker_wdqs-frontend_1 is up-to-date
wikibasedocker_wdqs-updater_1 is up-to-date

I waited an evening for the wdqs to pick up. Unfortunatly still not updates

Am I a missing a specific command?

So you can have a look at the log for everything (checkout the docker-compose logs command), a paste of those would help pin down what happened.
Although if you have already run 'down' then the containers have been removed and the logs won't exist any more.

When I then tried to run some SPARQL queries in the WDQS the WDQS gave a nginx error

What error exactly? Again you can see what was happening in cases like this by looking at the logs of the containers.

I waited an evening for the wdqs to pick up. Unfortunatly still not updates

It looks like you are running into the exact issue that T182394 was filed for (though that is a guess without seeing the logs).
Per T182394#3938346 the fix we need will be in 0.3.0 of the query service which is currently blocked on T178712

Also, how much memory does the instance have that is running all of the docker containers?

Addshore moved this task from Features to Misc on the Wikibase-Containers board.Apr 25 2018, 2:01 PM
Addshore triaged this task as Normal priority.Jun 26 2018, 3:52 PM

So, this should all be documented, but perhaps we need to more clearly link to the query service docs in the wdqs docker images readme and maybe make it explicit how to move the needed files between 2 containers.

I would like to re-open the ticket/issue.
I am running and/or have access to several wikibase instances. After loading data via a bot, no data appear to be querable at the wqds endpoint.
I tried different form of queries, even trying all possible form of prefixes modelling them on those available for Wikidata (a bit "out of desperation"). No query and for no prefix reported any result.
Even putting down all services and restarting the machine did not solve the problem on the wikibase instances I can control.

As for instances openly available, for example consider the query

https://orig-query.wmflabs.org/#PREFIX%20wb%3A%20%3Chttp%3A%2F%2Forig-query.wmflabs.org%2Fentity%2F%3E%0APREFIX%20p%3A%20%3Chttp%3A%2F%2Forig-query.wmflabs.org%2Fprop%2Fdirect%2F%3E%0A%0ADESCRIBE%20wb%3AQ2%0A

which does not return any result although the Q2 item exists in the instance.

Andrawaag raised the priority of this task from Normal to High.Jul 4 2018, 10:47 AM

@Addshore This is the issue I mentioned in Antwerp, which I could then not replicate.

Addshore lowered the priority of this task from High to Normal.Jul 10 2018, 8:36 AM
In T186161#4394184, @Considering.Different.Routes wrote:No query and for no prefix reported any result.

Note even the below?

SELECT * where { ?a ?b ?c }

I saw you linked to your query service but I seem to see tripples https://orig-query.wmflabs.org/#SELECT%20%2a%20WHERE%20%7B%20%3Fa%20%3Fb%20%3Fc%20%7D

If that returns nothing at all then the updater has not started running and you'll have to paste the logs of the updater here so we can see what happened.

As for instances openly available, for example consider the query
https://orig-query.wmflabs.org/#PREFIX%20wb%3A%20%3Chttp%3A%2F%2Forig-query.wmflabs.org%2Fentity%2F%3E%0APREFIX%20p%3A%20%3Chttp%3A%2F%2Forig-query.wmflabs.org%2Fprop%2Fdirect%2F%3E%0A%0ADESCRIBE%20wb%3AQ2%0A
which does not return any result although the Q2 item exists in the instance.

It looks like your prefixes were wrong, the below shows Q2:

https://orig-query.wmflabs.org/#PREFIX%20wb%3A%20%3Chttp%3A%2F%2Forig.wmflabs.org%2Fentity%2F%3E%0APREFIX%20p%3A%20%3Chttp%3A%2F%2Forig.wmflabs.org%2Fprop%2Fdirect%2F%3E%0A%0ADESCRIBE%20wb%3AQ2%0A

Hi, I am having the same issue. I installed the Wikibase and created new properties and items. But I am unable to query them and they do not appear in the query results. Is there any resolution to this problem?

SELECT * {?item ?prop ?val}

doesn't show the newly created properties and items.