Page MenuHomePhabricator

WDQS_Updater issue when Docker containers restart in Example (and likely otherwise)
Open, In Progress, Needs TriagePublic

Description

Currently: When the Example Docker containers are stopped and restarted without adding or resetting its data volumes, the following error appears and the container stops running:

12:55:33.443 [main] INFO  org.wikidata.query.rdf.tool.Update - Starting Updater 0.3.135 (993993ebef2c4c06d98aaa24767edef029bee287)
12:55:33.803 [main] INFO  o.w.q.r.t.change.ChangeSourceContext - Checking where we left off
12:55:33.803 [main] INFO  o.w.query.rdf.tool.rdf.RdfRepository - Checking for left off time from the updater
12:55:33.897 [main] INFO  o.w.query.rdf.tool.rdf.RdfRepository - Found left off time from the updater
12:55:33.897 [main] ERROR org.wikidata.query.rdf.tool.Update - Error during initialization.
java.lang.IllegalStateException: RDF store reports the last update time is before the minimum safe poll time.  You will have to reload from scratch or you might have missing data.
	at org.wikidata.query.rdf.tool.change.ChangeSourceContext.getStartTime(ChangeSourceContext.java:100)
	at org.wikidata.query.rdf.tool.Update.initialize(Update.java:144)
	at org.wikidata.query.rdf.tool.Update.main(Update.java:97)`

Goal: Find the root cause using the Example configuration, determine whether it is a general issue, and fix it. There have been reports of this same issue in the past in this ticket: https://phabricator.wikimedia.org/T182394. It was evidently resolved at that time but this looks to be possibly a regression.

Acceptance Criteria:

  • WDQS service remains stable and data accessible / uncorrupted upon container restarts in Example
  • WDQS service remains stable and data accessible / uncorrupted upon container restarts in general use

To Reproduce Issue:

Start and stop Example and start again without any data.

You should see this error appear, and the service will stop.

Event Timeline

lojo_wmde renamed this task from WDQS issue when Docker containers restart in Example (and likely otherwise) to WDQS_Updater issue when Docker containers restart in Example (and likely otherwise).Jan 3 2024, 1:00 PM
lojo_wmde updated the task description. (Show Details)
lojo_wmde updated the task description. (Show Details)

This part of our old documentation might be helpful as well.

https://github.com/wmde/wikibase-release-pipeline/blob/135f5d5d684d3a72c9f3b58e9d72c38a7469c086/docs/topics/test-system.md#updating
If the query service updater is restarting, it is likely due to updates not having happened in the past month.

Hi @roti_WMDE,

We are trying to set up local wiki instance for our institute but facing the updater restart issue.

Followed the instructions on https://github.com/wmde/wikibase-release-pipeline/blob/135f5d5d684d3a72c9f3b58e9d72c38a7469c086/docs/topics/test-system.md#updating to fix the restart issue: stopping updater, running with the current date, exit when "Sleeping for 10 secs" and restart updater
However, updater keeps restarting every minute. Can you please help? We installed https://www.mediawiki.org/wiki/Wikibase/Docker wmde 20. Thanks!

Dear @Padmalc,

Let me point you to the discussion in T352975, if you have not seen it already. I think the suggestions in there might be helpful for your case as well.

Especially:

[...] it could be interesting to experiment with the --start parameter for wdqs' runUpdate.sh script, if you have not tried that yet.
From the directory of your docker compose files (what is the example directory in the wikibase release pipeline):

# stop the currently running wdqs-updater service
docker compose -f docker-compose.yml -f docker-compose.extra.yml --env-file YOUR_ENV_FILE.env stop wdqs-updater

# start a manual update taking in all the changes since Jan 1, 2000
docker compose -f docker-compose.yml -f docker-compose.extra.yml --env-file YOUR_ENV_FILE.env run --rm wdqs-updater bash -c 
  '/wdqs/runUpdate.sh -h http://"$WDQS_HOST":"$WDQS_PORT" -- --wikibaseUrl "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST" --conceptUri "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST" --entityNamespaces "120,122" --init --start 20000101000000'

# start the wdqs-updater service again
docker compose -f docker-compose.yml -f docker-compose.extra.yml --env-file YOUR_ENV_FILE.env start wdqs-updater

[...]

Please let me know if that helped in your situation. If not, I would be very interested in some of the logs you get, especially from WDQS and MediaWiki.

Best,
Robert

roti_WMDE changed the task status from Open to In Progress.Thu, May 30, 2:05 PM
roti_WMDE claimed this task.