Page MenuHomePhabricator
Create Task
Maniphest T286334

`wdqs-frontend` and `wdqs` containers are constantly restarting using the wikibase-release-pipeline docker-compose example
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

Steps to reproduce

  1. Download the example from the wikibase-release-pipeline https://github.com/wmde/wikibase-release-pipeline/tree/main/example
  2. Move template.env to .env
  3. Uncomment the lines - ./extra-install.sh:/extra-install.sh, - MW_ELASTIC_HOST=elasticsearch.svc and - MW_ELASTIC_PORT=9200 in`docker-compose.yml`
  4. Run docker-compose -f docker-compose.yml -f docker-compose.extra.yml up -d
  5. Run docker ps

Expectations

The containers wdqs-frontend and wdqs work properly.

What happens instead

The containers wdqs-frontend and wdqs are permanently restarting:

CONTAINER ID        IMAGE                                    COMMAND                   CREATED              STATUS                         PORTS                  NAMES
f32bffcc17f5        wikibase/wdqs-frontend:wmde.1            "/entrypoint.sh ngin\u2026"    About a minute ago   Restarting (1) 9 seconds ago                          example_wdqs-frontend_1
7f8bc2952f8c        wikibase/wdqs:0.3.40-wmde.1              "/entrypoint.sh /run\u2026"    About a minute ago   Up About a minute                                     example_wdqs-updater_1
eb0ba87b74b2        wikibase/quickstatements:wmde.1          "/bin/bash /entrypoi\u2026"    About a minute ago   Up About a minute              0.0.0.0:8840->80/tcp   example_quickstatements_1
424fe4e45f3a        wikibase/wdqs-proxy:wmde.1               "/bin/sh -c \"/entryp\u2026"   About a minute ago   Up About a minute              80/tcp                 example_wdqs-proxy_1
08685e100423        wikibase/wikibase-bundle:1.35.2-wmde.1   "/bin/bash /entrypoi\u2026"    About a minute ago   Up About a minute              0.0.0.0:80->80/tcp     example_wikibase_1
38b68097c947        mariadb:10.3                             "docker-entrypoint.s\u2026"    About a minute ago   Up About a minute              3306/tcp               example_mysql_1
1dc5de5c1da1        wikibase/wdqs:0.3.40-wmde.1              "/entrypoint.sh /run\u2026"    About a minute ago   Restarting (1) 8 seconds ago                          example_wdqs_1
3f64e085c74c        wikibase/elasticsearch:6.5.4-wmde.1      "/usr/local/bin/dock\u2026"    About a minute ago   Up About a minute              9200/tcp, 9300/tcp     example_elasticsearch_1

Related Objects

Event Timeline

RShigapov created this task.Jul 8 2021, 9:16 AM
Restricted Application added a project: [DEPRECATED] wdwb-tech. · View Herald TranscriptJul 8 2021, 9:16 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Maintenance_bot added a project: Wikidata.Jul 8 2021, 9:45 AM
• Samantha_Alipio_WMDE added a project: Wikidata-Campsite.Jul 14 2021, 10:14 AM
• Samantha_Alipio_WMDE moved this task from Incoming to Unconnected Stories on the Wikidata-Campsite board.
Addshore set the point value for this task to 8.Jul 14 2021, 10:47 AM
Addshore moved this task from Unconnected Stories to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.Jul 14 2021, 11:03 AM
Addshore edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Jul 14 2021, 11:15 AM
Addshore moved this task from Inbox to Active on the [DEPRECATED] wdwb-tech board.Jul 14 2021, 11:15 AM
danshick-wmde subscribed.Jul 14 2021, 12:07 PM
• toan subscribed.Jul 16 2021, 10:34 AM

Hello @RShigapov,

Do you have any logs from this that you could provide for debugging this?

Removing the -d flag from the docker-compose up command would attach to the log output immediately, otherwise running docker-compose logs -f would attach afterwards.

I noticed this behaviour once while running this locally giving this output and the wdqs-updater restarting.

However, just starting over and

docker-compose -f docker-compose.yml -f docker-compose.extra.yml down --volumes
docker-compose -f docker-compose.yml -f docker-compose.extra.yml up

It would start normally and I don't get this problem again. I think adding T283833: Add CI for testing the wikibase-release-strategy example docker-compose might help us understanding this better.

wdqs-updater_1     | 10:16:46.501 [main] ERROR org.wikidata.query.rdf.tool.Update - Error during initialization.
wdqs-updater_1     | org.wikidata.query.rdf.tool.exception.ContainedException: Non-200 response from triple store:  HttpContentResponse[HTTP/1.1 500 Server Error - 8074 bytes] body=
wdqs-updater_1     | SPARQL-QUERY: queryStr=PREFIX schema: <http://schema.org/>
wdqs-updater_1     | SELECT * WHERE { <http://wikibase.svc> schema:dateModified ?date }
wdqs-updater_1     | java.util.concurrent.ExecutionException: java.lang.RuntimeException: off=0, len=702::namespace=wdq, timestamp=readOnly(1626430606490), readTime=readOnly(1620304255632)
wdqs-updater_1     | 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
wdqs-updater_1     | 	at java.util.concurrent.FutureTask.get(FutureTask.java:206)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.BigdataServlet.submitApiTask(BigdataServlet.java:292)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.QueryServlet.doSparqlQuery(QueryServlet.java:678)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.QueryServlet.doPost(QueryServlet.java:275)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.RESTServlet.doPost(RESTServlet.java:269)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doPost(MultiTenancyServlet.java:195)
wdqs-updater_1     | 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
wdqs-updater_1     | 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
wdqs-updater_1     | 	at org.wikidata.query.rdf.blazegraph.throttling.ThrottlingFilter.doFilter(ThrottlingFilter.java:320)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
wdqs-updater_1     | 	at org.wikidata.query.rdf.blazegraph.throttling.SystemOverloadFilter.doFilter(SystemOverloadFilter.java:82)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
wdqs-updater_1     | 	at ch.qos.logback.classic.helpers.MDCInsertingServletFilter.doFilter(MDCInsertingServletFilter.java:49)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
wdqs-updater_1     | 	at org.wikidata.query.rdf.blazegraph.filters.QueryEventSenderFilter.doFilter(QueryEventSenderFilter.java:86)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
wdqs-updater_1     | 	at org.wikidata.query.rdf.blazegraph.filters.ClientIPFilter.doFilter(ClientIPFilter.java:43)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
wdqs-updater_1     | 	at org.wikidata.query.rdf.blazegraph.filters.RealAgentFilter.doFilter(RealAgentFilter.java:33)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
wdqs-updater_1     | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
wdqs-updater_1     | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
wdqs-updater_1     | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
wdqs-updater_1     | 	at org.eclipse.jetty.server.Server.handle(Server.java:503)
wdqs-updater_1     | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
wdqs-updater_1     | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
wdqs-updater_1     | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
wdqs-updater_1     | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
wdqs-updater_1     | 	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
wdqs-updater_1     | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
wdqs-updater_1     | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
wdqs-updater_1     | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
wdqs-updater_1     | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
wdqs-updater_1     | 	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
wdqs-updater_1     | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
wdqs-updater_1     | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
wdqs-updater_1     | 	at java.lang.Thread.run(Thread.java:748)
wdqs-updater_1     | Caused by: java.lang.RuntimeException: off=0, len=702::namespace=wdq, timestamp=readOnly(1626430606490), readTime=readOnly(1620304255632)
wdqs-updater_1     | 	at com.bigdata.relation.locator.DefaultResourceLocator.locateResourceOn(DefaultResourceLocator.java:817)
wdqs-updater_1     | 	at com.bigdata.relation.locator.DefaultResourceLocator.locateResource(DefaultResourceLocator.java:586)
wdqs-updater_1     | 	at com.bigdata.relation.locator.DefaultResourceLocator.cacheMiss(DefaultResourceLocator.java:395)
wdqs-updater_1     | 	at com.bigdata.relation.locator.DefaultResourceLocator.locate(DefaultResourceLocator.java:347)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.<init>(BigdataSail.java:2067)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.BigdataSail$BigdataSailReadOnlyConnection.<init>(BigdataSail.java:5235)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.BigdataSail._getReadOnlyConnection(BigdataSail.java:1539)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.BigdataSail.getReadOnlyConnection(BigdataSail.java:1502)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.BigdataSailRepository.getReadOnlyConnection(BigdataSailRepository.java:140)
wdqs-updater_1     | 	at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:247)
wdqs-updater_1     | 	at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:221)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:746)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:695)
wdqs-updater_1     | 	at com.bigdata.rdf.task.ApiTaskForIndexManager.call(ApiTaskForIndexManager.java:68)
wdqs-updater_1     | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
wdqs-updater_1     | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
wdqs-updater_1     | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
wdqs-updater_1     | 	... 1 more
wdqs-updater_1     | Caused by: java.lang.RuntimeException: off=0, len=702
wdqs-updater_1     | 	at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:239)
wdqs-updater_1     | 	at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:207)
wdqs-updater_1     | 	at com.bigdata.sparse.ValueType.decode(ValueType.java:333)
wdqs-updater_1     | 	at com.bigdata.sparse.AbstractAtomicRowReadOrWrite.atomicRead(AbstractAtomicRowReadOrWrite.java:347)
wdqs-updater_1     | 	at com.bigdata.sparse.AbstractAtomicRowReadOrWrite.atomicRead(AbstractAtomicRowReadOrWrite.java:157)
wdqs-updater_1     | 	at com.bigdata.sparse.AtomicRowRead.apply(AtomicRowRead.java:98)
wdqs-updater_1     | 	at com.bigdata.sparse.AtomicRowRead.apply(AtomicRowRead.java:36)
wdqs-updater_1     | 	at com.bigdata.btree.AbstractBTree.submit(AbstractBTree.java:3263)
wdqs-updater_1     | 	at com.bigdata.sparse.SparseRowStore.read(SparseRowStore.java:537)
wdqs-updater_1     | 	at com.bigdata.sparse.SparseRowStore.read(SparseRowStore.java:420)
wdqs-updater_1     | 	at com.bigdata.relation.locator.DefaultResourceLocator.locateResourceOn(DefaultResourceLocator.java:807)
wdqs-updater_1     | 	... 17 more
wdqs-updater_1     | Caused by: com.bigdata.rdf.vocab.BaseVocabulary$VocabularyVersioningException
wdqs-updater_1     | 	at com.bigdata.rdf.vocab.BaseVocabulary.readVersion2(BaseVocabulary.java:680)
wdqs-updater_1     | 	at com.bigdata.rdf.vocab.BaseVocabulary.readExternal(BaseVocabulary.java:458)
wdqs-updater_1     | 	at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2118)
wdqs-updater_1     | 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
wdqs-updater_1     | 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
wdqs-updater_1     | 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
wdqs-updater_1     | 	at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:231)
wdqs-updater_1     | 	... 27 more
wdqs-updater_1     | 
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.rdf.client.RdfClient.execute(RdfClient.java:226)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.rdf.client.RdfClient.query(RdfClient.java:97)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.rdf.RdfRepository.dateFromQuery(RdfRepository.java:540)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.rdf.RdfRepository.fetchLeftOffTime(RdfRepository.java:509)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.change.ChangeSourceContext.getStartTime(ChangeSourceContext.java:92)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.Update.initialize(Update.java:145)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.Update.main(Update.java:98)
wdqs-updater_1     | Exception in thread "main" org.wikidata.query.rdf.tool.exception.ContainedException: Non-200 response from triple store:  HttpContentResponse[HTTP/1.1 500 Server Error - 8074 bytes] body=
wdqs-updater_1     | SPARQL-QUERY: queryStr=PREFIX schema: <http://schema.org/>
wdqs-updater_1     | SELECT * WHERE { <http://wikibase.svc> schema:dateModified ?date }
wdqs-updater_1     | java.util.concurrent.ExecutionException: java.lang.RuntimeException: off=0, len=702::namespace=wdq, timestamp=readOnly(1626430606490), readTime=readOnly(1620304255632)
wdqs-updater_1     | 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
wdqs-updater_1     | 	at java.util.concurrent.FutureTask.get(FutureTask.java:206)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.BigdataServlet.submitApiTask(BigdataServlet.java:292)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.QueryServlet.doSparqlQuery(QueryServlet.java:678)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.QueryServlet.doPost(QueryServlet.java:275)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.RESTServlet.doPost(RESTServlet.java:269)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doPost(MultiTenancyServlet.java:195)
wdqs-updater_1     | 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
wdqs-updater_1     | 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
wdqs-updater_1     | 	at org.wikidata.query.rdf.blazegraph.throttling.ThrottlingFilter.doFilter(ThrottlingFilter.java:320)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
wdqs-updater_1     | 	at org.wikidata.query.rdf.blazegraph.throttling.SystemOverloadFilter.doFilter(SystemOverloadFilter.java:82)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
wdqs-updater_1     | 	at ch.qos.logback.classic.helpers.MDCInsertingServletFilter.doFilter(MDCInsertingServletFilter.java:49)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
wdqs-updater_1     | 	at org.wikidata.query.rdf.blazegraph.filters.QueryEventSenderFilter.doFilter(QueryEventSenderFilter.java:86)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
wdqs-updater_1     | 	at org.wikidata.query.rdf.blazegraph.filters.ClientIPFilter.doFilter(ClientIPFilter.java:43)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
wdqs-updater_1     | 	at org.wikidata.query.rdf.blazegraph.filters.RealAgentFilter.doFilter(RealAgentFilter.java:33)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
wdqs-updater_1     | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
wdqs-updater_1     | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
wdqs-updater_1     | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
wdqs-updater_1     | 	at org.eclipse.jetty.server.Server.handle(Server.java:503)
wdqs-updater_1     | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
wdqs-updater_1     | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
wdqs-updater_1     | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
wdqs-updater_1     | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
wdqs-updater_1     | 	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
wdqs-updater_1     | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
wdqs-updater_1     | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
wdqs-updater_1     | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
wdqs-updater_1     | 	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
wdqs-updater_1     | 	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
wdqs-updater_1     | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
wdqs-updater_1     | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
wdqs-updater_1     | 	at java.lang.Thread.run(Thread.java:748)
wdqs-updater_1     | Caused by: java.lang.RuntimeException: off=0, len=702::namespace=wdq, timestamp=readOnly(1626430606490), readTime=readOnly(1620304255632)
wdqs-updater_1     | 	at com.bigdata.relation.locator.DefaultResourceLocator.locateResourceOn(DefaultResourceLocator.java:817)
wdqs-updater_1     | 	at com.bigdata.relation.locator.DefaultResourceLocator.locateResource(DefaultResourceLocator.java:586)
wdqs-updater_1     | 	at com.bigdata.relation.locator.DefaultResourceLocator.cacheMiss(DefaultResourceLocator.java:395)
wdqs-updater_1     | 	at com.bigdata.relation.locator.DefaultResourceLocator.locate(DefaultResourceLocator.java:347)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.<init>(BigdataSail.java:2067)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.BigdataSail$BigdataSailReadOnlyConnection.<init>(BigdataSail.java:5235)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.BigdataSail._getReadOnlyConnection(BigdataSail.java:1539)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.BigdataSail.getReadOnlyConnection(BigdataSail.java:1502)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.BigdataSailRepository.getReadOnlyConnection(BigdataSailRepository.java:140)
wdqs-updater_1     | 	at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:247)
wdqs-updater_1     | 	at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:221)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:746)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:695)
wdqs-updater_1     | 	at com.bigdata.rdf.task.ApiTaskForIndexManager.call(ApiTaskForIndexManager.java:68)
wdqs-updater_1     | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
wdqs-updater_1     | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
wdqs-updater_1     | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
wdqs-updater_1     | 	... 1 more
wdqs-updater_1     | Caused by: java.lang.RuntimeException: off=0, len=702
wdqs-updater_1     | 	at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:239)
wdqs-updater_1     | 	at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:207)
wdqs-updater_1     | 	at com.bigdata.sparse.ValueType.decode(ValueType.java:333)
wdqs-updater_1     | 	at com.bigdata.sparse.AbstractAtomicRowReadOrWrite.atomicRead(AbstractAtomicRowReadOrWrite.java:347)
wdqs-updater_1     | 	at com.bigdata.sparse.AbstractAtomicRowReadOrWrite.atomicRead(AbstractAtomicRowReadOrWrite.java:157)
wdqs-updater_1     | 	at com.bigdata.sparse.AtomicRowRead.apply(AtomicRowRead.java:98)
wdqs-updater_1     | 	at com.bigdata.sparse.AtomicRowRead.apply(AtomicRowRead.java:36)
wdqs-updater_1     | 	at com.bigdata.btree.AbstractBTree.submit(AbstractBTree.java:3263)
wdqs-updater_1     | 	at com.bigdata.sparse.SparseRowStore.read(SparseRowStore.java:537)
wdqs-updater_1     | 	at com.bigdata.sparse.SparseRowStore.read(SparseRowStore.java:420)
wdqs-updater_1     | 	at com.bigdata.relation.locator.DefaultResourceLocator.locateResourceOn(DefaultResourceLocator.java:807)
wdqs-updater_1     | 	... 17 more
wdqs-updater_1     | Caused by: com.bigdata.rdf.vocab.BaseVocabulary$VocabularyVersioningException
wdqs-updater_1     | 	at com.bigdata.rdf.vocab.BaseVocabulary.readVersion2(BaseVocabulary.java:680)
wdqs-updater_1     | 	at com.bigdata.rdf.vocab.BaseVocabulary.readExternal(BaseVocabulary.java:458)
wdqs-updater_1     | 	at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2118)
wdqs-updater_1     | 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
wdqs-updater_1     | 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
wdqs-updater_1     | 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
wdqs-updater_1     | 	at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:231)
wdqs-updater_1     | 	... 27 more
wdqs-updater_1     | 
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.rdf.client.RdfClient.execute(RdfClient.java:226)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.rdf.client.RdfClient.query(RdfClient.java:97)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.rdf.RdfRepository.dateFromQuery(RdfRepository.java:540)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.rdf.RdfRepository.fetchLeftOffTime(RdfRepository.java:509)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.change.ChangeSourceContext.getStartTime(ChangeSourceContext.java:92)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.Update.initialize(Update.java:145)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.Update.main(Update.java:98)
example_wdqs-updater_1 exited with code 1
Phaebz mentioned this in T215623: Problem with docker-compose2 (behind proxy).Jul 22 2021, 10:30 AM
RShigapov added a comment.Jul 23 2021, 10:02 AM

Hi @toan,

I did as you proposed: down -volumes and then up again. No changes. Still restarting.

At the same time my logs are quite clean, it's strange. Even no message from wdqs-updater:

wikibase_1  | 	Fetching Elasticsearch version...6.5.4...ok
wikibase_1  | 	Scanning available plugins...
wikibase_1  | 		experimental-highlighter, extra, ingest-geoip, ingest-user-agent
wikibase_1  | 	Picking analyzer...english
wikibase_1  | 	Inferring index identifier...my_wiki_archive_first
wikibase_1  | 	Creating index...ok
wikibase_1  | 		Validating number of shards...ok
wikibase_1  | 		Validating replica range...ok
wikibase_1  | 		Validating shard allocation settings...done
wikibase_1  | 		Validating max shards per node...ok
wikibase_1  | 	Validating analyzers...ok
wikibase_1  | 	Validating mappings...
wikibase_1  | 		Validating mapping...different...corrected
wikibase_1  | 	Validating aliases...
wikibase_1  | 		Validating my_wiki_archive alias...alias is free...corrected
wikibase_1  | 		Updating tracking indexes...done
wikibase_1  | + php /var/www/html/extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipParse
wikibase_1  | [             my_wiki] Indexed 1 pages ending at 1 at 12/second
wikibase_1  | Indexed a total of 1 pages at 12/second
wikibase_1  | + php /var/www/html/extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipLinks --indexOnSkip
wikibase_1  | [             my_wiki] Indexed 1 pages ending at 1 at 21/second
wikibase_1  | Indexed a total of 1 pages at 21/second
wikibase_1  | ++ bash /extra-install/QuickStatements.sh
wikibase_1  | + php /var/www/html/extensions/OAuth/maintenance/createOAuthConsumer.php --approve --callbackUrl 'http://quickstatements.svc # quickstatements.svc is the internal docker hostname, change this value to the public hostname:8840/api.php' --callbackIsPrefix true --user admin --name QuickStatements --description QuickStatements --version 1.0.1 --grants createeditmovepage --grants editpage --grants highvolume --jsonOnSuccess
wikibase_1  | + [[ -f /quickstatements/data/qs-oauth.json ]]
wikibase_1  | ++ grep -o '[^"]*$'
wikibase_1  | ++ grep -o '"key":"[^"]*' /quickstatements/data/qs-oauth.json
wikibase_1  | + OAUTH_CONSUMER_KEY=bla-bla-bla
wikibase_1  | ++ grep -o '"secret":"[^"]*' /quickstatements/data/qs-oauth.json
wikibase_1  | ++ grep -o '[^"]*$'
wikibase_1  | + OAUTH_CONSUMER_SECRET=bla-bla-bla
wikibase_1  | + export OAUTH_CONSUMER_KEY
wikibase_1  | + export OAUTH_CONSUMER_SECRET
wikibase_1  | + envsubst
wikibase_1  | + docker-php-entrypoint apache2-foreground
wikibase_1  | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using bla-bla-bla. Set the 'ServerName' directive globally to suppress this message
wikibase_1  | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using bla-bla-bla. Set the 'ServerName' directive globally to suppress this message
wikibase_1  | [Fri Jul 23 09:49:42.937942 2021] [mpm_prefork:notice] [pid 89] AH00163: Apache/2.4.38 (Debian) PHP/7.3.28 configured -- resuming normal operations
wikibase_1  | [Fri Jul 23 09:49:42.937984 2021] [core:notice] [pid 89] AH00094: Command line: 'apache2 -D FOREGROUND'
Addshore subscribed.Jul 26 2021, 1:20 PM
wdqs-updater_1     | org.wikidata.query.rdf.tool.exception.ContainedException: Non-200 response from triple store:  HttpContentResponse[HTTP/1.1 500 Server Error - 8074 bytes] body=

The key thing here is to try and see what the non 200 response from the triple store.

Do you see anything in the wdqs logs?

docker-compose -f docker-compose.yml -f docker-compose.extra.yml logs wdqs

The 500 error is probably coming from the fact that the wdqs service has fallen over, the question is why has it fallen over?

Addshore renamed this task from `wdqs-frontend` and `wdqs` containers are constantly restarting in wikibase-release-pipeline to `wdqs-frontend` and `wdqs` containers are constantly restarting using the wikibase-release-pipeline docker-compose example.Jul 26 2021, 1:20 PM
RShigapov added a comment.Jul 26 2021, 1:34 PM

wdqs logs return

wdqs_1             | /wdqs /wdqs
wdqs_1             | Running Blazegraph from /wdqs on :9999/bigdata
wdqs_1             | Error: Could not find or load main class #
Addshore mentioned this in T283810: WDQS from docker compose gives an nginx error [8 hours].Jul 27 2021, 6:50 AM
Addshore added a comment.Jul 27 2021, 7:08 AM

If everything were going okay you'd see this as your log output:

wdqs_1             | /wdqs /wdqs
wdqs_1             | Running Blazegraph from /wdqs on :9999/bigdata
wdqs_1             | 2021-07-22 13:58:42.181:INFO::main: Logging initialized @630ms to org.eclipse.jetty.util.log.StdErrLog
wdqs_1             | WARNING: jetty-runner is deprecated.
wdqs_1             |          See Jetty Documentation for startup options
wdqs_1             |          https://www.eclipse.org/jetty/documentation/
wdqs_1             | 2021-07-22 13:58:42.201:INFO:oejr.Runner:main: Runner
wdqs_1             | 2021-07-22 13:58:42.748:INFO:oejs.Server:main: jetty-9.4.12.v20180830; built: 2018-08-30T13:59:14.071Z; git: 27208684755d94a92186989f695db2d7b21ebc51; jvm 1.8.0_212-b04

The command that is trying to run and failing is https://github.com/wikimedia/wikidata-query-rdf/blob/query-service-parent-0.3.40/dist/src/script/runBlazegraph.sh#L103-L131

I can't help but notice that the # in your error message is the main class that java is trying to load.
This looks wrong, and the only place that I can imagine a # is unexpectedly appearing from in the command that gets run are the comments at the end of the lines in the template env file.

https://github.com/wmde/wikibase-release-pipeline/blob/main/example/template.env#L24
https://github.com/wmde/wikibase-release-pipeline/blob/main/example/template.env#L31

Could you try removing these and trying again?

Addshore claimed this task.Jul 27 2021, 7:09 AM
Addshore moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Restricted Application added a project: User-Addshore. · View Herald TranscriptJul 27 2021, 7:09 AM
RShigapov added a comment.Jul 27 2021, 12:05 PM

You are right! Those two inline comments in env file are the source of the problem! I removed those two comments and the query service works now as expected.

Addshore merged a task: T283810: WDQS from docker compose gives an nginx error [8 hours].Jul 27 2021, 6:50 PM
Addshore added a subscriber: jjkoehorst.
In T286334#7239230, @RShigapov wrote:

You are right! Those two inline comments in env file are the source of the problem! I removed those two comments and the query service works now as expected.

Great, I'll make a change to the example .env file moving those comments to a different line or getting rid of them entirely.

Addshore added a project: Patch-For-Review.Jul 27 2021, 6:53 PM
Addshore moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

https://github.com/wmde/wikibase-release-pipeline/pull/196

RShigapov added a comment.Jul 28 2021, 7:21 AM

Thank you!

Lucas_Werkmeister_WMDE moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jul 28 2021, 2:42 PM
Addshore closed this task as Resolved.Aug 2 2021, 10:38 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL