Page MenuHomePhabricator

wdqs-updater should start updating when installation is done
Open, Needs TriagePublic

Description

On initial startup wdqs-updater is currently only waiting for any response on the host/port of mediawiki.

This leads to Mediawiki/Wikibase still being in the installation phase when wdqs-updater starts polling. There is nothing dangerous in this in itself but looks bad and spams the log (which may look like an error for users)

wikibase_1            | done
wikibase_1            | Creating main page with default content
elasticsearch_1       | [2021-11-25T08:26:54,669][INFO ][o.e.g.GatewayService     ] [YVG0bXA] recovered [0] indices into cluster_state
wdqs-updater_1        | 08:26:54.675 [main] INFO  o.w.q.r.t.change.ChangeSourceContext - Defaulting start time to 90 days ago: 2021-08-27T08:26:54.675Z
wdqs-updater_1        | 08:26:54.763 [main] WARN  org.wikidata.query.rdf.tool.Updater - Retryable error fetching first batch.  Retrying.
wdqs-updater_1        | org.wikidata.query.rdf.tool.exception.RetryableException: Error fetching recent changes
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:352)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.doFetchRecentChanges(RecentChangesPoller.java:325)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.fetchRecentChanges(RecentChangesPoller.java:314)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.batch(RecentChangesPoller.java:338)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:162)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:38)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.Updater.run(Updater.java:149)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.Update.run(Update.java:175)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.Update.main(Update.java:99)
wdqs-updater_1        | Caused by: org.apache.http.NoHttpResponseException: 192.168.0.5:80 failed to respond
wdqs-updater_1        | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
wdqs-updater_1        | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
wdqs-updater_1        | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
wdqs-updater_1        | 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
wdqs-updater_1        | 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
wdqs-updater_1        | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
wdqs-updater_1        | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:84)
wdqs-updater_1        | 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
wdqs-updater_1        | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
wdqs-updater_1        | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.getJson(WikibaseRepository.java:508)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:344)
wdqs-updater_1        | 	... 8 common frames omitted
wdqs-updater_1        | 08:26:54.768 [main] WARN  org.wikidata.query.rdf.tool.Updater - Retryable error fetching first batch.  Retrying.
wdqs-updater_1        | org.wikidata.query.rdf.tool.exception.RetryableException: Error fetching recent changes
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:352)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.doFetchRecentChanges(RecentChangesPoller.java:325)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.fetchRecentChanges(RecentChangesPoller.java:314)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.batch(RecentChangesPoller.java:338)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:162)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:38)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.Updater.run(Updater.java:149)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.Update.run(Update.java:175)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.Update.main(Update.java:99)
wdqs-updater_1        | Caused by: org.apache.http.NoHttpResponseException: 192.168.0.5:80 failed to respond
wdqs-updater_1        | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
wdqs-updater_1        | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
wdqs-updater_1        | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
wdqs-updater_1        | 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
wdqs-updater_1        | 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
wdqs-updater_1        | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
wdqs-updater_1        | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:84)
wdqs-updater_1        | 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
wdqs-updater_1        | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
wdqs-updater_1        | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.getJson(WikibaseRepository.java:508)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:344)
wdqs-updater_1        | 	... 8 common frames omitted
wdqs-updater_1        | 08:26:54.772 [main] WARN  org.wikidata.query.rdf.tool.Updater - Retryable error fetching first batch.  Retrying.
wdqs-updater_1        | org.wikidata.query.rdf.tool.exception.RetryableException: Error fetching recent changes
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:352)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.doFetchRecentChanges(RecentChangesPoller.java:325)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.fetchRecentChanges(RecentChangesPoller.java:314)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.batch(RecentChangesPoller.java:338)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:162)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:38)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.Updater.run(Updater.java:149)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.Update.run(Update.java:175)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.Update.main(Update.java:99)
wdqs-updater_1        | Caused by: org.apache.http.NoHttpResponseException: 192.168.0.5:80 failed to respond
wdqs-updater_1        | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
wdqs-updater_1        | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
wdqs-updater_1        | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
wdqs-updater_1        | 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
wdqs-updater_1        | 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
wdqs-updater_1        | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
wdqs-updater_1        | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:84)
wdqs-updater_1        | 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
wdqs-updater_1        | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
wdqs-updater_1        | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.getJson(WikibaseRepository.java:508)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:344)
wdqs-updater_1        | 	... 8 common frames omitted
wdqs-updater_1        | 08:26:54.776 [main] ERROR org.wikidata.query.rdf.tool.Update - Error during updater run.
wdqs-updater_1        | java.lang.RuntimeException: java.net.SocketException: Connection reset
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:347)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.doFetchRecentChanges(RecentChangesPoller.java:325)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.fetchRecentChanges(RecentChangesPoller.java:314)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.batch(RecentChangesPoller.java:338)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:162)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:38)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.Updater.run(Updater.java:149)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.Update.run(Update.java:175)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.Update.main(Update.java:99)
wdqs-updater_1        | Caused by: java.net.SocketException: Connection reset
wdqs-updater_1        | 	at java.net.SocketInputStream.read(SocketInputStream.java:210)
wdqs-updater_1        | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
wdqs-updater_1        | 	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
wdqs-updater_1        | 	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
wdqs-updater_1        | 	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
wdqs-updater_1        | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
wdqs-updater_1        | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
wdqs-updater_1        | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
wdqs-updater_1        | 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
wdqs-updater_1        | 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
wdqs-updater_1        | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
wdqs-updater_1        | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
wdqs-updater_1        | 	at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:84)
wdqs-updater_1        | 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
wdqs-updater_1        | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
wdqs-updater_1        | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.getJson(WikibaseRepository.java:508)
wdqs-updater_1        | 	at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:344)
wdqs-updater_1        | 	... 8 common frames omitted
wikibase_1            | done

AC

  • WDQS-updater runUpdate script only starts polling when wikibase is ready

Maybe this can be resolved with some additional healthcheck / rework of the wait-for-it script?