Page MenuHomePhabricator

updater failed during ~40 minutes on wdqs1002
Closed, DeclinedPublic

Description

wdqs-updater stopped processing updates on wdqs1002 on August 25, from ~10:20 to 10:40 (UTC). Looking at the logs, there is no obvious errors except the RecentChangePoller "getting too close to the current stream" and a failed HTTP request just before the updater starts processing updates again. See P5920 for an extract of the logs.

Related Objects

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Similar issue happened again, between 5:50am and 8:50am UTC today (Sept 1). Again, updater recover after an HTTP exception. See logstash for details.

This seems to be the trigger:

Aug 25 10:35:25 wdqs1002 bash[18627]: 10:35:25.238 [main] INFO  o.w.query.rdf.tool.rdf.RdfRepository - HTTP request failed: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Buffering capacity exceeded, attempt 1, will retry

but I have no idea what that message means... This SO article provides some clues, but I'm not sure why RdfRepository produces overly large answer (over 2M) - generally we don't need something from it that is so large.

I can't find any other instance of such exception in the logs, so I'm closing this for now, since I can't really find the cause without recent occurrence in the logs.