Current hosts: wdqs1011, wdqs2006 are failing with the following
ryankemper@wdqs1011:~$ sudo systemctl status prometheus-blazegraph-exporter-wdqs-categories.service ● prometheus-blazegraph-exporter-wdqs-categories.service - Prometheus Blazegraph Exporter (wdqs-categories) Loaded: loaded (/lib/systemd/system/prometheus-blazegraph-exporter-wdqs-categories.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2020-12-10 17:10:22 UTC; 12min ago Process: 21672 ExecStart=/usr/local/bin/prometheus-blazegraph-exporter --listen :9194 --port 9990 (code=exited, status=1/FAILURE) Main PID: 21672 (code=exited, status=1/FAILURE) Dec 10 17:10:22 wdqs1011 systemd[1]: prometheus-blazegraph-exporter-wdqs-categories.service: Service RestartSec=100ms expired, scheduling restart. Dec 10 17:10:22 wdqs1011 systemd[1]: prometheus-blazegraph-exporter-wdqs-categories.service: Scheduled restart job, restart counter is at 5. Dec 10 17:10:22 wdqs1011 systemd[1]: Stopped Prometheus Blazegraph Exporter (wdqs-categories). Dec 10 17:10:22 wdqs1011 systemd[1]: prometheus-blazegraph-exporter-wdqs-categories.service: Start request repeated too quickly. Dec 10 17:10:22 wdqs1011 systemd[1]: prometheus-blazegraph-exporter-wdqs-categories.service: Failed with result 'exit-code'. Dec 10 17:10:22 wdqs1011 systemd[1]: Failed to start Prometheus Blazegraph Exporter (wdqs-categories).
This exporter relies on localhost:9194, which is refusing connections currently. It seems like the actual categories service on the "other side" is failing:
ryankemper@wdqs1011:~$ sudo systemctl status wdqs-categories.service
● wdqs-categories.service - Query Service - Blazegraph - wdqs-categories
Loaded: loaded (/lib/systemd/system/wdqs-categories.service; static; vendor preset: enabled)
Active: active (running) since Thu 2020-12-10 17:04:48 UTC; 19min ago
Main PID: 18590 (java)
Tasks: 114 (limit: 10000)
Memory: 2.3G
CGroup: /system.slice/wdqs-categories.service
└─18590 java -server -XX:+UseG1GC -Xmx8g -Xloggc:/var/log/wdqs/wdqs-categories_jvm_gc.%p-%t.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy -X
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: SELECT ( COUNT( * ) AS ?count ) { ?s ?p ?o }
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: } UNION {
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: SELECT * WHERE { <http://www.wikidata.org> schema:dateModified ?y }
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: } } req.requestURI=/bigdata/namespace/wdq/sparql, req.xForwardedFor=null, req.queryString=format=json&query=+prefix+schema%3A+%3Cht
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: com.bigdata.rdf.sail.webapp.DatasetNotFoundException: namespace=wdq
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: at com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.<init>(BigdataSail.java:2071)
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: Wrapped by: java.lang.RuntimeException: com.bigdata.rdf.sail.webapp.DatasetNotFoundException: namespace=wdq
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: at com.bigdata.rdf.sail.BigdataSail.getReadOnlyConnection(BigdataSail.java:1506)
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: Wrapped by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: com.bigdata.rdf.sail.webapp.DatasetNotFoundException: namespace=wdq
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: at java.util.concurrent.FutureTask.report(FutureTask.java:122)As a separate issue, wdqs2002 is returning 500 response code for curl localhost:9194 leading to a status of Unknown rather than Critical