User Details
User Details
- User Since
- Oct 23 2014, 3:02 PM (251 w, 2 d)
- Availability
- Available
- LDAP User
- Magnus Manske
- MediaWiki User
- Unknown
Wed, Jul 24
Wed, Jul 24
Magnus added a comment to T205549: book2scroll is not working (due to jquery 404?).
Because toolforge forgot the replica.conf again, see T166949. Webservice restarted, manually, yet again, works. For the next few minutes, probably.
Jul 2 2019
Jul 2 2019
Magnus added a comment to T223548: Item namespace issues in Factgrid.
Everyone, I own Reasonator, including the experimental version 2 which is used here (and should be better suited than the dated V1).
Jun 27 2019
Jun 27 2019
Jun 26 2019
Jun 26 2019
Magnus added a comment to T149410: For consistency MediaInfo serialization should use "claims" as key, rather than "statements".
Jun 25 2019
Jun 25 2019
Magnus updated subscribers of T149410: For consistency MediaInfo serialization should use "claims" as key, rather than "statements".
@Jdforrester-WMF Is that an official design decision (claims=>statements)? Where was this fundamentally breaking change announced to the public?
Jun 22 2019
Jun 22 2019
Magnus added a comment to T149410: For consistency MediaInfo serialization should use "claims" as key, rather than "statements".
FWIW, I have already changed my code to work with either claims or statements. Quick thoughts:
Jun 21 2019
Jun 21 2019
Magnus added a comment to T226084: Wikidata server slow.
On another note, the Reasonator example in my original post seems to load now. I'll check if the Rust code works as well now.
Magnus added a comment to T226084: Wikidata server slow.
Jun 20 2019
Jun 20 2019
Magnus added a comment to T226084: Wikidata server slow.
May I humbly suggest to have a look at the consistent 2min response time of the p99 server (in grafana), before deciding it's a problem outside WMFs control, no matter how convenient that may seem?
Jun 19 2019
Jun 19 2019
Magnus added a comment to T226084: Wikidata server slow.
No, sorry, issue remains.
Magnus added a comment to T226084: Wikidata server slow.
GET /w/api.php?callback=jQuery21303406678877236998_1560936691744&action=wbgetentities&ids=P2508%7CP2631%7CP2509%7CP4276%7CP272%7CP4529%7CP5032%7CP4947%7CP5786%7CP6145%7CP1609%7CP1230%7CP2896%7CP4730%7CP2093%7CP1844%7CP1813%7CP5396%7CQ1199348%7CP435%7CP3959%7CP747%7CP1274%7CP1085%7CP5331%7CP4839%7CP4969%7CP103%7CQ49088%7CP1648%7CQ19045189%7CP3793%7CP2847%7CP3035%7CP4389%7CP5062%7CP5508%7CP4264%7CP6698%7CP6617%7CP2241%7CQ44374960%7CQ4644021%7CQ839097%7CP1268%7CQ9624%7CQ8055775%7CQ210152%7CQ4642661%7CQ635616&props=info%7Caliases%7Clabels%7Cdescriptions%7Cclaims%7Csitelinks%7Cdatatype&format=json&_=1560936691745 HTTP/1.1 Host: www.wikidata.org User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0 Accept: */* Accept-Language: en-GB,en;q=0.7,de;q=0.3 Accept-Encoding: gzip, deflate, br Referer: https://tools.wmflabs.org/reasonator/?q=Q350 DNT: 1 Connection: keep-alive
Cookies redacted
Magnus added a comment to T226084: Wikidata server slow.
Timing details of that slow request:
Magnus added a comment to T226084: Wikidata server slow.
Response header from one of the slow requests:
HTTP/2.0 200 OK date: Wed, 19 Jun 2019 09:31:43 GMT content-type: text/javascript; charset=utf-8 server: mw1341.eqiad.wmnet x-powered-by: HHVM/3.18.6-dev mediawiki-login-suppressed: true cache-control: private, must-revalidate, max-age=0 content-disposition: inline; filename=api-result.js x-content-type-options: nosniff x-frame-options: DENY backend-timing: D=1129973 t=1560936702635082 vary: Accept-Encoding,Treat-as-Untrusted,X-Forwarded-Proto,Cookie,Authorization,X-Seven content-encoding: gzip x-varnish: 774984440, 505592114, 724292664 via: 1.1 varnish (Varnish/5.1), 1.1 varnish (Varnish/5.1), 1.1 varnish (Varnish/5.1) accept-ranges: bytes age: 0 x-cache: cp1081 pass, cp3032 pass, cp3041 pass x-cache-status: pass server-timing: cache;desc="pass" strict-transport-security: max-age=106384710; includeSubDomains; preload x-analytics: ns=-1;special=Badtitle;loggedIn=1;WMF-Last-Access=19-Jun-2019;WMF-Last-Access-Global=19-Jun-2019;https=1 x-client-ip: 2001:630:206:6204:cc46:3ce1:27e1:3062 X-Firefox-Spdy: h2
Magnus added a comment to T226084: Wikidata server slow.
Jun 6 2019
Jun 6 2019
May 31 2019
May 31 2019
remove site notice
separate DB fixes
changing to new auth DB
JSON oauth
May 23 2019
May 23 2019
Bstorm awarded T224155: Reduce size of referee db on toolsdb if at all possible a Party Time token.
Done.
May 14 2019
May 14 2019
Never mind, it's the multilingual string!
Mar 29 2019
Mar 29 2019
maxlag for bot mode
Mar 8 2019
Mar 8 2019
Magnus added a comment to T217894: Toolforge Stretch bastion occasionally closes SSH connection just before authentication succeeds.
Happened to me as well, yesterday (2019-03-08, 08:23UTC)
Mar 7 2019
Mar 7 2019
Magnus added a comment to T217141: ListeriaBot appear in recent changes although human (not bot) filter is active in arwiki.
Fixed Listeria.
Don't know anything about ASammourBot.
Mar 5 2019
Mar 5 2019
fixing slow start of new batches
Edit Groups UUID fix
formatting Edit Groups link
formatting Edit Groups link
Feb 28 2019
Feb 28 2019
Magnus added a comment to T215380: Content too big! Entity: Q27972199.
Removal is running.
Magnus added a comment to T215380: Content too big! Entity: Q27972199.
Update: Will remove them with QuickStatements now
Magnus added a comment to T215380: Content too big! Entity: Q27972199.
So here is what happens: I create(d) lots of gene/protein items (example) for various species. For many statements, I can create references, as I get them from the upstream source. That paper is one of the often-cited ones, about a determination method.
Feb 27 2019
Feb 27 2019
Feb 19 2019
Feb 19 2019
Magnus changed the status of T216153: ListeriaBot appears as a non bot despite of having a bot flag in euwiki recent changes from Invalid to Resolved.
I have added euwiki to the list of wikis where the bot flag is to be used.
Feb 12 2019
Feb 12 2019
That did the trick, thanks!
Feb 11 2019
Feb 11 2019
Sjoerddebruin awarded T215620: Tool "autodesc" - can't start webservice a The World Burns token.
Magnus added a comment to T215620: Tool "autodesc" - can't start webservice.
Tried that, also on login.tools.wmflabs.org (just to be sure). Both say "webservice is not running". Still won't start kubernetes.
Magnus added a comment to T215620: Tool "autodesc" - can't start webservice.
Thanks, I have rebuild and updated via npm on the kubernetes shell.
Feb 9 2019
Feb 9 2019
Magnus added a comment to T215620: Tool "autodesc" - can't start webservice.
I have run npm update in the kubernetes shell, but no joy.
Feb 8 2019
Feb 8 2019
Feb 1 2019
Feb 1 2019
Magnus added a comment to T215035: 502 Bad Gateway issue on Petscan.
Try it now...
Magnus added a comment to T215035: 502 Bad Gateway issue on Petscan.
Testing on dev now. Looks like frequent Lost connection to MySQL server during query errors from the DB replicas.
Magnus added a comment to T215035: 502 Bad Gateway issue on Petscan.
Also, I just clicked on the two examples. They took 133 and 173 seconds, and returned 50 and 1 results, respectively. No 502s, though I have seen those occasionally.
Magnus added a comment to T215035: 502 Bad Gateway issue on Petscan.
OK, so what appears to happen is that SQL queries timeout and take PetScan with them. Note:
- I wrote some code that re-arranges certain large queries into smaller ones, which cuts down on the timeouts; that code has been live for weeks
- That works fine on the dev machine but not reliably on the production machine
- The dev machine has less resources than production, but is otherwise identical (OS etc)
As this does not fail reproducibly, it's either some odd bug in my code, or some situation on the DB replicas.
Jan 21 2019
Jan 21 2019
Tagishsimon awarded T214278: Quickstatements, "backend is overloaded" a Heartbreak token.
Magnus added a comment to T214278: Quickstatements, "backend is overloaded".
Jan 17 2019
Jan 17 2019
Magnus added a comment to T193560: Tool keeps falling into permanent 500 error.
Looks like it's back to normal at the moment.
Jan 14 2019
Jan 14 2019
Magnus committed R2010:756382987e1a: fixes for Commons, also requires fixed wikidata.php! (authored by Magnus).
fixes for Commons, also requires fixed wikidata.php!
Magnus committed R2010:aa343fe8abeb: trying to add Commons access by making things more generic (authored by Magnus).
trying to add Commons access by making things more generic
Dec 4 2018
Dec 4 2018
quite option
one batch per job
Nov 30 2018
Nov 30 2018
Magnus awarded T210836: +2 for Lucas_Werkmeister_WMDE in mediawiki/* a Like token.
Nov 9 2018
Nov 9 2018
Magnus added a comment to T208940: Grid slow on Toolforge.
I have repeatedly asked if there is a "max concurrency" setting for jsub, as there is for other grid engines. I would consider it rather silly to force every user with such jobs to implement that on their own. For example, I could record it in the database when a job starts, but how do I know it is still running, and hasn't failed? Guess based on last action time? It makes vastly more sense to do that kind of thing in the job scheduler.
Nov 8 2018
Nov 8 2018
Magnus added a comment to T208940: Grid slow on Toolforge.
Checked out sourcemd. That is actually correct; one job for each of the "TODO" ones here:
https://tools.wmflabs.org/sourcemd/?action=batches
Magnus added a comment to T208940: Grid slow on Toolforge.
Looks like the grid caught up with them. One job type seems to be problematic, not sure if it's an aftereffect.
Magnus added a comment to T208940: Grid slow on Toolforge.
Using mix-n-match as an example, two of the cronjob commands:
0,5,10,15,20,25,30,35,40,45,50,55 * * * * cd /data/project/mix-n-match ; /usr/bin/jsub -quiet -mem 8g -N as_import -cwd ./autoscrape_import_bot.php 4,14,24,34,44,54 * * * * cd /data/project/mix-n-match ; /usr/bin/jsub -quiet -mem 6g -cwd -N mnm-microsync ./microsync.php random
Yes, they run often, but usually not for long, and have never had such an issue in the years before.
Magnus added a comment to T208940: Grid slow on Toolforge.
Up to 95 "qw" now...
Nov 7 2018
Nov 7 2018
Magnus added a comment to T208940: Grid slow on Toolforge.
(I picked mix-n-match as an example. Other tools are affected as well)
Magnus created T208940: Grid slow on Toolforge.
Nov 2 2018
Nov 2 2018
Magnus added a comment to T199228: Define an SLO for Wikidata Query Service public endpoint and communicate it.
In many cases, especially bot/background tasks (e.g. Listeria), a lag of hours is not critical. This is also true for many interactive tools, where the user gets some items matching certain criteria.
Nov 1 2018
Nov 1 2018
Magnus closed T208460: wikidata-todo is using 959GB in shared NFS area, a subtask of T206239: 2018-10-04: tools and NFS share cleanup (high usage), as Resolved.
Magnus added a comment to T208460: wikidata-todo is using 959GB in shared NFS area.
I have removed unused files /shared/dumps and altered the update code to only keep recent ones around.
Oct 25 2018
Oct 25 2018
Magnus committed R2010:d4bbd816e688: This patch set, along with an accompanying series for magnustools, adds… (authored by Magnus).
This patch set, along with an accompanying series for magnustools, adds…
Oct 23 2018
Oct 23 2018
Magnus added a comment to T207579: imagecopy_tests failing with IndexError.
Try it now please.
Oct 22 2018
Oct 22 2018
Magnus added a comment to T207579: imagecopy_tests failing with IndexError.
This should be fixed now in CommonsHelper.
Oct 20 2018
Oct 20 2018
Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.
Thanks @Jkbr, that seems to do the trick!
Oct 19 2018
Oct 19 2018
Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.
Can I give you (or can you just get) access to my VM? mixnmatch in the mix-n-match project.
Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.
As a sidenote, this may also contain the solution to T207132 ...
Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.
docker-compose.yml:
# Wikibase with Query Service # # This docker-compose example can be used to pull the images from docker hub. # # Examples: # # Access Wikibase via "http://localhost:8181" # (or "http://$(docker-machine ip):8181" if using docker-machine) # # Access Query Service via "http://localhost:8282" # (or "http://$(docker-machine ip):8282" if using docker-machine) version: '3'
Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.
wdqs log (last "entry"):
wdqs_1 | 08:20:16.620 [qtp1747585824-18] ERROR c.b.r.sail.webapp.BigdataRDFServlet IP:wikibase-docker_wdqs-proxy_1.wikibase-docker_default UA:Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:63.0) Gecko/20100101 Firefox/63.0 - cause=java.util.concurrent.ExecutionException: java.lang.RuntimeException: off=0, len=558::namespace=wdq, timestamp=readOnly(1539937216616), readTime=readOnly(1539701290285), query=SPARQL-QUERY: queryStr=prefix schema: <http://schema.org/> SELECT * WHERE {<http://www.wikidata.org> schema:dateModified ?y}
wdqs_1 | java.util.concurrent.ExecutionException: java.lang.RuntimeException: off=0, len=558::namespace=wdq, timestamp=readOnly(1539937216616), readTime=readOnly(1539701290285)
wdqs_1 | at java.util.concurrent.FutureTask.report(FutureTask.java:122)
wdqs_1 | at java.util.concurrent.FutureTask.get(FutureTask.java:206)
wdqs_1 | at com.bigdata.rdf.sail.webapp.BigdataServlet.submitApiTask(BigdataServlet.java:293)
wdqs_1 | at com.bigdata.rdf.sail.webapp.QueryServlet.doSparqlQuery(QueryServlet.java:654)
wdqs_1 | at com.bigdata.rdf.sail.webapp.QueryServlet.doGet(QueryServlet.java:288)
wdqs_1 | at com.bigdata.rdf.sail.webapp.RESTServlet.doGet(RESTServlet.java:240)
wdqs_1 | at com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doGet(MultiTenancyServlet.java:271)
wdqs_1 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
wdqs_1 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
wdqs_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
wdqs_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
wdqs_1 | at org.wikidata.query.rdf.blazegraph.throttling.ThrottlingFilter.doFilter(ThrottlingFilter.java:304)
wdqs_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
wdqs_1 | at ch.qos.logback.classic.helpers.MDCInsertingServletFilter.doFilter(MDCInsertingServletFilter.java:49)
wdqs_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
wdqs_1 | at org.wikidata.query.rdf.blazegraph.filters.ClientIPFilter.doFilter(ClientIPFilter.java:43)
wdqs_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
wdqs_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
wdqs_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
wdqs_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
wdqs_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
wdqs_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
wdqs_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
wdqs_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
wdqs_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
wdqs_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
wdqs_1 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
wdqs_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
wdqs_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
wdqs_1 | at org.eclipse.jetty.server.Server.handle(Server.java:497)
wdqs_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
wdqs_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
wdqs_1 | at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
wdqs_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
wdqs_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
wdqs_1 | at java.lang.Thread.run(Thread.java:748)
wdqs_1 | Caused by: java.lang.RuntimeException: off=0, len=558::namespace=wdq, timestamp=readOnly(1539937216616), readTime=readOnly(1539701290285)
wdqs_1 | at com.bigdata.relation.locator.DefaultResourceLocator.locateResourceOn(DefaultResourceLocator.java:817)
wdqs_1 | at com.bigdata.relation.locator.DefaultResourceLocator.locateResource(DefaultResourceLocator.java:586)
wdqs_1 | at com.bigdata.relation.locator.DefaultResourceLocator.cacheMiss(DefaultResourceLocator.java:395)
wdqs_1 | at com.bigdata.relation.locator.DefaultResourceLocator.locate(DefaultResourceLocator.java:347)
wdqs_1 | at com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.<init>(BigdataSail.java:2068)
wdqs_1 | at com.bigdata.rdf.sail.BigdataSail$BigdataSailReadOnlyConnection.<init>(BigdataSail.java:5236)
wdqs_1 | at com.bigdata.rdf.sail.BigdataSail._getReadOnlyConnection(BigdataSail.java:1540)
wdqs_1 | at com.bigdata.rdf.sail.BigdataSail.getReadOnlyConnection(BigdataSail.java:1503)
wdqs_1 | at com.bigdata.rdf.sail.BigdataSailRepository.getReadOnlyConnection(BigdataSailRepository.java:140)
wdqs_1 | at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:247)
wdqs_1 | at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:221)
wdqs_1 | at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:722)
wdqs_1 | at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:671)
wdqs_1 | at com.bigdata.rdf.task.ApiTaskForIndexManager.call(ApiTaskForIndexManager.java:68)
wdqs_1 | at java.util.concurrent.FutureTask.run(FutureTask.java:266)
wdqs_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
wdqs_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
wdqs_1 | ... 1 common frames omitted
wdqs_1 | Caused by: java.lang.RuntimeException: off=0, len=558
wdqs_1 | at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:239)
wdqs_1 | at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:207)
wdqs_1 | at com.bigdata.sparse.ValueType.decode(ValueType.java:333)
wdqs_1 | at com.bigdata.sparse.AbstractAtomicRowReadOrWrite.atomicRead(AbstractAtomicRowReadOrWrite.java:347)
wdqs_1 | at com.bigdata.sparse.AbstractAtomicRowReadOrWrite.atomicRead(AbstractAtomicRowReadOrWrite.java:157)
wdqs_1 | at com.bigdata.sparse.AtomicRowRead.apply(AtomicRowRead.java:98)
wdqs_1 | at com.bigdata.sparse.AtomicRowRead.apply(AtomicRowRead.java:36)
wdqs_1 | at com.bigdata.btree.AbstractBTree.submit(AbstractBTree.java:3263)
wdqs_1 | at com.bigdata.sparse.SparseRowStore.read(SparseRowStore.java:537)
wdqs_1 | at com.bigdata.sparse.SparseRowStore.read(SparseRowStore.java:420)
wdqs_1 | at com.bigdata.relation.locator.DefaultResourceLocator.locateResourceOn(DefaultResourceLocator.java:807)
wdqs_1 | ... 17 common frames omitted
wdqs_1 | Caused by: com.bigdata.rdf.vocab.BaseVocabulary$VocabularyVersioningException: null
wdqs_1 | at com.bigdata.rdf.vocab.BaseVocabulary.readVersion2(BaseVocabulary.java:680)
wdqs_1 | at com.bigdata.rdf.vocab.BaseVocabulary.readExternal(BaseVocabulary.java:458)
wdqs_1 | at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2116)
wdqs_1 | at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2065)
wdqs_1 | at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
wdqs_1 | at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
wdqs_1 | at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:231)
wdqs_1 | ... 27 common frames omittedOct 18 2018
Oct 18 2018
Magnus committed R2010:d5f673887ffa: make user oauth batch edits a configurable option, enabled by default (authored by Magnus).
make user oauth batch edits a configurable option, enabled by default
Magnus committed R2010:9e1e370e5977: patch by Lucas Werkmeister to run batch jobs under user OAuth account (authored by Magnus).
patch by Lucas Werkmeister to run batch jobs under user OAuth account
Magnus added a comment to T207370: Statistics of number of Wikidata edits with Magnus Manske's tools.
In the same timeframe:
- [[User:Reinheitsgebot]] (my other bot for ... various things) : 354,089 (Quarry)[https://quarry.wmflabs.org/query/30455]
- [[User:QuickStatementsBot]] : 3,885,709 (these are included in the ~6.9M figure from Sandra)
- "#mix'n'match" : 60,317
Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.
I set
services:
wikibase:
networks:
default:
aliases:
- wikibase.svc
- mixnmatch.wmflabs.orgOct 17 2018
Oct 17 2018
Magnus raised the priority of T207133: SPARQL service returns entities with broken URLs from Normal to Unbreak Now!.
So the tool still doesn't work, and I can't rollback the change...
Oct 16 2018
Oct 16 2018
Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.
I changed the configuration back to where it was, and restarted the docker containers. Still broken. Now the tool does not work anymore, and I can't fix it.
Magnus added a comment to T206767: wikibase-docker Quickstatements SSL error.
Yes, this is a hack I put in for ToolForge. I should probably remove it. It can be removed without consequences, AFAICT.
Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.
Did that now. No change. wikibase-docker_wdqs-updater_1 still keeps dying.
Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.
log files:
wdqs-updater_1 | Exception in thread "main" org.wikidata.query.rdf.tool.exception.ContainedException: Non-200 response from triple store: HttpContentResponse[HTTP/1.1 500 Server Error - 6429 bytes] body=
wdqs-updater_1 | SPARQL-QUERY: queryStr=PREFIX schema: <http://schema.org/>
wdqs-updater_1 | SELECT * WHERE { <http://mixnmatch.wmflabs.org> schema:dateModified ?date }
wdqs-updater_1 | java.util.concurrent.ExecutionException: java.lang.RuntimeException: off=0, len=558::namespace=wdq, timestamp=readOnly(1539696002531), readTime=readOnly(1539679638136)
wdqs-updater_1 | at java.util.concurrent.FutureTask.report(FutureTask.java:122)
wdqs-updater_1 | at java.util.concurrent.FutureTask.get(FutureTask.java:206)
wdqs-updater_1 | at com.bigdata.rdf.sail.webapp.BigdataServlet.submitApiTask(BigdataServlet.java:293)
wdqs-updater_1 | at com.bigdata.rdf.sail.webapp.QueryServlet.doSparqlQuery(QueryServlet.java:654)
wdqs-updater_1 | at com.bigdata.rdf.sail.webapp.QueryServlet.doPost(QueryServlet.java:273)
wdqs-updater_1 | at com.bigdata.rdf.sail.webapp.RESTServlet.doPost(RESTServlet.java:269)
wdqs-updater_1 | at com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doPost(MultiTenancyServlet.java:193)
wdqs-updater_1 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
wdqs-updater_1 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
wdqs-updater_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
wdqs-updater_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
wdqs-updater_1 | at org.wikidata.query.rdf.blazegraph.throttling.ThrottlingFilter.doFilter(ThrottlingFilter.java:304)
wdqs-updater_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
wdqs-updater_1 | at ch.qos.logback.classic.helpers.MDCInsertingServletFilter.doFilter(MDCInsertingServletFilter.java:49)
wdqs-updater_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
wdqs-updater_1 | at org.wikidata.query.rdf.blazegraph.filters.ClientIPFilter.doFilter(ClientIPFilter.java:43)
wdqs-updater_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
wdqs-updater_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
wdqs-updater_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
wdqs-updater_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
wdqs-updater_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
wdqs-updater_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
wdqs-updater_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
wdqs-updater_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
wdqs-updater_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
wdqs-updater_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
wdqs-updater_1 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
wdqs-updater_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
wdqs-updater_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
wdqs-updater_1 | at org.eclipse.jetty.server.Server.handle(Server.java:497)
wdqs-updater_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
wdqs-updater_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
wdqs-updater_1 | at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
wdqs-updater_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
wdqs-updater_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
wdqs-updater_1 | at java.lang.Thread.run(Thread.java:748)
wdqs-updater_1 | Caused by: java.lang.RuntimeException: off=0, len=558::namespace=wdq, timestamp=readOnly(1539696002531), readTime=readOnly(1539679638136)
wdqs-updater_1 | at com.bigdata.relation.locator.DefaultResourceLocator.locateResourceOn(DefaultResourceLocator.java:817)
wdqs-updater_1 | at com.bigdata.relation.locator.DefaultResourceLocator.locateResource(DefaultResourceLocator.java:586)
wdqs-updater_1 | at com.bigdata.relation.locator.DefaultResourceLocator.cacheMiss(DefaultResourceLocator.java:395)
wdqs-updater_1 | at com.bigdata.relation.locator.DefaultResourceLocator.locate(DefaultResourceLocator.java:347)
wdqs-updater_1 | at com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.<init>(BigdataSail.java:2068)
wdqs-updater_1 | at com.bigdata.rdf.sail.BigdataSail$BigdataSailReadOnlyConnection.<init>(BigdataSail.java:5236)
wdqs-updater_1 | at com.bigdata.rdf.sail.BigdataSail._getReadOnlyConnection(BigdataSail.java:1540)
wdqs-updater_1 | at com.bigdata.rdf.sail.BigdataSail.getReadOnlyConnection(BigdataSail.java:1503)
wdqs-updater_1 | at com.bigdata.rdf.sail.BigdataSailRepository.getReadOnlyConnection(BigdataSailRepository.java:140)
wdqs-updater_1 | at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:247)
wdqs-updater_1 | at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:221)
wdqs-updater_1 | at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:722)
wdqs-updater_1 | at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:671)
wdqs-updater_1 | at com.bigdata.rdf.task.ApiTaskForIndexManager.call(ApiTaskForIndexManager.java:68)
wdqs-updater_1 | at java.util.concurrent.FutureTask.run(FutureTask.java:266)
wdqs-updater_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
wdqs-updater_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
wdqs-updater_1 | ... 1 more
wdqs-updater_1 | Caused by: java.lang.RuntimeException: off=0, len=558
wdqs-updater_1 | at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:239)
wdqs-updater_1 | at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:207)
wdqs-updater_1 | at com.bigdata.sparse.ValueType.decode(ValueType.java:333)
wdqs-updater_1 | at com.bigdata.sparse.AbstractAtomicRowReadOrWrite.atomicRead(AbstractAtomicRowReadOrWrite.java:347)
wdqs-updater_1 | at com.bigdata.sparse.AbstractAtomicRowReadOrWrite.atomicRead(AbstractAtomicRowReadOrWrite.java:157)
wdqs-updater_1 | at com.bigdata.sparse.AtomicRowRead.apply(AtomicRowRead.java:98)
wdqs-updater_1 | at com.bigdata.sparse.AtomicRowRead.apply(AtomicRowRead.java:36)
wdqs-updater_1 | at com.bigdata.btree.AbstractBTree.submit(AbstractBTree.java:3263)
wdqs-updater_1 | at com.bigdata.sparse.SparseRowStore.read(SparseRowStore.java:537)
wdqs-updater_1 | at com.bigdata.sparse.SparseRowStore.read(SparseRowStore.java:420)
wdqs-updater_1 | at com.bigdata.relation.locator.DefaultResourceLocator.locateResourceOn(DefaultResourceLocator.java:807)
wdqs-updater_1 | ... 17 more
wdqs-updater_1 | Caused by: com.bigdata.rdf.vocab.BaseVocabulary$VocabularyVersioningException
wdqs-updater_1 | at com.bigdata.rdf.vocab.BaseVocabulary.readVersion2(BaseVocabulary.java:680)
wdqs-updater_1 | at com.bigdata.rdf.vocab.BaseVocabulary.readExternal(BaseVocabulary.java:458)
wdqs-updater_1 | at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2116)
wdqs-updater_1 | at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2065)
wdqs-updater_1 | at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
wdqs-updater_1 | at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
wdqs-updater_1 | at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:231)
wdqs-updater_1 | ... 27 more
wdqs-updater_1 |
wdqs-updater_1 | at org.wikidata.query.rdf.tool.rdf.RdfRepository.execute(RdfRepository.java:726)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.rdf.RdfRepository.query(RdfRepository.java:679)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.rdf.RdfRepository.dateFromQuery(RdfRepository.java:750)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.rdf.RdfRepository.fetchLeftOffTime(RdfRepository.java:636)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.Update.buildRecentChangePollerChangeSource(Update.java:160)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.Update.buildChangeSource(Update.java:141)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.Update.main(Update.java:65)Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.
Maybe more detail:
I changed the three occurrences of WIKIBASE_HOST in docker-composer.yml to "mixnmatch.wmflabs.org", then did stop, then up -d. It rebuild and startedthe wdqs containers, but the updater_1 keeps dying.
Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.
I did that (docker-compose stop / sudo docker-compose up -d).