User Details
- User Since
- Oct 23 2014, 3:02 PM (207 w, 6 d)
- Availability
- Available
- LDAP User
- Magnus Manske
- MediaWiki User
- Unknown
Today
So the tool still doesn't work, and I can't rollback the change...
Yesterday
I changed the configuration back to where it was, and restarted the docker containers. Still broken. Now the tool does not work anymore, and I can't fix it.
Yes, this is a hack I put in for ToolForge. I should probably remove it. It can be removed without consequences, AFAICT.
Did that now. No change. wikibase-docker_wdqs-updater_1 still keeps dying.
log files:
wdqs-updater_1 | Exception in thread "main" org.wikidata.query.rdf.tool.exception.ContainedException: Non-200 response from triple store: HttpContentResponse[HTTP/1.1 500 Server Error - 6429 bytes] body=
wdqs-updater_1 | SPARQL-QUERY: queryStr=PREFIX schema: <http://schema.org/>
wdqs-updater_1 | SELECT * WHERE { <http://mixnmatch.wmflabs.org> schema:dateModified ?date }
wdqs-updater_1 | java.util.concurrent.ExecutionException: java.lang.RuntimeException: off=0, len=558::namespace=wdq, timestamp=readOnly(1539696002531), readTime=readOnly(1539679638136)
wdqs-updater_1 | at java.util.concurrent.FutureTask.report(FutureTask.java:122)
wdqs-updater_1 | at java.util.concurrent.FutureTask.get(FutureTask.java:206)
wdqs-updater_1 | at com.bigdata.rdf.sail.webapp.BigdataServlet.submitApiTask(BigdataServlet.java:293)
wdqs-updater_1 | at com.bigdata.rdf.sail.webapp.QueryServlet.doSparqlQuery(QueryServlet.java:654)
wdqs-updater_1 | at com.bigdata.rdf.sail.webapp.QueryServlet.doPost(QueryServlet.java:273)
wdqs-updater_1 | at com.bigdata.rdf.sail.webapp.RESTServlet.doPost(RESTServlet.java:269)
wdqs-updater_1 | at com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doPost(MultiTenancyServlet.java:193)
wdqs-updater_1 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
wdqs-updater_1 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
wdqs-updater_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
wdqs-updater_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
wdqs-updater_1 | at org.wikidata.query.rdf.blazegraph.throttling.ThrottlingFilter.doFilter(ThrottlingFilter.java:304)
wdqs-updater_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
wdqs-updater_1 | at ch.qos.logback.classic.helpers.MDCInsertingServletFilter.doFilter(MDCInsertingServletFilter.java:49)
wdqs-updater_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
wdqs-updater_1 | at org.wikidata.query.rdf.blazegraph.filters.ClientIPFilter.doFilter(ClientIPFilter.java:43)
wdqs-updater_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
wdqs-updater_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
wdqs-updater_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
wdqs-updater_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
wdqs-updater_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
wdqs-updater_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
wdqs-updater_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
wdqs-updater_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
wdqs-updater_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
wdqs-updater_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
wdqs-updater_1 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
wdqs-updater_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
wdqs-updater_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
wdqs-updater_1 | at org.eclipse.jetty.server.Server.handle(Server.java:497)
wdqs-updater_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
wdqs-updater_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
wdqs-updater_1 | at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
wdqs-updater_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
wdqs-updater_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
wdqs-updater_1 | at java.lang.Thread.run(Thread.java:748)
wdqs-updater_1 | Caused by: java.lang.RuntimeException: off=0, len=558::namespace=wdq, timestamp=readOnly(1539696002531), readTime=readOnly(1539679638136)
wdqs-updater_1 | at com.bigdata.relation.locator.DefaultResourceLocator.locateResourceOn(DefaultResourceLocator.java:817)
wdqs-updater_1 | at com.bigdata.relation.locator.DefaultResourceLocator.locateResource(DefaultResourceLocator.java:586)
wdqs-updater_1 | at com.bigdata.relation.locator.DefaultResourceLocator.cacheMiss(DefaultResourceLocator.java:395)
wdqs-updater_1 | at com.bigdata.relation.locator.DefaultResourceLocator.locate(DefaultResourceLocator.java:347)
wdqs-updater_1 | at com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.<init>(BigdataSail.java:2068)
wdqs-updater_1 | at com.bigdata.rdf.sail.BigdataSail$BigdataSailReadOnlyConnection.<init>(BigdataSail.java:5236)
wdqs-updater_1 | at com.bigdata.rdf.sail.BigdataSail._getReadOnlyConnection(BigdataSail.java:1540)
wdqs-updater_1 | at com.bigdata.rdf.sail.BigdataSail.getReadOnlyConnection(BigdataSail.java:1503)
wdqs-updater_1 | at com.bigdata.rdf.sail.BigdataSailRepository.getReadOnlyConnection(BigdataSailRepository.java:140)
wdqs-updater_1 | at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:247)
wdqs-updater_1 | at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:221)
wdqs-updater_1 | at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:722)
wdqs-updater_1 | at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:671)
wdqs-updater_1 | at com.bigdata.rdf.task.ApiTaskForIndexManager.call(ApiTaskForIndexManager.java:68)
wdqs-updater_1 | at java.util.concurrent.FutureTask.run(FutureTask.java:266)
wdqs-updater_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
wdqs-updater_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
wdqs-updater_1 | ... 1 more
wdqs-updater_1 | Caused by: java.lang.RuntimeException: off=0, len=558
wdqs-updater_1 | at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:239)
wdqs-updater_1 | at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:207)
wdqs-updater_1 | at com.bigdata.sparse.ValueType.decode(ValueType.java:333)
wdqs-updater_1 | at com.bigdata.sparse.AbstractAtomicRowReadOrWrite.atomicRead(AbstractAtomicRowReadOrWrite.java:347)
wdqs-updater_1 | at com.bigdata.sparse.AbstractAtomicRowReadOrWrite.atomicRead(AbstractAtomicRowReadOrWrite.java:157)
wdqs-updater_1 | at com.bigdata.sparse.AtomicRowRead.apply(AtomicRowRead.java:98)
wdqs-updater_1 | at com.bigdata.sparse.AtomicRowRead.apply(AtomicRowRead.java:36)
wdqs-updater_1 | at com.bigdata.btree.AbstractBTree.submit(AbstractBTree.java:3263)
wdqs-updater_1 | at com.bigdata.sparse.SparseRowStore.read(SparseRowStore.java:537)
wdqs-updater_1 | at com.bigdata.sparse.SparseRowStore.read(SparseRowStore.java:420)
wdqs-updater_1 | at com.bigdata.relation.locator.DefaultResourceLocator.locateResourceOn(DefaultResourceLocator.java:807)
wdqs-updater_1 | ... 17 more
wdqs-updater_1 | Caused by: com.bigdata.rdf.vocab.BaseVocabulary$VocabularyVersioningException
wdqs-updater_1 | at com.bigdata.rdf.vocab.BaseVocabulary.readVersion2(BaseVocabulary.java:680)
wdqs-updater_1 | at com.bigdata.rdf.vocab.BaseVocabulary.readExternal(BaseVocabulary.java:458)
wdqs-updater_1 | at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2116)
wdqs-updater_1 | at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2065)
wdqs-updater_1 | at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
wdqs-updater_1 | at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
wdqs-updater_1 | at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:231)
wdqs-updater_1 | ... 27 more
wdqs-updater_1 |
wdqs-updater_1 | at org.wikidata.query.rdf.tool.rdf.RdfRepository.execute(RdfRepository.java:726)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.rdf.RdfRepository.query(RdfRepository.java:679)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.rdf.RdfRepository.dateFromQuery(RdfRepository.java:750)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.rdf.RdfRepository.fetchLeftOffTime(RdfRepository.java:636)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.Update.buildRecentChangePollerChangeSource(Update.java:160)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.Update.buildChangeSource(Update.java:141)
wdqs-updater_1 | at org.wikidata.query.rdf.tool.Update.main(Update.java:65)Maybe more detail:
I changed the three occurrences of WIKIBASE_HOST in docker-composer.yml to "mixnmatch.wmflabs.org", then did stop, then up -d. It rebuild and startedthe wdqs containers, but the updater_1 keeps dying.
I did that (docker-compose stop / sudo docker-compose up -d).
OK, another sudo docker-compose up --no-build -d did it. Thanks.
End of the logfile:
wdqs-frontend_1 | 10.68.21.68 - - [16/Oct/2018:07:17:15 +0000] "GET /proxy/wdqs/bigdata/namespace/wdq/sparql?query=prefix%20schema:%20%3Chttp://schema.org/%3E%20SELECT%20*%20WHERE%20%7B%3Chttp://www.wikidata.org%3E%20schema:dateModified%20?y%7D&nocache=25661236 HTTP/1.1" 502 173 "http://mixnmatch-query.wmflabs.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:63.0) Gecko/20100101 Firefox/63.0" "193.62.205.8" wdqs-frontend_1 | 10.68.21.68 - - [16/Oct/2018:07:24:55 +0000] "GET /proxy/wdqs/bigdata/namespace/wdq/sparql?format=json&query=SELECT+%3Fq+%3FqLabel+%7B+%3Fq+wdt%3AP2+wd%3AQ2+.+SERVICE+wikibase%3Alabel+%7B+bd%3AserviceParam+wikibase%3Alanguage+%27%5BAUTO_LANGUAGE%5D%2Cen%27+%7D+%7D HTTP/1.1" 502 173 "-" "-" "172.16.1.251" wdqs-frontend_1 | 10.68.21.68 - - [16/Oct/2018:07:24:58 +0000] "GET /proxy/wdqs/bigdata/namespace/wdq/sparql?format=json&query=SELECT+%3Fq+%3FqLabel+%7B+%3Fq+wdt%3AP2+wd%3AQ2+.+SERVICE+wikibase%3Alabel+%7B+bd%3AserviceParam+wikibase%3Alanguage+%27%5BAUTO_LANGUAGE%5D%2Cen%27+%7D+%7D HTTP/1.1" 502 173 "-" "-" "172.16.1.251" wdqs-frontend_1 | 10.68.21.68 - - [16/Oct/2018:07:25:33 +0000] "GET /proxy/wdqs/bigdata/namespace/wdq/sparql?format=json&query=SELECT+%3Fq+%3FqLabel+%7B+%3Fq+wdt%3AP2+wd%3AQ2+.+SERVICE+wikibase%3Alabel+%7B+bd%3AserviceParam+wikibase%3Alanguage+%27%5BAUTO_LANGUAGE%5D%2Cen%27+%7D+%7D HTTP/1.1" 502 173 "-" "-" "172.16.1.251" wdqs-frontend_1 | 10.68.21.68 - - [16/Oct/2018:07:25:43 +0000] "GET /proxy/wdqs/bigdata/namespace/wdq/sparql?format=json&query=SELECT+%3Fq+%3FqLabel+%7B+%3Fq+wdt%3AP2+wd%3AQ2+.+SERVICE+wikibase%3Alabel+%7B+bd%3AserviceParam+wikibase%3Alanguage+%27%5BAUTO_LANGUAGE%5D%2Cen%27+%7D+%7D HTTP/1.1" 502 173 "-" "-" "172.16.1.251" wdqs-frontend_1 | 2018/10/16 07:26:38 [emerg] 1#1: host not found in upstream "wikibase.svc" in /etc/nginx/conf.d/default.conf:8 wdqs-frontend_1 | nginx: [emerg] host not found in upstream "wikibase.svc" in /etc/nginx/conf.d/default.conf:8
I'll try the proxy.
Mon, Oct 15
Maybe also relevant: The data used for the tool that @Lydia_Pintscher linked to are avilable on Toolforge in the s53867__referee_p database (readable with any toolforge account), especially the statements table.
The data in https://tools.wmflabs.org/mix-n-match/#/catalog/528 was a one-time import, and is not updated automatically. I can try and set up that I call an auto-scraper for that catalog if you like.
Sat, Oct 13
In this context, I would also appreciate a pointer as to enabling second precision on my own wikibase installation.
Tue, Oct 9
To reproduce, I just click "random item" a few times, one will show up very quickly.
Ah, I see a hackaround is on https://github.com/wmde/wikibase-docker/blob/master/quickstatements/README.md but still...
Addendum: Also added $wgEnableEmail=true; to LocalSettings.php, no joy.
Wed, Oct 3
Well, if we want to put the creation time of a file into a statement, and we have a (sub-)second-precision EXIF time, it seems silly to round it to a day...
Tue, Oct 2
I can further limit the item number the bot accepts (though I have other people asking me to _increase_ it...).
Fri, Sep 28
Many of the tools in Hay's list have links to the source repo:
https://tools.wmflabs.org/hay/directory/?search=wikidata#/search/wikidata
I have fixed the jQuery issue, but the underlying problem is that this code is very old (as in Toolserver-old), and only works with some site naming schemes. Other languages don't work either for many titles.
Tue, Sep 25
Mon, Sep 24
@Sjoerddebruin: Happens logged out as well, browser cache cleared, so no scripts/gadgets.
Happens on the same page in Safari Version 12.0 (13606.2.11) / Mac.
Happens to me on https://www.wikidata.org/wiki/Q29201051 even after I logged out and cleared the browser cache.
The effect is persistent on specific items, e.g. https://www.wikidata.org/wiki/Q29201051
I am using Firefox 62/Mac
Hitting "Random item", I can reproduce the effect on roughly 1 in 3 pages. On any given item page, this effect shows either for no statement, or for all of them.
Tue, Sep 18
Sep 17 2018
Both the production and the development instance have been upgraded to Debian Stretch.
Sep 10 2018
Confirmed, test account created:
https://structured-commons.wmflabs.org/wiki/User:Reinheitsgebot
Could someone please add that to "trusted", if necessary?
Sep 7 2018
I have seen this a few times, but I'm not exactly sure where it comes from.
Aug 29 2018
Aug 24 2018
Applied. Untested.
Aug 23 2018
Applied, thanks. This will not work in "browser mode" (new_item not exchanged via API). Untested in "bot mode".
Aug 9 2018
Jul 26 2018
Thanks, I have added an interation counter that will limit infinite recursion. Odd that this should happen though, unless Wikidata is closed for business...
Given that one can never exclude that, I don't think so. The code has been working for years. I now added a 30 seconds time limit to index.php, that should solve any infinite loop problems with my code, if they exists.
Is there anything I can do? Anything I should have done?
The problem is back. Same symptoms.
Jul 25 2018
Yup, seems to have self-healed while I drove home. Not sure if feature or scary signs of self-awareness.
Update: managed to start the kubernetes webengine again. Still "forever load".
tools.widar@tools-bastion-03:~$ webservice --backend=gridengine stop Your webservice is not running
Update: Now webservice won't start anymore, says:
Disease items where at least one of the 10 languages is missing, sorted by "importance":
Jul 24 2018
Why is patching with Differential so crappy?
Jul 16 2018
Action required: Make MediaWiki use OAuth as a login alternative to local user accounts.
Jul 3 2018
Jul 2 2018
Jun 28 2018
Excellent, that means my existing code should Just Work (tm).
Jun 26 2018
I run PetScan.
Jun 22 2018
@WMDE-leszek I think some tools could actually use SPARQL, if you look for a precise label with known language for example. But I mentioned SPARQL only as one of the other ways we currently have. wbsearchentities is another, and so is the fulltext search API. And wbgetentities of course, if you know the item and just want the label. They all have some potential for certain tasks, but especially when dealing with lots of labels to check, none of them scale up.
Found another one: Mix'n'match "automatch" (preliminary matches by name, that need to be confirmed by a user). This quickly checks tens/hundreds of thousands of names against Wikidata labels on a regular basis. I am quite certain this would not work with SPARQL, and though it might actually yield better quality using the search API, it's one http(s) query per name, and I don't want to DOS Wikidata, not even considering speed...
Jun 21 2018
Jun 20 2018
Labs currently lists 132 tools of mine: https://tools.wmflabs.org/admin/tools#!/author/Magnus%20Manske
Just within the "wikidata-todo" tool, these sub-tools use wb_terms in various ways:
- add_name_labels.php
- artwork_images.php
- awarder.php
- beacon.php
- cloudy_concept.php
- creator.php
- dupe_finder.php
- firstnamebasis.php
- get_item_names.php
- important_blank_items.php
- index.php
- label_no_instance.php
- no_statements.php
- project_stats.php
- relabel.php
- related_properties.php
- sexer.php
- sparql_rc.php
- translate_items_with_property.php
- user_edits.php
- wdq2graph.php
- wdq_feed.php
- Filter for items with all/any/no labels in a set of languages
- Find items by label
- Label Wikidata items in result set
Oh well, there go half my Wikidata-related tools...
Jun 15 2018
May 23 2018
Works again for me.
May 22 2018
