So here is what happens: I create(d) lots of gene/protein items (example) for various species. For many statements, I can create references, as I get them from the upstream source. That paper is one of the often-cited ones, about a determination method.

Feb 28 2019, 3:29 PM · Performance, Wikidata-Campsite, Wikidata, Wikimedia-production-error

Feb 27 2019

Magnus added projects to T217246: cronjob error mails: Tools, Cloud-VPS.

Feb 27 2019, 2:42 PM · Tools

Magnus created T217246: cronjob error mails.

Feb 27 2019, 2:41 PM · Tools

Feb 19 2019

Magnus changed the status of T216153: ListeriaBot appears as a non bot despite of having a bot flag in euwiki recent changes from Invalid to Resolved.

I have added euwiki to the list of wikis where the bot flag is to be used.

Feb 19 2019, 12:35 PM

Feb 12 2019

Magnus closed T215620: Tool "autodesc" - can't start webservice as Resolved.

That did the trick, thanks!

Feb 12 2019, 3:45 PM · Wikidata, Toolforge

Feb 11 2019

Sjoerddebruin awarded T215620: Tool "autodesc" - can't start webservice a The World Burns token.

Feb 11 2019, 7:54 PM · Wikidata, Toolforge

Magnus added a comment to T215620: Tool "autodesc" - can't start webservice.

Tried that, also on login.tools.wmflabs.org (just to be sure). Both say "webservice is not running". Still won't start kubernetes.

Feb 11 2019, 12:47 PM · Wikidata, Toolforge

Magnus added a comment to T215620: Tool "autodesc" - can't start webservice.

Thanks, I have rebuild and updated via npm on the kubernetes shell.

Feb 11 2019, 9:48 AM · Wikidata, Toolforge

Feb 9 2019

Magnus added a comment to T215620: Tool "autodesc" - can't start webservice.

I have run npm update in the kubernetes shell, but no joy.

Feb 9 2019, 8:45 AM · Wikidata, Toolforge

Feb 8 2019

Magnus created T215620: Tool "autodesc" - can't start webservice.

Feb 8 2019, 3:09 PM · Wikidata, Toolforge

Feb 1 2019

Magnus added a comment to T215035: 502 Bad Gateway issue on Petscan.

Try it now...

Feb 1 2019, 2:40 PM · VPS-Projects

Magnus added a comment to T215035: 502 Bad Gateway issue on Petscan.

Testing on dev now. Looks like frequent Lost connection to MySQL server during query errors from the DB replicas.

Feb 1 2019, 1:10 PM · VPS-Projects

Magnus added a comment to T215035: 502 Bad Gateway issue on Petscan.

Also, I just clicked on the two examples. They took 133 and 173 seconds, and returned 50 and 1 results, respectively. No 502s, though I have seen those occasionally.

Feb 1 2019, 9:51 AM · VPS-Projects

Magnus added a comment to T215035: 502 Bad Gateway issue on Petscan.

OK, so what appears to happen is that SQL queries timeout and take PetScan with them. Note:

I wrote some code that re-arranges certain large queries into smaller ones, which cuts down on the timeouts; that code has been live for weeks
That works fine on the dev machine but not reliably on the production machine
The dev machine has less resources than production, but is otherwise identical (OS etc)

As this does not fail reproducibly, it's either some odd bug in my code, or some situation on the DB replicas.

Feb 1 2019, 9:49 AM · VPS-Projects

Jan 21 2019

Tagishsimon awarded T214278: Quickstatements, "backend is overloaded" a Heartbreak token.

Jan 21 2019, 2:56 PM · Tools, Toolforge

Magnus added a comment to T214278: Quickstatements, "backend is overloaded".

In T214278#4896618, @Daniel_Mietchen wrote:

As far as I can see, this is not limited to QuickStatements - when I'm logged out (as just happened again - my "usual" time is at 21 min into the hour), I cannot access SourceMD either. Access normally comes back around 37 min into the hour, even if I press the "log in" button immediately after being logged out.

Jan 21 2019, 2:54 PM · Tools, Toolforge

Magnus created T214278: Quickstatements, "backend is overloaded".

Jan 21 2019, 8:25 AM · Tools, Toolforge

Jan 17 2019

Magnus added a comment to T193560: Tool keeps falling into permanent 500 error.

Looks like it's back to normal at the moment.

Jan 17 2019, 1:32 PM · cloud-services-team (Kanban), Toolforge

Jan 14 2019

Magnus committed R2010:779f5692926e: bugfix (authored by Magnus).

bugfix

Jan 14 2019, 8:24 PM

Magnus committed R2010:756382987e1a: fixes for Commons, also requires fixed wikidata.php! (authored by Magnus).

fixes for Commons, also requires fixed wikidata.php!

Jan 14 2019, 11:19 AM

Magnus committed R2010:574b1e48ccfe: bugfix (authored by Magnus).

bugfix

Jan 14 2019, 10:20 AM

Magnus committed R2010:aa343fe8abeb: trying to add Commons access by making things more generic (authored by Magnus).

trying to add Commons access by making things more generic

Jan 14 2019, 10:20 AM

Dec 4 2018

Magnus committed R2010:84fb87586d23: typo (authored by Magnus).

typo

Dec 4 2018, 11:04 AM

Magnus committed R2010:c5953eeeafa0: quite option (authored by Magnus).

quite option

Dec 4 2018, 10:58 AM

Magnus committed R2010:68f49e60870e: misc (authored by Magnus).

misc

Dec 4 2018, 9:43 AM

Magnus committed R2010:ed7c36eb5be4: one batch per job (authored by Magnus).

one batch per job

Dec 4 2018, 9:31 AM

Nov 30 2018

Magnus awarded T210836: +2 for Lucas_Werkmeister_WMDE in mediawiki/* a Like token.

Nov 30 2018, 2:24 PM · User-Ladsgroup, Repository-Ownership-Requests

Nov 9 2018

Magnus added a comment to T208940: Grid slow on Toolforge.

I have repeatedly asked if there is a "max concurrency" setting for jsub, as there is for other grid engines. I would consider it rather silly to force every user with such jobs to implement that on their own. For example, I could record it in the database when a job starts, but how do I know it is still running, and hasn't failed? Guess based on last action time? It makes vastly more sense to do that kind of thing in the job scheduler.

Nov 9 2018, 8:23 AM · cloud-services-team (Kanban), Toolforge

Nov 8 2018

Magnus added a comment to T208940: Grid slow on Toolforge.

Checked out sourcemd. That is actually correct; one job for each of the "TODO" ones here:
https://tools.wmflabs.org/sourcemd/?action=batches

Nov 8 2018, 9:48 PM · cloud-services-team (Kanban), Toolforge

Magnus added a comment to T208940: Grid slow on Toolforge.

Looks like the grid caught up with them. One job type seems to be problematic, not sure if it's an aftereffect.

Nov 8 2018, 9:44 PM · cloud-services-team (Kanban), Toolforge

Magnus added a comment to T208940: Grid slow on Toolforge.

Using mix-n-match as an example, two of the cronjob commands:

0,5,10,15,20,25,30,35,40,45,50,55 * * * * cd /data/project/mix-n-match ; /usr/bin/jsub -quiet -mem 8g -N as_import -cwd ./autoscrape_import_bot.php
4,14,24,34,44,54 * * * * cd /data/project/mix-n-match ; /usr/bin/jsub -quiet -mem 6g -cwd  -N  mnm-microsync ./microsync.php random

Yes, they run often, but usually not for long, and have never had such an issue in the years before.

Nov 8 2018, 3:46 PM · cloud-services-team (Kanban), Toolforge

Magnus added a comment to T208940: Grid slow on Toolforge.

Up to 95 "qw" now...

Nov 8 2018, 8:25 AM · cloud-services-team (Kanban), Toolforge

Nov 7 2018

Magnus added a comment to T208940: Grid slow on Toolforge.

(I picked mix-n-match as an example. Other tools are affected as well)

Nov 7 2018, 10:54 AM · cloud-services-team (Kanban), Toolforge

Magnus created T208940: Grid slow on Toolforge.

Nov 7 2018, 10:53 AM · cloud-services-team (Kanban), Toolforge

Nov 2 2018

Magnus added a comment to T199228: Define an SLO for Wikidata Query Service public endpoint and communicate it.

In many cases, especially bot/background tasks (e.g. Listeria), a lag of hours is not critical. This is also true for many interactive tools, where the user gets some items matching certain criteria.

Nov 2 2018, 2:28 PM · Operations, Discovery-Wikidata-Query-Service-Sprint, Wikidata, Wikidata-Query-Service

Nov 1 2018

Magnus closed T208460: wikidata-todo is using 959GB in shared NFS area as Resolved.

Nov 1 2018, 11:01 AM · Data-Services, Toolforge

Magnus closed T208460: wikidata-todo is using 959GB in shared NFS area, a subtask of T206239: 2018-10-04: tools and NFS share cleanup (high usage), as Resolved.

Nov 1 2018, 11:01 AM · cloud-services-team (Kanban)

Magnus added a comment to T208460: wikidata-todo is using 959GB in shared NFS area.

I have removed unused files /shared/dumps and altered the update code to only keep recent ones around.

Nov 1 2018, 10:58 AM · Data-Services, Toolforge

Oct 25 2018

Magnus committed R2010:d4bbd816e688: This patch set, along with an accompanying series for magnustools, adds… (authored by Magnus).

This patch set, along with an accompanying series for magnustools, adds…

Oct 25 2018, 5:13 PM

Oct 23 2018

Magnus added a comment to T207579: imagecopy_tests failing with IndexError.

Try it now please.

Oct 23 2018, 7:27 PM · Patch-For-Review, Tools, Upstream, Pywikibot-tests, Pywikibot

Oct 22 2018

Magnus added a comment to T207579: imagecopy_tests failing with IndexError.

This should be fixed now in CommonsHelper.

Oct 22 2018, 6:40 PM · Patch-For-Review, Tools, Upstream, Pywikibot-tests, Pywikibot

Oct 20 2018

Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.

Thanks @Jkbr, that seems to do the trick!

Oct 20 2018, 9:51 AM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Oct 19 2018

Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.

Can I give you (or can you just get) access to my VM? mixnmatch in the mix-n-match project.

Oct 19 2018, 1:57 PM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.

As a sidenote, this may also contain the solution to T207132 ...

Oct 19 2018, 8:32 AM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.

docker-compose.yml:

# Wikibase with Query Service
#
# This docker-compose example can be used to pull the images from docker hub.
#
# Examples:
#
# Access Wikibase via "http://localhost:8181"
#   (or "http://$(docker-machine ip):8181" if using docker-machine)
#
# Access Query Service via "http://localhost:8282"
#   (or "http://$(docker-machine ip):8282" if using docker-machine)
version: '3'

Oct 19 2018, 8:28 AM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.

wdqs log (last "entry"):

wdqs_1             | 08:20:16.620 [qtp1747585824-18] ERROR c.b.r.sail.webapp.BigdataRDFServlet IP:wikibase-docker_wdqs-proxy_1.wikibase-docker_default UA:Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:63.0) Gecko/20100101 Firefox/63.0 - cause=java.util.concurrent.ExecutionException: java.lang.RuntimeException: off=0, len=558::namespace=wdq, timestamp=readOnly(1539937216616), readTime=readOnly(1539701290285), query=SPARQL-QUERY: queryStr=prefix schema: <http://schema.org/> SELECT * WHERE {<http://www.wikidata.org> schema:dateModified ?y}
wdqs_1             | java.util.concurrent.ExecutionException: java.lang.RuntimeException: off=0, len=558::namespace=wdq, timestamp=readOnly(1539937216616), readTime=readOnly(1539701290285)
wdqs_1             | 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
wdqs_1             | 	at java.util.concurrent.FutureTask.get(FutureTask.java:206)
wdqs_1             | 	at com.bigdata.rdf.sail.webapp.BigdataServlet.submitApiTask(BigdataServlet.java:293)
wdqs_1             | 	at com.bigdata.rdf.sail.webapp.QueryServlet.doSparqlQuery(QueryServlet.java:654)
wdqs_1             | 	at com.bigdata.rdf.sail.webapp.QueryServlet.doGet(QueryServlet.java:288)
wdqs_1             | 	at com.bigdata.rdf.sail.webapp.RESTServlet.doGet(RESTServlet.java:240)
wdqs_1             | 	at com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doGet(MultiTenancyServlet.java:271)
wdqs_1             | 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
wdqs_1             | 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
wdqs_1             | 	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
wdqs_1             | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
wdqs_1             | 	at org.wikidata.query.rdf.blazegraph.throttling.ThrottlingFilter.doFilter(ThrottlingFilter.java:304)
wdqs_1             | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
wdqs_1             | 	at ch.qos.logback.classic.helpers.MDCInsertingServletFilter.doFilter(MDCInsertingServletFilter.java:49)
wdqs_1             | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
wdqs_1             | 	at org.wikidata.query.rdf.blazegraph.filters.ClientIPFilter.doFilter(ClientIPFilter.java:43)
wdqs_1             | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
wdqs_1             | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
wdqs_1             | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
wdqs_1             | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
wdqs_1             | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
wdqs_1             | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
wdqs_1             | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
wdqs_1             | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
wdqs_1             | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
wdqs_1             | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
wdqs_1             | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
wdqs_1             | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
wdqs_1             | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
wdqs_1             | 	at org.eclipse.jetty.server.Server.handle(Server.java:497)
wdqs_1             | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
wdqs_1             | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
wdqs_1             | 	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
wdqs_1             | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
wdqs_1             | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
wdqs_1             | 	at java.lang.Thread.run(Thread.java:748)
wdqs_1             | Caused by: java.lang.RuntimeException: off=0, len=558::namespace=wdq, timestamp=readOnly(1539937216616), readTime=readOnly(1539701290285)
wdqs_1             | 	at com.bigdata.relation.locator.DefaultResourceLocator.locateResourceOn(DefaultResourceLocator.java:817)
wdqs_1             | 	at com.bigdata.relation.locator.DefaultResourceLocator.locateResource(DefaultResourceLocator.java:586)
wdqs_1             | 	at com.bigdata.relation.locator.DefaultResourceLocator.cacheMiss(DefaultResourceLocator.java:395)
wdqs_1             | 	at com.bigdata.relation.locator.DefaultResourceLocator.locate(DefaultResourceLocator.java:347)
wdqs_1             | 	at com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.<init>(BigdataSail.java:2068)
wdqs_1             | 	at com.bigdata.rdf.sail.BigdataSail$BigdataSailReadOnlyConnection.<init>(BigdataSail.java:5236)
wdqs_1             | 	at com.bigdata.rdf.sail.BigdataSail._getReadOnlyConnection(BigdataSail.java:1540)
wdqs_1             | 	at com.bigdata.rdf.sail.BigdataSail.getReadOnlyConnection(BigdataSail.java:1503)
wdqs_1             | 	at com.bigdata.rdf.sail.BigdataSailRepository.getReadOnlyConnection(BigdataSailRepository.java:140)
wdqs_1             | 	at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:247)
wdqs_1             | 	at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:221)
wdqs_1             | 	at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:722)
wdqs_1             | 	at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:671)
wdqs_1             | 	at com.bigdata.rdf.task.ApiTaskForIndexManager.call(ApiTaskForIndexManager.java:68)
wdqs_1             | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
wdqs_1             | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
wdqs_1             | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
wdqs_1             | 	... 1 common frames omitted
wdqs_1             | Caused by: java.lang.RuntimeException: off=0, len=558
wdqs_1             | 	at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:239)
wdqs_1             | 	at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:207)
wdqs_1             | 	at com.bigdata.sparse.ValueType.decode(ValueType.java:333)
wdqs_1             | 	at com.bigdata.sparse.AbstractAtomicRowReadOrWrite.atomicRead(AbstractAtomicRowReadOrWrite.java:347)
wdqs_1             | 	at com.bigdata.sparse.AbstractAtomicRowReadOrWrite.atomicRead(AbstractAtomicRowReadOrWrite.java:157)
wdqs_1             | 	at com.bigdata.sparse.AtomicRowRead.apply(AtomicRowRead.java:98)
wdqs_1             | 	at com.bigdata.sparse.AtomicRowRead.apply(AtomicRowRead.java:36)
wdqs_1             | 	at com.bigdata.btree.AbstractBTree.submit(AbstractBTree.java:3263)
wdqs_1             | 	at com.bigdata.sparse.SparseRowStore.read(SparseRowStore.java:537)
wdqs_1             | 	at com.bigdata.sparse.SparseRowStore.read(SparseRowStore.java:420)
wdqs_1             | 	at com.bigdata.relation.locator.DefaultResourceLocator.locateResourceOn(DefaultResourceLocator.java:807)
wdqs_1             | 	... 17 common frames omitted
wdqs_1             | Caused by: com.bigdata.rdf.vocab.BaseVocabulary$VocabularyVersioningException: null
wdqs_1             | 	at com.bigdata.rdf.vocab.BaseVocabulary.readVersion2(BaseVocabulary.java:680)
wdqs_1             | 	at com.bigdata.rdf.vocab.BaseVocabulary.readExternal(BaseVocabulary.java:458)
wdqs_1             | 	at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2116)
wdqs_1             | 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2065)
wdqs_1             | 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
wdqs_1             | 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
wdqs_1             | 	at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:231)
wdqs_1             | 	... 27 common frames omitted

Oct 19 2018, 8:25 AM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Oct 18 2018

Magnus accepted D1119: Run new batches as the submitting user.

Oct 18 2018, 3:28 PM · Wikidata

Magnus committed R2010:d5f673887ffa: make user oauth batch edits a configurable option, enabled by default (authored by Magnus).

make user oauth batch edits a configurable option, enabled by default

Oct 18 2018, 3:26 PM

Magnus committed R2010:9e1e370e5977: patch by Lucas Werkmeister to run batch jobs under user OAuth account (authored by Magnus).

patch by Lucas Werkmeister to run batch jobs under user OAuth account

Oct 18 2018, 3:22 PM

Magnus added a comment to T207370: Statistics of number of Wikidata edits with Magnus Manske's tools.

In the same timeframe:

[[User:Reinheitsgebot]] (my other bot for ... various things) : 354,089 (Quarry)[https://quarry.wmflabs.org/query/30455]
[[User:QuickStatementsBot]] : 3,885,709 (these are included in the ~6.9M figure from Sandra)
"#mix'n'match" : 60,317

Oct 18 2018, 2:15 PM · GLAM, Wikidata

Magnus committed R2010:cc5a7d738b7a: misc (authored by Magnus).

misc

Oct 18 2018, 1:57 PM

Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.

I set

services:
  wikibase:
    networks:
      default:
        aliases:
         - wikibase.svc
         - mixnmatch.wmflabs.org

Oct 18 2018, 9:11 AM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Oct 17 2018

Magnus raised the priority of T207133: SPARQL service returns entities with broken URLs from Normal to Unbreak Now!.

So the tool still doesn't work, and I can't rollback the change...

Oct 17 2018, 10:13 AM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Oct 16 2018

Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.

I changed the configuration back to where it was, and restarted the docker containers. Still broken. Now the tool does not work anymore, and I can't fix it.

Oct 16 2018, 2:49 PM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Magnus added a comment to T206767: wikibase-docker Quickstatements SSL error.

Yes, this is a hack I put in for ToolForge. I should probably remove it. It can be removed without consequences, AFAICT.

Oct 16 2018, 2:29 PM · Wikibase-Containers, Wikidata

Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.

Did that now. No change. wikibase-docker_wdqs-updater_1 still keeps dying.

Oct 16 2018, 2:05 PM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.

log files:

wdqs-updater_1     | Exception in thread "main" org.wikidata.query.rdf.tool.exception.ContainedException: Non-200 response from triple store:  HttpContentResponse[HTTP/1.1 500 Server Error - 6429 bytes] body=
wdqs-updater_1     | SPARQL-QUERY: queryStr=PREFIX schema: <http://schema.org/>
wdqs-updater_1     | SELECT * WHERE { <http://mixnmatch.wmflabs.org> schema:dateModified ?date }
wdqs-updater_1     | java.util.concurrent.ExecutionException: java.lang.RuntimeException: off=0, len=558::namespace=wdq, timestamp=readOnly(1539696002531), readTime=readOnly(1539679638136)
wdqs-updater_1     | 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
wdqs-updater_1     | 	at java.util.concurrent.FutureTask.get(FutureTask.java:206)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.BigdataServlet.submitApiTask(BigdataServlet.java:293)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.QueryServlet.doSparqlQuery(QueryServlet.java:654)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.QueryServlet.doPost(QueryServlet.java:273)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.RESTServlet.doPost(RESTServlet.java:269)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.MultiTenancyServlet.doPost(MultiTenancyServlet.java:193)
wdqs-updater_1     | 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
wdqs-updater_1     | 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
wdqs-updater_1     | 	at org.wikidata.query.rdf.blazegraph.throttling.ThrottlingFilter.doFilter(ThrottlingFilter.java:304)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
wdqs-updater_1     | 	at ch.qos.logback.classic.helpers.MDCInsertingServletFilter.doFilter(MDCInsertingServletFilter.java:49)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
wdqs-updater_1     | 	at org.wikidata.query.rdf.blazegraph.filters.ClientIPFilter.doFilter(ClientIPFilter.java:43)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
wdqs-updater_1     | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
wdqs-updater_1     | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
wdqs-updater_1     | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
wdqs-updater_1     | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
wdqs-updater_1     | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
wdqs-updater_1     | 	at org.eclipse.jetty.server.Server.handle(Server.java:497)
wdqs-updater_1     | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
wdqs-updater_1     | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
wdqs-updater_1     | 	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
wdqs-updater_1     | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
wdqs-updater_1     | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
wdqs-updater_1     | 	at java.lang.Thread.run(Thread.java:748)
wdqs-updater_1     | Caused by: java.lang.RuntimeException: off=0, len=558::namespace=wdq, timestamp=readOnly(1539696002531), readTime=readOnly(1539679638136)
wdqs-updater_1     | 	at com.bigdata.relation.locator.DefaultResourceLocator.locateResourceOn(DefaultResourceLocator.java:817)
wdqs-updater_1     | 	at com.bigdata.relation.locator.DefaultResourceLocator.locateResource(DefaultResourceLocator.java:586)
wdqs-updater_1     | 	at com.bigdata.relation.locator.DefaultResourceLocator.cacheMiss(DefaultResourceLocator.java:395)
wdqs-updater_1     | 	at com.bigdata.relation.locator.DefaultResourceLocator.locate(DefaultResourceLocator.java:347)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.<init>(BigdataSail.java:2068)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.BigdataSail$BigdataSailReadOnlyConnection.<init>(BigdataSail.java:5236)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.BigdataSail._getReadOnlyConnection(BigdataSail.java:1540)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.BigdataSail.getReadOnlyConnection(BigdataSail.java:1503)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.BigdataSailRepository.getReadOnlyConnection(BigdataSailRepository.java:140)
wdqs-updater_1     | 	at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:247)
wdqs-updater_1     | 	at com.bigdata.rdf.task.AbstractApiTask.getQueryConnection(AbstractApiTask.java:221)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:722)
wdqs-updater_1     | 	at com.bigdata.rdf.sail.webapp.QueryServlet$SparqlQueryTask.call(QueryServlet.java:671)
wdqs-updater_1     | 	at com.bigdata.rdf.task.ApiTaskForIndexManager.call(ApiTaskForIndexManager.java:68)
wdqs-updater_1     | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
wdqs-updater_1     | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
wdqs-updater_1     | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
wdqs-updater_1     | 	... 1 more
wdqs-updater_1     | Caused by: java.lang.RuntimeException: off=0, len=558
wdqs-updater_1     | 	at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:239)
wdqs-updater_1     | 	at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:207)
wdqs-updater_1     | 	at com.bigdata.sparse.ValueType.decode(ValueType.java:333)
wdqs-updater_1     | 	at com.bigdata.sparse.AbstractAtomicRowReadOrWrite.atomicRead(AbstractAtomicRowReadOrWrite.java:347)
wdqs-updater_1     | 	at com.bigdata.sparse.AbstractAtomicRowReadOrWrite.atomicRead(AbstractAtomicRowReadOrWrite.java:157)
wdqs-updater_1     | 	at com.bigdata.sparse.AtomicRowRead.apply(AtomicRowRead.java:98)
wdqs-updater_1     | 	at com.bigdata.sparse.AtomicRowRead.apply(AtomicRowRead.java:36)
wdqs-updater_1     | 	at com.bigdata.btree.AbstractBTree.submit(AbstractBTree.java:3263)
wdqs-updater_1     | 	at com.bigdata.sparse.SparseRowStore.read(SparseRowStore.java:537)
wdqs-updater_1     | 	at com.bigdata.sparse.SparseRowStore.read(SparseRowStore.java:420)
wdqs-updater_1     | 	at com.bigdata.relation.locator.DefaultResourceLocator.locateResourceOn(DefaultResourceLocator.java:807)
wdqs-updater_1     | 	... 17 more
wdqs-updater_1     | Caused by: com.bigdata.rdf.vocab.BaseVocabulary$VocabularyVersioningException
wdqs-updater_1     | 	at com.bigdata.rdf.vocab.BaseVocabulary.readVersion2(BaseVocabulary.java:680)
wdqs-updater_1     | 	at com.bigdata.rdf.vocab.BaseVocabulary.readExternal(BaseVocabulary.java:458)
wdqs-updater_1     | 	at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2116)
wdqs-updater_1     | 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2065)
wdqs-updater_1     | 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
wdqs-updater_1     | 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
wdqs-updater_1     | 	at com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:231)
wdqs-updater_1     | 	... 27 more
wdqs-updater_1     |
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.rdf.RdfRepository.execute(RdfRepository.java:726)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.rdf.RdfRepository.query(RdfRepository.java:679)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.rdf.RdfRepository.dateFromQuery(RdfRepository.java:750)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.rdf.RdfRepository.fetchLeftOffTime(RdfRepository.java:636)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.Update.buildRecentChangePollerChangeSource(Update.java:160)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.Update.buildChangeSource(Update.java:141)
wdqs-updater_1     | 	at org.wikidata.query.rdf.tool.Update.main(Update.java:65)

Oct 16 2018, 1:27 PM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.

Maybe more detail:
I changed the three occurrences of WIKIBASE_HOST in docker-composer.yml to "mixnmatch.wmflabs.org", then did stop, then up -d. It rebuild and startedthe wdqs containers, but the updater_1 keeps dying.

Oct 16 2018, 1:24 PM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Magnus added a comment to T207133: SPARQL service returns entities with broken URLs.

I did that (docker-compose stop / sudo docker-compose up -d).

Oct 16 2018, 1:19 PM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Magnus created T207133: SPARQL service returns entities with broken URLs.

Oct 16 2018, 8:07 AM · User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Magnus created T207132: Fulltext/ElasticSearch does not work.

Oct 16 2018, 8:03 AM · Wikidata, Wikibase-Containers

Magnus closed T207131: wikibase-docker_wdqs-frontend_1 stopped working as Resolved.

Oct 16 2018, 7:58 AM · Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Magnus added a comment to T207131: wikibase-docker_wdqs-frontend_1 stopped working.

OK, another sudo docker-compose up --no-build -d did it. Thanks.

Oct 16 2018, 7:58 AM · Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Magnus added a comment to T207131: wikibase-docker_wdqs-frontend_1 stopped working.

End of the logfile:

wdqs-frontend_1    | 10.68.21.68 - - [16/Oct/2018:07:17:15 +0000] "GET /proxy/wdqs/bigdata/namespace/wdq/sparql?query=prefix%20schema:%20%3Chttp://schema.org/%3E%20SELECT%20*%20WHERE%20%7B%3Chttp://www.wikidata.org%3E%20schema:dateModified%20?y%7D&nocache=25661236 HTTP/1.1" 502 173 "http://mixnmatch-query.wmflabs.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:63.0) Gecko/20100101 Firefox/63.0" "193.62.205.8"
wdqs-frontend_1    | 10.68.21.68 - - [16/Oct/2018:07:24:55 +0000] "GET /proxy/wdqs/bigdata/namespace/wdq/sparql?format=json&query=SELECT+%3Fq+%3FqLabel+%7B+%3Fq+wdt%3AP2+wd%3AQ2+.+SERVICE+wikibase%3Alabel+%7B+bd%3AserviceParam+wikibase%3Alanguage+%27%5BAUTO_LANGUAGE%5D%2Cen%27+%7D+%7D HTTP/1.1" 502 173 "-" "-" "172.16.1.251"
wdqs-frontend_1    | 10.68.21.68 - - [16/Oct/2018:07:24:58 +0000] "GET /proxy/wdqs/bigdata/namespace/wdq/sparql?format=json&query=SELECT+%3Fq+%3FqLabel+%7B+%3Fq+wdt%3AP2+wd%3AQ2+.+SERVICE+wikibase%3Alabel+%7B+bd%3AserviceParam+wikibase%3Alanguage+%27%5BAUTO_LANGUAGE%5D%2Cen%27+%7D+%7D HTTP/1.1" 502 173 "-" "-" "172.16.1.251"
wdqs-frontend_1    | 10.68.21.68 - - [16/Oct/2018:07:25:33 +0000] "GET /proxy/wdqs/bigdata/namespace/wdq/sparql?format=json&query=SELECT+%3Fq+%3FqLabel+%7B+%3Fq+wdt%3AP2+wd%3AQ2+.+SERVICE+wikibase%3Alabel+%7B+bd%3AserviceParam+wikibase%3Alanguage+%27%5BAUTO_LANGUAGE%5D%2Cen%27+%7D+%7D HTTP/1.1" 502 173 "-" "-" "172.16.1.251"
wdqs-frontend_1    | 10.68.21.68 - - [16/Oct/2018:07:25:43 +0000] "GET /proxy/wdqs/bigdata/namespace/wdq/sparql?format=json&query=SELECT+%3Fq+%3FqLabel+%7B+%3Fq+wdt%3AP2+wd%3AQ2+.+SERVICE+wikibase%3Alabel+%7B+bd%3AserviceParam+wikibase%3Alanguage+%27%5BAUTO_LANGUAGE%5D%2Cen%27+%7D+%7D HTTP/1.1" 502 173 "-" "-" "172.16.1.251"
wdqs-frontend_1    | 2018/10/16 07:26:38 [emerg] 1#1: host not found in upstream "wikibase.svc" in /etc/nginx/conf.d/default.conf:8
wdqs-frontend_1    | nginx: [emerg] host not found in upstream "wikibase.svc" in /etc/nginx/conf.d/default.conf:8

I'll try the proxy.

Oct 16 2018, 7:51 AM · Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Magnus created T207131: wikibase-docker_wdqs-frontend_1 stopped working.

Oct 16 2018, 7:37 AM · Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata, Wikibase-Containers

Oct 15 2018

Magnus added a comment to T156389: automatic sourcing of existing Wikidata statements (expanding an existing scripts).

Maybe also relevant: The data used for the tool that @Lydia_Pintscher linked to are avilable on Toolforge in the s53867__referee_p database (readable with any toolforge account), especially the statements table.

Oct 15 2018, 8:45 AM · Wikimedia-Hackathon-2018, Wikimedia-Hackathon-2017, Addwiki, patch-welcome, Wikidata

Magnus added a comment to T205060: Meeting in Ramsele with SVAR?.

The data in https://tools.wmflabs.org/mix-n-match/#/catalog/528 was a one-time import, and is not updated automatically. I can try and set up that I call an auto-scraper for that catalog if you like.

Oct 15 2018, 8:13 AM · WMSE-Riksarkivet-TORA, WMSE-Riksarkivet-NAD

Oct 13 2018

Magnus added a comment to T57755: Allow time values more precise than day.

In this context, I would also appreciate a pointer as to enabling second precision on my own wikibase installation.

Oct 13 2018, 10:19 AM · Wikidata, MediaWiki-extensions-WikibaseRepository

Oct 9 2018

Magnus added a comment to T205556: Missing labels on items (statements show item IDs instead of labels even though labels are defined).

To reproduce, I just click "random item" a few times, one will show up very quickly.

Oct 9 2018, 12:00 PM · Regression, User-Addshore, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Wikidata

Magnus added a comment to T206517: Sendmail not working. mail() function required in wikibase docker image.

Ah, I see a hackaround is on https://github.com/wmde/wikibase-docker/blob/master/quickstatements/README.md but still...

Oct 9 2018, 10:20 AM · MediaWiki-Containers, Wikibase-Containers, Wikidata

Magnus added a comment to T206517: Sendmail not working. mail() function required in wikibase docker image.

Addendum: Also added $wgEnableEmail=true; to LocalSettings.php, no joy.

Oct 9 2018, 10:10 AM · MediaWiki-Containers, Wikibase-Containers, Wikidata

Magnus created T206517: Sendmail not working. mail() function required in wikibase docker image.

Oct 9 2018, 10:10 AM · MediaWiki-Containers, Wikibase-Containers, Wikidata

Oct 3 2018

Magnus added a comment to T57755: Allow time values more precise than day.

Well, if we want to put the creation time of a file into a statement, and we have a (sub-)second-precision EXIF time, it seems silly to round it to a day...

Oct 3 2018, 7:55 PM · Wikidata, MediaWiki-extensions-WikibaseRepository

Magnus created T206162: Request creation of mix-n-match VPS project.

Oct 3 2018, 7:49 PM · Cloud-VPS (Project-requests)

Oct 2 2018

Magnus added a comment to T205876: Specific user subpage on dewiki shows a PHP fatal error timeout.

I can further limit the item number the bot accepts (though I have other people asking me to _increase_ it...).

Oct 2 2018, 8:43 AM · Wikidata, Performance, Wikimedia-General-or-Unknown

Sep 28 2018

Magnus added a comment to T205035: Discovery of source code for Wikibase tools not facilitated by the Wikidata live implementations.

Many of the tools in Hay's list have links to the source repo:
https://tools.wmflabs.org/hay/directory/?search=wikidata#/search/wikidata

Sep 28 2018, 12:48 PM · patch-welcome, Wikidata, Documentation, Federated-Wikibase-Workshops@NewYork-2018

Magnus added a comment to T205549: book2scroll is not working (due to jquery 404?).

I have fixed the jQuery issue, but the underlying problem is that this code is very old (as in Toolserver-old), and only works with some site naming schemes. Other languages don't work either for many titles.

Sep 28 2018, 7:39 AM · Tools, Wikisource

Sep 25 2018

Magnus committed R2010:e6f2205ec0b5: trying to re-run failed command for slow DB (authored by Magnus).

trying to re-run failed command for slow DB

Sep 25 2018, 10:26 AM

Magnus committed R2010:cc93b46c9e46: trying to re-run failed command for slow DB (authored by Magnus).

trying to re-run failed command for slow DB

Sep 25 2018, 10:26 AM

Sep 24 2018

Magnus added a comment to T205330: Adding a value to a statement prompts for a property.

@Sjoerddebruin: Happens logged out as well, browser cache cleared, so no scripts/gadgets.

Sep 24 2018, 8:35 PM · wikidata-tech-focus, Patch-For-Review, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), User-Addshore, Wikidata

Magnus added a comment to T205330: Adding a value to a statement prompts for a property.

Happens on the same page in Safari Version 12.0 (13606.2.11) / Mac.

Sep 24 2018, 8:25 PM · wikidata-tech-focus, Patch-For-Review, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), User-Addshore, Wikidata

Magnus added a comment to T205330: Adding a value to a statement prompts for a property.

Happens to me on https://www.wikidata.org/wiki/Q29201051 even after I logged out and cleared the browser cache.

Sep 24 2018, 8:24 PM · wikidata-tech-focus, Patch-For-Review, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), User-Addshore, Wikidata

Magnus added a comment to T205330: Adding a value to a statement prompts for a property.

The effect is persistent on specific items, e.g. https://www.wikidata.org/wiki/Q29201051

Sep 24 2018, 8:22 PM · wikidata-tech-focus, Patch-For-Review, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), User-Addshore, Wikidata

Magnus added a comment to T205330: Adding a value to a statement prompts for a property.

I am using Firefox 62/Mac

Sep 24 2018, 8:18 PM · wikidata-tech-focus, Patch-For-Review, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), User-Addshore, Wikidata

Magnus added a comment to T205330: Adding a value to a statement prompts for a property.

Hitting "Random item", I can reproduce the effect on roughly 1 in 3 pages. On any given item page, this effect shows either for no statement, or for all of them.

Sep 24 2018, 8:18 PM · wikidata-tech-focus, Patch-For-Review, Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), User-Addshore, Wikidata