• inspect differences between the repo and https://github.com/judaicadh/wikibaseopenrefine to see if there are things worth porting

Looks like it has been created at https://gitlab.wikimedia.org/repos/wikibase-reconciliation via T424637. I'll add my patches there when it's been populated with source code.

That sounds like a great venue for folks to hack on the Wikidata reconciliation service, but it doesn't look like they support virtual participation. I see @Abbe98 and @Ainali on the participants list who are both familiar with OpenRefine (and @Abbe98 has contributed patches to the reconciliation service).

I think it would probably be better to host the repository for the source code of the service on https://gitlab.wikimedia.org/, as it would ease the involvement of the Wikimedia community for the maintenance of the service.

As @Pxxlhxslxn said, clarity is important. It sounds like @Loz.ross is willing to commit their organization to hosting the repo for the Wikibase reconciliation SOFTWARE, but it's not clear to me who, if anyone, is committing to support the Wikidata reconciliation SERVICE. Those are the people who will be in charge of customizing the home page to link to the appropriate issue tracker, customizing the User-Agent to include the correct contact info, etc as well as configuring and maintaining the various bits of ancillary network topology (load balancer, DNS settings, etc). If this were a dev ops team, there would probably be a separate repo with their Terraform/Puppet/Docker/whatever scripts specific to configuring that one instance.

That's a useful diagram. One thing it doesn't include is the current topology of two different reconciliation services, on different hosts, one of which is outside of WMCS, both fronted by a load balancer on the private server at https://wikidata.reconci.link/

It is hosted on a personal server, serving as a load balancer which forwards the queries to https://wikidata-reconciliation.wmcloud.org/ by default, and to its own local instance of the reconciliation when https://wikidata-reconciliation.wmcloud.org/ is not reachable.

The original incident which had been going on for "several days" according to users, resolved itself shortly after this ticket was created, but Wikidata appears to have turned off the spigot again yesterday, so things are broken again (and not in an intermittent fashion).

Looking longer term, I wonder if a shared multi-user unauthenticated service will continue to be feasible as Wikidata tightens the screws on rate limits. We may need to have the users authenticate in some way to get more permissive rate limits which would require changes to both the client (OpenRefine) and reconciliation service.

I don't know where the source repo for the deployed service lives (there seem to be three different candidates), but I had a brief look at this yesterday when it was erroring and I think there are bugs in both the reconciliation service and OpenRefine.

In T400119#11133180, @Tgr wrote:

In T400119#11133146, @tfmorris wrote:

Note that returning a plain text error message to a JSON API request is going to make it very likely to get swallowed by a JSON parse failure in the client library and never seen by the end user.

That seems like a bug in the client library? If the response code is 4xx/5xx and the content type isn't JSON, there is no reason to try to parse it as such.

Note that returning a plain text error message to a JSON API request is going to make it very likely to get swallowed by a JSON parse failure in the client library and never seen by the end user. This obscured the problem for OpenRefine users (and all other clients using the Wikidata Toolkit).

Ideally someone else would replicate the benchmark results

I'm surprised that this hasn't received any attention in 15 months. As an update to @Nikki 's numbers there are now on the order of 2.5 BILLION of these bot generated descriptions. The top 5 alone represent over 2 billion triples. That's a huge waste of resources!

@Manuel when you write:

I have a theory as to where a big chunk of the machine generated descriptions are from. They are the phrase "Wikimedia category" in hundreds of languages as a textual transcription of the triple instanceOf Q4167836. For example, Catégorie:Naissance à Seri Menanti has a single label in French and the P31 instanceOf claim which together occupy 802 bytes. Then two bots (Mr.Ibrahembot and Emijrpbot) came along and added another 11.5K (!) of static text (not even anything templated) in 129 languages, none of which have labels for the category.

Is triple count the only important parameter? It seems likely that the descriptions could be larger, on average, than labels.

How does one discover what the resolution was? (Apologies if this should be obvious, but I'm used to bug trackers which link the commits back to the issue.)

I vote for full URLs. Also, HTTPS URLs should probably be used throughout in preference to HTTP URLs to save naive clients from the extra latency of a redirect.

https://isa.toolforge.org/ and https://wikishootme.toolforge.org/ were also down about the same time (11:03 Eastern US).

I'm surprised that a private third party proxying such a significant segment of the traffic to Wikidata hasn't prompted the Wikidata Engineering team to take this more seriously.

In T240442#5851541, @Addshore wrote:

In T240442#5834866, @Ladsgroup wrote:

Very broad idea, feel free to discard, I think using industry-wide standards for throttling like token bucket, leaky bucket, fixed-window counter or sliding-window counter might help here.

One of the primary questions we need to answer is do we want to keep doing this client side self throttling, or switch to something more server side.

The documentation claims INCRBYFLOAT was introduced in Redis 2.6.0

If the manifest has to be constructed by hand, it seems like YAML would be a better format than JSON. They are equivalent from a structural and informational point of view, but YAML is much easier to edit without creating invalid documents.

The so-called "Freebase" dataset is actually a mix of data from Freebase and a bunch of URLs that were pulled from Google web crawls by an intern as potential "evidence." They don't have anything to do with the provenance of the data that was in Freebase, which was recorded for every item of data that was written there. Of course it would be silly to suggest a blacklisted site, but I don't believe the intern was provided with a blacklist ahead of time and as the blacklist was developed after the fact it hasn't been used to filter what's presented to users.

It would seem like the 2018-03-13 spreadsheet should be adequate to call this task complete. I would recommend including some qualitative understanding of the source of the Freebase data in addition to just pure curation ratio when making judgements about how to use which data. Things like MusicBrainz IDs and ISFDB IDs went through a heavily QA'd reconciliation process and are going to be high quality. Films, and to a lesser extent TV shows, were an area of focus for the Freebase team, so will generally be both high quality and relatively complete.

It seems bizarre that the utility of this is debated. The solution suggested by Bene sounds simple, straightforward, and useful.

I think these are likely two different bugs. Has anyone looked at either of them in the last 4 months?