We should export the triples from a production journal to try to understand where are the differences. To do this we need to copy a journal and run some tools provided by blazegraph.

Some numbers extracted from a dump:

• number of values: 20,659,551
• number of unique values: 11,028,526
• number of references: 60,078,314
• number of unique references: 58,876,057

References and values are identified by a hash computed over their properties. It is not a stable ID as it is always generated on the fly when extracting the entity data.
The current RDF projection makes it a resource that is referenced from other triples. The problem is that when some of the properties of the value/reference are changed the hash is changed and the hash will be different as well. Leaving potential orphans.
Other complexity is that the hashes may be used by other entities making the cleanup even harder since we to cleanup the values/refs only when they're never used elsewhere. This cleanup currently done in realtime whenever an entity is updated.
The query is as follow:

select ?s ?p ?o WHERE {
  VALUES ?s { list of values/references }
  # Since values are shared we can only clear the values on them when they are no longer used
  # anywhere else.

raising prio as we would like to deploy a new version today

In T239414#5702018, @Lucas_Werkmeister_WMDE wrote:

But in general, are these triples interesting in wdqs? Since they're never used as subject there are no way to use them directly, it seems the only thing that we can do is to display them (T173248).

I’m not sure what you mean, to be honest… but they’re used in some useful queries, e. g. heads of state and government (“end time” qualifier absent or unknown) or anonymous paintings (unknown creator). You can find more by searching for isBlank on Wikidata project pages (most results use !isBlank to filter out blank nodes, but there are some non-negated uses too).

unrelated but the doc seems incorrect, p:P2 should be used to link the reified statement ID not its value, wdt:P2 should be used for direct value links.
As to this particular issue I feel that it is slightly incorrect to use a different bnode since when this value will be discovered it cannot be different since it is the same thing.

@JAllemandou no, I moved this done too early and it was closed before being available.
The code is not yet deployed. I'll ping you once that is available.

Raising prio as the sole person of the team able to merge on this repo is absent :)

I'd need access as well, username: dcausse
Thanks!

Nothing is broken if we are OK having Special:EntityData/M123 linking back to the File page, the only thing that I don't know how to do is to have direct reference to a particular statement (like the statement id anchor on wikibase entity (e.g. https://www.wikidata.org/wiki/Q1#Q1$8983b0ea-4a9c-0902-c0db-785db33f767c)

Problem is that Special:EntityData does not support displaying the content of non main slot, it seems to just redirect to the File page: http://commons.wikimedia.org/wiki/Special:EntityData/M1916 which is not particularly useful because these Concept URI links simply link back to the same File page. I guess the question is more how to make Special:EntityData displays the structured the MediaInfo data in the browser. JSON/RDF/TTL format are supported for MediaInfo entities but I can't seem to find how to make it visible in html.
@Lucas_Werkmeister_WMDE, @Cparle what do you think Special:EntityData should do when given a MediaInfo entity?

Thanks!

We do already index terms in multiple ways but I don't think this is a good use of server resources to duplicate a field just for letting users to select how apostrophes should be handled. It's why I suggest to re-frame this ticket in a simple Actual behavior/Expected behavior form and point that apostrophe is the likely cause of this. Framed as it is at the moment it somewhat dictates a solution that we are unlikely to implement.

@Eevans I think this is the way forward, as for schemas elastic is promoting what they call ECS (elastic common schemas). This might be a good source of inspiration if we want to follow this route.

Adding a new preference is not really possible as this would mean we will index the data twice which we generally don't do for such narrow usecases.
When you say "simple search" do you mean running a full text search in the Quick Search input box? This again I'm afraid is not possible either, fulltext searches are too costly to be run on every character typed.
Concerning to meet the majority of users' expectations do you have any numbers or studies we could look at that support your statement?
About "Works like Google search" while I believe most users expect any search engine to be as smart and powerful as google, this statement is too vague to be useful in this ticket.

Indeed we used to display a timeout warning and it seems to have disappeared in this case, thanks for the report.
As for your particular usecase I'm afraid that if you don't provide more hints (filters) to the search engine it will have to scan all pages sequentially and will be as slow as running a grep on a dump. Once the timeout issue is resolved I suggest that we merge this task into T211824 as it is definitely a perf issue and I'm afraid the regex search mechanism does not have enough information to avoid a fullscan.

Tested sending an event to eventgate in beta generated by the new code:

curl -XPOST -H"Content-Type: application/json; charset=UTF-8" http://deployment-eventgate-3.deployment-prep.eqiad.wmflabs:8192/v1/events -d '[{"namespace":"ns123","query":"select","format":"xml","params":{"foo":"bar","baz":"bat,bats\\,\\\\qux"},"meta":{"id":"00000000-0000-0001-0000-000000000002","dt":"1970-01-01T00:00:00Z","request_id":"the_request_id","domain":"hostname.domain.local","stream":"mystream.name"},"http":{"method":"POST","clientIp":"10.1.2.3","request_headers":{"Accept":"application/json","X-Request-Id":"the_request_id","X-Custom":"one,two\\,t\\\\hree"},"status_code":200,"has_cookies":true},"backend_host":"hostname.domain.local","query_time":1000,"$schema":"/sparql/query/1.0.0"}]'

Not sure if related or need another ticket: related to the browsertests module (enabled when browser_tests is set to true when calling mediawiki::extension):