Expose wikidata query service LDF endpoint in a scalable and available way
Open, MediumPublic
Actions

Assigned To

None

Authored By

	Gehel
	Mar 23 2017, 7:29 PM

Description

T159574 has been fixed by sending all LDF traffic to a single server. This has obvious impacts on availability and scalability. This has to be addressed in some way.

The application has to present consistent responses to work correctly behind LVS. Any kind of client affinity can be put in place for optimisation, but not for correctness. It has to account for transient failures, routing quirks, ...

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T161240 Expose wikidata query service LDF endpoint in a scalable and available way
		Open		None	T256948 Evaluate using LinkedDataFragments/Server.js instead of custom blazegraph integration

Event Timeline

Gehel created this task.Mar 23 2017, 7:29 PM

Restricted Application added projects: Wikidata, Discovery-ARCHIVED. · View Herald TranscriptMar 23 2017, 7:29 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Gehel mentioned this in T159574: LDF endpoint ordering is not stable between servers when paging.Mar 23 2017, 7:29 PM

Smalyshev removed a project: Discovery-Wikidata-Query-Service-Sprint.Mar 23 2017, 8:02 PM

Smalyshev moved this task from Incoming to Need investigation on the Wikidata-Query-Service board.

The problem is as follows:

Blazegraph stores the triples in BTree. Btree does have an ordering, and it is stable, but what is stored are not strings but terms generated from strings, which seem to be identified and ordered by essentially running ID - i.e. Term(1), Term(2), etc.
Since different servers are loaded in different times from different dump/update combinations, same URIs/strings get different Term IDs - same URI may be Term(123) on one server and Term(3456) on another. This means order of the triples in indexes will be different on different servers.
When paginating through results, different page requests may be directed to different servers, thus creating completely wrong picture in the summary data.
Due to the way we do load balancing (IP-based kernel balancing) we can not ensure any request affinity, as the balancer does not even look inside the packets.
Using client IP for balancing (pretty much the only thing we have that is not inside the packet) is not possible since we have varnish in front of LVS, which means client IPs are always the ones from Varnish servers. We do have real client IPs, but the are also inside the packets, so LVS can't use them.

Lydia_Pintscher moved this task from incoming to monitoring on the Wikidata board.May 5 2017, 2:40 PM

Smalyshev moved this task from Need investigation to Operations/SRE on the Wikidata-Query-Service board.May 25 2017, 1:24 AM

Smalyshev triaged this task as Medium priority.Feb 12 2018, 8:02 AM

The recent crash of wdqs1004 (T188045) had an impact on the LDF service, which was hosted on wdqs1004 at the time of the crash. The LDF service has been manually routed to wdqs1005, but this raises the concern again of the stability of this service.