Maniphest T425770

WDQS token bucket throttling logic should not apply to the the streaming-updater-consumer
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	gmodena
	May 8 2026, 11:49 AM

Project Tags

Referenced Files

None

Subscribers

View All 9 Subscribers

Description

WDQS Blazegraph applies token-bucket based throttling per (ip, UA) pairs.

Under service load, the streaming-updater-consumer client gets throttled (status code 429) too: https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-default-1-7.0.0-1-2026.05.08?id=qz_pBp4BIM7hdzulB9Wh.

An agents.txt file groups known clients into shared buckets. Requests that do not match any pattern fall back to a generic (ip, UA) bucket. The consumer updater UA is not currently included in agents.txt, but is treated as an unclassified client and still throttled.

We should handle the consumer IP (localhost) and User-Agent (default Jetty string?) more explicitly, and prioritize it over read-heavy query traffic.

Related.

T226471: WDQS bans its own monitoring due to bad user agent

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	throttling: exclude localhost from token buckets	Gmodena	wikidata/query/rdf	master	+77 -1

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		BTullis	T425758 Investigate rdf-streaming-updater consumer failures in eqiad
		Resolved		gmodena	T425770 WDQS token bucket throttling logic should not apply to the the streaming-updater-consumer

Event Timeline

gmodena created this task.May 8 2026, 11:49 AM

gmodena updated the task description. (Show Details)

Change #1285417 had a related patch set uploaded (by Gmodena; author: Gmodena):

[wikidata/query/rdf@master] throttling: exclude localhost from token buckets

https://gerrit.wikimedia.org/r/1285417

gerritbot added a project: Patch-For-Review.May 8 2026, 6:18 PM

gmodena changed the task status from Open to In Progress.May 8 2026, 6:44 PM

gmodena moved this task from Incoming to Sprint 05 (2026/05/05) on the Wikidata Platform Team board.

gmodena edited projects, added: Wikidata Platform Team (Sprint 05 (2026/05/05)); removed: Wikidata Platform Team.

gmodena moved this task from Backlog to In Code Review on the Wikidata Platform Team (Sprint 05 (2026/05/05)) board.

gmodena added a project: Essential-Work.

RKemper subscribed.May 8 2026, 7:13 PM

gmodena mentioned this in T425758: Investigate rdf-streaming-updater consumer failures in eqiad.May 11 2026, 3:15 PM

bking subscribed.May 11 2026, 3:50 PM

Gehel edited projects, added: Data-Platform-SRE (2026-04-24 - 2026-05-15); removed: Data-Platform-SRE.May 12 2026, 12:15 PM

Gehel moved this task from Backlog - project to Needs Review on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.

Change #1285417 merged by jenkins-bot:

[wikidata/query/rdf@master] throttling: exclude localhost from token buckets

https://gerrit.wikimedia.org/r/1285417

Maintenance_bot removed a project: Patch-For-Review.May 13 2026, 1:31 PM

gmodena moved this task from In Code Review to Needs Sign-off on the Wikidata Platform Team (Sprint 05 (2026/05/05)) board.May 19 2026, 8:57 AM

DSantamaria closed this task as Resolved.May 20 2026, 1:59 PM

Gehel moved this task from Needs Review to Done on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.May 22 2026, 9:36 AM

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.May 22 2026, 9:49 AM