Page MenuHomePhabricator
Create Task
Maniphest T431253

[DE2.4.2] WDQS v2 Scaling
Open, Needs TriagePublic
Actions

Description

FY2627 Q1

If we apply the learnings from our pilot cohort migration and implement improvements not on the critical path to migration readiness, we will be able to scale our QLever implementation, prepare for general release to the community, and set our future goals for the migration.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
wdqs: pool backend on index loadGmodenaoperations/deployment-chartsmaster+6 -0
wdqs: Update docker image versions + config values for debuggingTruegoperations/deployment-chartsmaster+69 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpengmodenaT431253 [DE2.4.2] WDQS v2 Scaling
 StalledgmodenaT422179 Set up regular bulk data ingestion and indexing pipeline
 ResolvedlericksonT422675 Design the bulk ingestion/indexing pipeline for the new triplestore
 ResolvedlericksonT425035 data-reload: Set up Wikidata Platform team to use Ceph
 DeclinedNoneT425141 data-reload: preprocess the dump
 DeclinedNoneT425137 data-reload: set up RDF tables in Ceph, refactor DDL
 ResolvedtruegT425119 data-reload: implement a quality check step on the wikidata dump
 ResolvedtruegT430115 data-reload: Improved dump verification
 ResolvedlericksonT425036 data-reload: implement wikidata dump to Ceph S3
 DeclinedNoneT425467 data-reload: Tune Spark params for the processing stage
 ResolvedbkingT427319 request: s3 access for wdqs::alternatives
 ResolvedNoneT428235 [NEEDS GROOMING] data-reload: create a QLever index using an ephemeral instance
 ResolvedtruegT428455 wdqs-backend-init: Support for S3 bucket sub-path
 ResolvedlericksonT427030 data-reload: get qlever index into S3, bypassing pipeline
 ResolvedgmodenaT428237 data-reload: spark processing jobs need to be orchestrated in the wikidata dag.
 ResolvedgmodenaT429248 data-reload: munge_dumps task reads TTL data from S3 and writes to HDFS
 ResolvedgmodenaT429250 data-reload: implement gen_rev_map task
 ResolvedgmodenaT429252 data-relaod: implement split_subgraphs task
 ResolvedgmodenaT429253 data-reload: implement export_n3 task
 ResolvedgmodenaT429254 data-reload: keep data pipeline tables DDL under version control
 StalledNoneT435502 [NEEDS GROOMING] implement a data retention policy for the data pipeline
 ResolvedtruegT433138 data-reload: DAG is failing the integrity check
 ResolvedlericksonT421200 wdqs-proxy: event platform integration for query logs
 ResolvedgmodenaT427787 wdqs: design and declare an event platform schema for WDQS v2 logs.
 ResolvedlericksonT429380 wdqs: integrate with eventgate
 ResolvedlericksonT429407 wdqs-proxy: post events to event streams
 ResolvedlericksonT431482 wdqs-proxy: events posted to EventGate should contain the stream (configurable)
 ResolvedtruegT431796 sparql/query/2.0.0: deperecated forwarded_query from event schema
 ResolvedtruegT431483 wdqs-proxy: fill `meta` in events posted to EventGate
 ResolvedtruegT431480 Events: add top-level `dt` field to the event schema, and fill it.
 ResolvedtruegT431485 wdqs-proxy: fill http.client_ip in QueryEvents posted to EventGate
 ResolvedlericksonT431481 wdqs-proxy: events posted to EventGate should contain the graph name.
 ResolvedgmodenaT422056 wdqs: database node logs should be pushed to logstash
 ResolvedtruegT425007 Helm chart for wdqs-qlever and wdqs-streaming-consumer
 ResolvedtruegT424338 Provide a helm chart and dse-k8s helmfiles for wdqs-proxy.
 ResolvedtruegT428925 DNS entries for the WDQS deployments
 ResolvedtruegT429313 Ingress configuration for the wdqs(-next) DSE namespaces
 ResolvedbrouberolT427348 S3 credentials for the WDQS K8S deployment
 ResolvedtruegT428880 wdqs-streaming-consumer: fix prefixes in lexemes query
 ResolvedtruegT427785 wdqs-proxy: Allow the usage of the Qlever pathSearch SERVICE
 OpenNoneT429173 wdqs-backend-init: support for optional "files" section in the index metadata file
 ResolvedbrouberolT429150 WDQS v2 subdomain for public endpoints
 OpenNoneT431271 [SPIKE] understand, document and tune QLever memory behaviour when real time updates are applied.
 DuplicateNoneT431052 consumer: collect QLever memory allocation metrics with different batch sizes
 ResolvedtruegT431310 wdqs-proxy: disable query rewrite code path
Restricted Task
 ResolvedtruegT431390 We need the capability to build and publish qlever from source
 OpenNoneT428960 WDQS on K8s: test common operations/failure scenarios
 OpenNoneT431395 [SPIKE][NEEDS GROOMING] resource allocation for co-located wdqs-qlever instances
 ResolvedBTracy-WMFT431392 [SPIKE] evaluate TopoLVM viability as the storage for the indexing pipeline
 OpenNoneT431589 [NEEDS GROOMING] Setup qlever build instance on wdp-gitlab-runner
 OpenNoneT431690 [NEEDS GROOMING][SPIKE] what is required to enable distributed tracing on WDQSv2?
 ResolvedBUG REPORTlericksonT432336 [BUG] wdqs-proxy v0.4.0 fails to init EventGate client.
 ResolvedtruegT432348 wdqs v2: deploy wdqs-next
 OpenNoneT436018 [NEEDS GROOMING] Register WDQS v2 with AlertManager
 OpenNoneT436373 We need the capability to route Wikibase Quality Constraint Checker traffic to wdqs v2
 OpenNoneT436375 [NEEDS GROOMING] Deploy WDQS v2 internal releases
 OpenNoneT436768 [NEEDS GROOMING][SPIKE] Automate rollout gating and rollback for WDQS v2
 OpenNoneT433499 [NEEDS GROOMING] wdqs-backend-init: hardening

Event Timeline

gmodena created this task.Jul 6 2026, 8:48 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 6 2026, 8:48 AM
gmodena added a subtask: T422179: Set up regular bulk data ingestion and indexing pipeline.Jul 6 2026, 9:20 AM
gmodena added a subtask: T421200: wdqs-proxy: event platform integration for query logs.
gmodena added a subtask: T422056: wdqs: database node logs should be pushed to logstash.Jul 6 2026, 9:24 AM
gmodena added a subtask: T425007: Helm chart for wdqs-qlever and wdqs-streaming-consumer.Jul 6 2026, 9:35 AM
gmodena added a subtask: T424338: Provide a helm chart and dse-k8s helmfiles for wdqs-proxy..
gmodena added a subtask: T428925: DNS entries for the WDQS deployments.
gmodena added a subtask: T429313: Ingress configuration for the wdqs(-next) DSE namespaces.Jul 6 2026, 9:37 AM
gmodena added a subtask: T427785: wdqs-proxy: Allow the usage of the Qlever pathSearch SERVICE.
gmodena added a subtask: T429173: wdqs-backend-init: support for optional "files" section in the index metadata file.
gmodena added a subtask: T429150: WDQS v2 subdomain for public endpoints.Jul 6 2026, 9:39 AM
gmodena added a subtask: T424336: Document how truncated responses should be handled in WDQS v2.
gmodena created subtask T431271: [SPIKE] understand, document and tune QLever memory behaviour when real time updates are applied..Jul 6 2026, 10:52 AM
gmodena created subtask T431310: wdqs-proxy: disable query rewrite code path.Jul 6 2026, 2:28 PM
gmodena added a subtask: Restricted Task.Jul 7 2026, 8:31 AM
gmodena created subtask T431390: We need the capability to build and publish qlever from source.Jul 7 2026, 8:49 AM
gmodena created subtask T431392: [SPIKE] evaluate TopoLVM viability as the storage for the indexing pipeline.Jul 7 2026, 9:44 AM
gmodena created subtask T431395: [SPIKE][NEEDS GROOMING] resource allocation for co-located wdqs-qlever instances.Jul 7 2026, 9:51 AM
gmodena added a subtask: T428960: WDQS on K8s: test common operations/failure scenarios.Jul 7 2026, 10:00 AM
BTracy-WMF closed subtask T424338: Provide a helm chart and dse-k8s helmfiles for wdqs-proxy. as Resolved.Jul 7 2026, 1:36 PM
BTracy-WMF closed subtask T425007: Helm chart for wdqs-qlever and wdqs-streaming-consumer as Resolved.
BTracy-WMF closed subtask T429313: Ingress configuration for the wdqs(-next) DSE namespaces as Resolved.Jul 7 2026, 1:38 PM
BTracy-WMF mentioned this in T428097: [DE2.4.2] WDQS v2 Scaling.Jul 7 2026, 2:48 PM
BTracy-WMF closed subtask T428925: DNS entries for the WDQS deployments as Resolved.Jul 7 2026, 2:56 PM
gmodena created subtask T431589: [NEEDS GROOMING] Setup qlever build instance on wdp-gitlab-runner.Jul 8 2026, 3:00 PM
gmodena added a subtask: T431690: [NEEDS GROOMING][SPIKE] what is required to enable distributed tracing on WDQSv2?.Jul 9 2026, 11:30 AM
brouberol changed the status of subtask T429150: WDQS v2 subdomain for public endpoints from Open to In Progress.Jul 9 2026, 12:57 PM
trueg changed the status of subtask T431390: We need the capability to build and publish qlever from source from Open to In Progress.Jul 9 2026, 4:25 PM
gerritbot added a comment.Jul 10 2026, 8:33 AM

Change #1309597 had a related patch set uploaded (by Gmodena; author: Gmodena):

[operations/deployment-charts@master] wdqs: pool backend on index load

https://gerrit.wikimedia.org/r/1309597

gerritbot added a project: Patch-For-Review.Jul 10 2026, 8:33 AM
gerritbot added a comment.Jul 10 2026, 8:56 AM

Change #1309606 had a related patch set uploaded (by Trueg; author: Trueg):

[operations/deployment-charts@master] wdqs: Update docker image versions + config values for debugging

https://gerrit.wikimedia.org/r/1309606

gerritbot added a comment.Jul 10 2026, 9:04 AM

Change #1309606 merged by jenkins-bot:

[operations/deployment-charts@master] wdqs: Update docker image versions + config values for debugging

https://gerrit.wikimedia.org/r/1309606

gerritbot added a comment.Jul 10 2026, 9:32 AM

Change #1309597 merged by jenkins-bot:

[operations/deployment-charts@master] wdqs: pool backend on index load

https://gerrit.wikimedia.org/r/1309597

Maintenance_bot removed a project: Patch-For-Review.Jul 10 2026, 10:31 AM
trueg changed the status of subtask T431310: wdqs-proxy: disable query rewrite code path from Open to In Progress.Jul 13 2026, 12:15 PM
trueg changed the status of subtask T427785: wdqs-proxy: Allow the usage of the Qlever pathSearch SERVICE from Open to In Progress.Jul 13 2026, 1:34 PM
brouberol closed subtask T429150: WDQS v2 subdomain for public endpoints as Resolved.Jul 13 2026, 1:39 PM
ArthurTaylor subscribed.Jul 14 2026, 9:27 AM
gmodena changed the status of subtask T422179: Set up regular bulk data ingestion and indexing pipeline from Open to Stalled.Jul 15 2026, 11:01 AM
gmodena created subtask T432336: [BUG] wdqs-proxy v0.4.0 fails to init EventGate client..Jul 16 2026, 11:59 AM
gmodena created subtask T432348: wdqs v2: deploy wdqs-next.Jul 16 2026, 1:59 PM
ArthurTaylor mentioned this in T432585: Access to staging k8s `wikidata-query-gui` service instance during deploy.Jul 20 2026, 10:30 AM
trueg changed the status of subtask T432348: wdqs v2: deploy wdqs-next from Open to In Progress.Jul 21 2026, 8:34 AM
AudreyPenven_WMDE subscribed.Aug 3 2026, 10:09 AM
BTracy-WMF closed subtask T431310: wdqs-proxy: disable query rewrite code path as Resolved.Aug 3 2026, 8:25 PM
BTracy-WMF closed subtask T432348: wdqs v2: deploy wdqs-next as Resolved.
BTracy-WMF closed subtask T427785: wdqs-proxy: Allow the usage of the Qlever pathSearch SERVICE as Resolved.
BTracy-WMF closed subtask T432336: [BUG] wdqs-proxy v0.4.0 fails to init EventGate client. as Resolved.
BTracy-WMF closed subtask T431390: We need the capability to build and publish qlever from source as Resolved.Aug 3 2026, 8:28 PM
BTracy-WMF removed a subtask: T424336: Document how truncated responses should be handled in WDQS v2.Mon, Aug 17, 2:32 PM
BTracy-WMF closed subtask T431392: [SPIKE] evaluate TopoLVM viability as the storage for the indexing pipeline as Resolved.Tue, Aug 18, 8:00 PM
BTracy-WMF closed subtask T421200: wdqs-proxy: event platform integration for query logs as Resolved.
gmodena created subtask T436018: [NEEDS GROOMING] Register WDQS v2 with AlertManager.Tue, Aug 25, 8:25 PM
gmodena added a subtask: T436373: We need the capability to route Wikibase Quality Constraint Checker traffic to wdqs v2.Fri, Aug 28, 1:52 PM
gmodena created subtask T436375: [NEEDS GROOMING] Deploy WDQS v2 internal releases.Fri, Aug 28, 2:02 PM
gmodena created subtask T436768: [NEEDS GROOMING][SPIKE] Automate rollout gating and rollback for WDQS v2.Wed, Sep 2, 10:34 AM
gmodena closed subtask T422056: wdqs: database node logs should be pushed to logstash as Resolved.Wed, Sep 2, 11:25 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits