Page MenuHomePhabricator
Create Task
Maniphest T422522

WE2.5.8 WDQS v2 technical build
Open, Needs TriagePublic
Actions

Description

If we start to build the decoupled architecture described in our design doc, we should be able incrementally deliver value throughout the quarter and stand by a service that is able to support the pilot cohort by FY27 Q1.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Update the k8s deployment tokens for wdqs namespacesoperations/puppetproduction+6 -14
Switch from 4 wdqs namespaces to 2operations/deployment-chartsmaster+2 -46
Enable higher resource limits for wdqs-* namespacesoperations/deployment-chartsmaster+88 -4
Move the definition of wdqs-v2 namespaces to the dse-k8s common valuesoperations/deployment-chartsmaster+6 -8
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpengmodenaT422522 WE2.5.8 WDQS v2 technical build
 OpenNoneT418595 WDQS Label service proxy performance tests for Qlever and Virtuoso
 OpenNoneT418805 [NEEDS GROOMING] Determine appropriate rate limits for new backend candidates
 OpenNoneT422179 Set up regular bulk data ingestion and indexing pipeline
 ResolvedlericksonT422675 Design the bulk ingestion/indexing pipeline for the new triplestore
 ResolvedlericksonT425035 data-reload: Set up Wikidata Platform team to use Ceph
 OpenNoneT425141 data-reload: preprocess the dump
 OpenNoneT425137 data-reload: set up RDF tables in Ceph, refactor DDL
 In ProgresstruegT425119 data-reload: implement a quality check step on the wikidata dump
 ResolvedlericksonT425036 data-reload: implement wikidata dump to Ceph S3
 OpenNoneT425466 data-reload: SPIKE: investigate Spark/Kubernetes/Ceph performance
 OpenNoneT425467 data-reload: Tune Spark params for the processing stage
 ResolvedbkingT427319 request: s3 access for wdqs::alternatives
 OpenNoneT428235 [NEEDS GROOMING] data-reload: create a QLever index using an ephemeral instance
 ResolvedtruegT428455 wdqs-backend-init: Support for S3 bucket sub-path
 ResolvedlericksonT427030 data-reload: get qlever index into S3, bypassing pipeline
 OpengmodenaT428237 data-reload: spark processing jobs need to be orchestrated in the wikidata dag.
 OpengmodenaT429248 data-reload: munge_dumps task reads TTL data from S3 and writes to HDFS
 OpengmodenaT429250 data-reload: implement gen_rev_map task
 OpengmodenaT429252 data-relaod: implement split_subgraphs task
 OpengmodenaT429253 data-reload: implement export_n3 task
 OpengmodenaT429254 data-reload: keep data pipeline tables DDL under version control
 ResolvedtruegT422151 wdqs-streaming-consumer: report update operation metrics
 ResolvedgmodenaT422118 wdqs-proxy: Use wmf java parent pom
 ResolvedtruegT422116 wdqs-proxy: Improved gitlab pipeline
 ResolvedlericksonT421201 wdqs-proxy: ECS logging
 OpenlericksonT421200 wdqs-proxy: event platform integration for query logs
 OpengmodenaT427787 wdqs: design and declare an event platform schema for WDQS v2 logs.
 OpenlericksonT429380 wdqs: integrate with eventgate
 ResolvedtruegT421199 wdqs-proxy: Query federation policy filter
 ResolvedtruegT424208 wdqs federation allow list: Create a new gitlab repository
 OpengmodenaT422056 wdqs: database node logs should be pushed to logstash
 ResolvedtruegT422057 rdf-streaming-consumer: Consider deployment simplification
 ResolvedtruegT417402 Split WDQS git repository
 ResolvedtruegT422053 rdf-streaming-updater: Stand-alone git repository and docker image creation
 ResolvedlericksonT422054 wdqs-streaming-consumer: Stand-alone git repository
 ResolvedtruegT423153 wdqs-common: Stand-alone git repository
 ResolvedtruegT423385 rdf-spark-tools: stand-alone git repository
 ResolvedlericksonT423104 wdqs-proxy: write protection (reject SPARQL update queries)
 DuplicateNoneT423166 [NEEDS GROOMING] Curate a list of representative queries using the label service and no other special features
 ResolvedBTracy-WMFT414847 Deep dive category service
 ResolvedgmodenaT424844 wdqs-proxy should integrate with prometheus
Restricted Task
 ResolvedtruegT424670 Qlever docker image for WDQS
 In ProgresstruegT425007 Helm chart for wdqs-qlever and wdqs-streaming-consumer
 OpenbrouberolT427348 S3 credentials for the WDQS K8S deployment
 ResolvedtruegT428880 wdqs-streaming-consumer: fix prefixes in lexemes query
 ResolvedBTullisT425653 Add the wdqs::alternative nodes to the dse-k8s clusters - with a taint to avoid normal jobs being scheduled
 ResolvedbrouberolT428036 Ceph-csi daemonset pods do not run on the dedicated wdws k8s workers
 OpenBTullisT423312 Q4:rack/setup/install dse-k8s-wdqs200[1-4] (formerly wdqs20[28-31])
 ResolvedJclark-ctrT423314 Q4:rack/setup/install dse-k8s-wdqs100[1-3] (formerly wdqs103[6-8])
 In ProgresstruegT420727 New k8s service request: wdqs-proxy
 In ProgresstruegT424338 Provide a helm chart and dse-k8s helmfiles for wdqs-proxy.
 OpentruegT428925 DNS entries for the WDQS deployments
 In ProgresstruegT429313 Ingress configuration for the wdqs(-next) DSE namespaces
 ResolvedgmodenaT424896 request: index wikidata-platform gitlab repos
 DuplicategmodenaT426187 database node metrics should be available in Prometheus
 ResolvedtruegT425702 WDQS backend init image which prepares the triple-store index
 ResolvedtruegT425636 Java openjdk 25 base production docker images
 ResolvedlericksonT424218 wdqs-proxy: Improved error handling apart from HTTP response codes
 ResolvedtruegT426748 wdqs-backend-health-monitor
 ResolvedtruegT427414 wdqs-backend-init: Init container needs to determine the most recent successful index to read from ceph S3
 OpenNoneT427785 wdqs-proxy: Allow the usage of the Qlever pathSearch SERVICE
 OpenNoneT424621 wdqs-streaming-consumer: Replace the JMX Prometheus Agent
 ResolvedtruegT428228 wdqs-streaming-consumer: Update the Prometheus JMX metric agent
 OpenNoneT424336 Document how truncated responses should be handled in WDQS v2
 ResolvedtruegT428236 wdqs-streaming-consumer: Expose the number of update events still to apply as a metric
 DuplicateNoneT429151 WDQS v2 subdomain for public endpoints
 OpenNoneT429150 WDQS v2 subdomain for public endpoints
 OpenNoneT429173 wdqs-backend-init: support for optional "files" section in the index metadata file

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 7 2026, 3:40 PM
DSantamaria added a subtask: T418595: WDQS Label service proxy performance tests for Qlever and Virtuoso.Apr 8 2026, 12:55 PM
DSantamaria added a subtask: T418805: [NEEDS GROOMING] Determine appropriate rate limits for new backend candidates.Apr 8 2026, 1:03 PM
DSantamaria added a subtask: T422179: Set up regular bulk data ingestion and indexing pipeline.Apr 8 2026, 1:16 PM
DSantamaria added a subtask: T422151: wdqs-streaming-consumer: report update operation metrics.
DSantamaria added a subtask: T422118: wdqs-proxy: Use wmf java parent pom.
DSantamaria added a subtask: T422116: wdqs-proxy: Improved gitlab pipeline.Apr 8 2026, 1:21 PM
DSantamaria added a subtask: T421201: wdqs-proxy: ECS logging.Apr 8 2026, 1:24 PM
trueg added a subtask: T421200: wdqs-proxy: event platform integration for query logs.Apr 8 2026, 1:25 PM
DSantamaria added a subtask: T421199: wdqs-proxy: Query federation policy filter.Apr 8 2026, 1:25 PM
trueg added a subtask: T422056: wdqs: database node logs should be pushed to logstash.Apr 8 2026, 1:26 PM
DSantamaria added a subtask: T422057: rdf-streaming-consumer: Consider deployment simplification.Apr 8 2026, 1:27 PM
trueg added a subtask: T417402: Split WDQS git repository.Apr 8 2026, 1:27 PM
gmodena added a subtask: T423104: wdqs-proxy: write protection (reject SPARQL update queries).Apr 13 2026, 11:31 AM
DSantamaria changed the status of subtask T422118: wdqs-proxy: Use wmf java parent pom from Open to In Progress.Apr 13 2026, 1:22 PM
lerickson changed the status of subtask T421201: wdqs-proxy: ECS logging from Open to In Progress.Apr 17 2026, 6:18 PM
trueg changed the status of subtask T421199: wdqs-proxy: Query federation policy filter from Open to In Progress.Apr 20 2026, 2:59 PM
lerickson mentioned this in T423166: [NEEDS GROOMING] Curate a list of representative queries using the label service and no other special features.Apr 20 2026, 9:57 PM
DSantamaria changed the status of subtask T422116: wdqs-proxy: Improved gitlab pipeline from Open to In Progress.Apr 21 2026, 2:46 PM
DSantamaria added a subtask: T423166: [NEEDS GROOMING] Curate a list of representative queries using the label service and no other special features.
DSantamaria added a subtask: T414847: Deep dive category service.
DSantamaria closed subtask T422116: wdqs-proxy: Improved gitlab pipeline as Resolved.Apr 21 2026, 2:49 PM
gmodena changed the status of subtask T423166: [NEEDS GROOMING] Curate a list of representative queries using the label service and no other special features from Open to Stalled.Apr 23 2026, 12:44 PM
AWesterinen-WMF created subtask T424304: [SPIKE] Design architecture for collection and rewriting of queries and comparison against different backends.Apr 24 2026, 4:07 AM
AWesterinen-WMF created subtask T424305: [SPIKE] Define SPARQL query characterization.Apr 24 2026, 4:15 AM
AWesterinen-WMF removed a subtask: T424304: [SPIKE] Design architecture for collection and rewriting of queries and comparison against different backends.Apr 24 2026, 4:19 AM
AWesterinen-WMF removed a subtask: T424305: [SPIKE] Define SPARQL query characterization.
trueg changed the status of subtask T422057: rdf-streaming-consumer: Consider deployment simplification from Open to In Progress.Apr 28 2026, 1:23 PM
BTracy-WMF closed subtask T422057: rdf-streaming-consumer: Consider deployment simplification as Resolved.Apr 29 2026, 1:15 PM
trueg closed subtask T421199: wdqs-proxy: Query federation policy filter as Resolved.Apr 29 2026, 2:12 PM
gmodena closed subtask T422118: wdqs-proxy: Use wmf java parent pom as Resolved.May 4 2026, 11:15 AM
gmodena added a subtask: T424844: wdqs-proxy should integrate with prometheus.May 4 2026, 12:27 PM
BTracy-WMF closed subtask T421201: wdqs-proxy: ECS logging as Resolved.May 4 2026, 8:10 PM
BTracy-WMF changed the status of subtask T417402: Split WDQS git repository from In Progress to Stalled.May 5 2026, 3:30 PM
BTracy-WMF edited projects, added: Wikidata Platform Team (Sprint 05 (2026/05/05)); removed: Wikidata Platform Team (Sprint 04 (2026/04/08)).
DSantamaria added a subtask: T425035: data-reload: Set up Wikidata Platform team to use Ceph.May 5 2026, 3:55 PM
DSantamaria added a subtask: Restricted Task.
DSantamaria added a subtask: T425141: data-reload: preprocess the dump.May 5 2026, 3:57 PM
DSantamaria added a subtask: T425137: data-reload: set up RDF tables in Ceph, refactor DDL.
DSantamaria added a subtask: T425119: data-reload: implement a quality check step on the wikidata dump.
DSantamaria added a subtask: T420727: New k8s service request: wdqs-proxy.
lerickson changed the status of subtask T425035: data-reload: Set up Wikidata Platform team to use Ceph from Open to In Progress.May 5 2026, 5:52 PM
lerickson added a subtask: T425466: data-reload: SPIKE: investigate Spark/Kubernetes/Ceph performance.May 5 2026, 6:26 PM
lerickson added a subtask: T425467: data-reload: Tune Spark params for the processing stage.
trueg changed the status of subtask T422151: wdqs-streaming-consumer: report update operation metrics from Open to In Progress.May 6 2026, 9:01 AM
gmodena changed the status of subtask T424844: wdqs-proxy should integrate with prometheus from Open to In Progress.May 6 2026, 10:51 AM
DSantamaria added a subtask: T424896: request: index wikidata-platform gitlab repos.May 6 2026, 3:06 PM
trueg closed subtask T417402: Split WDQS git repository as Resolved.May 6 2026, 3:16 PM
lerickson changed the status of subtask T423104: wdqs-proxy: write protection (reject SPARQL update queries) from Open to In Progress.May 11 2026, 6:36 PM
gmodena created subtask T426187: database node metrics should be available in Prometheus.May 13 2026, 12:11 PM
DSantamaria added a subtask: T425702: WDQS backend init image which prepares the triple-store index.May 13 2026, 1:06 PM
DSantamaria added a subtask: T425636: Java openjdk 25 base production docker images.May 13 2026, 1:10 PM
BTracy-WMF closed subtask T414847: Deep dive category service as Resolved.May 18 2026, 2:27 PM
DSantamaria closed subtask T425636: Java openjdk 25 base production docker images as Resolved.Wed, May 20, 1:21 PM
DSantamaria closed subtask T422151: wdqs-streaming-consumer: report update operation metrics as Resolved.
DSantamaria closed subtask T425702: WDQS backend init image which prepares the triple-store index as Resolved.
DSantamaria closed subtask T425035: data-reload: Set up Wikidata Platform team to use Ceph as Resolved.
DSantamaria closed subtask T424844: wdqs-proxy should integrate with prometheus as Resolved.Wed, May 20, 1:59 PM
DSantamaria closed subtask T423104: wdqs-proxy: write protection (reject SPARQL update queries) as Resolved.Wed, May 27, 3:09 PM
DSantamaria added a subtask: T424218: wdqs-proxy: Improved error handling apart from HTTP response codes.
DSantamaria added a subtask: T426748: wdqs-backend-health-monitor.Wed, May 27, 3:12 PM
lerickson changed the status of subtask T424218: wdqs-proxy: Improved error handling apart from HTTP response codes from Open to In Progress.Wed, May 27, 7:01 PM
DSantamaria added a subtask: T427414: wdqs-backend-init: Init container needs to determine the most recent successful index to read from ceph S3.Thu, May 28, 2:51 PM
DSantamaria closed subtask T427414: wdqs-backend-init: Init container needs to determine the most recent successful index to read from ceph S3 as Resolved.Tue, Jun 2, 1:49 PM
DSantamaria closed subtask T426748: wdqs-backend-health-monitor as Resolved.
DSantamaria added a subtask: T427030: data-reload: get qlever index into S3, bypassing pipeline.Tue, Jun 2, 1:54 PM
DSantamaria moved this task from Sprint 05 (2026/05/05) to Sprint 06 (2026/06/02) on the Wikidata Platform Team board.Tue, Jun 2, 3:19 PM
DSantamaria edited projects, added: Wikidata Platform Team (Sprint 06 (2026/06/02)); removed: Wikidata Platform Team (Sprint 05 (2026/05/05)).
DSantamaria added a subtask: T427785: wdqs-proxy: Allow the usage of the Qlever pathSearch SERVICE.Tue, Jun 2, 3:40 PM
DSantamaria added a subtask: T424621: wdqs-streaming-consumer: Replace the JMX Prometheus Agent.
DSantamaria added a subtask: T424336: Document how truncated responses should be handled in WDQS v2.Tue, Jun 2, 3:44 PM
gerritbot added a comment.Thu, Jun 4, 2:14 PM

Change #1297708 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Move the definition of wdqs-v2 namespaces to the dse-k8s common values

https://gerrit.wikimedia.org/r/1297708

gerritbot added a project: Patch-For-Review.Thu, Jun 4, 2:14 PM
gerritbot added a comment.Thu, Jun 4, 2:19 PM

Change #1297709 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Enable higher resource limites for wdqs-* namespaces

https://gerrit.wikimedia.org/r/1297709

gerritbot added a comment.Thu, Jun 4, 2:35 PM

Change #1297708 merged by jenkins-bot:

[operations/deployment-charts@master] Move the definition of wdqs-v2 namespaces to the dse-k8s common values

https://gerrit.wikimedia.org/r/1297708

gerritbot added a comment.Thu, Jun 4, 2:42 PM

Change #1297709 merged by jenkins-bot:

[operations/deployment-charts@master] Enable higher resource limits for wdqs-* namespaces

https://gerrit.wikimedia.org/r/1297709

Maintenance_bot removed a project: Patch-For-Review.Thu, Jun 4, 3:31 PM
trueg added a subtask: T428236: wdqs-streaming-consumer: Expose the number of update events still to apply as a metric.Fri, Jun 5, 9:17 AM
trueg changed the status of subtask T428236: wdqs-streaming-consumer: Expose the number of update events still to apply as a metric from Open to In Progress.Fri, Jun 5, 9:27 AM
gerritbot added a comment.Fri, Jun 5, 3:08 PM

Change #1298307 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Switch from 4 wdqs namespaces to 2

https://gerrit.wikimedia.org/r/1298307

gerritbot added a project: Patch-For-Review.Fri, Jun 5, 3:08 PM
gerritbot added a comment.Fri, Jun 5, 3:13 PM

Change #1298308 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update the k8s deployment tokens for wdqs namespaces

https://gerrit.wikimedia.org/r/1298308

gerritbot added a comment.Tue, Jun 9, 9:08 AM

Change #1298307 merged by jenkins-bot:

[operations/deployment-charts@master] Switch from 4 wdqs namespaces to 2

https://gerrit.wikimedia.org/r/1298307

gerritbot added a comment.Tue, Jun 9, 12:34 PM

Change #1298308 merged by Btullis:

[operations/puppet@production] Update the k8s deployment tokens for wdqs namespaces

https://gerrit.wikimedia.org/r/1298308

Maintenance_bot removed a project: Patch-For-Review.Tue, Jun 9, 1:32 PM
bking subscribed.Thu, Jun 11, 7:26 PM
bking mentioned this in T428960: WDQS on K8s: test common operations/failure scenarios.Thu, Jun 11, 8:22 PM
gmodena created subtask T429150: WDQS v2 subdomain for public endpoints.Mon, Jun 15, 10:02 AM
gmodena created subtask T429151: WDQS v2 subdomain for public endpoints.
trueg added a subtask: T429173: wdqs-backend-init: support for optional "files" section in the index metadata file.Mon, Jun 15, 1:24 PM
BTracy-WMF closed subtask T428236: wdqs-streaming-consumer: Expose the number of update events still to apply as a metric as Resolved.Tue, Jun 16, 12:48 PM
BTracy-WMF closed subtask T427030: data-reload: get qlever index into S3, bypassing pipeline as Resolved.Tue, Jun 16, 12:53 PM
BTracy-WMF closed subtask T424218: wdqs-proxy: Improved error handling apart from HTTP response codes as Resolved.Tue, Jun 16, 12:57 PM
trueg changed the status of subtask T420727: New k8s service request: wdqs-proxy from Open to In Progress.Wed, Jun 17, 9:53 AM
trueg changed the status of subtask T425119: data-reload: implement a quality check step on the wikidata dump from Open to In Progress.Wed, Jun 17, 2:28 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits