Page MenuHomePhabricator

[BLZMI] Set wbqc live on query-next.wikidata.org
Open, Needs TriagePublic

Description

This is a config change, affecting these settings:

  • wmgWBRepoSettingsSparqlEndpoint -> for wikidatawiki, set to the new endpoint
  • wgWBQualityConstraintsSparqlHasWikibaseSupport -> set to false. At time of writing, in wmf-config, this defaults to true. It is unclear whether we should change the default to false or to leave the default as-is and set only wikidatawiki to false. The answer depends on whether any other wikibases affected by this config are using Blazegraph. I assume it's only wikidata.org and this can safely default to false. We need to check that assumption.
  • wgWBQualityConstraintsAdditionalSparqlEndpoints - this is an array of other endpoints to query, and was added to handle the graph-split for scholarly articles. However, at time of writing, the default value is [ 'http://localhost:6042/sparql' ] and there is nothing set for wikidata specifically. It's unclear whether we should leave this config alone, or add an array with the scholarly articles endpoint.

Rollout plan addendum

This adds the deployment sequence, owners, and validation steps to the existing scope of T433710. It excludes the current WBQC code change.

Approach: switch scholarly first, validate it under production traffic, then switch main only if those checks pass.

Provisional date: September 29, 2026, subject to deployment-window and participant availability. All Berlin times below are CEST (UTC+2).

Owners:

  • WMDE: WBQC configuration, deployment, and constraint-result validation. Confirm the individual deployer.
  • WMF (Wikimedia Foundation): clusters, routing, capacity, and backend monitoring. Confirm the individual contact.

1. Prepare the clusters and routes - WMF

  • Prepare separate main and scholarly services behind the WDQS proxy, provisionally named wdqs-internal-main-next and wdqs-internal-scholarly-next.
  • Confirm the actual endpoint URLs, correct datasets, data freshness, and reachability from MediaWiki.
  • Confirm capacity for the expected traffic and keep the existing Blazegraph routes available for rollback.

Routing dependency: T436373. The names above are provisional, not confirmed endpoint URLs.

2. Prepare two configuration changes - WMDE

Prepare separate changes in operations/mediawiki-config, in wmf-config/InitialiseSettings.php, scoped to wikidatawiki:

StageSettingChange
ScholarlywgWBQualityConstraintsAdditionalSparqlEndpointsReplace the scholarly entry with the WMF-confirmed scholarly proxy URL; leave main on Blazegraph.
MainwgWBQualityConstraintsSparqlEndpointReplace the primary endpoint with the WMF-confirmed main proxy URL after scholarly passes validation.

Clarification to the original description: WBQC reads wgWBQualityConstraintsSparqlEndpoint, not wmgWBRepoSettingsSparqlEndpoint.

  • Confirm prefix injection for scholarly and the required wgWBQualityConstraintsSparqlHasWikibaseSupport setting. Disabling this flag also removes main's Blazegraph query hint, so validate the mixed configuration.
  • For main, configure WBQC to avoid Blazegraph-specific hints.
  • Record previous endpoint and flag settings for independent rollback of each stage. Preserve defaults and overrides for other wikis.

3. Prepare and deploy scholarly - WMDE + WMF

Proposed window: September 29, 09:00-11:00 Berlin time (CEST), 07:00-09:00 UTC.

  • Confirm the deployment window and both contacts' availability for monitoring and rollback.
  • Agree a shared coordination channel. Proposed: #wikimedia-operations on Libera IRC. Post monitoring findings, go/no-go decisions, and rollback updates there, not in Slack DMs.
  • Save baseline results for representative uniqueness checks: existing duplicates, no duplicates, qualifiers/references, no-value separators, deprecated statements, and recently changed duplicates.
  • Record baseline performance and agree rollback thresholds before deployment. No numeric thresholds have been agreed yet.
  • Test the scholarly config on debug servers, then deploy using scap backport <config-change>.

CI validates the patch; an authorized deployer initiates Scap and confirms progression from test servers to production. Merging alone does not switch traffic.

4. Validate scholarly before proceeding - WMDE + WMF

Observe for at least two hours after production deployment, longer if traffic is insufficient. If deployment completes at 11:00 CEST, the earliest main decision is 13:00 CEST.

  • Confirm scholarly traffic reaches the proxy and main remains on Blazegraph.
  • Purge selected items with action=purge, rerun checks, and compare constraint outcomes with the saved results.
  • Investigate every unexplained outcome difference, accounting for data freshness and result ordering/limits. Existing violations should remain visible.
  • Compare request success, timeouts, check completion, and latency with the baseline, using available metrics and sample checks. Confirm any missing monitoring with WMF.
  • Check backend load, queues, memory, data freshness, and logs for parsing errors, query timeouts, and restarts.
MonitoringOwner
WBQC dashboardWMDE
Existing WDQS dashboardWMDE + WMF
WDQS v2 dashboard, plus target-specific metrics supplied by WMFWMF
LogstashWMDE + WMF

Fewer warnings alone do not prove success: failed checks can be absent from cached warning lists. Zero unexplained differences applies to the validation sample, not every item in Wikidata.

5. If scholarly passes, repeat with main - WMDE + WMF

Proposed window: September 29, 13:00-15:00 CEST, 11:00-13:00 UTC.

  • Record the explicit go/no-go decision in the shared channel.
  • Proceed only if scholarly checks pass, WMF confirms main is ready for its workload, and both contacts remain available.
  • Test and deploy the separate main config change.
  • Repeat the checks above, adding type/subclass checks.
  • Monitor for at least two hours after deployment, provisionally through 17:00 CEST / 15:00 UTC. If deployment finishes later than 15:00 CEST, agree extended monitoring or defer main.

If scholarly needs further investigation, defer main. Do not shorten observation to finish both stages in one day.

Rollback and follow-up

If results differ unexpectedly or performance exceeds the thresholds agreed before deployment, revert the affected configuration stage. Normal rollback: scap backport --revert <config-change>.

If main fails, retain scholarly only if it remains healthy. After reverting, purge the selected validation items and verify recovery; assess whether additional targeted cache cleanup is needed. Do not routinely flush all constraint caches.

Record the outcome in T433710, name the monitoring contact, and review overnight traffic on September 30, 09:00-10:00 CEST / 07:00-08:00 UTC. A quiet observation period does not establish peak-load capacity.

Deployment and rollback procedure
Google Docs

Event Timeline