Summary
EventGate is rejecting mediawiki.api-request events with schema ValidationErrors because Extension:OAuth adds four context.oauth_consumer_* fields to MediaWiki's global logging context, which are then flattened onto the api-request event as unknown top-level properties. The schema /mediawiki/api/request/1.0.0 is additionalProperties: false, so every affected event is dropped.
This is the same failure mode as T411074 (context.ab_tests), now recurring with a different context.* field. T411074 was fixed by removing the field from the logging context and explicitly deferred "the correct fix".
These fields were added in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/OAuth/+/1308180
Impact
- Continuous loss of mediawiki.api-request events for OAuth-authenticated requests (e.g. bots such as InternetArchiveBot). Rejected events are diverted to eventgate-analytics.error.validation and never reach the primary stream / Hadoop.
- Steady rate ~0.3–0.5 errors/s on eqiad, with bursts to ~3.7/s during OAuth-bot activity.
- Fires the EventgateValidationErrors critical alert (data-engineering) whenever a burst pushes the 15m rate over the eventgate-analytics threshold of 1/s.
- From my understanding, if the change is going out with the deployment train, it will roll out to more wikis this week and may cause more validation errors and alerts.
Timeline (UTC)
- 2026-07-28 ~18:45 — onset, step to sustained ~0.45/s (eqiad only). Correlates with an OAuth-extension rollout — to confirm against SAL / train.
- 2026-07-29 00:05–00:10 — burst to ~3.7/s (OAuth-bot traffic); 15m rate crosses >1/s.
- 2026-07-29 00:06 — EventgateValidationErrors critical fires.
- 2026-07-29 ~00:15 — burst subsides; rate returns to ~0.3/s baseline; alert auto-resolves
- Ongoing — baseline ~0.3/s still present as of 02:30.
Evidence
Rejected event (from eventgate-analytics.error.validation, schema /error/2.1.0):
error_type: ValidationError errored_schema_uri: /mediawiki/api/request/1.0.0 errored_stream_name: mediawiki.api-request message: '' should NOT have additional properties (x4)
The offending event carried four flat, dotted top-level keys:
"context.oauth_consumer_id": "562", "context.oauth_consumer_key": "ad8e33572688dd300d2b726bee409f5d", "context.oauth_consumer_name": "InternetArchiveBot AUTH v2", "context.oauth_consumer_version": "1.0"
Each unknown property produces one '' should NOT have additional properties ('' = root).
Root cause
Extension:OAuth src/SessionProvider.php, sessionWasAttachedToRequest():
LoggerFactory::getContext()->add( [
'context.oauth_consumer_id' => (string)$consumer->getId(),
'context.oauth_consumer_key' => $consumer->getConsumerKey(),
'context.oauth_consumer_name' => $consumer->getName(),
'context.oauth_consumer_version' => $consumer->getVersion(),
] );This adds the fields to the global request logging context so they appear in Logstash. Per T411074, the api-request Monolog channel has an eventbus logger, so the entire logging context is injected into the event sent to EventGate (call chain: ApiMain::logRequest() -> wfDebugLog('api-request', ...) -> Monolog ContextProcessor -> LoggingContext::get()). The keys are copied verbatim as flat dotted top-level properties; the closed schema rejects them.
Solution
- If we want this data in data lake, the schema should be evolved
- If not, these fields may need to be moved elsewhere; OR the fields should be filtered out
References
- Prior identical incident & call-chain analysis: T411074
- Related: T409965 (experiment enrollment in the Action API)
- Runbook: https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate
- Dashboard: https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?var-service=eventgate-analytics
- Alert rule: operations/alerts team-data-engineering/eventgate_local.yaml
