ApiEchoUnreadNotificationPages::getUnreadNotificationPagesFromForeign: Unexpected API response from {wiki}
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	Sgs
	Jun 29 2022, 7:21 AM

Description

Error

mwversion: 1.39.0-wmf.17
reqId: 8f9960d6-a6a6-433d-864d-7946934d9c95
Find reqId in Logstash

normalized_message

ApiEchoUnreadNotificationPages::getUnreadNotificationPagesFromForeign: Unexpected API response from {wiki}

exception.trace

Impact

~ 22 events in the last day

Notes

The data returned by the error log contains "code": "badtoken" in all events

"data": {
      "servedby": "mw1400",
      "docref": "См. https://ru.wikinews.org/w/api.php для использования API. Подпишитесь на электронную рассылку MediaWiki API на &lt;https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/&gt;, чтобы получать информацию о неподдерживаемых функциях и ломающих изменениях.",
      "errors": [
        {
          "code": "badtoken",
          "html": "Токен CentralAuth недействителен."
        }
      ]
    }

Some more context for the logged data rECHO36c65d8fa835: Log ApiEchoUnreadNotificationPages bad responses

Details

Request URL: https://commons.wikimedia.org/w/api.php?format=*&formatversion=*&errorformat=*&errorsuselocal=*&action=query&meta=*&unplimit=*&unpwikis=*

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		aaron	T88445 MediaWiki active/active datacenter investigation and work (tracking)
Resolved		Krinkle	T270223 FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking)
Resolved		tstarling	T267270 Determine multi-dc strategy for CentralAuth
Resolved		aaron	T278392 Storage solution for cross-datacenter tokens
Resolved	PRODUCTION ERROR	tstarling	T311590 ApiEchoUnreadNotificationPages::getUnreadNotificationPagesFromForeign: Unexpected API response from {wiki}

Event Timeline

Sgs created this task.Jun 29 2022, 7:21 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 29 2022, 7:22 AM

Sgs updated the task description. (Show Details)Jun 29 2022, 7:24 AM

Sgs moved this task from Inbox to Up next Sprint on the Growth-Team board.Jun 29 2022, 7:28 AM

MediaWiki-extensions-CentralAuth seems implicated.

@tstarling Hey, timing wise this seems to match up with https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/683465/, could yiu take a look?

taavi added a parent task: T278392: Storage solution for cross-datacenter tokens.Jun 29 2022, 1:30 PM

• Zabe subscribed.Jun 29 2022, 2:11 PM

thcipriani moved this task from Untriaged to Jun 2022 on the Wikimedia-production-error board.Jun 30 2022, 3:24 PM

A burst of these errors was an expected effect of the switchover to a new store, since the new store starts empty, we didn't copy it in from Redis. We considered that to be an acceptable degradation. Within a few hours the rates returned to the usual background level seen before the switchover, with 4 errors since June 30.

ApiEchoUnreadNotificationPages::getUnreadNotificationPagesFromForeign: Unexpected API response from {wiki}Closed, ResolvedPublicPRODUCTION ERRORActions