Page MenuHomePhabricator

Create a schema for fetch failures
Closed, ResolvedPublic5 Estimated Story Points

Description

When enriching the events the call to the MW API might fail in different ways. Even if we attempt retries there might be situations in which we have to abandon the process to avoid stalling the whole pipeline. In these cases we might have to record such problems in a "side-output". This side-output will produce events to a kafka topic and thus we have to design a schema for describing such events.

AC:

  • the schema defines all the required information to understand the cause of the failure and to eventually "reconcile" this item at a later time
  • a schema is added to schema.wikimedia.org

Related Objects

Event Timeline

Change 854572 had a related patch set uploaded (by DCausse; author: DCausse):

[schemas/event/primary@master] [WIP] cirrussearch: add fetch_failure schema

https://gerrit.wikimedia.org/r/854572

Change 858364 had a related patch set uploaded (by DCausse; author: DCausse):

[search/cirrus-streaming-updater@master] Convert and route fetch failures to a kafka sink

https://gerrit.wikimedia.org/r/858364

Change 858364 merged by jenkins-bot:

[search/cirrus-streaming-updater@master] Convert and route fetch failures to a kafka sink

https://gerrit.wikimedia.org/r/858364

Change 854572 merged by jenkins-bot:

[schemas/event/primary@master] cirrussearch: add fetch_failure schema

https://gerrit.wikimedia.org/r/854572

Change 960616 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[operations/mediawiki-config@master] add search update pipeline streams (update + fetch_error)

https://gerrit.wikimedia.org/r/960616

Change 961000 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@master] Add version suffix to mediawiki.cirrussearch_page_rerender

https://gerrit.wikimedia.org/r/961000

Change 961065 had a related patch set uploaded (by Gmodena; author: Gmodena):

[eventgate-wikimedia@master] blubber: bump schema repo version.

https://gerrit.wikimedia.org/r/961065

Change 961065 merged by jenkins-bot:

[eventgate-wikimedia@master] blubber: bump schema repo version.

https://gerrit.wikimedia.org/r/961065

Change 960616 merged by jenkins-bot:

[operations/mediawiki-config@master] add search update pipeline streams (update + fetch_error)

https://gerrit.wikimedia.org/r/960616

Mentioned in SAL (#wikimedia-operations) [2023-09-26T13:25:56Z] <lucaswerkmeister-wmde@deploy2002> Started scap: Backport for [[gerrit:960616|add search update pipeline streams (update + fetch_error) (T317609)]]

Mentioned in SAL (#wikimedia-operations) [2023-09-26T13:27:19Z] <lucaswerkmeister-wmde@deploy2002> pfischer and lucaswerkmeister-wmde: Backport for [[gerrit:960616|add search update pipeline streams (update + fetch_error) (T317609)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)

Mentioned in SAL (#wikimedia-operations) [2023-09-26T13:37:51Z] <lucaswerkmeister-wmde@deploy2002> Finished scap: Backport for [[gerrit:960616|add search update pipeline streams (update + fetch_error) (T317609)]] (duration: 11m 54s)

Change 961000 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Add version suffix to mediawiki.cirrussearch_page_rerender

https://gerrit.wikimedia.org/r/961000