Change Details

As a **WME Engineer** i want to synchronize ondemand state with snapshots state. At the moment they are out of sync. **Root Cause** On the 28th of April we merged a schema change that has impacted ondemand service avro deserialization. Our complex system of submodules, and lack of detection of out of sync….ondemand for the base product has been losing events since then. This means ondemand api for base product and structured contents is now out of sync with the snapshots. The issue doesn't impact structured contents, so the focus should be only on articles. **Solution** As discussed with team, the decision is to synchronize ondemand state only, by resetting ondemand consumer offsets and replay traffic. **TODO** [] scale ondemand services to 0 [] wait for consumer to go from STABLE to EMPTY state [] reset the consumer offsets, consumers: * ondemand-articles-compacted-v2 * ondemand-categories-compacted-v2 * ondemand-files-compacted-v2 * ondemand-templates-compacted-v2 [] scale ondemand to it's original scaling size [] Monitor logs for serialization errors, until consumer lag is down to 0 **Note** You can execute the procedure one service at a time, starting maybe by files. **Acceptance Criteria** * Consumer lag down to 0 or close * No errors during re-ingestion.

As a **WME Engineer** i want to synchronize ondemand state with snapshots state. At the moment they are out of sync. **Root Cause** A schema change has impacted ondemand service avro deserialization. Our complex system of submodules, and lack of detection of out of sync….ondemand for the base product has been losing events since then. This means ondemand api for base product and structured contents is now out of sync with the snapshots. The issue doesn't impact structured contents, so the focus should be only on articles. **Solution** As discussed with team, the decision is to synchronize ondemand state only, by resetting ondemand consumer offsets and replay traffic. **TODO** [] scale ondemand services to 0 [] wait for consumer to go from STABLE to EMPTY state [] reset the consumer offsets, consumers: * ondemand-articles-compacted-v2 * ondemand-categories-compacted-v2 * ondemand-files-compacted-v2 * ondemand-templates-compacted-v2 [] scale ondemand to it's original scaling size [] Monitor logs for serialization errors, until consumer lag is down to 0 **Note** You can execute the procedure one service at a time, starting maybe by files. **Acceptance Criteria** * Consumer lag down to 0 or close * No errors during re-ingestion.

As a **WME Engineer** i want to synchronize ondemand state with snapshots state. At the moment they are out of sync. **Root Cause** On the 28th of April we merged aA schema change that has impacted ondemand service avro deserialization. Our complex system of submodules, and lack of detection of out of sync….ondemand for the base product has been losing events since then. This means ondemand api for base product and structured contents is now out of sync with the snapshots. The issue doesn't impact structured contents, so the focus should be only on articles. **Solution** As discussed with team, the decision is to synchronize ondemand state only, by resetting ondemand consumer offsets and replay traffic. **TODO** [] scale ondemand services to 0 [] wait for consumer to go from STABLE to EMPTY state [] reset the consumer offsets, consumers: * ondemand-articles-compacted-v2 * ondemand-categories-compacted-v2 * ondemand-files-compacted-v2 * ondemand-templates-compacted-v2 [] scale ondemand to it's original scaling size [] Monitor logs for serialization errors, until consumer lag is down to 0 **Note** You can execute the procedure one service at a time, starting maybe by files. **Acceptance Criteria** * Consumer lag down to 0 or close * No errors during re-ingestion.