Seemingly, I clicked the Merge button! 🚀

QA is good in dev.

E2E integration tests pass in dv

In Snapshot, NDJSON files

Version 4.0 is in Kafka compacted topics

Working in dev - on-demand

I will pick up 7-9 Oct, after bulk ingestion has started. We'll see if it can be ready for the next SC ingestion

Closing this ticket, the root cause was due to old invalid Kafka messages in our dev backend. Thank you @cscott for helping me.

de.wikinews has 3 mismatches after a S3 cleanup.

anpwiki, these 5 have mismatches. I don't see any redirects, only edits that occurred within 5 minute period

I've found one issue with our setup, old s3 files are not getting cleaned in our articles folder. However, there is still some mismatch in older articles (from July and August) that get redirected to newer article names.

I reran our ingestion today and got these mismatches in https://da.wiktionary.org

We are seeing a mismatch in the w/rest.php/v1/revision/%s/html API calls

huwiki errors to flag to WMF:

12 rows (out of 661601) in huwiki snapshot have mismatching revision id in the HTML field.

Checked dewikinews snapshot after dev ingestion. There were 35+ mismatches before this fix in dewikinews. After the fix, only one document has a mismatching HTML rev id, but this seems to be from the snapshot service, see JSON duplicate below. Checking a larger project huwiki`

Latency for articleUpdate service. Mostly the GetRevisionHTML is causing the delay, but not very often

I'm going to test on a different set of projects: ['dawiktionary', 'dewikinews', 'huwiki', 'cswikibooks']

One error related to log message: "wmf api parsoid error", caused by RequestTimeout for this ZH page: https://zh.wikipedia.org/wiki/Template:Recent_changes_article_requests/list. (could be because it's a Template page?)

OK, will do

Probably best to move the last sub-task to the next ticket where we do the full bulk ingestion in PROD

Airflow logs in dev:

See screenshot of DLQ with CS Voyage message: