Per pairing session today, the above script needs a small bit of work to fetch the entire document. We'd use it to compare the pages flowing to relforge (via the cirrus streaming updater test) to the ES documents in production. We (as in SREs) should be able to handle this, so moving this to our column.

@EBernhardson @dcausse based on chatter in Wikimedia-Search , it seems like we're already past this point? Like, we're already doing these same tests in cloudelastic? Let us know if we still need to do this, feel free to close ticket if not.

I've run this a few times, it claims the indices in relforge match the ones in production. I'm still a bit suspicious that it passed on the first try, maybe could try harder to see what is broken. But we've done the testing and what we have so far claims to work.

@pfischer Wanted to ask you since Erik is out for the holidays...what is your level of confidence that the indices are created correctly? In other words, can we close this one or do we need to leave it open so we can continue testing?

Per conversation with @pfischer yesterday, here are the latest numbers. The mismatches have decreased by an order of magnitude since the last run.

I did have a few questions that didn't seem to be answered by the script output:

• What is the target amount of consistency, and
• How can we measure it?

For example, we might want more detail about revision mismatches. Which one is newer? How many revisions behind is the trailing index, and how does that compare to the rate of edits on the article?

Apologies if this is already being discussed, just a few thoughts I had from reading the script output.

We can also use Saneitizer on Cloudelastic to see how much it diverges from other clusters (will be tracked on another ticket)

We followed this data over time and it seemed to stay in line. We've now progressed from relforge to a cloudelastic deployment and we can probably consider this complete.