Per IRC conversation with @EBernhardson, some of our common operations depend a little too much on human vigilance and could benefit from some simple verification scripts. Some suggested improvements:

• OpenSearch plugins: Verify the plugins are installed on all instances after an update, see this comment for an example.
• Reindexing: we occasionally miss an index.
• config drift quorum failures in T400160 were caused by forgetting to remove settings that should've only been in place during a migration operation. We should monitor the cluster settings and warn when they deviate from what's set in Puppet for more than a few days.

Creating this ticket to:

• Create subtickets for specific operations
• Create verification scripts
• Document verification scripts on Wikitech, preferably as part of a runbook.
• Socialize these changes so Search Platform and Data Platform SRE are aware that the scripts exist.