Creating this ticket based on an IRC discussion between myself, @ssingh and @CDanis from earlier this year.
We (Search Platform SRE) have inadvertently created invalid pybal configuration via a Puppet patch at least twice. Pybal configuration is protected by a schema, so this does not break Pybal.
However, invalid Pybal config does set off errors such as ConfdResourceFailed: confd resource _srv_config-master_pybal_eqiad_search-https.toml . Additionally, it requires rolling back Puppet patches and other operationally expensive actions.
Since we already have a schema, it makes sense to use the schema in CI so we can catch these errors before they are live in production.
Creating this ticket to:
- Add pybal schema validation to CI
- Verify operation