The configuration mechanism in relforge has worked ok so far, but the fairly flat structure is limiting. Additionally our validation is relatively ad-hoc and doesn't cover all our cases. Something to consider might be yaml config files with jsonschema for validation. Using jsonschema removes some of the power user features of yaml, but those often lead to confusion anyways.

Some problems:

• Top level names are unclear, the set of tests to run is implied elsewhere
• For the tests to run in particular, a list of objects would make more sense
• It's hard to represent our nested ssh query (ssh to labs machine, vagrant ssh into MWV container, finally run search) and have everything escaped properly. Some nested structure that runs shell escaping at each level would be more obvious but hard to represent in the .ini format