Page MenuHomePhabricator

Build a black-box httpd testing framework
Open, Needs TriagePublic

Description

Our Apache configs are hard to understand, which makes it difficult to make changes with confidence that the result is as intended. An erroneous Apache config change can cause a global outage.

Currently, we use apache-fast-test to make a rough sanity check on a running httpd by sending it requests for a hardcoded list of URLs. Its effectiveness is limited, though: an SRE deploying a config change must run apache-fast-test manually after deploying to one host, and (because the tool is only configured with the list of URLs and no other information) must visually inspect the output and check that it looks correct.

The intended replacement, tentatively named httpbb (for "HTTP Black Box testing") will have a more expressive config: instead of a list of URLs, it's configured with a list of test cases. Each consists of a URL, and optionally other request parameters like arbitrary headers, followed by one or more assertions about the response (e.g. specific response codes; expected header values; a regular expression that the body text should match). httpbb will check each of those assertions against the actual response, emitting a pass-fail result. That will let us write a much more extensive list of test cases, building up much better test coverage of the full Apache config -- without making it more difficult to manually verify the results. In turn, that will let us make config changes with greater confidence, including simplifying refactors. Black-box testing also means the tests are agnostic to the underlying server technology, so they would also let us test for consistent behavior across reimages, upgrades and architecture changes.

In the initial workflow, an SRE will use httpbb manually in place of apache-fast-test: deploy a config change to one host, and run httpbb against that host. On a "pass" result, they'll continue with the deployment. In the long run, we may be able to automate that work away, and even use httpbb in CI for config changes.

Event Timeline

RLazarus created this task.Mon, Oct 28, 3:26 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Oct 28, 3:26 PM

Change 545689 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/software/httpbb@master] Initial version of httpbb, the HTTP black box testing tool.

https://gerrit.wikimedia.org/r/545689

elukey added a subscriber: elukey.Mon, Oct 28, 4:07 PM

Change 546661 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[integration/config@master] Add a new project for operations/software/httpbb.

https://gerrit.wikimedia.org/r/546661

Change 546661 merged by jenkins-bot:
[integration/config@master] Add a new project for operations/software/httpbb.

https://gerrit.wikimedia.org/r/546661

Mentioned in SAL (#wikimedia-releng) [2019-10-28T19:04:38Z] <James_F> Zuul: Add CI for operations/software/httpbb T236699

Dzahn added a subscriber: Dzahn.Wed, Oct 30, 1:31 AM

Change 545689 merged by jenkins-bot:
[operations/software/httpbb@master] Initial version of httpbb, the HTTP black box testing tool.

https://gerrit.wikimedia.org/r/545689

Change 548461 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] httpbb: Create a new Puppet module for httpbb.

https://gerrit.wikimedia.org/r/548461

Change 548461 merged by RLazarus:
[operations/puppet@production] httpbb: Create a new Puppet module for httpbb.

https://gerrit.wikimedia.org/r/548461

Change 549116 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] httpbb: Add httpbb module and baseurls test suite cribbed from apache-fast-test.

https://gerrit.wikimedia.org/r/549116

Change 549116 merged by RLazarus:
[operations/puppet@production] httpbb: Add httpbb module and baseurls test suite cribbed from apache-fast-test.

https://gerrit.wikimedia.org/r/549116

Change 550750 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] Install httpbb on cluster-management hosts.

https://gerrit.wikimedia.org/r/550750

Change 550750 merged by RLazarus:
[operations/puppet@production] Install httpbb on cluster-management hosts.

https://gerrit.wikimedia.org/r/550750

Change 550864 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/software/httpbb@master] Code against python3-attr 16.3.0-1, since that's what we have in stretch.

https://gerrit.wikimedia.org/r/550864

Change 550864 merged by jenkins-bot:
[operations/software/httpbb@master] Code against python3-attr 16.3.0-1, since that's what we have in stretch.

https://gerrit.wikimedia.org/r/550864

We use scap to deploy Netbox, a possible use-case would be to run httpbb as the last step to verify that apache is configured and working correctly and that the steps of the deploy completed correctly.

We use scap to deploy Netbox, a possible use-case would be to run httpbb as the last step to verify that apache is configured and working correctly and that the steps of the deploy completed correctly.

To be clear: The last step of the scap deploy, so a check in checks/ that does it.

Change 550877 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] httpbb: Avoid using flow mappings with URL keys.

https://gerrit.wikimedia.org/r/550877

Change 550877 merged by RLazarus:
[operations/puppet@production] httpbb: Avoid using flow mappings with URL keys.

https://gerrit.wikimedia.org/r/550877

CDanis added a subscriber: CDanis.Thu, Nov 14, 5:38 PM

Change 550927 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] httpbb: Correct baseurls.yaml for in-prod behavior.

https://gerrit.wikimedia.org/r/550927

Change 550927 merged by RLazarus:
[operations/puppet@production] httpbb: Correct baseurls.yaml for in-prod behavior.

https://gerrit.wikimedia.org/r/550927

Change 551249 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/puppet@production] httpbb: Install python3-requests-toolbelt.

https://gerrit.wikimedia.org/r/551249

Change 551250 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/software/httpbb@master] Verify SSL certs against the domain in the Host: header.

https://gerrit.wikimedia.org/r/551250

Change 551283 had a related patch set uploaded (by RLazarus; owner: RLazarus):
[operations/software/httpbb@master] Refactor the state shared between test cases into a TestHarness class.

https://gerrit.wikimedia.org/r/551283

jijiki moved this task from Backlog to Doing on the serviceops board.Wed, Nov 20, 6:54 PM