Adapt scap's testing strategy to mw-on-k8s
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Clement_Goubert
	Feb 21 2024, 4:22 PM

Description

scap does swagger checks on bare metal canaries. To replicate that for mw-on-k8s, we need to find a way to route requests to the canary releases directly while also keeping them as part of the normal traffic path.
Currently, canaries use routed_via: main

However, the bigger number of replicas used for canaries in mw-on-k8s compared to bare-metal means swagger checks would end up testing only one or two of the many canary pods. As such, we've decided to use httpbb to do more in-depth testing of mwdebug before proceeding to canary deployment, and rely on logstash error-rate detection for canary testing.

Details

	Subject	Repo	Branch	Lines +/-
	scap.cfg.erb: Set testservers_check_cmd_* in production	operations/puppet	production	+2 -3

Customize query in gerrit

Title	Reference	Author	Source Branch	Dest Branch
exp/files/php/scap.cfg: Set testservers_check_cmd_* for deploy container	repos/releng/train-dev!48	dancy	main-I3c22e6a1f9cd1195ac0c612b5d7573c010202779	main
Dockerfile.deploy: Add httpbb	repos/releng/train-dev!47	dancy	main-I67368cd5d0f00406bf4c35f62c1b1f423eb0493b	main
scap sync-world: Add support for testserver checks	repos/releng/scap!230	dancy	master-I617844a5319d104e8d9a8303dcf29f8ea34c05b7	master

Customize query in GitLab

Related Objects
Search...

Status	Assigned	Task
Stalled	None	T255792 Quibble runs core:unit tests twice!
Open	None	T328919 Upgrade to PHPUnit 10
Open	None	T338103 Micro-optimize ApiResult::isMetadataKey with str_starts_with once we support PHP8+
Open	None	T328921 Drop PHP 7.4 support from MediaWiki
Stalled	None	T334726 Use return type `never` in Wikibase
Open	None	T328922 Drop PHP 8.0 support from MediaWiki
Stalled	None	T319055 Upgrade to psr/container 2.x
Stalled	Krinkle	T319432 Migrate WMF production from PHP 7.4 to PHP 8.1
Open	None	T291916 Tracking task for Bullseye migrations in production
Stalled	None	T356293 Migrate MW appservers' base images to bullseye
Open	None	T290536 Serve production traffic via Kubernetes
Resolved	dancy	T357402 Scap should check errors coming from mw-on-k8s canaries during deployments
Resolved	Clement_Goubert	T358117 Adapt scap's testing strategy to mw-on-k8s

Event Timeline

Clement_Goubert triaged this task as Medium priority.Feb 21 2024, 4:22 PM

Clement_Goubert created this task.

Clement_Goubert moved this task from Incoming 🐫 to this.quarter 🍕 on the serviceops board.

Clement_Goubert updated the task description. (Show Details)Feb 21 2024, 4:29 PM

We've talked this over, and while doing swagger checks made sense when there were just a few canaries on bare-metal servers, these checks would now hit a random pod from ~3% of each deployment, making them of little potential value compared to the actual traffic being sent immediately to the canaries on deployment, the validity of which would be checked through the logstash error rate.

In T358117#9566949, @Clement_Goubert wrote:

We've talked this over, and while doing swagger checks made sense when there were just a few canaries on bare-metal servers, these checks would now hit a random pod from ~3% of each deployment, making them of little potential value compared to the actual traffic being sent immediately to the canaries on deployment, the validity of which would be checked through the logstash error rate.

I agree the checks don't provide much value in their current format—we know very quickly if something is wrong via the logstash checks on those canaries.

Also the swagger checks are only a subset of what httpbb (as I understand it) checks.

But a swagger/httpbb check would be useful if it first ran on a server without any production traffic. Then a deploy that immediately breaks all wikis would never serve any production traffic before we know that something is wrong.

Running httpbb against an mwdebug server before rolling to canaries would provide some additional safety to deploys that the swagger checks in their current format do not provide.

In T358117#9570020, @thcipriani wrote:

...
Running httpbb against an mwdebug server before rolling to canaries would provide some additional safety to deploys that the swagger checks in their current format do not provide.

Agreed, scap could run the httpbb appserver test suite on mwdebug.discovery.wmnet:4444 before proceeding instead of the swagger check. This would add about 15s.

so httpbb /srv/deployment/httpbb-tests/appserver/* --hosts=mwdebug.discovery.wmnet --https_port=4444 from the deploy server after deploying to mwdebug should work?

@dancy what do you think of ^ ?

In T358117#9572625, @thcipriani wrote:

so httpbb /srv/deployment/httpbb-tests/appserver/* --hosts=mwdebug.discovery.wmnet --https_port=4444 from the deploy server after deploying to mwdebug should work?

@dancy what do you think of ^ ?

Looks straightforward. I tested that command on the deploy server.

Yep, that's the check we also do hourly on all non-jobrunner releases.

Clement_Goubert renamed this task from Find a way to address canary releases directly to Adapt scap's testing strategy to mw-on-k8s.Feb 26 2024, 9:27 AM

Clement_Goubert changed the task status from Open to In Progress.

Clement_Goubert updated the task description. (Show Details)

dancy moved this task from Backlog to In progress on the Release-Engineering-Team (Now this 🫠) board.Feb 29 2024, 9:18 PM

dancy opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/230

Optionally run checks after deploying to testservers

Change 1007956 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] scap.cfg.erb: Set testservers_check_cmd to httppb in production

https://gerrit.wikimedia.org/r/1007956

@Clement_Goubert We have some questions:

Does mwdebug.discovery.wmnet resolve to a random bare-metal/k8s target?
Do you anticipate a case where you'd want to check only k8s testservers (e.g., when using scap sync-world --k8s-only), or only bare metal testservers? If so, what's the best way to achieve this?

In T358117#9598846, @dancy wrote:

@Clement_Goubert We have some questions:

Does mwdebug.discovery.wmnet resolve to a random bare-metal/k8s target?

It's an active/active discovery record, ~~so it resolves to the closest mw-debug deployment~~, k8s only.
Edit: Actually, it is active/active, but only pooled in the primary datacentre.

Do you anticipate a case where you'd want to check only k8s testservers (e.g., when using scap sync-world --k8s-only), or only bare metal testservers? If so, what's the best way to achieve this?

I don't really anticipate a case where I would want to check bare-metal specifically, but someone may. Testing only mwdebug.discovery.wmnet when doing --k8s-only would be a good idea, and would test only mw-on-k8s.