Investigate causes of Catalyst slowness
Open, Needs TriagePublic5 Estimated Story Points
Actions

Assigned To

Authored By

	thcipriani
	Tue, Feb 17, 6:59 PM

Description

Symptoms

CI jobs timing out waiting for response from Catalyst for 10 mins
Demo creation taking longer
Demo deletion taking longer

Possible problems

Catalyst API logs mention slow sql queries
K8s API bottleneck

Needs: go code profiling, sql query eval, checking traffic levels/crawlers, digging through logs during slowness, ...did we do apache bench tests?

Details

Related Changes in GitLab:

Title	Reference	Author	Source Branch	Dest Branch
batch Server-Sent Events when streaming logs	repos/test-platform/catalyst/catalyst-api!166	jnuche	T417689	main
use more relaxed SLOW SQL threshold when dropping env's DB schema	repos/test-platform/catalyst/catalyst-api!165	jnuche	T417689	main
tweak configuration values for DB connections	repos/test-platform/catalyst/catalyst-api!163	jnuche	T417689	main
reduce critical section in `environment.WaitForDeploymentToSucceed`	repos/test-platform/catalyst/catalyst-api!161	jnuche	T417689	main
remove `latestStatus` query param from Catalyst requests	repos/test-platform/catalyst/patchdemo!269	jnuche	T417689	main
set Deployment's revision history limit to 2	repos/test-platform/catalyst/ci-charts!119	jnuche	T417689	main
add User Agent when calling Gerrit	repos/test-platform/catalyst/patchdemo!268	jnuche	T417689	main
add explicit history max limit to all Helm deployments	repos/test-platform/catalyst/patchdemo!266	jnuche	T417689	main
add explicit history max limit to all Helm deployments	repos/test-platform/catalyst/catalyst-api!159	jnuche	T417689	main

Customize query in GitLab

Event Timeline

thcipriani created this task.Tue, Feb 17, 6:59 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Feb 17, 6:59 PM

jnuche claimed this task.Thu, Feb 19, 3:32 PM

jnuche moved this task from Backlog to In progress on the Catalyst (Luka Ijo Pimeja Jan) board.

@jnuche mentions helm has a bunch of old secrets: one path he's running down.

Some of the helm commands had started slowing to a crawl; as it turns out we had 912 secrets in the cat-env namespace. A majority of those were originating in helm history revisions. A lot of those revisions could be safely removed and after doing that helm commands became noticeably more responsive again. Plus env creation times have gone down again to the levels we had a few months ago:

helm's CLI sets a maximum of 10 revisions per release by default but the go library doesn't set a value. One of the envs had 246 live revisions:

sh.helm.release.v1.wiki-6c16706061-3362.v1     helm.sh/release.v1   1      56d
[...]
sh.helm.release.v1.wiki-6c16706061-3362.v246   helm.sh/release.v1   1      41h

We don't really rely on the revision history for our workflows and the impact on performance of each one seems to add up pretty quickly, so I'm setting a limit of 3 for our components all across the board.

jnuche opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-api/-/merge_requests/159

add explicit history max limit to all Helm deployments

jnuche merged https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-api/-/merge_requests/159

add explicit history max limit to all Helm deployments

Maintenance_bot removed a project: Patch-For-Review.Fri, Feb 20, 3:31 PM

jnuche opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/266

add explicit history max limit to all Helm deployments

jnuche merged https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/266

add explicit history max limit to all Helm deployments

Maintenance_bot removed a project: Patch-For-Review.Fri, Feb 20, 4:31 PM

jnuche opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/268

add User Agent when calling Gerrit

jnuche merged https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/268

add User Agent when calling Gerrit

Similarly to what happened with secrets, recreated envs have been leaving behind a significant number of replica sets behind:

kubectl -n cat-env get rs | grep 750c4d946d
wiki-750c4d946d-3895-mediawiki-66bccd4d77   0         0         0       15d
wiki-750c4d946d-3895-mediawiki-8669b898df   0         0         0       15d
wiki-750c4d946d-3895-mediawiki-668466c6b8   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-c556c45c5    0         0         0       14d
wiki-750c4d946d-3895-mediawiki-76c5fb6d87   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-6968d99c4d   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-6b5b86ff6f   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-68cd949987   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-75749d974b   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-5d46bb4dc5   0         0         0       14d
wiki-750c4d946d-3895-mediawiki-68c5994db9   1         1         1       11d

I'm going to proceed to remove many of those old replica sets + add configuration to prevent this for future envs. Note that the configuration will not be updated for existing envs, so these won't get the fix

jnuche opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/ci-charts/-/merge_requests/119

set Deployment's revision history limit to 2

jnuche merged https://gitlab.wikimedia.org/repos/test-platform/catalyst/ci-charts/-/merge_requests/119

set Deployment's revision history limit to 2

Maintenance_bot removed a project: Patch-For-Review.Mon, Feb 23, 3:31 PM

jnuche opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/269

remove latestStatus query param from Catalyst requests

jnuche merged https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/269

remove latestStatus query param from Catalyst requests

Maintenance_bot removed a project: Patch-For-Review.Tue, Feb 24, 12:30 PM

jnuche added projects: Essential-Work, Release-Engineering-Team (Doing 😎).Tue, Feb 24, 12:31 PM

jnuche opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-api/-/merge_requests/161