| cjming | |
| Jul 2 2026, 5:55 PM |
| F93804049: Screenshot 2026-07-15 at 16.38.20.png | |
| Wed, Jul 15, 2:38 PM |
| F93762539: Screenshot 2026-07-15 at 12.37.21.png | |
| Wed, Jul 15, 10:37 AM |
| F93732701: Screenshot 2026-07-15 at 09.47.21.png | |
| Wed, Jul 15, 7:49 AM |
| Restricted File | |
| Wed, Jul 15, 7:46 AM |
| F93730420: ChatGPT-2026-07-15(1).html | |
| Wed, Jul 15, 7:41 AM |
| F91583217: Screenshot 2026-07-02 at 11.54.06 AM.png | |
| Jul 2 2026, 5:55 PM |
Experiment Platform team started noticing very sluggish load times for the GrowthBook staging server
Periodically, GrowthBook staging appears to throw the following error:
Slack thread for reference: https://wikimedia.slack.com/archives/C055QGPTC69/p1783008875606879
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Disable Checklist completion | repos/data-engineering/test-kitchen!371 | sfaci | disable-drafts_validation-by-default | main |
I've restarted the backend pod and things look better. I realized that was an issue due to the growthbook_next_ldap_sync DAG failing consistently since the slowness was observed.
Seems like the issue is still manifesting, even after the backend pod restart. Re-opening.
The PG master pod seems to be taking some time to delete
postgresql-growthbook-next-2 0/1 Terminating 0 46d
I'm sensing this pod was in a weird state, which could be linked to the issue we've been seeing.
I bounced the database pods as well as the ferretdb one as well, and the app seems to have recovered.
No more timeouts from Test Kitchen side when making API requests to GB since your last action. Looks good
Closing!
We've investigated yesterday with the help of a clanker. I'm attaching the full debugging session as a standalone html file, and will also attach a summary.
A PostgreSQL CPU saturation issue was investigated on a cluster used by FerretDB to provide a MongoDB-compatible interface for GrowthBook.
The investigation identified that the CPU load is caused by a high-concurrency workload against the MongoDB agendaJobs collection, specifically repeated findAndModify operations used by Agenda job workers.
The workload path is:
GrowthBook
|
| MongoDB protocol
v
FerretDB
|
| PostgreSQL calls
v
documentdb_api.find_and_modify()
|
v
documentdb_data.documents_50 (agendaJobs)PostgreSQL:
PostgreSQL 17.6 Debian package x86_64
Installed extensions:
documentdb_core 0.106-0 documentdb 0.106-0 pg_cron 1.6 vector 0.8.0 postgis 3.5.2 rum 1.3
The PostgreSQL activity view showed many concurrent queries:
SELECT p_result::bytea, p_success FROM documentdb_api.find_and_modify($1, $2::bytea)
Initially these appeared as idle connections, but during the CPU incident they were active:
Query state: active
Runtime: several seconds
No blocking transaction IDs
High concurrency
The active sessions were traced to FerretDB:
SELECT client_addr, application_name, count(*) FROM pg_stat_activity WHERE query LIKE '%find_and_modify%' GROUP BY client_addr, application_name;
Result:
client_addr application_name count ------------ ----------------- ----- 10.67.31.119 FerretDB 50
Conclusion:
There were 50 concurrent MongoDB-style findAndModify operations executing simultaneously.
The active sessions showed:
LWLock BufferMapping IPC BufferIo Timeout SpinDelay
No sessions showed PostgreSQL lock blocking:
pg_blocking_pids(pid) = {}Interpretation:
The physical DocumentDB table was identified:
SELECT * FROM documentdb_api_catalog.collections WHERE collection_id = 50;
Result:
database_name: app collection_name: agendaJobs collection_id: 50
The physical table:
documentdb_data.documents_50
maps to the MongoDB collection: agendaJobs
The logical collection contained:
SELECT count(*) FROM documentdb_data.documents_50;
Result:
159580 documents
The collection contains many Agenda job documents.
Example documents showed fields typical of Agenda:
nextRunAt priority lockedAt lastRunAt lastFinishedAt
The MongoDB catalog showed:
findAndLockNextJobIndex (name, nextRunAt, priority, lockedAt, disabled)
The physical PostgreSQL indexes were:
collection_pk_50 documents_rum_index_49 (name, nextRunAt, priority, lockedAt, disabled) documents_rum_index_79 (lastFinishedAt, nextRunAt)
Conclusion:
The expected Agenda index exists.
A missing-index problem is unlikely.
GrowthBook uses Agenda-style job scheduling.
Agenda workers repeatedly execute operations equivalent to:
With 50 concurrent workers:
Worker 1 -> findAndModify
Worker 2 -> findAndModify
Worker 3 -> findAndModify
...
Worker 50 -> findAndModify
All workers compete on the same:
collection
DocumentDB RUM index
frequently updated fields (lockedAt, lastRunAt, etc.)
This produces:
The first diagnostic and mitigation step is to reduce the number of concurrent workers.
Suggested test:
Expected outcome:
The value strongly suggests a configured pool or worker limit:
Possible sources:
Determine whether many jobs are eligible to run.
A large backlog causes all workers to continuously compete for jobs.
Check:
Agenda is designed around MongoDB atomic job locking behavior.
A high-concurrency job queue can be a challenging workload for a MongoDB compatibility layer backed by PostgreSQL.
If reducing concurrency is insufficient, consider:
The investigation points to:
*High-concurrency Agenda job locking through FerretDB and the Microsoft DocumentDB PostgreSQL extension causing contention on the agendaJobs collection.*
The evidence does not indicate:
The primary mitigation to test is reducing Agenda worker concurrency and observing the effect on PostgreSQL CPU and find_and_modify activity.
the concurrency limit set to 50 comes from https://github.com/FerretDB/FerretDB/blob/799235dab9e350655e72e65a0b24d849f7d68143/internal/documentdb/pool_new.go#L91
I created a port-forward + ssh-tunnel to be able to have access to the database from my laptop, on which I could install pymongo
brouberol@deploy2003:~$ kubectl port-forward ferretdb-ferretdb-growthbook-next-5569546cb7-zszmn 27017 Forwarding from 127.0.0.1:27017 -> 27017 Forwarding from [::1]:27017 -> 27017
~/wmf/airflow-devenv main *2 ❯ ssh -N deployment.eqiad.wmnet -L 127.0.0.1:27017:127.0.0.1:27017 Confirm user presence for key ED25519-SK SHA256:4qn2xf6Y3giE15e05wI9r00ORBgJ6NpxppW3ScPjjwQ User presence confirmed
>>> MONGO_URI = "mongodb://app:[REDACTED]@127.0.0.1:27017/app" >>> client = MongoClient(MONGO_URI) >>> db = client["app"] >>> collection = db["agendaJobs"] >>> collection.count_documents({}) 158909 >>> print("\nJob names:") ... for item in collection.aggregate([ ... {"$group": {"_id": "$name", "count": {"$sum": 1}}}, ... {"$sort": {"count": -1}} ... ]): ... print(item) Job names: {'_id': 'eventCreated', 'count': 158792} <---- {'_id': 'updateLicenses', 'count': 1} {'_id': 'queueExperimentUpdates', 'count': 1} {'_id': 'queueScheduledExperimentStatusUpdates', 'count': 1} {'_id': 'queueMetricUpdates', 'count': 1} {'_id': 'queueDashboardUpdates', 'count': 1} {'_id': 'queueScheduledHoldoutUpdates', 'count': 1} {'_id': 'deleteOldAgendaJobs', 'count': 1} {'_id': 'queueScheduledFeatureUpdates', 'count': 1} {'_id': 'queueAutoSliceUpdates', 'count': 1} {'_id': 'queueRampScheduleAdvances', 'count': 1} {'_id': 'queueSafeRolloutSnapshotUpdates', 'count': 1} {'_id': 'expireOldQueries', 'count': 1} >>> collection.find_one({'name': 'eventCreated'}) {'_id': ObjectId('6a4e3ddf3d9dab8e160369c0'), 'data': {'eventId': 'event-ab36c246-7f13-4746-8983-3851a3b3e170'}, 'name': 'eventCreated', 'priority': 0, 'shouldSaveResult': False, 'type': 'normal', 'nextRunAt': None, 'lastModifiedBy': None, 'lockedAt': None, 'lastRunAt': datetime.datetime(2026, 7, 8, 12, 19, 47, 611000), 'lastFinishedAt': datetime.datetime(2026, 7, 8, 12, 19, 47, 624000)}
These events don't seem to tell us much.
At that point I decided to attempt to clear that collection of all jobAgenda documents:
>>> collection.delete_many({'name': 'eventCreated'}) DeleteResult({'n': 158792, 'ok': 1.0}, acknowledged=True) >>> print("\nJob names:") ... for item in collection.aggregate([ ... {"$group": {"_id": "$name", "count": {"$sum": 1}}}, ... {"$sort": {"count": -1}} ... ]): ... print(item) ... Job names: {'_id': 'queueSafeRolloutSnapshotUpdates', 'count': 1} {'_id': 'updateLicenses', 'count': 1} {'_id': 'queueExperimentUpdates', 'count': 1} {'_id': 'queueScheduledExperimentStatusUpdates', 'count': 1} {'_id': 'queueMetricUpdates', 'count': 1} {'_id': 'queueDashboardUpdates', 'count': 1} {'_id': 'queueScheduledHoldoutUpdates', 'count': 1} {'_id': 'deleteOldAgendaJobs', 'count': 1} {'_id': 'queueScheduledFeatureUpdates', 'count': 1} {'_id': 'queueAutoSliceUpdates', 'count': 1} {'_id': 'queueRampScheduleAdvances', 'count': 1} {'_id': 'expireOldQueries', 'count': 1}
After a while, we could see new events trickling in:
Job names:
{'_id': 'eventCreated', 'count': 76}
{'_id': 'expireOldQueries', 'count': 1}
...
{'_id': 'updateLicenses', 'count': 1}These many jobs should have been deleted by https://github.com/growthbook/growthbook/blob/main/packages/back-end/src/jobs/deleteOldAgendaJobs.ts but somehow didn't.
This deletion had a drastic impact on the CPU load: {F93732437}
We're now ~24h later and we can see that the load increased/decreased many times and then was pegged back to 100%:
The number of agendaJobs crept back up
>>> collection.count_documents({'name': 'eventCreated'})
25744Now, I'd like to know whether these are considered pending or completed.
>>> collection.count_documents({'name': 'eventCreated', 'nextRunAt': {'$ne': None}})
5
>>> collection.count_documents({'name': 'eventCreated', 'nextRunAt': {'$eq': None}})
25795Only 5 seem to be pending, the others were completed.
Obviously, none of the events would be garbage collected by https://github.com/growthbook/growthbook/blob/main/packages/back-end/src/jobs/deleteOldAgendaJobs.ts because they were all created less than a week ago
>>> from datetime import datetime, timedelta >>> a_week_ago = datetime.now() - timedelta(days=7) >>> collection.count_documents({'lastFinishedAt': {'$lt': a_week_ago}, 'nextRunAt': None}) 0
Let's see if we're doing anything weird with the growthbook API
brouberol@deploy2003:~$ k logs --tail=10000 growthbook-backend-staging-7b9f7b67fd-dc5xv| jq '.req | "\(.method) \(.url)"' | sort | uniq -c | grep -v auth | sort -nr
Defaulted container "growthbook-backend-staging" out of: growthbook-backend-staging, growthbook-backend-staging-renew-kerberos-token, growthbook-backend-staging-tls-proxy
915 "GET /api/v1/experiments?limit=100&offset=0"
197 "POST /api/v1/experiments/exp_zu4cinmqzgayc6/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_zu4cinmqyeg69u/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_zu4cinmqs7ms43/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_zu4cinmqjjduyc/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_zu4cinmqjidy0i/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_zu4cinmqgns1qz/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_zu4cinmqgnnavk/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_zu4cinmqffiwy0/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_zu4cinmq9i29qu/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_zu4cinmq9bwjy9/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_zu4cinmq8poa7l/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_hq7mummr38kb6r/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_hq7mummr14yqjp/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_5sicznmplfjwj7/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_5sicznmpgxslpy/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_1momn7k1s2f/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_13xroyxdmmrf4byl6/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_13xroyxdmmrbx8uwb/start-checklist/manual/complete"
197 "POST /api/v1/experiments/exp_13xroyxdmmrarbasr/start-checklist/manual/complete"
30 "GET /user"
29 "OPTIONS /user"
25 "GET /sdk-connections"
25 "GET /experiments?includeArchived=&project=&type="
24 "GET /templates"
23 "OPTIONS /templates"
23 "OPTIONS /sdk-connections"
23 "OPTIONS /experiments?includeArchived=&project=&type="
23 "GET /revision/count"
23 "GET /organization/definitions"
23 "GET /organization"
23 "GET /holdout?includeArchived=&project="
23 "GET /experiments?includeArchived=&project=prj_org_42b6b1mmisylysh_demo-datasource-project&type="
23 "GET /experiments?includeArchived=&project=prj_2Cb9wHcLGmgGaxzr9RdPyu&type=standard"
22 "OPTIONS /revision/count"
22 "OPTIONS /organization/definitions"
22 "OPTIONS /organization"
22 "OPTIONS /holdout?includeArchived=&project="
22 "OPTIONS /experiments?includeArchived=&project=prj_org_42b6b1mmisylysh_demo-datasource-project&type="
22 "OPTIONS /experiments?includeArchived=&project=prj_2Cb9wHcLGmgGaxzr9RdPyu&type=standard"
13 "OPTIONS /user/history"
13 "OPTIONS /safe-rollout"
13 "OPTIONS /revision/feature?sparse=true"
13 "OPTIONS /organization/feature-exp-usage"
13 "GET /user/history"
13 "GET /safe-rollout"
13 "GET /revision/feature?sparse=true"
13 "GET /organization/feature-exp-usage"
7 "GET /api/v1/projects?limit=100&offset=0"
7 "GET /api/v1/members?limit=100&offset=0"These POST /api/v1/experiments/exp_xxxx/start-checklist/manual/complete jump at me.
The production growthbook instance display none of these issues atm. Let's inspect the size of their agendaJobs collection
>>> collection.count_documents({'name': 'eventCreated'})
455Looking at these backend log patterns, we're not seeing these POST /api/v1/experiments/exp_xxxxx/start-checklist/manual/complete" entries.
brouberol@deploy2003:~$ k logs --tail=10000 growthbook-backend-production-5f7c87f866-rwk8w| jq '.req | "\(.method) \(.url)"' | sort | uniq -c | grep -v auth | sort -nr
Defaulted container "growthbook-backend-production" out of: growthbook-backend-production, growthbook-backend-production-renew-kerberos-token, growthbook-backend-production-tls-proxy
1400 "GET /api/v1/experiments?limit=100&offset=0&status=running"
37 "null null"
12 "GET /api/v1/projects?limit=100&offset=0"
12 "GET /api/v1/members?limit=100&offset=0"
...@Sfaci I would like to disable whatever Test-Kitchen is doing on growthbook, to see if these numbers start growing again.
My current thinking is
So, I'd like to try 2 things:
My thinking is that we're calling the GB API assuming that it is idempotent, but calling it as much seems to create these agendaJobs faster than we can clear them up, which then causes 50 concurrent queries on ~the same document to be executed, which causes locking and contention, itself spiking the DB CPU usage (
Change #1311001 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] test-kitchen-next: temporarily disable the GB experiment validation feature
Change #1311001 merged by Brouberol:
[operations/deployment-charts@master] test-kitchen-next: temporarily disable the GB experiment validation feature
We have redeployed test-kitchen-next without the growthbook integration enabled, and have deleted the existing agendaJobs documents:
>>> collection.delete_many({'nextRunAt': None}) DeleteResult({'n': 2774, 'ok': 1.0}, acknowledged=True)
Now we keep an eye on the load of the postgresql-growthbook-next PG master.
We can see that TK was making multiple requests/s to Growthbook
Do we need that many queries?
2026-07-15 10:40:31.118063+00:00: 0 agendaJobs 2026-07-15 10:45:31.292799+00:00: 0 agendaJobs 2026-07-15 11:33:59.335492+00:00: 0 agendaJobs 2026-07-15 12:22:25.993956+00:00: 0 agendaJobs 2026-07-15 12:27:26.236828+00:00: 0 agendaJobs 2026-07-15 12:32:26.491785+00:00: 0 agendaJobs 2026-07-15 12:37:26.664893+00:00: 0 agendaJobs 2026-07-15 12:42:26.856274+00:00: 0 agendaJobs 2026-07-15 12:47:27.024309+00:00: 0 agendaJobs 2026-07-15 12:52:27.318257+00:00: 0 agendaJobs 2026-07-15 12:57:27.574369+00:00: 1 agendaJobs 2026-07-15 13:02:27.755714+00:00: 1 agendaJobs 2026-07-15 13:07:27.944903+00:00: 1 agendaJobs 2026-07-15 13:12:28.127571+00:00: 1 agendaJobs 2026-07-15 14:37:01.382294+00:00: 1 agendaJobs
I think we can conclude that we've identified the source of the load.
@Sfaci Is that ok if I re-assign the task to you now, to figure out how to avoid creating these events in the first place?
@Sfaci Is that ok if I re-assign the task to you now, to figure out how to avoid creating these events in the first place?
That's ok!
We are already discussing whether we should disable the feature that is causing the issue while waiting for a new feature in GrowthBook that will allow us to have inline validation. That would allow us to get rid of the one that is causing this. But in the meantime, let's keep this on our side while we decide what to do.
Thank you very much for all your investigation!!
sfaci opened https://gitlab.wikimedia.org/repos/data-engineering/test-kitchen/-/merge_requests/371
Disable Checklist completion
From our side, Experiment Platform team, we have already disabled the feature that caused this issue in both staging and production environments.
In addition to that, a MR has been pushed to deactivate the feature directly in the poller as a way to avoid that any of us enable it when testing something locally (running Test Kitchen UI locally) while using the GrowthBook instance we are running on staging
phuedx merged https://gitlab.wikimedia.org/repos/data-engineering/test-kitchen/-/merge_requests/371
Disable Checklist completion
Change #1311413 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):
[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.4.8 release to staging
Change #1311414 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):
[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.4.9 release to production
Change #1311413 merged by jenkins-bot:
[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.4.9 release to staging
Change #1311414 merged by jenkins-bot:
[operations/deployment-charts@master] Test Kitchen UI: Deploy v1.4.9 release to production