catalyst-daily-core's only two failures (build 22, build 26) are internal_api_error_DBQueryError from the shared envdb during API setup calls in page.js. Different envs, separate build/runs, so infrastructure flake rather than a code / test issue.
https://integration.wikimedia.org/ci/view/selenium-daily/job/catalyst-daily-core/26/console P94846
https://integration.wikimedia.org/ci/view/selenium-daily/job/catalyst-daily-core/22/console P94847
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| selenium: Retry transient API errors in MwApiHttpClient | Vaughn Walters | mediawiki/core | master | +9 -1 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| In Progress | vaughnwalters | T424411 [EPIC] Run selenium daily tests against a catalyst env (extensions with existing daily betacluster runs) | |||
| Resolved | vaughnwalters | T432171 Transient DBQueryError from Catalyst envdb fails catalyst-daily-core selenium tests |
Change #1310680 had a related patch set uploaded (by Vaughn Walters; author: Vaughn Walters):
[mediawiki/core@master] selenium: Retry transient API errors in MwApiHttpClient
Change #1310680 merged by jenkins-bot:
[mediawiki/core@master] selenium: Retry transient API errors in MwApiHttpClient
Derp... I was incorrect here, because this is core it doesn't need to wait for new package release. Moving back to In Validation to watch it for a week or so and see if this helps absorb any more transient errors that may occur before marking as done.
This has already caught one example of the DB error and worked succesfully on retry https://integration.wikimedia.org/ci/job/catalyst-daily-core/34/consoleText P94863
... [0-9] PASSED in chrome - file:///tests/selenium/docs/Stack/specs/mocha.js [0-1] [API] API Error: internal_api_error_DBQueryError: [e20b39fa2296734edc9c3a64] Exception caught: A database query error has occurred. This may indicate a bug in the software. [0-1] [API] Retrying request (attempt 2 of 3) [0-4] PASSED in chrome - file:///tests/selenium/specs/temporaryuseraccount.js ...
I am not sure of the underlying reason for this, but the fix is now proven to catch this on retry.
The error seems to be coming from the MW persistence layer. It may be flaky infrastructure, but could also be an actual issue in the code (e.g. a deadlock) — the error is too generic to be able to tell.
We can try to help with troubleshooting, but we would need some guidance on how to debug that part of MW code (at least I personally would)
Hey jaime since this workaround is functioning in catalyst-daily-core tests now, I marked this one as done, and for fixing the underlying bug I created T432284: Catalyst envdb intermittently returns DBQueryError to MediaWiki environments. I'm resolving this one since the workaround is, well... working around 😆 . But please feel free to merge these two ticket tho if you think that makes more sense - the other ticket is more comprehensive.