Page MenuHomePhabricator

Gerrit keeps returning 503 errors
Closed, ResolvedPublic

Description

I keep getting errors like

An error occurred
Error 503: upstream connect error or disconnect/reset before headers. reset reason: connection timeout
Endpoint: /changes/*~*/revisions/*/robotcomments

image.png (947×310 px, 41 KB)

This is also causing CI to fail:

https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-vendor-mysql-php83/40342/console

01:38:46 stderr: 'error: RPC failed; HTTP 503 curl 22 The requested URL returned error: 503

Event Timeline

Jelto claimed this task.
Jelto added a project: collaboration-services.
Jelto added subscribers: ABran-WMF, Jelto.

Hi, thanks for reporting the issue!

Traffic against Gerrit was increased for the last hours which overloaded the host.

We mitigated the issue and Gerrit should be more responsive again.

I'll resolve the task, please re-open if you see the issue again. Thank you!

Hi! I saw similar 503 errors after you resolved the task.

For patch Ie3b068 (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CheckUser/+/1300279):

https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83/84149/console:

00:06:58.752   stderr: 'remote: upstream connect error or disconnect/reset before headers. reset reason: connection timeout
00:06:58.752 fatal: unable to access 'https://gerrit.wikimedia.org/r/mediawiki/extensions/CheckUser/': The requested URL returned error: 503'

https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83-selenium/60431/console:

00:07:11.353   stderr: 'error: RPC failed; HTTP 503 curl 22 The requested URL returned error: 503
00:07:11.353 fatal: expected 'acknowledgments''

https://integration.wikimedia.org/ci/job/mwext-phan-vendor-php83/5277/console:

00:06:59.107   stderr: 'remote: upstream connect error or disconnect/reset before headers. reset reason: connection timeout
00:06:59.107 fatal: unable to access 'https://gerrit.wikimedia.org/r/mediawiki/extensions/FlaggedRevs/': The requested URL returned error: 503'

https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-vendor-mysql-php83/40562/console:

00:06:58.741   stderr: 'remote: upstream connect error or disconnect/reset before headers. reset reason: connection timeout
00:06:58.741 fatal: unable to access 'https://gerrit.wikimedia.org/r/mediawiki/extensions/CirrusSearch/': The requested URL returned error: 503'

https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php83/38961/console:

00:06:58.605   stderr: 'error: RPC failed; HTTP 503 curl 22 The requested URL returned error: 503
00:06:58.605 fatal: expected flush after ref listing'
00:06:58.904 Build step 'Execute shell' marked build as failure

Please let me know what you advise and if what I am seeing is related to this task. Thank you.