mobileapps-periodic-test failing since 2019-12-04 01:00 UTC due to failing git fetches
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Mholloway
	Dec 4 2019, 1:51 PM

Description

Since 2019-12-04 01:00 UTC, the mobileapps-periodic-test CI job is failing due to a server 500 error when attempting to fetch the mobileapps repo from Phabricator.

20:03:04 + git fetch --quiet --depth 2 https://phabricator.wikimedia.org/diffusion/GMOA master
20:03:04 fatal: unable to access 'https://phabricator.wikimedia.org/diffusion/GMOA/': The requested URL returned error: 500

The timing of the first of these failures corresponds with the switchover described in T238956: switch prod Phabricator from phab1003 to phab1001.

Details

	Subject	Repo	Branch	Lines +/-
	jjb: [mobileapps-periodic-test] Point at gerrit, not diffusion	integration/config	master	+2 -2

Customize query in gerrit

Related Objects

Mentioned Here: T238956: switch prod Phabricator from phab1003 to phab1001

Event Timeline

• Mholloway created this task.Dec 4 2019, 1:51 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 4 2019, 1:51 PM

• Jhernandez added a project: Mobile-Content-Service.Dec 4 2019, 4:34 PM

• Jhernandez moved this task from Needs triage to Tracking on the Product-Infrastructure-Team-Backlog-Deprecated board.

• Jhernandez added a subscriber: MSantos.

• Mholloway edited projects, added Page Content Service; removed Mobile-Content-Service.Dec 4 2019, 4:43 PM

Something is wrong with this repo. I spent a couple of hours trying to figure this out and the issue comes down to this: shallow clone of the repo fails while a normal clone works fine.

Is there a reason why this is even cloning from phabricator at all? The CI job should probably be set up to clone from gerrit-mirror

Thanks for investigating, @mmodell. That's strange; a shallow clone from Gerrit works fine. In any case, I don't know of any specific reason this was set up to clone from Phabricator.

If the fix is any more involved than changing a clone URI, maybe it's best to wait until we've had a chance to reassess the usefulness of this CI job before sinking any more time into it.

It should be as simple as switching the url to gerrit. Generally I think it's a bad idea to have CI rely on phabricator which is sometimes out of date due to replication lag and phabricator's git hosting is essentially maintained as a best-effort service rather than a critical service that must have ~100% uptime.

I obviously can't comment on the value of this CI job.

tldr;

There is a real issue with phabricator that I am still working on, however, the fastest route to getting this job running again is to switch the url to gerrit.

In T239815#5713380, @mmodell wrote:

It should be as simple as switching the url to gerrit. Generally I think it's a bad idea to have CI rely on phabricator which is sometimes out of date due to replication lag and phabricator's git hosting is essentially maintained as a best-effort service rather than a critical service that must have ~100% uptime.

I obviously can't comment on the value of this CI job.

tldr;

There is a real issue with phabricator that I am still working on, however, the fastest route to getting this job running again is to switch the url to gerrit.

I can help with redirecting.

Change 554595 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] jjb: [mobileapps-periodic-test] Point at gerrit, not diffusion

https://gerrit.wikimedia.org/r/554595

gerritbot added a project: Patch-For-Review.Dec 4 2019, 7:21 PM

Jdforrester-WMF added a project: Release-Engineering-Team-TODO (201912).Dec 4 2019, 7:23 PM

Change 554595 merged by jenkins-bot:
[integration/config@master] jjb: [mobileapps-periodic-test] Point at gerrit, not diffusion

https://gerrit.wikimedia.org/r/554595