debian-glue jobs have a 2 minutes delay when building against Jessie
Closed, DeclinedPublic
Actions

Assigned To

Authored By

	hashar
	Mar 12 2020, 8:25 AM

Description

labs/toollabs.git has a debian glue job for each of unstable, jessie, stretch and buster. The Jessie build has a two minutes delay:

debian-glue-unstable SUCCESS	44s
debian-glue-jessie SUCCESS	3m 11s
debian-glue-stretch SUCCESS	36s
debian-glue-buster SUCCESS	42s

https://gerrit.wikimedia.org/r/#/c/labs/toollabs/+/104917/

With relative time since start of build:

00:00:09.224 Hit http://mirrors.wikimedia.org jessie Release
00:00:10.949 Hit http://mirrors.wikimedia.org jessie/main amd64 Packages

00:02:07.465 Err http://security.debian.org jessie/updates InRelease
00:02:07.465 Err http://security.debian.org jessie/updates Release.gpg
00:02:07.465   Cannot initiate the connection to webproxy.eqiad.wmnet:8080 (2620:0:861:1:208:80:154:32). - connect (101: Network is unreachable) [IP: 2620:0:861:1:208:80:154:32 8080]

00:02:07.471 Reading package lists...
00:02:09.836 W: Failed to fetch http://security.debian.org/debian-security/dists/jessie/updates/InRelease  
00:02:09.837 
00:02:09.837 W: Failed to fetch http://security.debian.org/debian-security/dists/jessie/updates/Release.gpg  Cannot initiate the connection to webproxy.eqiad.wmnet:8080 (2620:0:861:1:208:80:154:32). - connect (101: Network is unreachable) [IP: 2620:0:861:1:208:80:154:32 8080]
00:02:09.837 
00:02:09.837 W: Some index files failed to download. They have been ignored, or old ones used instead.
00:02:09.839 I: user script /srv/pbuilder/build/cow.12869/tmp/hooks/D01security finished

I am tempted to think the webproxy connect failure is a red hearing and the root cause is jessie/updates is no more available.

Details

	Subject	Repo	Branch	Lines +/-
	layout: [labs/toollabs] Stop running the glue job for Jessie	integration/config	master	+0 -2
	package_builder: do not set webproxy by default	operations/puppet	production	+116 -6

Customize query in gerrit

Event Timeline

hashar created this task.Mar 12 2020, 8:25 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 12 2020, 8:25 AM

Ah it is more complicated. webproxy.eqiad.wmnet:8080 is not reachable from WMCS instances. The cause is that our pbuilderrc enables security updates for Jessie:

modules/package_builder/templates/pbuilderrc.erb

# Enable security backports by default on jessie
if [ "${DIST%-*}" = "jessie" ]; then
  SECURITY_UPDATES=${SECURITY_UPDATES:-"yes"}
else
  SECURITY_UPDATES=${SECURITY_UPDATES:-"no"}
fi
export SECURITY_UPDATES

And the D01Security hook then enable the proxy:

modules/package_builder/templates/D01security.erb

PROXY=http://webproxy.<%= @site %>.wmnet:8080
# buster does not have security updates yet
if [ "${SECURITY_UPDATES}" = "yes" -a "${DIST}" != "buster" ]; then
	cat > /etc/apt/sources.list.d/security.list <<-'EOF'
	deb http://security.debian.org/debian-security <%= @distribution %>/updates  main contrib non-free
	deb-src http://security.debian.org/debian-security <%= @distribution %>/updates main contrib non-free
EOF
	printf "Acquire::http::Proxy::security.debian.org \"${PROXY}\";\n" >> /etc/apt/apt.conf.d/01Proxy
	printf "Acquire::http::Proxy::security-cdn.debian.org \"${PROXY}\";\n" >> /etc/apt/apt.conf.d/01Proxy
	apt-get update
fi

Which causes the timeout :)

security updates should be disabled on Jessie since they are no more available?
the webproxy should not be enabled on WMCS instances

Change 579231 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] package_builder: do not set webproxy on WMCS

https://gerrit.wikimedia.org/r/579231

gerritbot added a project: Patch-For-Review.Mar 12 2020, 9:34 AM

hashar claimed this task.Mar 12 2020, 9:34 AM

hashar triaged this task as Medium priority.

I think we can probably drop the Jessie build for the labs/toollabs.git package. I don't think we need to be updating jsub for an OS we aren't running on the grid anymore. @Bstorm does that seem right to you?

If this is limited to just jessie, I 'd advise to not waste more time against this. There is no point, jessie is slated for removal pretty soon[1]

[1] https://wikitech.wikimedia.org/wiki/Operating_system_upgrade_policy

Change 579231 abandoned by Hashar:
package_builder: do not set webproxy by default

https://gerrit.wikimedia.org/r/579231

hashar closed this task as Declined.Mar 20 2020, 5:46 PM

Maintenance_bot removed a project: Patch-For-Review.Mar 20 2020, 6:10 PM

Change 594566 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Remove debian-glue-jessie

https://gerrit.wikimedia.org/r/594566

gerritbot added a project: Patch-For-Review.May 5 2020, 8:24 PM

Change 596260 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: [labs/toollabs] Stop running the glue job for Jessie

https://gerrit.wikimedia.org/r/596260

Change 596260 merged by jenkins-bot:
[integration/config@master] layout: [labs/toollabs] Stop running the glue job for Jessie

https://gerrit.wikimedia.org/r/596260

Mentioned in SAL (#wikimedia-releng) [2020-05-13T18:35:48Z] <James_F> Zuul: [labs/toollabs] Stop running the glue job for Jessie T247496

debian-glue jobs have a 2 minutes delay when building against JessieClosed, DeclinedPublicActions

Description

Details

Event Timeline

debian-glue jobs have a 2 minutes delay when building against Jessie
Closed, DeclinedPublic
Actions