Page MenuHomePhabricator

Containers on releases hosts cannot update apt cache from non-WMF sources
Closed, ResolvedPublic

Description

Access to http://security.debian.org is currently restricted on releases* (and other) hosts. This isn't an issue for the host itself as it has an apt proxy configured.

However, containers that run there do not have proxies configured and as a result cannot update their apt caches from security.debian.org during builds.

Event Timeline

Notes from #mw-on-k8s for one possible solution.

  1. add apt.proxy support to blubber
  2. add policy for our blubberoid instance to limit it to http://webproxy.eqiad.wmnet:8080
  3. have pipelinelib enforce default blubber config, in this case just the apt.proxy setting

Another option might just be to set an http_proxy environment variable in mediawiki-config's .pipeline/config.yaml. It seems apt-get will respect that for all of its http transfers. @Legoktm does that seem reasonable?

I could see this solution getting a bit cumbersome if we needed to build more projects on releases hosts, but as of now it's just the one, and my other proposal is a lot more work.

Another option might just be to set an http_proxy environment variable in mediawiki-config's .pipeline/config.yaml. It seems apt-get will respect that for all of its http transfers.

Ok, I just realized that you can't currently set environment variables for the build, only at runtime. Maybe this isn't viable.

Change 671199 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[blubber@master] apt: Support configuration of http/https proxies

https://gerrit.wikimedia.org/r/671199

Change 671211 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[integration/pipelinelib@master] Provide httpProxy context variables if a proxy is configured

https://gerrit.wikimedia.org/r/671211

Change 671211 merged by jenkins-bot:
[integration/pipelinelib@master] Provide setup.httpProxy context variable if a proxy is configured

https://gerrit.wikimedia.org/r/671211

Change 671199 merged by jenkins-bot:
[blubber@master] apt: Support configuration of http/https proxies

https://gerrit.wikimedia.org/r/671199

Change 673332 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[blubber@master] apt: Implement merging apt.proxies config

https://gerrit.wikimedia.org/r/673332

Change 673332 merged by jenkins-bot:
[blubber@master] apt: Implement merging apt.proxies config

https://gerrit.wikimedia.org/r/673332

Change 673375 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[operations/mediawiki-config@master] pipeline: Use build environment HTTP proxy for APT sources

https://gerrit.wikimedia.org/r/673375

Change 673375 merged by jenkins-bot:
[operations/mediawiki-config@master] pipeline: Use build environment HTTP proxy for APT sources

https://gerrit.wikimedia.org/r/673375

Mentioned in SAL (#wikimedia-operations) [2021-03-18T23:25:39Z] <dduvall@deploy1002> Synchronized .pipeline: config: [[gerrit:673375|Use build environment HTTP proxy for APT sources (T277109)]] (duration: 01m 02s)