Page MenuHomePhabricator

Manage apt sources via puppet
Closed, ResolvedPublic

Description

/etc/apt/sources.list is currently set up by d-i, but not managed via puppet. This leaves room to all kinds of inconsistencies, e.g. some hosts are using external mirrors (e.g. multatuli is using ftp.nl.debian.org instead of mirrors.wikimedia.org) and crucial apt sources can even be omitted (e.g. planet2001 currently has the security apt sources uncommented (which made me notice this and file this task)).

apt.wikimedia.org and backports are already added via /etc/apt/sources.list.d/wikimedia.list and /etc/apt/sources.list.d/debian-backports.list (and optionally also the experimental component).

I think that at least on Debian systems we should also integrate mirrors.wikimedia.org, security.debian.org and jessie|stretch-updates via sub files in /etc/apt/sources.list.d and simply stub /etc/apt/sources.list with a comment like "# managed via puppetised sub files in /etc/apt/sources/list.d".

Opinions?

Event Timeline

apt::repository has a comment_old option that comments out the line in sources.list, and Apt::Repository[wikimedia] sets that to true, so this should be the case already. If it's not, it's probably a bug and not intentional.

(for esams, note that using ftp.nl is probably a better idea than mirrors.wikimedia.org due to network proximity -- but it doesn't matter much anyway)

Ottomata triaged this task as Low priority.
Ottomata added a subscriber: Ottomata.

+1 in general, but yeah, I think this should basically already be happening.

Perhaps manually installing our own sources.list via an ERb template would be better than commenting lines?

@MoritzMuehlenhoff, I'm just triaging, feel free to re-assign as needed.

Another case I found: ms-be2013-ms-be2021 were unable to install the systemd update that was released via stretch-updates and it turned that that stretch-updates was missing in /etc/apt/sources.list. Those were among the first stretch hosts, so this was probably a one-off issue in early installations only. I've fixed up the apt config on the affected systems.

Change 548230 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] d-i: add contrib component to d-i configuration

https://gerrit.wikimedia.org/r/548230

Change 548230 abandoned by Jbond:
d-i: add contrib component to d-i configuration

Reason:
moritz beat me to it

https://gerrit.wikimedia.org/r/548230

This task has been assigned to the same task owner for more than two years. Resetting task assignee due to inactivity, to decrease task cookie-licking and to get a slightly more realistic overview of plans. Please feel free to assign this task to yourself again if you still realistically work or plan to work on this task - it would be welcome!

For tips how to manage individual work in Phabricator (noisy notifications, lists of task, etc.), see https://phabricator.wikimedia.org/T228575#6237124 for available options.
(For the records, two emails were sent to assignee addresses before resetting assignees. See T228575 for more info and for potential feedback. Thanks!)

MoritzMuehlenhoff renamed this task from Manage apt sources via puppet? to Manage apt sources via puppet.Sep 11 2020, 9:08 AM
MoritzMuehlenhoff claimed this task.
MoritzMuehlenhoff raised the priority of this task from Low to Medium.

Change 626693 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Manage /etc/apt/sources.list via Puppet (WIP)

https://gerrit.wikimedia.org/r/626693

Mentioned in SAL (#wikimedia-operations) [2020-09-25T10:28:26Z] <moritzm> reimaging sretest1002 to validate puppetised sources.list with a new installation T158562

Mentioned in SAL (#wikimedia-operations) [2020-09-25T11:10:15Z] <moritzm> reimaging sretest1001 to validate puppetised sources.list with a new installation T158562

I did a test installation with the new setting as I had a hunch there would be issues in early install and turns out I was right: The installer writes out an initial /etc/apt/sources.list and the puppetised /etc/apt/sources.list does not contain apt.wikimedia.org (it's handled via /etc/apt/sources.list.d/wikimedia). Then there's some Puppet logic to comment out entries from /etc/apt/sources.list in the apt::repository define once the sub sources list is added and the current processing order installation ferm and librdkafka in the stock Debian versions instead of the customised ones from apt.wikimedia.org

But I think this can be fixed by making the puppetised sources.list depend on apt::repository{'wikimedia'}.

An alternative and more generic way would be to only apply the puppetised version once the installer is complete, the reimage script could write out a file like /var/lib/wmf-installation-complete (which could be useful for other purposes as well) and then the puppetised sources.list would depend on the presence of /var/lib/wmf-installation-complete. But I'm not too fond of that, as it creates special cases and from my PoV Puppet should behave identical independent of the current installation process.

Change 630179 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Have the puppetised sources.list depend on the wikimedia repository

https://gerrit.wikimedia.org/r/630179

Change 630179 merged by Muehlenhoff:
[operations/puppet@production] Have the puppetised sources.list depend on the wikimedia repository

https://gerrit.wikimedia.org/r/630179

Mentioned in SAL (#wikimedia-operations) [2020-09-28T13:19:15Z] <moritzm> reimaging sretest1001 to validate puppetised sources.list with a new installation T158562

I did a test installation with the new setting as I had a hunch there would be issues in early install and turns out I was right: The installer writes out an initial /etc/apt/sources.list and the puppetised /etc/apt/sources.list does not contain apt.wikimedia.org (it's handled via /etc/apt/sources.list.d/wikimedia). Then there's some Puppet logic to comment out entries from /etc/apt/sources.list in the apt::repository define once the sub sources list is added and the current processing order installation ferm and librdkafka in the stock Debian versions instead of the customised ones from apt.wikimedia.org

This is fixed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/630179, confirmed in a reimage of sretest1001.

Change 630785 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enabled managed sources.list for ulsfo

https://gerrit.wikimedia.org/r/630785

Change 630785 merged by Muehlenhoff:
[operations/puppet@production] Enabled managed sources.list for ulsfo

https://gerrit.wikimedia.org/r/630785

Change 630876 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] autoinstall: Also use mirrors.wikimedia.org for publi/esams

https://gerrit.wikimedia.org/r/630876

Change 630879 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enabled managed sources.list for esams/ulsfo

https://gerrit.wikimedia.org/r/630879

Change 630879 merged by Muehlenhoff:
[operations/puppet@production] Enabled managed sources.list for esams/eqsin

https://gerrit.wikimedia.org/r/630879

Change 631139 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enable managed sources.list for codfw

https://gerrit.wikimedia.org/r/631139

Change 631139 merged by Muehlenhoff:
[operations/puppet@production] Enable managed sources.list for codfw

https://gerrit.wikimedia.org/r/631139

Change 631396 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Enabled managed sources.list for all of production

https://gerrit.wikimedia.org/r/631396

Change 631396 merged by Muehlenhoff:
[operations/puppet@production] Enabled managed sources.list for all of production

https://gerrit.wikimedia.org/r/631396

Change 630876 merged by Muehlenhoff:
[operations/puppet@production] autoinstall: Also use mirrors.wikimedia.org for public/esams

https://gerrit.wikimedia.org/r/630876

/etc/apt/sources.list is managed by Puppet since a few weeks in production, closing the task (for Cloud VPS it's being considered to also enabled it in a separate task).