Page MenuHomePhabricator
Create Task
Maniphest T224576

Upgrade install servers to Buster
Closed, ResolvedPublic
Actions

Description

These act as TFTP/DHCP/boot servers, but also host apt.wikimedia.org

  • assign new IP addresses for replacement servers (private IPs, unlike before!)
  • create Ganeti VMs
  • install OS (buster) on new VMs
  • create new puppet role for installserver minus APT repo (role::installserver::light)
  • create new puppet role for APT repo minus DHCP/TFTP (role::apt_repo)
  • apply both puppet roles and check/fix all errors (add buster support)
  • rsync data APT repo data (done from install1002 to apt1001 and apt2001, added to cron jobs)
  • switch DNS CNAME for webproxy / switch squids to buster
  • switch DHCP_relay config on routers
  • switch TFTP server IPs in DHCP config
  • switch apt.wikimedia.org to the new backend apt1001 in DNS after making some tests
  • install1002.wikimedia.org replaced with install1003.eqiad.wmnet in all config files
  • install2002.wikimedia.org replaced with install2003.codfw.wmnet in all config files
  • remove old servers from DNS and decom fully with cookbook

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+0 -5Remove install[12]002 site.pp entries
operations/dnsmaster+0 -8remove install1002/install2002
operations/puppetproduction+0 -5apt_repo: stop including migration rsync class
operations/dnsmaster+2 -0add install.wikimedia.org CNAME to install1003
operations/puppetproduction+6 -6install_server: replace apt.wm.org with install.wm.org in autoinstall URLs
operations/puppetproduction+1 -2installserver: include preseed in apt role, not in light role
operations/puppetproduction+1 -1installserver::migration: fix rsync dir name
operations/puppetproduction+44 -0installserver: rsync home dir data for migration
operations/puppetproduction+5 -0Add s-nail to send mails after package imports
operations/puppetproduction+13 -9aptrepo: remove reprepro package on inactive servers
operations/dnsmaster+1 -1switch apt.wikimedia.org from install1002 to apt1001
operations/puppetproduction+35 -0aptrepo: add MOTD warning to not use old install servers for APT
operations/puppetproduction+1 -3hiera/apt.wikimedia.org: switch from install1002 to apt1001
operations/puppetproduction+1 -1adjust my .bash_profile to set base dir for reprepro on apt*
operations/puppetproduction+2 -0tftp,bastionhost: Add missing ensure_service hiera key
operations/puppetproduction+5 -18DHCP: remove old install servers and use new servers as next-server
operations/puppetproduction+6 -0tftp: if service is told to be stopped, do not monitor it
operations/puppetproduction+3 -1installserver/tftp: add missing ensure_service parameter
operations/puppetproduction+9 -3installserver: stop tftp service on old install servers
operations/puppetproduction+26 -26install_server: switch TFTP servers in DHCP to new install servers
operations/puppetproduction+1 -1stop DHCP service on jessie install servers
operations/puppetproduction+1 -1start DHCP service on new installserver::light servers
operations/homer/publicmaster+47 -47replace DCHP relays with new installservers
operations/puppetproduction+2 -1installserver: disable squid monitoring if service is absent
operations/puppetproduction+7 -3installserver: allow configuring squid as absent in Hiera
operations/puppetproduction+0 -1squid: remove obsolete hierarchy_stoplist config directive
operations/dnsmaster+5 -5switch webproxy CNAMEs to new install servers
operations/puppetproduction+4 -4hiera/aptrepo: rename install_server variables to aptrepo_server
operations/puppetproduction+2 -2switch install_server and failover in hieradata/common.yaml
operations/puppetproduction+4 -4DHCP: switch install1003/2003 to public IP, new VMs, update MACs
operations/puppetproduction+1 -1site: switch install1003/2003 to public IPs
operations/dnsmaster+8 -0re-add install1002/install2003 with public IPs
operations/dnsmaster+0 -8remove install1003/2003 again to recreate with public IPs
operations/puppetproduction+14 -1installserver: ensure interface for DHCP server is configured
operations/puppetproduction+0 -0site: add installserver::light role on new install servers
operations/puppetproduction+12 -6installserver: allow stopping DHCP server in Hiera, apply new role
operations/puppetproduction+19 -8installserver/apt: allow setting gpg_user different from reprepro user
operations/puppetproduction+7 -1installserver/apt_repo: add homedir parameter, move dirs to Hiera
operations/puppetproduction+2 -0aptrepo: puppetize gpg sec and pub keys for apt.wm.org
operations/puppetproduction+5 -1aptrepo: puppetize REPREPRO_BASE_DIR env variable
operations/puppetproduction+12 -3install_server: allow rsyncing from active to replacement servers
operations/dnsmaster+4 -4fix IP address for apt2001.wikimedia.org
operations/puppetproduction+1 -0add apt1001 to install_servers failover for rsyncing APT repo data
operations/puppetproduction+0 -3apt: remove (duplicate) OCSP stapling config and RSA cert
operations/puppetproduction+4 -0aptrepo: add support for buster, install python-apt package
operations/puppetproduction+1 -1site: add apt[12]001.wikimedia.org with role::apt_repo
operations/puppetproduction+10 -0DHCP: add apt1002 and apt2001
operations/puppetproduction+5 -0site: add apt1001/apt2001 with spare role
operations/puppetproduction+9 -6aptrepo/install: move https monitoring to aptrepo profile
operations/puppetproduction+16 -11installserver: allow for multiple failover servers at once
operations/puppetproduction+5 -5wmflib: replace install1002 with bast1002 in ipresolve_spec
operations/puppetproduction+11 -0introduce new role to install nginx and APT repo without DHCP/TFTP
operations/puppetproduction+8 -8installserver: create new role without HTTP/APT, rename existing role
operations/dnsmaster+1 -1switch apt.wikimedia.org from install1002 to install1003
operations/puppetproduction+5 -0site: add new install servers with private IP, spare role
operations/puppetproduction+0 -2install: remove next-server for new install servers for OS install
operations/puppetproduction+15 -0DHCP: add install1003/install2003, using current install servers
operations/dnsmaster+8 -0add private IPs for new install servers
operations/dnsmaster+8 -0add IP addresses for new install servers on buster
operations/puppetproduction+1 -1install_server: add install[12]003 to partman recipe regex
operations/puppetproduction+9 -0Create separate role for repository servers
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Dzahn mentioned this in T242602: Sort out plan for install* servers in edge sites.Mar 4 2020, 12:24 AM
gerritbot added a comment.Mar 4 2020, 5:12 PM

Change 572394 abandoned by Dzahn:
site: add installserver::light role on new install servers

Reason:
rebased to nothing as it was done in https://gerrit.wikimedia.org/r/c/operations/puppet/ /576460

https://gerrit.wikimedia.org/r/572394

gerritbot added a comment.Mar 4 2020, 6:24 PM

Change 576479 merged by Dzahn:
[operations/puppet@production] installserver: ensure interface for DHCP server is configured

https://gerrit.wikimedia.org/r/576479

gerritbot added a comment.Mar 6 2020, 6:36 PM

Change 577640 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] remove install1003/2003 again to recreate with public IPs

https://gerrit.wikimedia.org/r/577640

gerritbot added a comment.Mar 6 2020, 6:37 PM

Change 577641 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: switch install1003/2003 to public IPs

https://gerrit.wikimedia.org/r/577641

ops-monitoring-bot added a comment.Mar 6 2020, 6:46 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: install1003.eqiad.wmnet

  • install1003.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
ops-monitoring-bot added a comment.Mar 6 2020, 6:52 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: install2003.codfw.wmnet

  • install2003.codfw.wmnet (FAIL)
    • Host steps raised exception: Empty Management Password

ERROR: some step on some host failed, check the bolded items above

ops-monitoring-bot added a comment.Mar 6 2020, 6:54 PM

cookbooks.sre.hosts.decommission executed by dzahn@cumin1001 for hosts: install2003.codfw.wmnet

  • install2003.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
gerritbot added a comment.Mar 6 2020, 7:03 PM

Change 577640 merged by Dzahn:
[operations/dns@master] remove install1003/2003 again to recreate with public IPs

https://gerrit.wikimedia.org/r/577640

gerritbot added a comment.Mar 6 2020, 7:10 PM

Change 577649 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] re-add install1002/install2003 with public IPs

https://gerrit.wikimedia.org/r/577649

Dzahn mentioned this in T210038: upgrade install servers to stretch.Mar 6 2020, 7:21 PM
Dzahn merged a task: T210038: upgrade install servers to stretch.
Dzahn added a subscriber: Dzahn.
gerritbot added a comment.Mar 6 2020, 7:39 PM

Change 577649 merged by Dzahn:
[operations/dns@master] re-add install1002/install2003 with public IPs

https://gerrit.wikimedia.org/r/577649

Dzahn added a comment.Mar 6 2020, 7:55 PM

install1003.eqiad.wmnet and install2003.codfw.wmnet have been removed entirely with the decom cookbook.

DNS has been adjusted and then VMs with identical specs but public IPs have been created with the makevm cookbook.

install1003.wikimedia.org and install2003.wikimedia.org

gerritbot added a comment.Mar 6 2020, 8:00 PM

Change 577663 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] DHCP: switch install1003/2003 to public IP, new VMs, update MACs

https://gerrit.wikimedia.org/r/577663

gerritbot added a comment.Mar 6 2020, 8:01 PM

Change 577641 merged by Dzahn:
[operations/puppet@production] site: switch install1003/2003 to public IPs

https://gerrit.wikimedia.org/r/577641

gerritbot added a comment.Mar 6 2020, 8:06 PM

Change 577663 merged by Dzahn:
[operations/puppet@production] DHCP: switch install1003/2003 to public IP, new VMs, update MACs

https://gerrit.wikimedia.org/r/577663

Dzahn added a comment.Mar 7 2020, 12:02 AM

@Muehlenhoff @ayounsi

Tested squid on re-installed install1003.wikimedia.org:

[planet1001:~] $ http_proxy="http://install1003.wikimedia.org:8080; export HTTPS_PROXY=http://install1003.wikimedia.org:8080" curl foobar.org

and works for me.

Also see that in squid/access.log and the prometheus squid exporter has been added.

gerritbot added a comment.Mar 7 2020, 12:57 AM

Change 569687 abandoned by Dzahn:
switch install_server and failover in hieradata/common.yaml

Reason:
wrong, this is only used by aptrepo which is now a separate role, will rename this variable

https://gerrit.wikimedia.org/r/569687

gerritbot added a comment.Mar 7 2020, 1:02 AM

Change 577701 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] hiera/aptrepo: rename install_server variables to aptrepo_server

https://gerrit.wikimedia.org/r/577701

gerritbot added a comment.Mar 10 2020, 5:55 PM

Change 577701 merged by Dzahn:
[operations/puppet@production] hiera/aptrepo: rename install_server variables to aptrepo_server

https://gerrit.wikimedia.org/r/577701

gerritbot added a comment.Mar 12 2020, 12:21 AM

Change 569680 merged by Dzahn:
[operations/dns@master] switch webproxy CNAMEs to new install servers

https://gerrit.wikimedia.org/r/569680

gerritbot added a comment.Mar 12 2020, 12:53 AM

Change 579106 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] installserver: allow configuring squid as running/stopped in Hiera

https://gerrit.wikimedia.org/r/579106

gerritbot added a comment.Mar 12 2020, 6:59 PM

Change 579373 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] squid: remove obsolete hierarchy_stoplist config directive

https://gerrit.wikimedia.org/r/579373

gerritbot added a comment.Mar 13 2020, 5:08 PM

Change 579373 merged by Dzahn:
[operations/puppet@production] squid: remove obsolete hierarchy_stoplist config directive

https://gerrit.wikimedia.org/r/579373

gerritbot added a comment.Mar 13 2020, 5:15 PM

Change 579106 merged by Dzahn:
[operations/puppet@production] installserver: allow configuring squid as absent in Hiera

https://gerrit.wikimedia.org/r/579106

Stashbot added a comment.Mar 13 2020, 5:21 PM

Mentioned in SAL (#wikimedia-operations) [2020-03-13T17:21:12Z] <mutante> removed squid from install1002/install2002 (formerly webproxy.(eqiad|codfw).wmnet until 2 days ago, replaced by install1003/install2003) T224576

gerritbot added a comment.Mar 13 2020, 7:11 PM

Change 579610 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] installserver: disable squid monitoring if service is absent

https://gerrit.wikimedia.org/r/579610

gerritbot added a comment.Mar 13 2020, 7:16 PM

Change 579610 merged by Dzahn:
[operations/puppet@production] installserver: disable squid monitoring if service is absent

https://gerrit.wikimedia.org/r/579610

Dzahn updated the task description. (Show Details)Mar 13 2020, 10:59 PM
Dzahn added a parent task: T247045: Migrate all of production metal and VMs to Buster or later.Mar 14 2020, 12:19 AM
hashar added a subscriber: hashar.Mar 16 2020, 2:42 PM

When building a docker container on contint1001.wikimedia.org with docker-pkg, pip gets proxy timeout error when using http://webproxy.eqiad.wmnet:8080.

I have manually switched to the codfw one (http://webproxy.codfw.wmnet:8080) and it worked fine.

So I guess install1003.wikimedia.org has an issue of some sort?

ayounsi added a comment.Mar 16 2020, 3:05 PM

I opened T247759 to track this issue.

hashar mentioned this in T247759: eqiad squid performances issue.Mar 17 2020, 11:44 AM
gerritbot added a comment.Mar 31 2020, 3:50 PM

Change 584963 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/homer/public@master] replace DCHP relays with new installservers

https://gerrit.wikimedia.org/r/584963

Dzahn updated the task description. (Show Details)Mar 31 2020, 4:03 PM
hashar removed a subscriber: hashar.Apr 1 2020, 7:19 AM
gerritbot added a comment.Apr 1 2020, 9:31 AM

Change 584963 merged by jenkins-bot:
[operations/homer/public@master] replace DCHP relays with new installservers

https://gerrit.wikimedia.org/r/584963

Stashbot added a comment.Apr 1 2020, 9:34 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-01T09:34:33Z] <mutante> install_servers: DHCP_relay in routers and TFTP server in DHCP server config have been switched from install1002/2002 to install1003/2003 - doing a test install, but if any issues report on T224576

Stashbot added a comment.Apr 1 2020, 9:35 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-01T09:35:02Z] <XioNoX> Update install servers IPs (dhcp helpers + firewall rules) - T224576

gerritbot added a comment.Apr 1 2020, 10:46 AM

Change 585194 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] start DHCP service on new installserver::light servers

https://gerrit.wikimedia.org/r/585194

gerritbot added a comment.Apr 1 2020, 10:48 AM

Change 585194 merged by Dzahn:
[operations/puppet@production] start DHCP service on new installserver::light servers

https://gerrit.wikimedia.org/r/585194

gerritbot added a comment.Apr 1 2020, 11:10 AM

Change 585199 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] stop DHCP service on jessie install servers

https://gerrit.wikimedia.org/r/585199

gerritbot added a comment.Apr 1 2020, 11:14 AM

Change 585199 merged by Dzahn:
[operations/puppet@production] stop DHCP service on jessie install servers

https://gerrit.wikimedia.org/r/585199

Dzahn updated the task description. (Show Details)Apr 1 2020, 1:09 PM
gerritbot added a comment.Apr 1 2020, 1:12 PM

Change 569684 merged by Dzahn:
[operations/puppet@production] install_server: switch TFTP servers in DHCP to new install servers

https://gerrit.wikimedia.org/r/569684

gerritbot added a comment.Apr 1 2020, 1:23 PM

Change 585221 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] installserver: allow stopping tftp service with parameter

https://gerrit.wikimedia.org/r/585221

gerritbot added a comment.Apr 1 2020, 1:36 PM

Change 585221 merged by Dzahn:
[operations/puppet@production] installserver: stop tftp service on old install servers

https://gerrit.wikimedia.org/r/585221

gerritbot added a comment.Apr 1 2020, 1:45 PM

Change 585233 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] installserver/tftp: add missing ensure_service parameter

https://gerrit.wikimedia.org/r/585233

gerritbot added a comment.Apr 1 2020, 1:50 PM

Change 585233 merged by Dzahn:
[operations/puppet@production] installserver/tftp: add missing ensure_service parameter

https://gerrit.wikimedia.org/r/585233

gerritbot added a comment.Apr 1 2020, 2:04 PM

Change 585238 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] installserver/tftp: if service is ensured stop, do not monitor it

https://gerrit.wikimedia.org/r/585238

gerritbot added a comment.Apr 1 2020, 2:22 PM

Change 585238 merged by Dzahn:
[operations/puppet@production] tftp: if service is told to be stopped, do not monitor it

https://gerrit.wikimedia.org/r/585238

gerritbot added a comment.Apr 1 2020, 2:30 PM

Change 569686 merged by Dzahn:
[operations/puppet@production] DHCP: remove old install servers and use new servers as next-server

https://gerrit.wikimedia.org/r/569686

gerritbot added a comment.Apr 1 2020, 2:50 PM

Change 585245 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] hiera/apt.wikimedia.org: switch from install1002 to apt1001

https://gerrit.wikimedia.org/r/585245

gerritbot added a comment.Apr 2 2020, 5:14 AM

Change 585370 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] tftp,bastionhost: Add missing ensure_service hiera key

https://gerrit.wikimedia.org/r/585370

gerritbot added a comment.Apr 2 2020, 5:24 AM

Change 585370 merged by Vgutierrez:
[operations/puppet@production] tftp,bastionhost: Add missing ensure_service hiera key

https://gerrit.wikimedia.org/r/585370

gerritbot added a comment.Apr 2 2020, 11:51 AM

Change 585485 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] adjust my .bash_profile to set base dir for reprepro on apt*

https://gerrit.wikimedia.org/r/585485

gerritbot added a comment.Apr 2 2020, 12:08 PM

Change 585485 merged by Dzahn:
[operations/puppet@production] adjust my .bash_profile to set base dir for reprepro on apt*

https://gerrit.wikimedia.org/r/585485

gerritbot added a comment.Apr 2 2020, 3:44 PM

Change 585528 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add s-nail to send mails after package imports

https://gerrit.wikimedia.org/r/585528

gerritbot added a comment.Apr 3 2020, 6:11 AM

Change 585528 merged by Muehlenhoff:
[operations/puppet@production] Add s-nail to send mails after package imports

https://gerrit.wikimedia.org/r/585528

MoritzMuehlenhoff added a comment.Apr 3 2020, 6:57 AM

I successfully tested import and removals of a dummy build (hello) and fixed up the sending of status mails.

I've also synched the public repo keys from install1002 to apt1001, "reprepro checkup" against external repos now also works fine and I could successfully update the external Tor repo for stretch.

ayounsi removed a subscriber: ayounsi.Apr 3 2020, 7:06 AM
gerritbot added a comment.Apr 3 2020, 10:10 AM

Change 585245 merged by Dzahn:
[operations/puppet@production] hiera/apt.wikimedia.org: switch from install1002 to apt1001

https://gerrit.wikimedia.org/r/585245

gerritbot added a comment.Apr 3 2020, 10:35 AM

Change 585722 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] aptrepo: add MOTD warning to not use old install servers for APT

https://gerrit.wikimedia.org/r/585722

gerritbot added a comment.Apr 3 2020, 10:47 AM

Change 585723 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] aptrepo: remove reprepro package on inactive servers

https://gerrit.wikimedia.org/r/585723

gerritbot added a comment.Apr 3 2020, 10:47 AM

Change 585722 merged by Dzahn:
[operations/puppet@production] aptrepo: add MOTD warning to not use old install servers for APT

https://gerrit.wikimedia.org/r/585722

gerritbot added a comment.Apr 3 2020, 11:01 AM

Change 575404 merged by Dzahn:
[operations/dns@master] switch apt.wikimedia.org from install1002 to apt1001

https://gerrit.wikimedia.org/r/575404

gerritbot added a comment.Apr 3 2020, 11:48 AM

Change 585723 abandoned by Dzahn:
aptrepo: remove reprepro package on inactive servers

https://gerrit.wikimedia.org/r/585723

Dzahn updated the task description. (Show Details)Apr 3 2020, 12:08 PM
gerritbot added a comment.Apr 3 2020, 3:02 PM

Change 585765 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] installserver: rsync home dir data for migration

https://gerrit.wikimedia.org/r/585765

gerritbot added a comment.Apr 3 2020, 3:33 PM

Change 585765 merged by Dzahn:
[operations/puppet@production] installserver: rsync home dir data for migration

https://gerrit.wikimedia.org/r/585765

gerritbot added a comment.Apr 3 2020, 5:50 PM

Change 585789 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] installserver::migration: fix rsync dir name

https://gerrit.wikimedia.org/r/585789

gerritbot added a comment.Apr 3 2020, 5:54 PM

Change 585789 merged by Dzahn:
[operations/puppet@production] installserver::migration: fix rsync dir name

https://gerrit.wikimedia.org/r/585789

elukey added a subscriber: elukey.Apr 8 2020, 4:06 PM

There seems to be a problem with PXE installs, today I tried to live-hack a partman recipe from install1003/2003 but it didn't work, and after a while I found the following in the squid logs:

GET http://apt.wikimedia.org/autoinstall/partman/custom/kafka-jumbo.cfg

After live hacking the recipe on apt1001 I was finally able to see results. I also noticed that /srv/autoinstall on apt1001 seems not managed by puppet. A quick grep in the puppet repo shows stuff like:

preseed/url=http://apt.wikimedia.org/autoinstall/preseed.cfg

 d-i	preseed/early_command	string	wget -O /tmp/early_command http://apt.wikimedia.org/autoinstall/scripts/early_command.sh && sh /tmp/early_command

etc..

We should review all occurrences of apt.wikimedia.org in puppet and see if they need to be migrated to another DNS/alias.

• Marostegui added a subscriber: • Marostegui.Apr 9 2020, 6:20 AM
• Marostegui added a comment.Apr 9 2020, 6:24 AM

I have also experienced autoinstall/ not being managed by puppet issues :-(

root@install1002:/srv/autoinstall# grep -iR "no-srv-format.cfg" * | grep pc2007
netboot.cfg:        db1[01][0-9][0-9]|dbstore100[3-5]|db2[01][0-9][0-9]|es101[1-9]|es201[1-9]|pc100[7-9]|pc2007|pc2009|pc[12]010|labsdb1009|labsdb101[0-2]|dbprov[12]00[12]) echo partman/custom/no-srv-format.cfg ;; \
preseed.cfg:        db1[01][0-9][0-9]|dbstore100[3-5]|db2[01][0-9][0-9]|es101[1-9]|es201[1-9]|pc100[7-9]|pc2007|pc2009|pc[12]010|labsdb1009|labsdb101[0-2]|dbprov[12]00[12]) echo partman/custom/no-srv-format.cfg ;; \
root@install1002:/srv/autoinstall#
root@apt1001:/srv/autoinstall# grep -iR "no-srv-format.cfg" * | grep pc2007
root@apt1001:/srv/autoinstall#
gerritbot added a comment.Apr 9 2020, 8:36 AM

Change 587698 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] add install.wikimedia.org CNAME to install1003

https://gerrit.wikimedia.org/r/587698

gerritbot added a comment.Apr 9 2020, 8:41 AM

Change 587699 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] install_server: replace apt.wm.org with install.wm.org in autoinstall URLs

https://gerrit.wikimedia.org/r/587699

gerritbot added a comment.Apr 9 2020, 8:50 AM

Change 587701 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] installserver: include preseed in apt role, not in light role

https://gerrit.wikimedia.org/r/587701

gerritbot added a comment.Apr 9 2020, 9:11 AM

Change 587701 merged by Dzahn:
[operations/puppet@production] installserver: include preseed in apt role, not in light role

https://gerrit.wikimedia.org/r/587701

gerritbot added a comment.Apr 9 2020, 10:56 AM

Change 587699 abandoned by Dzahn:
install_server: replace apt.wm.org with install.wm.org in autoinstall URLs

Reason:
preferring other solution https://gerrit.wikimedia.org/r/c/operations/puppet/ /587701

https://gerrit.wikimedia.org/r/587699

gerritbot added a comment.Apr 9 2020, 10:58 AM

Change 587698 abandoned by Dzahn:
add install.wikimedia.org CNAME to install1003

Reason:
https://gerrit.wikimedia.org/r/c/operations/puppet/ /587701

https://gerrit.wikimedia.org/r/587698

• Marostegui added a comment.Apr 15 2020, 5:13 AM

Looks like the puppet issue isn't yet fixed, no? (I thought it was, so commenting here just to make sure)

I pushed a change to netboot.cfg which does show on a puppet run on install1002 but not on install1003.

Dzahn added a comment.Apr 15 2020, 6:52 AM

@Marostegui Please go to apt1001.wikimedia.org and you should see your change there. The "preseed" profile which includes /srv/autoinstall files is applied on the same hosts that have the APT repo and the webserver. install1003 is only TFTP/DHCP/proxy but not a webserver.

• Marostegui added a comment.Apr 15 2020, 6:53 AM

Yeah, it was applied there indeed. Thanks for the clarification!

Dzahn added subtasks: T251131: decommission install1002.wikimedia.org, T251130: decommission install2002.wikimedia.org.Apr 27 2020, 1:39 PM
Dzahn closed subtask T251130: decommission install2002.wikimedia.org as Invalid.Apr 27 2020, 1:42 PM
Dzahn closed subtask T251131: decommission install1002.wikimedia.org as Invalid.
gerritbot added a comment.Apr 27 2020, 2:00 PM

Change 592662 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] apt_repo: stop including migration rsync class

https://gerrit.wikimedia.org/r/592662

gerritbot added a comment.Apr 27 2020, 2:05 PM

Change 592662 merged by Dzahn:
[operations/puppet@production] apt_repo: stop including migration rsync class

https://gerrit.wikimedia.org/r/592662

gerritbot added a comment.Apr 27 2020, 2:14 PM

Change 569683 merged by Dzahn:
[operations/dns@master] remove install1002/install2002

https://gerrit.wikimedia.org/r/569683

Dzahn closed this task as Resolved.Apr 27 2020, 2:14 PM
Dzahn updated the task description. (Show Details)
gerritbot added a comment.May 22 2020, 3:49 PM

Change 598071 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Remove install[12]002 site.pp entries

https://gerrit.wikimedia.org/r/598071

gerritbot added a comment.May 24 2020, 2:01 PM

Change 598071 merged by Alexandros Kosiaris:
[operations/puppet@production] Remove install[12]002 site.pp entries

https://gerrit.wikimedia.org/r/598071

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL