Page MenuHomePhabricator
Create Task
Maniphest T132757

Split carbon's install/mirror roles, provision install1001
Closed, ResolvedPublic
Actions

Description

OK, so we have a few things in-flight right now (e.g. jessie upgrades, T132450/HTTPS for carbon etc.), so it's about time to clean this all up.

This would be my suggestion:

  • Upgrade install2001 to jessie, make sure our manifests work there (trivial)
  • Deploy a new install1001 VM
  • move install_server (DHCP, TFTP, preseed web) to install1001
  • move reprepro/apt (remember /root/.gnupg/!) to install1001
  • Remove all install_server-related roles out of carbon and designate it as a mirror server (T84817); reinstall carbon with jessie (T123733)
  • (optionally) split the reprepro role out of install_server in puppet, they're really different roles
  • Upgrade install2001 to match install1001 (i.e. add DHCP, point codfw's DHCP relays to install2001) (T84380)
  • Set up apt on install2001 too (sync done with rsync in cron)
  • use a reprepro hooks to sync apt across and possibly make apt.wikimedia.org HA (perhaps with gdnsd). -> (T158022)

Any takers? @Dzahn maybe?

related patches: https://gerrit.wikimedia.org/r/#/q/topic:installserver+%28status:open+OR+status:merged%29

Details

Related Gerrit Patches:
operations/puppet : productionlet install1002 be the new source for APT data rsync
operations/puppet : productioninstall: enable Letsencrypt on install1002
operations/puppet : productioninstall: copy/move apt.wm.org setup to aptrepo module
operations/dns : masterswitch apt.wm.org from carbon to install1002
operations/puppet : productionremove install1001/install2001 from site.pp
operations/puppet : productioninstall/DHCP/TFTP: use install1002 and install2002 as next-servers
operations/puppet : productionaptrepo: disable autoconfigured EUI64 addresses
operations/puppet : productioninstallserver: add firewall hole for rsync also for IPv6
operations/puppet : productionaptrepo: add cron to rsync APT data automatically
operations/puppet : productioninstallserver: add install1002/2002 to hiera
operations/puppet : productionDHCP: add install1001,install2001
operations/dns : mastermove install1002 to lower free IP address
operations/dns : masteradd install1002/2002 to replace install1001/2001
operations/puppet : productionadd install1002/2002 to replace 1001/2001
operations/puppet : productionaptrepo: add second rsync module for entire /srv/
operations/puppet : productionaptrepo: fix rsyncd 'hosts allow' syntax
operations/puppet : productionswitch install_server to carbon for initial rsync of APT repo data
operations/puppet : productionaptrepo: setup rsync between 2 APT servers
operations/puppet : productionganglia: switch eqiad aggregator from carbon to install1001
operations/puppet : productionopenstack: switch tftp server from carbon to install1001
operations/puppet : productionmove ganglia aggregator eqiad from carbon to install1001
operations/puppet : productioninstall/dhcp: switch all "next-server" from carbon to install1001
operations/puppet : productiondhcp: switch private-c-eqiad to install1001 as tftp
operations/puppet : productioninstall: add http & proxy roles on install2001
operations/puppet : productioninstall: add hiera override to skip Letsencrypt cert creation
operations/puppet : productioninstall: add http & proxy roles on install1001
operations/puppet : productioninstall: add 'preseed'-role to install1001
operations/puppet : productioninstall: (re)move remaining "role::installserver"
operations/puppet : productioninstall: rename tftp_server role to just tftp and further cleanup
operations/puppet : productioninstallserver: move firewall include to http/proxy classes
operations/puppet : productioninstallserver: split squid proxy to own class
operations/puppet : productioninstallserver: move http to own class (kill carbon WIP)
operations/puppet : productionput installserver::dhcp on install1001, install2001
operations/puppet : productioninstallserver: split DHCP part out into own role
operations/puppet : productionDHCP: make configurable in Hiera which is the running server
operations/puppet : productionaptrepo: make owners of incoming dir configurable
operations/puppet : productionaptrepo: set separate incoming conf for releases
operations/puppet : productionaptrepo: make conf/incoming configurable parameter
operations/puppet : productionaptrepo: mv wikimedia-specific distributions file to role
operations/puppet : productioninstall1001, add new aptrepo role like carbon
operations/puppet : productionaptrepo: un-change conf/incoming
operations/puppet : productioninstall_server: split out reprepro to module aptrepo
operations/puppet : productioninstall/aptrepo: move distributions file
operations/puppet : productioninstall_server: move mirrors stuff to own role
operations/puppet : productioninstall_server: move tftp role to autoloader layout
operations/puppet : productionadd install_server::tftp role on install1001
operations/puppet : productionnetboot: add install1001, use partman for VMs
operations/puppet : productionadd install1001 to site.pp and DHCP
operations/dns : masteradd install1001.wikimedia.org using 208.80.154.83
operations/puppet : productioninstall_server: switch install2001 to jessie

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Dec 7 2016, 6:07 AM

Change 325739 had a related patch set uploaded (by Dzahn):
install: add http & proxy roles on install1001

https://gerrit.wikimedia.org/r/325739

Dzahn added a comment.Dec 7 2016, 6:14 AM
node 'carbon.wikimedia.org' {
    role(installserver::tftp,
        installserver::dhcp,
        installserver::http,
        installserver::proxy,
        installserver::preseed,
        aptrepo::wikimedia)
gerritbot added a comment.Dec 7 2016, 6:20 AM

Change 325743 had a related patch set uploaded (by Dzahn):
install: add http & proxy roles on install2001

https://gerrit.wikimedia.org/r/325743

gerritbot added a comment.Dec 7 2016, 11:46 PM

Change 325737 merged by Dzahn:
install: add 'preseed'-role to install1001

https://gerrit.wikimedia.org/r/325737

gerritbot added a comment.Dec 8 2016, 12:45 AM

Change 325864 had a related patch set uploaded (by Dzahn):
install: copy/move apt.wm.org setup to aptrepo module

https://gerrit.wikimedia.org/r/325864

gerritbot added a comment.Dec 20 2016, 10:14 PM

Change 325864 abandoned by Dzahn:
install: copy/move apt.wm.org setup to aptrepo module

Reason:
wrong approach after rethinking

https://gerrit.wikimedia.org/r/325864

gerritbot added a comment.Dec 20 2016, 10:14 PM

Change 325864 restored by Dzahn:
install: copy/move apt.wm.org setup to aptrepo module

https://gerrit.wikimedia.org/r/325864

gerritbot added a comment.Dec 20 2016, 10:21 PM

Change 325739 merged by Dzahn:
install: add http & proxy roles on install1001

https://gerrit.wikimedia.org/r/325739

gerritbot added a comment.Dec 20 2016, 10:41 PM

Change 328429 had a related patch set uploaded (by Dzahn):
install: add hiera override to skip Letsencrypt cert creation

https://gerrit.wikimedia.org/r/328429

gerritbot added a comment.Dec 20 2016, 10:51 PM

Change 328429 merged by Dzahn:
install: add hiera override to skip Letsencrypt cert creation

https://gerrit.wikimedia.org/r/328429

gerritbot added a comment.Dec 20 2016, 10:54 PM

Change 325743 merged by Dzahn:
install: add http & proxy roles on install2001

https://gerrit.wikimedia.org/r/325743

gerritbot added a comment.Dec 20 2016, 11:35 PM

Change 328439 had a related patch set uploaded (by Dzahn):
install/dhcp: switch all "next-server" from carbon to install1001

https://gerrit.wikimedia.org/r/328439

gerritbot added a comment.Dec 21 2016, 1:23 AM

Change 328448 had a related patch set uploaded (by Dzahn):
dhcp: switch private-c-eqiad to install1001 as tftp

https://gerrit.wikimedia.org/r/328448

gerritbot added a comment.Dec 21 2016, 1:25 AM

Change 328448 merged by Dzahn:
dhcp: switch private-c-eqiad to install1001 as tftp

https://gerrit.wikimedia.org/r/328448

gerritbot added a comment.Dec 21 2016, 3:02 AM

Change 328450 had a related patch set uploaded (by Dzahn):
move ganglia aggregator eqiad from carbon to install1001

https://gerrit.wikimedia.org/r/328450

gerritbot added a comment.Dec 21 2016, 7:12 PM

Change 328439 merged by Dzahn:
install/dhcp: switch all "next-server" from carbon to install1001

https://gerrit.wikimedia.org/r/328439

Dzahn added a subscriber: akosiaris.Dec 21 2016, 8:45 PM

so current status is now:

if in public1-b-eqiad or public1-c-eqiad then install1001 is used for everything (DHCP -> TFTP -> serves installer), if in other subnets we still use carbon as DHCP but then TFTP is also install1001 already

Dzahn added a comment.Dec 21 2016, 8:48 PM

other things we need:

  • move ganglia aggregator
  • change ACLs / ferm rules for webproxy
    • (it's still webproxy 1H IN CNAME carbon.wikimedia.org. in DNS)
  • rsync apt repo again
  • confirm if we still want to reinstall carbon as a "role mirror" as the ticket says or if that is already done because sodium is now a mirror and we just decom carbon
  • ?
gerritbot added a comment.Dec 21 2016, 11:55 PM

Change 328597 had a related patch set uploaded (by Dzahn):
openstack: switch tftp server from carbon to install1001

https://gerrit.wikimedia.org/r/328597

Dzahn updated the task description. (Show Details)Dec 22 2016, 12:35 AM
akosiaris added a comment.Dec 22 2016, 7:55 AM
In T132757#2894515, @Dzahn wrote:
  • @akosiaris configured switches so that public1-b-eqiad and public1-c-eqiad are using install1001 as DHCP

Correction: private1-c-eqiad, private1-b-codfw

so current status is now:
if in public1-b-eqiad or public1-c-eqiad then install1001 is used for everything (DHCP -> TFTP -> serves installer), if in other subnets we still use carbon as DHCP but then TFTP is also install1001 already

Same correction here as well.

Dzahn added a comment.Dec 22 2016, 4:17 PM

oops, yea, of course, private(!)-eqiad, consider that a typo

gerritbot added a comment.Dec 22 2016, 8:03 PM

Change 328450 abandoned by Dzahn:
move ganglia aggregator eqiad from carbon to install1001

Reason:
duplicate of https://gerrit.wikimedia.org/r/#/c/328599/

https://gerrit.wikimedia.org/r/328450

gerritbot added a comment.Dec 22 2016, 8:06 PM

Change 328599 had a related patch set uploaded (by Dzahn):
ganglia: switch eqiad aggregator from carbon to install1001

https://gerrit.wikimedia.org/r/328599

gerritbot added a comment.Jan 5 2017, 12:48 AM

Change 328597 merged by Dzahn:
openstack: switch tftp server from carbon to install1001

https://gerrit.wikimedia.org/r/328597

gerritbot added a comment.Jan 5 2017, 11:08 PM

Change 328599 merged by Dzahn:
ganglia: switch eqiad aggregator from carbon to install1001

https://gerrit.wikimedia.org/r/328599

gerritbot added a comment.Jan 23 2017, 8:46 PM

Change 333676 had a related patch set uploaded (by Dzahn):
aptrepo: setup rsync between 2 APT servers

https://gerrit.wikimedia.org/r/333676

Dzahn updated the task description. (Show Details)Jan 23 2017, 8:47 PM
gerritbot added a comment.Jan 24 2017, 2:00 AM

Change 333676 merged by Dzahn:
aptrepo: setup rsync between 2 APT servers

https://gerrit.wikimedia.org/r/333676

Dzahn added a comment.Jan 24 2017, 7:45 AM

I moved the eqiad Ganglia aggregator from carbon to install1001 today. This part is unblocked.

gerritbot added a comment.Jan 26 2017, 12:12 AM

Change 334221 had a related patch set uploaded (by Dzahn):
switch install_server to carbon for initial rsync of APT repo data

https://gerrit.wikimedia.org/r/334221

gerritbot added a comment.Jan 26 2017, 12:26 AM

Change 334221 merged by Dzahn:
switch install_server to carbon for initial rsync of APT repo data

https://gerrit.wikimedia.org/r/334221

gerritbot added a comment.Jan 26 2017, 3:16 AM

Change 334237 had a related patch set uploaded (by Dzahn):
aptrepo: fix rsyncd 'hosts allow' syntax

https://gerrit.wikimedia.org/r/334237

gerritbot added a comment.Jan 26 2017, 3:24 AM

Change 334237 merged by Dzahn:
aptrepo: fix rsyncd 'hosts allow' syntax

https://gerrit.wikimedia.org/r/334237

gerritbot added a comment.Jan 27 2017, 1:01 AM

Change 334465 had a related patch set uploaded (by Dzahn):
aptrepo: add second rsync module for entire /srv/

https://gerrit.wikimedia.org/r/334465

gerritbot added a comment.Jan 27 2017, 10:23 PM

Change 334465 merged by Dzahn:
aptrepo: add second rsync module for entire /srv/

https://gerrit.wikimedia.org/r/334465

gerritbot added a comment.Jan 31 2017, 11:15 PM

Change 335372 had a related patch set uploaded (by Dzahn):
add install1002/2002 to replace 1001/2001

https://gerrit.wikimedia.org/r/335372

gerritbot added a comment.Jan 31 2017, 11:35 PM

Change 335372 merged by Dzahn:
add install1002/2002 to replace 1001/2001

https://gerrit.wikimedia.org/r/335372

gerritbot added a comment.Jan 31 2017, 11:46 PM

Change 335376 had a related patch set uploaded (by Dzahn):
add install1002/2002 to replace install1001/2001

https://gerrit.wikimedia.org/r/335376

gerritbot added a comment.Jan 31 2017, 11:54 PM

Change 335376 merged by Papaul:
add install1002/2002 to replace install1001/2001

https://gerrit.wikimedia.org/r/335376

gerritbot added a comment.Feb 1 2017, 12:02 AM

Change 335379 had a related patch set uploaded (by Dzahn):
move install1002 to lower free IP address

https://gerrit.wikimedia.org/r/335379

gerritbot added a comment.Feb 1 2017, 12:16 AM

Change 335379 merged by Dzahn:
move install1002 to lower free IP address

https://gerrit.wikimedia.org/r/335379

Stashbot added a subscriber: Stashbot.Feb 1 2017, 1:15 AM

Mentioned in SAL (#wikimedia-operations) [2017-02-01T01:15:26Z] <mutante> ganeti: install1001 - remove virtual disk 1 from instance | create instance install1002 instead (T132757)

gerritbot added a comment.Feb 1 2017, 2:17 AM

Change 335386 had a related patch set uploaded (by Dzahn):
DHCP: add install1001,install2001

https://gerrit.wikimedia.org/r/335386

gerritbot added a comment.Feb 1 2017, 2:33 AM

Change 335386 merged by Dzahn:
DHCP: add install1001,install2001

https://gerrit.wikimedia.org/r/335386

Stashbot added a comment.Feb 1 2017, 2:48 AM

Mentioned in SAL (#wikimedia-operations) [2017-02-01T02:48:05Z] <mutante> install1002, install2002 - install jessie, sign puppet certs, initial puppet run (T132757, T156440)

gerritbot added a comment.Feb 1 2017, 3:01 AM

Change 335388 had a related patch set uploaded (by Dzahn):
installserver: add install1002/2002 to hiera

https://gerrit.wikimedia.org/r/335388

gerritbot added a comment.Feb 1 2017, 3:03 AM

Change 335388 merged by Dzahn:
installserver: add install1002/2002 to hiera

https://gerrit.wikimedia.org/r/335388

Stashbot added a comment.Feb 1 2017, 10:54 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-01T22:54:51Z] <mutante> carbon - rsyncing /srv/ data to install1002 (T132757)

gerritbot added a comment.Feb 2 2017, 12:23 AM

Change 334241 had a related patch set uploaded (by Dzahn):
aptrepo: add cron to rsync APT data automatically

https://gerrit.wikimedia.org/r/334241

gerritbot added a comment.Feb 2 2017, 12:35 AM

Change 334241 merged by Dzahn:
aptrepo: add cron to rsync APT data automatically

https://gerrit.wikimedia.org/r/334241

gerritbot added a comment.Feb 2 2017, 1:42 AM

Change 335585 had a related patch set uploaded (by Dzahn):
installserver: add firewall hole for rsync also for IPv6

https://gerrit.wikimedia.org/r/335585

gerritbot added a comment.Feb 2 2017, 1:44 AM

Change 335585 merged by Dzahn:
installserver: add firewall hole for rsync also for IPv6

https://gerrit.wikimedia.org/r/335585

Stashbot added a comment.Feb 2 2017, 2:02 AM

Mentioned in SAL (#wikimedia-operations) [2017-02-02T02:02:09Z] <mutante> carbon - remove unmapped IPv6 address making ferm rules fail, use only the _mapped_ IP (ip addr del 2620:0:861:1:7a2b:cbff:fe09:ea0/64 dev eth0) (T84380 T132757)

gerritbot added a comment.Feb 2 2017, 3:03 AM

Change 335594 had a related patch set uploaded (by Dzahn):
aptrepo: disable autoconfigured EUI64 addresses

https://gerrit.wikimedia.org/r/335594

gerritbot added a comment.Feb 2 2017, 11:40 PM

Change 335594 abandoned by Dzahn:
aptrepo: disable autoconfigured EUI64 addresses

Reason:
it just affects precise so we can just ignore the issue on carbon until it's down. (rsync -4 to work around it for example)

https://gerrit.wikimedia.org/r/335594

gerritbot added a comment.Feb 3 2017, 12:18 AM

Change 335734 had a related patch set uploaded (by Dzahn):
switch apt.wm.org from carbon to install1002

https://gerrit.wikimedia.org/r/335734

Dzahn updated the task description. (Show Details)Feb 3 2017, 12:25 AM
gerritbot added a comment.Feb 10 2017, 8:58 PM

Change 336959 had a related patch set uploaded (by Dzahn):
install/TFTP: use install1002 and install2002 as next-servers

https://gerrit.wikimedia.org/r/336959

gerritbot added a comment.Feb 10 2017, 8:58 PM

Change 337084 had a related patch set uploaded (by Dzahn):
remove install1001/install2001 from site.pp

https://gerrit.wikimedia.org/r/337084

gerritbot added a comment.Feb 10 2017, 9:02 PM

Change 336959 merged by Dzahn:
install/DHCP/TFTP: use install1002 and install2002 as next-servers

https://gerrit.wikimedia.org/r/336959

gerritbot added a comment.Feb 10 2017, 9:09 PM

Change 337084 merged by Dzahn:
remove install1001/install2001 from site.pp

https://gerrit.wikimedia.org/r/337084

gerritbot added a comment.Feb 10 2017, 9:15 PM

Change 337093 had a related patch set uploaded (by Dzahn):
remove install1001 and install2001, keep 2001 mgmt

https://gerrit.wikimedia.org/r/337093

Stashbot added a comment.Feb 10 2017, 9:18 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-10T21:18:27Z] <mutante> install1001, install2001 - revoke puppet certs, puppet node deactivate, delete salt keys (T84380, T132757)

Stashbot added a comment.Feb 10 2017, 9:27 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-10T21:27:47Z] <mutante> install1001, install2001 - removed from Icinga, shutting down (T84380, T132757)

Stashbot added a comment.Feb 10 2017, 10:17 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-10T22:17:16Z] <mutante> install1001 - shutdown ganeti instance and deleting it and its disk (T132757)

Stashbot added a comment.Feb 10 2017, 10:29 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-10T22:29:17Z] <mutante> carbon - stopping puppet and most services, adding deprecation warning to motd, rsyncing data one last time (T132757)

gerritbot added a comment.Feb 10 2017, 11:56 PM

Change 335734 merged by Dzahn:
switch apt.wm.org from carbon to install1002

https://gerrit.wikimedia.org/r/335734

Dzahn added a comment.Feb 11 2017, 12:01 AM

16:02 < mutante> !log switching apt.wikimedia.org from carbon to install1002 - there might be a short time until the LE SSL cert is also adjusted

gerritbot added a comment.Feb 11 2017, 12:22 AM

Change 337197 had a related patch set uploaded (by Dzahn):
install: remove carbon from puppet and netboot

https://gerrit.wikimedia.org/r/337197

gerritbot added a comment.Feb 11 2017, 1:01 AM

Change 325864 abandoned by Dzahn:
install: copy/move apt.wm.org setup to aptrepo module

Reason:
thinking about it again i guess these things should stay in the "webserver" class even though "aptrepo" might sound like it sets up apt.wikimedia.org

https://gerrit.wikimedia.org/r/325864

Stashbot added a comment.Feb 13 2017, 7:38 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-13T19:38:04Z] <mutante> carbon - synced /srv/ data to install1002/2002 for the last time, switching apt.wikimedia.org CNAME to install1002 - carbon deprecated (T132757)

gerritbot added a comment.Feb 13 2017, 8:05 PM

Change 337443 had a related patch set uploaded (by Dzahn):
install: enable Letsencrypt on install1002

https://gerrit.wikimedia.org/r/337443

gerritbot added a comment.Feb 13 2017, 8:09 PM

Change 337443 merged by Dzahn:
install: enable Letsencrypt on install1002

https://gerrit.wikimedia.org/r/337443

gerritbot added a comment.Feb 13 2017, 8:47 PM

Change 337198 had a related patch set uploaded (by Dzahn):
let install1002 be the new source for APT data rsync

https://gerrit.wikimedia.org/r/337198

gerritbot added a comment.Feb 13 2017, 8:50 PM

Change 337198 merged by Dzahn:
let install1002 be the new source for APT data rsync

https://gerrit.wikimedia.org/r/337198

Stashbot added a comment.Feb 13 2017, 8:51 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-13T20:51:57Z] <mutante> carbon/install - adjusted Letsencrypt cert creation, deactivated reprepro to protect from accidental use, switching rsync direction from install1002->install2002, disabled cron on carbon (T132757)

Dzahn updated the task description. (Show Details)Feb 13 2017, 9:52 PM
Dzahn updated the task description. (Show Details)
Dzahn closed this task as Resolved.Feb 14 2017, 1:07 AM

all the things originally listed in this ticket have been done - except the "make APT HA" one which has been split out into T158022

decom of carbon is in T158020

based on that i am resolving this

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL