Page MenuHomePhabricator
Create Task
Maniphest T204271

Grow frack-administration-codfw to /28
Closed, ResolvedPublic
Actions

Description

Follow up from T204079#4579043

10.195.0.72/29 - frack-administration-codfw is running out of IPs

The agreed over IRC plan is to move the only host in 10.195.0.64/29 - frack-bastion-codfw (frbast2001) to a new subnet, and grow frack-administration-codfw to 10.195.0.64/28, this should cause minimum downtime.

  1. Move frbast2001
  2. Reserve 10.195.0.128/29 (and future IPs) in DNS for frack-bastion-codfw
  3. Configure new subnet on pfw3-codfw (on same interface)
  4. Update firewall policies to have both old/new bastion subnet
  5. Downtime monitoring for frbast2001
  6. Change IP config of frbast2001
  7. Update NAT rule on pfw3-codfw to point to new IP
  8. Re-enable monitoring for frbast2001
  9. Remove old IPs from DNS
  10. Remove old IPs from firewall policies
  1. Grow frack-administration-codfw
  2. Assign 10.195.0.65/28 to pfw3-codfw:reth0.2134 (in addition to current .73/29)
  3. Change netmask of frack-administration hosts to /28 and gateway to .65
  4. Remove .73/29 IP from pfw3-codfw:reth0.2134
  5. Update DNS

Details

SubjectRepoBranchLines +/-
Update data.yaml for frack-(administration|bastion)-codfw subnet changesoperations/puppetproduction+2 -2
Remove old frack-bastion-codfw grow frack-administration-codfw to /28operations/dnsmaster+2 -6
Reserve new frack-bastion-codfw subnetoperations/dnsmaster+5 -0
Customize query in gerrit

Related Objects

Event Timeline

ayounsi triaged this task as Medium priority.Sep 13 2018, 6:27 PM
ayounsi created this task.
Restricted Application added a project: SRE. · View Herald TranscriptSep 13 2018, 6:27 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
gerritbot subscribed.Oct 2 2018, 4:43 PM

Change 463983 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Reserve new frack-bastion-codfw subnet

https://gerrit.wikimedia.org/r/463983

gerritbot added a project: Patch-For-Review.Oct 2 2018, 4:43 PM
gerritbot added a comment.Oct 2 2018, 4:46 PM

Change 463983 merged by Ayounsi:
[operations/dns@master] Reserve new frack-bastion-codfw subnet

https://gerrit.wikimedia.org/r/463983

Stashbot subscribed.Oct 2 2018, 4:49 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-02T16:49:06Z] <XioNoX> assign 10.195.0.129/29 to pfw3-codfw:reth0.2133 - T204271

ayounsi updated the task description. (Show Details)Oct 2 2018, 4:49 PM
gerritbot added a comment.Oct 2 2018, 4:53 PM

Change 463990 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Remove old frack-bastion-codfw grow frack-administration-codfw to /28

https://gerrit.wikimedia.org/r/463990

Stashbot added a comment.Oct 2 2018, 5:22 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-02T17:22:40Z] <XioNoX> update fw policies on pfw3-codfw - T204271

Stashbot added a comment.Oct 2 2018, 5:25 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-02T17:25:28Z] <XioNoX> update fw policies on pfw3-eqiad - T204271

ayounsi updated the task description. (Show Details)Oct 2 2018, 5:26 PM
Stashbot added a comment.Oct 2 2018, 5:37 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-02T17:37:52Z] <XioNoX> update NAT for frbast2001 on pfw3-codfw - T204271

ayounsi updated the task description. (Show Details)Oct 2 2018, 5:39 PM
Stashbot added a comment.Oct 2 2018, 6:26 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-02T18:26:30Z] <XioNoX> remove old 10.195.0.65/29 from pfw3-codfw - T204271

Stashbot added a comment.Oct 2 2018, 6:39 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-02T18:39:15Z] <XioNoX> replace 10.195.0.73/29 with 10.195.0.65/28 on pfw3-codfw - T204271

gerritbot added a comment.Oct 2 2018, 6:46 PM

Change 463990 merged by Ayounsi:
[operations/dns@master] Remove old frack-bastion-codfw grow frack-administration-codfw to /28

https://gerrit.wikimedia.org/r/463990

ayounsi updated the task description. (Show Details)Oct 2 2018, 6:48 PM
Stashbot added a comment.Oct 2 2018, 7:19 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-02T19:19:54Z] <XioNoX> update fw policies on pfw3-codfw - T204271

Stashbot added a comment.Oct 2 2018, 7:21 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-02T19:21:24Z] <XioNoX> update fw policies on pfw3-eqiad - T204271

Stashbot added a comment.Oct 2 2018, 7:50 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-02T19:50:46Z] <XioNoX> update prefix-list fundraising-codfw-internal4 to /24 on pfw3-codfw - T204271

ayounsi closed this task as Resolved.Oct 2 2018, 9:41 PM
ayounsi updated the task description. (Show Details)

An oversight prevented frbast2001 to reach eqiad:
codfw only advertised 10.195.0.0/25 to eqiad over ipsec.
Making it a /24 fixed the issue.

gerritbot added a comment.Oct 10 2018, 2:17 PM

Change 465635 had a related patch set uploaded (by Jgreen; owner: Jgreen):
[operations/puppet@production] Update data.yaml for frack-(administration|bastion)-codfw subnet changes

https://gerrit.wikimedia.org/r/465635

gerritbot added a comment.Oct 10 2018, 2:18 PM

Change 465635 merged by Jgreen:
[operations/puppet@production] Update data.yaml for frack-(administration|bastion)-codfw subnet changes

https://gerrit.wikimedia.org/r/465635

Jgreen mentioned this in T206637: icinga reports frbast2001.frack.eqiad.wmnet as host down.Oct 10 2018, 3:29 PM
Dwisehaupt moved this task from Triage to Done on the fundraising-tech-ops board.Feb 13 2020, 9:43 PM
Maintenance_bot removed a project: Patch-For-Review.Feb 13 2020, 10:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits