Page MenuHomePhabricator

Grow frack-administration-codfw to /28
Closed, ResolvedPublic

Description

Follow up from T204079#4579043

10.195.0.72/29 - frack-administration-codfw is running out of IPs

The agreed over IRC plan is to move the only host in 10.195.0.64/29 - frack-bastion-codfw (frbast2001) to a new subnet, and grow frack-administration-codfw to 10.195.0.64/28, this should cause minimum downtime.

  1. Move frbast2001
  2. Reserve 10.195.0.128/29 (and future IPs) in DNS for frack-bastion-codfw
  3. Configure new subnet on pfw3-codfw (on same interface)
  4. Update firewall policies to have both old/new bastion subnet
  5. Downtime monitoring for frbast2001
  6. Change IP config of frbast2001
  7. Update NAT rule on pfw3-codfw to point to new IP
  8. Re-enable monitoring for frbast2001
  9. Remove old IPs from DNS
  10. Remove old IPs from firewall policies
  1. Grow frack-administration-codfw
  2. Assign 10.195.0.65/28 to pfw3-codfw:reth0.2134 (in addition to current .73/29)
  3. Change netmask of frack-administration hosts to /28 and gateway to .65
  4. Remove .73/29 IP from pfw3-codfw:reth0.2134
  5. Update DNS

Event Timeline

ayounsi triaged this task as Medium priority.Sep 13 2018, 6:27 PM
ayounsi created this task.
Restricted Application added a project: Operations. · View Herald TranscriptSep 13 2018, 6:27 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 463983 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Reserve new frack-bastion-codfw subnet

https://gerrit.wikimedia.org/r/463983

Change 463983 merged by Ayounsi:
[operations/dns@master] Reserve new frack-bastion-codfw subnet

https://gerrit.wikimedia.org/r/463983

Mentioned in SAL (#wikimedia-operations) [2018-10-02T16:49:06Z] <XioNoX> assign 10.195.0.129/29 to pfw3-codfw:reth0.2133 - T204271

ayounsi updated the task description. (Show Details)Oct 2 2018, 4:49 PM

Change 463990 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/dns@master] Remove old frack-bastion-codfw grow frack-administration-codfw to /28

https://gerrit.wikimedia.org/r/463990

Mentioned in SAL (#wikimedia-operations) [2018-10-02T17:22:40Z] <XioNoX> update fw policies on pfw3-codfw - T204271

Mentioned in SAL (#wikimedia-operations) [2018-10-02T17:25:28Z] <XioNoX> update fw policies on pfw3-eqiad - T204271

ayounsi updated the task description. (Show Details)Oct 2 2018, 5:26 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-02T17:37:52Z] <XioNoX> update NAT for frbast2001 on pfw3-codfw - T204271

ayounsi updated the task description. (Show Details)Oct 2 2018, 5:39 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-02T18:26:30Z] <XioNoX> remove old 10.195.0.65/29 from pfw3-codfw - T204271

Mentioned in SAL (#wikimedia-operations) [2018-10-02T18:39:15Z] <XioNoX> replace 10.195.0.73/29 with 10.195.0.65/28 on pfw3-codfw - T204271

Change 463990 merged by Ayounsi:
[operations/dns@master] Remove old frack-bastion-codfw grow frack-administration-codfw to /28

https://gerrit.wikimedia.org/r/463990

ayounsi updated the task description. (Show Details)Oct 2 2018, 6:48 PM

Mentioned in SAL (#wikimedia-operations) [2018-10-02T19:19:54Z] <XioNoX> update fw policies on pfw3-codfw - T204271

Mentioned in SAL (#wikimedia-operations) [2018-10-02T19:21:24Z] <XioNoX> update fw policies on pfw3-eqiad - T204271

Mentioned in SAL (#wikimedia-operations) [2018-10-02T19:50:46Z] <XioNoX> update prefix-list fundraising-codfw-internal4 to /24 on pfw3-codfw - T204271

ayounsi closed this task as Resolved.Oct 2 2018, 9:41 PM
ayounsi updated the task description. (Show Details)

An oversight prevented frbast2001 to reach eqiad:
codfw only advertised 10.195.0.0/25 to eqiad over ipsec.
Making it a /24 fixed the issue.

Change 465635 had a related patch set uploaded (by Jgreen; owner: Jgreen):
[operations/puppet@production] Update data.yaml for frack-(administration|bastion)-codfw subnet changes

https://gerrit.wikimedia.org/r/465635

Change 465635 merged by Jgreen:
[operations/puppet@production] Update data.yaml for frack-(administration|bastion)-codfw subnet changes

https://gerrit.wikimedia.org/r/465635

Dwisehaupt moved this task from Triage to Done on the fundraising-tech-ops board.Feb 13 2020, 9:43 PM