⚓ T253173 Some clusters do not have DNS for IPv6 addresses (TRACKING TASK)

Subject	Repo	Branch	Lines +/-
profile/manifests/dumps: enable ipv6 drop ferm rule for 443	operations/puppet	production	+2 -6
AAAA: for flerovium and furud	operations/dns	master	+4 -1
sretest: add AAAA records	operations/dns	master	+5 -1
mongodb: enable ipv6	operations/puppet	production	+5 -4
theemin.codfw.wmnet: add AAAA	operations/dns	master	+3 -0
puppetmaster2003: add AAAA	operations/dns	master	+3 -0

Status	Assigned	Task
Open	None	T253173 Some clusters do not have DNS for IPv6 addresses (TRACKING TASK)
Open	None	T253986 update bacula-sd config so that it listens on IPv6
Resolved	• crusnov	T271133 Some Analytics clusters apparently do not support IPv6
Declined	None	T271134 Some Machine Learning clusters do not support IPv6
Open	MoritzMuehlenhoff	T271136 Some Foundation clusters do not appear to support IPv6
Open	MoritzMuehlenhoff	T379890 Enable ipv6 on ganeti2019-ganeti2024
Resolved	fgiunchedi	T271138 Some Observability clusters do not support IPv6.
Open	None	T271139 Some WMCS clusters apparently do not support IPv6
Open	None	T271140 Some Data Persistence clusters apparently do not support IPv6
Open	None	T270101 Grants not working with DB hosts with to ipv6
Resolved	akosiaris	T271142 Some Service Operations clusters apparently do not support IPv6
Resolved	bking	T271143 Some elastic hosts do not have IPv6 DNS records
Resolved	BCornwall	T271144 Some Traffic clusters apparently do not support IPv6
Resolved	• crusnov	T271148 Some Data Persistence Backup clusters apparently do not support IPv6
Resolved	brouberol	T312555 Some Search clusters have inconsistent AAAA DNS records for the primary IPv6 of the hosts
Open	None	T312556 Some Core Platform clusters have inconsistent AAAA DNS records for the primary IPv6 of the hosts
Resolved	Volans	T312557 Some WMCS clusters have inconsistent AAAA DNS records for the primary IPv6 of the hosts
Open	None	T369308 Decommission clouddb2002-dev.codfw.wmnet
Open	None	T378260 Retire labtestwiki
Declined	None	T380254 The mwdebug cluster has inconsistent AAAA DNS records for the primary IPv6 of the hosts
Open	MoritzMuehlenhoff	T380256 The maps cluster is missing AAAA DNS records for the primary IPv6 of the hosts

MoritzMuehlenhoff subscribed.May 29 2020, 9:34 PM

Change 599779 merged by Jbond:
[operations/dns@master] AAAA: for flerovium and furud

https://gerrit.wikimedia.org/r/599779

jbond updated the task description. (Show Details)Jun 1 2020, 9:49 AM

Change 599783 merged by Jbond:
[operations/puppet@production] profile/manifests/dumps: enable ipv6 drop ferm rule for 443

https://gerrit.wikimedia.org/r/599783

Maintenance_bot removed a project: Patch-For-Review.Jun 1 2020, 10:10 AM

jbond updated the task description. (Show Details)Jun 1 2020, 11:46 AM

jbond moved this task from Unsorted 💣 to Friday tasks on the User-jbond board.Jun 2 2020, 2:24 PM

tungsten has been decom'ed today. checked the box because it's gone

Volans updated the task description. (Show Details)Oct 1 2020, 7:07 AM

@jbond now that everything is in Netbox, would it make sens to have a Netbox report that shows the hosts that have a primary v4 IP and primary v6 without DNS name? Eventually grouping them by prefix (types).
Non-alerting of course.

Volans mentioned this in T270101: Grants not working with DB hosts with to ipv6.Dec 14 2020, 2:36 PM

I have rejiggered the list in the task description to cross-reference with responsible teams. @Dzahn @Volans could you fill in some gaps as to who is responsible for the ??? hosts? Thanks.

• crusnov updated the task description. (Show Details)Dec 18 2020, 5:41 PM

• crusnov updated the task description. (Show Details)Dec 18 2020, 5:51 PM

I have gone through and checked each of the clusters and updated their status. Most have IPv6 but no DNS. Some have been fixed in the intervening time.

scandium is a testing rig for parsoid, so it can move to serviceops (where parse*/wtp* servers are it belongs as well)

heze is a backup server (offsite), so I think that would be data persistence.

maps .. <unknown value>

• crusnov updated the task description. (Show Details)Jan 4 2021, 6:21 PM

• crusnov mentioned this in T271133: Some Analytics clusters apparently do not support IPv6.Jan 4 2021, 6:33 PM

• crusnov mentioned this in T271134: Some Machine Learning clusters do not support IPv6.Jan 4 2021, 6:38 PM

• crusnov mentioned this in T271138: Some Observability clusters do not support IPv6..Jan 4 2021, 6:45 PM

• crusnov mentioned this in T271139: Some WMCS clusters apparently do not support IPv6.Jan 4 2021, 6:48 PM

• crusnov mentioned this in T271140: Some Data Persistence clusters apparently do not support IPv6.Jan 4 2021, 6:51 PM

• crusnov mentioned this in T271142: Some Service Operations clusters apparently do not support IPv6.Jan 4 2021, 6:55 PM

• crusnov mentioned this in T271143: Some elastic hosts do not have IPv6 DNS records.Jan 4 2021, 6:59 PM

• crusnov mentioned this in T271144: Some Traffic clusters apparently do not support IPv6.

Aklapper added a project: IPv6.Jan 4 2021, 7:37 PM

LSobanski mentioned this in T271148: Some Data Persistence Backup clusters apparently do not support IPv6.Jan 4 2021, 7:42 PM

LSobanski added a subtask: T271148: Some Data Persistence Backup clusters apparently do not support IPv6.

• crusnov updated the task description. (Show Details)Jan 4 2021, 9:56 PM

Can you please add the steps needed in Netbox to make it generate a DNS record for a server which handles IPv6 fine to the task description? We use enable_ip6_mapped by default for practically all servers these days. Is this just a matter of adding of setting "DNS Name" in the interface to the FQDN and running the sre.dns.netbox cookbook or is there more to it?

In T253173#6721846, @MoritzMuehlenhoff wrote:

Is this just a matter of adding of setting "DNS Name" in the interface to the FQDN and running the sre.dns.netbox cookbook or is there more to it?

It's exactly just that. When all the data was imported from the existing DNS into netbox, the DNS Name for IPv6 that didn't have an AAAA/PTR record were skipped to make sure to generate from Netbox the same data the manual DNS repo had.

If you do that make sure that the service is correctly configured to handle traffic on v6 (listen, ferm, grants, etc...).

• ayounsi updated the task description. (Show Details)Jan 5 2021, 9:46 AM

• ayounsi unsubscribed.Jan 5 2021, 9:49 AM

In T253173#6560445, @ayounsi wrote:

@jbond now that everything is in Netbox, would it make sens to have a Netbox report that shows the hosts that have a primary v4 IP and primary v6 without DNS name? Eventually grouping them by prefix (types).
Non-alerting of course.

Sorry missed this but yes i think it would

Is this just a matter of adding of setting "DNS Name" in the interface to the FQDN and running the sre.dns.netbox cookbook or is there more to it?

Just to clarify that yes this is correct however the risk is that not all services are listening on the ipv6 addresses

Ack, added to T222931#6722271

• crusnov moved this task from Discussion / Design / Consensus Making to Work in Progress / Tasks to Do on the netbox board.Jan 8 2021, 7:32 PM

akosiaris closed subtask T271134: Some Machine Learning clusters do not support IPv6 as Declined.Jan 20 2021, 11:06 AM

• crusnov closed subtask T271133: Some Analytics clusters apparently do not support IPv6 as Resolved.Mar 26 2021, 11:45 PM

• crusnov updated the task description. (Show Details)

• crusnov updated the task description. (Show Details)Mar 26 2021, 11:50 PM

• crusnov updated the task description. (Show Details)Mar 26 2021, 11:52 PM

• crusnov updated the task description. (Show Details)Mar 27 2021, 12:00 AM

• crusnov closed subtask T271148: Some Data Persistence Backup clusters apparently do not support IPv6 as Resolved.Mar 27 2021, 12:02 AM

• crusnov updated the task description. (Show Details)Mar 27 2021, 12:10 AM

• crusnov updated the task description. (Show Details)Mar 31 2021, 3:22 PM

Since the maps servers are being replaced? I think? Perhaps we can cross them off for this project. Am I right in that this is happening?

elukey closed subtask T271136: Some Foundation clusters do not appear to support IPv6 as Resolved.Apr 28 2021, 7:20 AM

Aklapper added a project: Infrastructure-Foundations.Jun 21 2021, 8:59 PM

joanna_borun moved this task from Backlog to In Progress on the Infrastructure-Foundations board.Aug 9 2021, 3:44 PM

joanna_borun changed the task status from Open to In Progress.Sep 21 2021, 4:03 PM

Volans changed the task status from In Progress to Open.Dec 6 2021, 5:37 PM

Volans moved this task from In Progress to Blocked on the Infrastructure-Foundations board.

Gehel updated the task description. (Show Details)Dec 21 2021, 10:18 AM

fgiunchedi closed subtask T271138: Some Observability clusters do not support IPv6. as Declined.Mar 23 2022, 1:35 PM

FYI, the full list up to date is on https://netbox.wikimedia.org/extras/reports/network.Network/

bking closed subtask T271143: Some elastic hosts do not have IPv6 DNS records as Resolved.May 17 2022, 10:24 PM

Volans reopened subtask T271143: Some elastic hosts do not have IPv6 DNS records as Open.May 18 2022, 8:28 AM

bking closed subtask T271143: Some elastic hosts do not have IPv6 DNS records as Resolved.Jun 23 2022, 1:35 PM

Volans reopened subtask T271136: Some Foundation clusters do not appear to support IPv6 as Open.Jul 5 2022, 7:56 AM

Volans reopened subtask T271138: Some Observability clusters do not support IPv6. as Open.Jul 7 2022, 2:41 PM

BCornwall closed subtask T271144: Some Traffic clusters apparently do not support IPv6 as Resolved.Jul 7 2022, 6:31 PM

BCornwall updated the task description. (Show Details)

speaking for scandium.eqiad.wmnet. that is a parsoid test server and while used by the parsoid team, is not critical for production. imho it can just be added. the other servers listed for serviceops I'm not sure if they are actually owned by serviceops.

lmata subscribed.Aug 1 2022, 8:21 PM

colewhite closed subtask T271138: Some Observability clusters do not support IPv6. as Resolved.Sep 15 2022, 3:39 PM

fgiunchedi reopened subtask T271138: Some Observability clusters do not support IPv6. as Open.Sep 26 2022, 7:47 AM

Andrew closed subtask T312557: Some WMCS clusters have inconsistent AAAA DNS records for the primary IPv6 of the hosts as Resolved.Nov 17 2022, 9:43 PM

dcaro reopened subtask T312557: Some WMCS clusters have inconsistent AAAA DNS records for the primary IPv6 of the hosts as Stalled.Nov 24 2022, 10:31 AM

dcaro closed subtask T312557: Some WMCS clusters have inconsistent AAAA DNS records for the primary IPv6 of the hosts as Resolved.Nov 29 2022, 9:22 AM

dcaro reopened subtask T312557: Some WMCS clusters have inconsistent AAAA DNS records for the primary IPv6 of the hosts as Stalled.

joanna_borun moved this task from Blocked to Backlog on the Infrastructure-Foundations board.May 26 2023, 2:13 PM

fgiunchedi closed subtask T271138: Some Observability clusters do not support IPv6. as Resolved.Oct 18 2023, 1:38 PM

fgiunchedi updated the task description. (Show Details)

Dzahn unsubscribed.Nov 3 2023, 5:37 PM

@MoritzMuehlenhoff I see that ganeti[2009-2024] and ganeti[1009-1022] are lacking AAAA records while the rest have it. Can we add them to the rest of the cluster?

In T253173#9375719, @Volans wrote:

@MoritzMuehlenhoff I see that ganeti[2009-2024] and ganeti[1009-1022] are lacking AAAA records while the rest have it. Can we add them to the rest of the cluster?

I don't see why not. Best to first start with codfw only, maybe.

From sudo cumin ganeti[1009-1022].eqiad.wmnet 'ip -6 addr | grep "scope global" | grep -v dynamic (and the same in codfw).

Those hosts only have a SLAAC v6 IP: ganeti[2010,2016,2020].codfw.wmnet so the same procedure as on T353254 needs to be done first.

In T253173#9412420, @ayounsi wrote:

Those hosts only have a SLAAC v6 IP: ganeti[2010,2016,2020].codfw.wmnet so the same procedure as on T353254 needs to be done first.

I'll convert these in January. When that is done, we can proceed with the AAAA records.

akosiaris closed subtask T271142: Some Service Operations clusters apparently do not support IPv6 as Resolved.Jun 17 2024, 1:52 PM

akosiaris updated the task description. (Show Details)

• ayounsi removed a project: netbox.Aug 6 2024, 12:53 PM

brouberol closed subtask T312555: Some Search clusters have inconsistent AAAA DNS records for the primary IPv6 of the hosts as Resolved.Aug 27 2024, 2:30 PM

Volans added a subtask: T380254: The mwdebug cluster has inconsistent AAAA DNS records for the primary IPv6 of the hosts.Tue, Nov 19, 9:28 AM

Volans closed subtask T312557: Some WMCS clusters have inconsistent AAAA DNS records for the primary IPv6 of the hosts as Resolved.Tue, Nov 19, 9:31 AM

akosiaris closed subtask T380254: The mwdebug cluster has inconsistent AAAA DNS records for the primary IPv6 of the hosts as Declined.Fri, Nov 22, 9:57 AM