openstack network problems (November 2024)
Open, HighPublic
Actions

Assigned To

None

Authored By

	aborrero
	Nov 26 2024, 4:14 PM

Description

This ticket is to track an openstack network problem that we believe it may be the cause of 2 outages.

T380827: tools-nfs outage 2024-11-25
T380844: 2024-11-26 Toolforge DNS incident

And possibly the reason for other DNS issues reported in the past few weeks:

T381078: cloudgw: suspected network problems
T379927: Puppet removed "nameserver" line from /etc/resolv.conf
T374830: Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for our own hosts such as gerrit.wikimedia.org

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T380882 openstack network problems (November 2024)
Resolved		aborrero	T380827 tools-nfs outage 2024-11-25
Open		None	T380829 rsyslogd tls issues on tools-nfs-2
Resolved		None	T380832 [jobs-api] crashing
Resolved		aborrero	T380844 2024-11-26 Toolforge DNS incident
Open		None	T380886 openstack: increase virtual network observability
Open		None	T380892 [infra,k8s,o11y] introduce additional observability for calico and general networking
Resolved		aborrero	T380972 openstack: prevent puppet from restarting neutron-openvswitch-agent
Open		None	T380833 [harbor] some artifacts and projects seems to have gone missing
Open		None	T380980 openstack: introduce additional DNS monitoring and alerting
Open		None	T380985 [infra,k8s,o11y] Introduce worker checks
Open	BUG REPORT	ssingh	T379927 Puppet removed "nameserver" line from /etc/resolv.conf
Resolved		fnegri	T381092 2024-11-25 ProjectProxyMainProxyDown
Open		Andrew	T374830 Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for our own hosts such as gerrit.wikimedia.org
Open		None	T381078 cloudgw: suspected network problems
In Progress		fnegri	T381373 Restrict outbound connectivity from PAWS hosts
Resolved		fnegri	T382220 KernelError Server cloudgw1002 may have kernel errors
Resolved		fnegri	T382290 ProbeDown
Resolved		fnegri	T382222 ProbeDown Service wan.cloudgw.eqiad1.wikimediacloud.org:0 has failed probes (icmp_wan_cloudgw_eqiad1_wikimediacloud_org_ip4)
Open		fnegri	T382356 replace cloudgw100[12] with spare 'second region' dev servers cloudnet100[78]-dev
Open		None	T382412 Relocate cloudnet1007-dev and cloudnet1008-dev to new racks and rename
Resolved		fnegri	T380692 ProbeDown Service virt.cloudgw.eqiad1.wikimediacloud.org:0 has failed probes (icmp_virt_cloudgw_eqiad1_wikimediacloud_org_ip4)