Page MenuHomePhabricator
Create Task
Maniphest T380882

openstack network problems (November 2024)
Closed, ResolvedPublic
Actions

Description

This ticket is to track an openstack network problem that we believe it may be the cause of 2 outages.

T380827: tools-nfs outage 2024-11-25
T380844: 2024-11-26 Toolforge DNS incident

And possibly the reason for other DNS issues reported in the past few weeks:

T381078: cloudgw: suspected network problems
T379927: Puppet removed "nameserver" line from /etc/resolv.conf
T374830: Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for our own hosts such as gerrit.wikimedia.org

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• aborreroT380882 openstack network problems (November 2024)
 Resolved• aborreroT380827 tools-nfs outage 2024-11-25
 ResolvedNoneT380832 [jobs-api] crashing
 Resolved• aborreroT380844 2024-11-26 Toolforge DNS incident
 OpenNoneT380886 openstack: increase virtual network observability
 OpenNoneT380892 [infra,k8s,o11y] introduce additional observability for calico and general networking
 Resolved• aborreroT380972 openstack: prevent puppet from restarting neutron-openvswitch-agent
 OpenNoneT380833 [harbor] some artifacts and projects seems to have gone missing
 OpenNoneT380980 openstack: introduce additional DNS monitoring and alerting
 OpenNoneT380985 [infra,k8s,o11y] Introduce worker checks
 ResolvedBUG REPORTssinghT379927 Puppet removed "nameserver" line from /etc/resolv.conf
 ResolvedfnegriT381092 2024-11-25 ProjectProxyMainProxyDown
 ResolvedAndrewT374830 Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for our own hosts such as gerrit.wikimedia.org
 Resolved• aborreroT381078 cloudgw: suspected network problems
 ResolvedfnegriT381373 Restrict outbound connectivity from PAWS hosts
 ResolvedfnegriT383261 Remove hardcoded NFT rules related to PAWS workers
 ResolvedfnegriT382220 KernelError Server cloudgw1002 may have kernel errors
 ResolvedfnegriT382290 ProbeDown
ResolvedfnegriT382222 ProbeDown Service wan.cloudgw.eqiad1.wikimediacloud.org:0 has failed probes (icmp_wan_cloudgw_eqiad1_wikimediacloud_org_ip4)
 Resolved• aborreroT382356 replace cloudgw100[12] with spare 'second region' dev servers cloudnet100[78]-dev
 ResolvedAndrewT382412 Relocate cloudnet1007-dev and cloudnet1008-dev to new racks and rename
 Invalid• aborreroT385600 Cloud DNS: investigate weird graph during cloudgw replacement operation
 ResolvedRequestVRiley-WMFT386810 decommission cloudgw100[12]
 ResolvedfnegriT380692 ProbeDown Service virt.cloudgw.eqiad1.wikimediacloud.org:0 has failed probes (icmp_virt_cloudgw_eqiad1_wikimediacloud_org_ip4)

Event Timeline

• aborrero created this task.Nov 26 2024, 4:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 26 2024, 4:14 PM
• aborrero triaged this task as High priority.Nov 26 2024, 4:14 PM
• aborrero added a subtask: T380827: tools-nfs outage 2024-11-25.
• aborrero added a subtask: T380844: 2024-11-26 Toolforge DNS incident.
• aborrero added a project: User-aborrero.
dcaro subscribed.Nov 26 2024, 4:16 PM
• aborrero moved this task from Backlog to Radar/observer on the User-aborrero board.Nov 26 2024, 4:16 PM
fnegri added a project: Cloud-VPS.Nov 26 2024, 4:20 PM
fnegri subscribed.
fnegri moved this task from Unsorted to Network on the Cloud-VPS board.Nov 26 2024, 5:09 PM
• aborrero changed the status of subtask T380972: openstack: prevent puppet from restarting neutron-openvswitch-agent from Open to In Progress.Nov 27 2024, 11:56 AM
• aborrero added a subtask: T380833: [harbor] some artifacts and projects seems to have gone missing.Nov 27 2024, 12:54 PM
• aborrero closed subtask T380972: openstack: prevent puppet from restarting neutron-openvswitch-agent as Resolved.Nov 27 2024, 1:10 PM
• aborrero renamed this task from 2024-11-16 openstack network problems to 2024-11-26 openstack network problems.Nov 27 2024, 1:23 PM
• aborrero closed subtask T380844: 2024-11-26 Toolforge DNS incident as Resolved.
• aborrero closed subtask T380827: tools-nfs outage 2024-11-25 as Resolved.
fnegri mentioned this in T380991: Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27).Nov 27 2024, 4:28 PM
fnegri added subtasks: T379927: Puppet removed "nameserver" line from /etc/resolv.conf, T374830: Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for our own hosts such as gerrit.wikimedia.org.Nov 27 2024, 4:34 PM
fnegri renamed this task from 2024-11-26 openstack network problems to openstack network problems.Nov 27 2024, 4:38 PM
fnegri updated the task description. (Show Details)
Aklapper renamed this task from openstack network problems to openstack network problems (November 2024).Nov 27 2024, 5:20 PM
fnegri updated the task description. (Show Details)Dec 4 2024, 3:29 PM
RhinosF1 subscribed.Jan 3 2025, 4:02 PM
• aborrero closed subtask T381078: cloudgw: suspected network problems as Resolved.Feb 19 2025, 1:28 PM
Andrew closed subtask T374830: Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for our own hosts such as gerrit.wikimedia.org as Resolved.Feb 25 2025, 12:30 PM
• aborrero closed this task as Resolved.Mar 3 2025, 12:00 PM
• aborrero claimed this task.

I think the most important subtasks have been resolved, so I will close this ticket. We can keep working on the few subtasks that remain opened.

Andrew closed subtask T379927: Puppet removed "nameserver" line from /etc/resolv.conf as Resolved.May 2 2025, 12:52 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits