Page MenuHomePhabricator
Create Task
Maniphest T294955

cloud network: improve automated testing & monitoring
Closed, ResolvedPublic
Actions

Description

Per parent task, our cloud network resilience could be improved by adding automated tests & more monitoring.

Idea here: P15659

Details

SubjectRepoBranchLines +/-
openstack: networktests: fix toolforge.org IP addressoperations/puppetproduction+1 -1
openstack: networktests: replace curl silent argumentoperations/puppetproduction+1 -1
openstack: networktests: update eqiad1 bastionoperations/puppetproduction+1 -1
openstack: networktests: introduce in eqiad1operations/puppetproduction+45 -0
openstack: networktests: fix timer job interval specificationoperations/puppetproduction+1 -1
openstack: networktests: add systemd timer job to run the test suiteoperations/puppetproduction+38 -15
openstack: networktests: run as dedicated useroperations/puppetproduction+19 -0
wmcs: add openstack network tests cookbookoperations/cookbookswmcs+145 -0
openstack: monitor: cmd-checklist-runner: exit with a different return codeoperations/puppetproduction+13 -2
wikimediacloud.org: add A records for cloudgw2001-dev/2002-devoperations/dnsmaster+2 -1
cloud: networktests: rework some of the raw icmp checksoperations/puppetproduction+23 -11
cloud: networktests: fix some testcasesoperations/puppetproduction+10 -7
cloud: networktests: use -qoperations/puppetproduction+1 -1
cloud: networktests: ssh: use -qoperations/puppetproduction+0 -0
cloud: networktests: runner: use expanded_cmdoperations/puppetproduction+1 -1
cloud: networktests: add missing ROUTING_SOURCE_IP envvaroperations/puppetproduction+1 -1
openstack: networktests: correct some problemsoperations/puppetproduction+22 -21
cloud: introduce network testsoperations/puppetproduction+522 -0
secret: add openstack networktests sshkeys placeholderslabs/privatemaster+9 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

aborrero renamed this task from cloud network: improve monitoring to cloud network: improve automated testing & monitoring.Nov 3 2021, 5:17 PM
aborrero created this task.
aborrero updated the task description. (Show Details)
aborrero triaged this task as Medium priority.Nov 3 2021, 6:41 PM
gerritbot added a comment.Nov 4 2021, 4:42 PM

Change 736819 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: introduce network tests

https://gerrit.wikimedia.org/r/736819

gerritbot added a project: Patch-For-Review.Nov 4 2021, 4:42 PM
aborrero claimed this task.Nov 5 2021, 12:20 PM
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
Stashbot added a comment.Nov 8 2021, 10:34 AM

Mentioned in SAL (#wikimedia-cloud) [2021-11-08T10:34:35Z] <arturo> create service account srv-networktests following https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Service_accounts for T294955

Stashbot added a comment.Nov 8 2021, 10:54 AM

Mentioned in SAL (#wikimedia-cloud) [2021-11-08T10:54:38Z] <arturo> [codfw1dev] create service account srv-networktests following https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Service_accounts for T294955

gerritbot added a comment.Nov 8 2021, 11:06 AM

Change 737346 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[labs/private@master] secret: add openstack networktests sshkeys placeholders

https://gerrit.wikimedia.org/r/737346

gerritbot added a comment.Nov 8 2021, 11:06 AM

Change 737346 merged by Arturo Borrero Gonzalez:

[labs/private@master] secret: add openstack networktests sshkeys placeholders

https://gerrit.wikimedia.org/r/737346

aborrero mentioned this in rLPRI07376ef58bc7: secret: add openstack networktests sshkeys placeholders.Nov 8 2021, 11:24 AM
gerritbot added a comment.Nov 8 2021, 12:31 PM

Change 736819 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: introduce network tests

https://gerrit.wikimedia.org/r/736819

Maintenance_bot removed a project: Patch-For-Review.Nov 8 2021, 1:10 PM
gerritbot added a comment.Nov 8 2021, 1:47 PM

Change 737392 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: correct some problems

https://gerrit.wikimedia.org/r/737392

gerritbot added a project: Patch-For-Review.Nov 8 2021, 1:47 PM
gerritbot added a comment.Nov 8 2021, 1:49 PM

Change 737392 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: correct some problems

https://gerrit.wikimedia.org/r/737392

Maintenance_bot removed a project: Patch-For-Review.Nov 8 2021, 2:11 PM
gerritbot added a comment.Nov 8 2021, 3:59 PM

Change 737418 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: networktests: add missing ROUTING_SOURCE_IP envvar

https://gerrit.wikimedia.org/r/737418

gerritbot added a project: Patch-For-Review.Nov 8 2021, 3:59 PM

Change 737418 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: networktests: add missing ROUTING_SOURCE_IP envvar

https://gerrit.wikimedia.org/r/737418

Maintenance_bot removed a project: Patch-For-Review.Nov 8 2021, 4:10 PM
gerritbot added a comment.Nov 9 2021, 7:56 AM

Change 737613 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: networktests: runner: use expanded_cmd

https://gerrit.wikimedia.org/r/737613

gerritbot added a project: Patch-For-Review.Nov 9 2021, 7:56 AM

Change 737614 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: networktests: ssh: use -q

https://gerrit.wikimedia.org/r/737614

gerritbot added a comment.Nov 9 2021, 7:58 AM

Change 737613 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: networktests: runner: use expanded_cmd

https://gerrit.wikimedia.org/r/737613

gerritbot added a comment.Nov 9 2021, 7:59 AM

Change 737614 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: networktests: ssh: use -q

https://gerrit.wikimedia.org/r/737614

gerritbot added a comment.Nov 9 2021, 8:01 AM

Change 737615 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: networktests: use -q

https://gerrit.wikimedia.org/r/737615

gerritbot added a comment.Nov 9 2021, 8:02 AM

Change 737615 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: networktests: use -q

https://gerrit.wikimedia.org/r/737615

Maintenance_bot removed a project: Patch-For-Review.Nov 9 2021, 8:10 AM
gerritbot added a comment.Nov 9 2021, 8:31 AM

Change 737620 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: networktests: fix some testcases

https://gerrit.wikimedia.org/r/737620

gerritbot added a project: Patch-For-Review.Nov 9 2021, 8:31 AM
gerritbot added a comment.Nov 9 2021, 9:09 AM

Change 737620 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: networktests: fix some testcases

https://gerrit.wikimedia.org/r/737620

Maintenance_bot removed a project: Patch-For-Review.Nov 9 2021, 9:10 AM
ayounsi closed subtask T295288: cr-codfw: set up static route for 185.15.57.8/30 as Resolved.Nov 9 2021, 10:01 AM
gerritbot added a comment.Nov 9 2021, 11:06 AM

Change 737642 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: networktests: rework some of the raw icmp checks

https://gerrit.wikimedia.org/r/737642

gerritbot added a project: Patch-For-Review.Nov 9 2021, 11:06 AM
gerritbot added a comment.Nov 9 2021, 11:09 AM

Change 737642 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: networktests: rework some of the raw icmp checks

https://gerrit.wikimedia.org/r/737642

gerritbot added a comment.Nov 9 2021, 11:17 AM

Change 737648 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/dns@master] wikimediacloud.org: add A records for cloudgw2001-dev/2002-dev

https://gerrit.wikimedia.org/r/737648

gerritbot added a comment.Nov 9 2021, 11:19 AM

Change 737648 merged by Arturo Borrero Gonzalez:

[operations/dns@master] wikimediacloud.org: add A records for cloudgw2001-dev/2002-dev

https://gerrit.wikimedia.org/r/737648

gerritbot added a comment.Nov 9 2021, 1:05 PM

Change 737667 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/cookbooks@wmcs] wmcs: add openstack network tests cookbook

https://gerrit.wikimedia.org/r/737667

gerritbot added a comment.Nov 9 2021, 4:41 PM

Change 737741 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: monitor: cmd-checklist-runner: exit with a different return code

https://gerrit.wikimedia.org/r/737741

gerritbot added a comment.Nov 9 2021, 4:42 PM

Change 737741 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: monitor: cmd-checklist-runner: exit with a different return code

https://gerrit.wikimedia.org/r/737741

gerritbot added a comment.Nov 10 2021, 2:00 PM

Change 737667 merged by jenkins-bot:

[operations/cookbooks@wmcs] wmcs: add openstack network tests cookbook

https://gerrit.wikimedia.org/r/737667

gerritbot added a comment.Nov 11 2021, 8:33 AM

Change 738068 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: add systemd timer job to run the test suite

https://gerrit.wikimedia.org/r/738068

gerritbot added a comment.Nov 11 2021, 8:49 AM

Change 738070 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: run as dedicated user

https://gerrit.wikimedia.org/r/738070

gerritbot added a comment.Nov 11 2021, 8:53 AM

Change 738070 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: run as dedicated user

https://gerrit.wikimedia.org/r/738070

gerritbot added a comment.Nov 11 2021, 9:06 AM

Change 738068 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: add systemd timer job to run the test suite

https://gerrit.wikimedia.org/r/738068

gerritbot added a comment.Nov 11 2021, 9:29 AM

Change 738180 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: fix timer job interval specification

https://gerrit.wikimedia.org/r/738180

gerritbot added a comment.Nov 11 2021, 9:35 AM

Change 738180 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: fix timer job interval specification

https://gerrit.wikimedia.org/r/738180

gerritbot added a comment.Nov 11 2021, 10:08 AM

Change 738191 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: introduce in eqiad1

https://gerrit.wikimedia.org/r/738191

gerritbot added a comment.Nov 11 2021, 10:12 AM

Change 738191 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: introduce in eqiad1

https://gerrit.wikimedia.org/r/738191

Stashbot added a comment.Nov 11 2021, 10:47 AM

Mentioned in SAL (#wikimedia-cloud) [2021-11-11T10:47:09Z] <arturo> add user srv-networktests as project user (T294955)

Stashbot added a comment.Nov 11 2021, 10:50 AM

Mentioned in SAL (#wikimedia-cloud) [2021-11-11T10:50:28Z] <arturo> add user srv-networktests as project user (T294955)

Stashbot added a comment.Nov 11 2021, 10:50 AM

Mentioned in SAL (#wikimedia-cloud) [2021-11-11T10:50:53Z] <arturo> add user srv-networktests as project user (T294955)

gerritbot added a comment.Nov 11 2021, 12:37 PM

Change 738211 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: update eqiad1 bastion

https://gerrit.wikimedia.org/r/738211

gerritbot added a comment.Nov 11 2021, 12:39 PM

Change 738211 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: update eqiad1 bastion

https://gerrit.wikimedia.org/r/738211

gerritbot added a comment.Nov 11 2021, 12:48 PM

Change 738212 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: replace curl silent argument

https://gerrit.wikimedia.org/r/738212

gerritbot added a comment.Nov 11 2021, 12:49 PM

Change 738212 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: replace curl silent argument

https://gerrit.wikimedia.org/r/738212

gerritbot added a comment.Nov 11 2021, 12:52 PM

Change 738214 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] openstack: networktests: fix toolforge.org IP address

https://gerrit.wikimedia.org/r/738214

gerritbot added a comment.Nov 11 2021, 12:53 PM

Change 738214 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] openstack: networktests: fix toolforge.org IP address

https://gerrit.wikimedia.org/r/738214

aborrero closed this task as Resolved.Nov 11 2021, 1:33 PM

Got to a nice stopping point.

  • tests have been deployed to both codfw1dev and eqiad1
  • a spicerack cookbook has been created to help with automated usage
  • a periodic job has been setup to help us monitor the health of the network
  • however, the periodic job depends on icinga monitoring systemd services, which by the time of this writing is disabled for eqiad1
  • anyway, not sure yet if we want to be paged by errors reported by the testsuite (not sure yet how stable it will be)
  • in any case, I'm leaving the systemd timer job enabled to at least we can see the logs and know if the network wasn't stable at some point
  • some docs have been created https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Network/Tests

TODO:

  • decide if we want to be paged by this
  • extend with more checks!
taavi mentioned this in T298501: Email flood due to a some email issue and a full disk on tools prometheus.Jan 6 2022, 12:01 PM
Krinkle edited projects, added Sustainability (Incident Followup); removed Patch-For-Review.Jan 10 2022, 6:05 PM
nskaggs mentioned this in T294853: 2021-11-02 Cloud VPS network outage.Sep 30 2022, 4:21 PM
aborrero closed subtask T294956: Avoid unnecessary keepalived flap after rebooting servers as Resolved.Oct 31 2022, 4:20 PM
dcaro mentioned this in rCCKB8930240a3eaa: wmcs: add openstack network tests cookbook.Dec 14 2022, 3:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL