Page MenuHomePhabricator

Stop some services before and healthcheck labstore1004/5 following row C network change
Closed, ResolvedPublic

Description

This is the tools NFS and home/project NFS for everything but maps.

  • With puppet disabled (will be disabled fleetwide during network changes anyway) shut down maintain-dbusers before the network changes.
  • After the changes verify that sudo drbd-overview looks good
  • Check that NFS is back everywhere

Event Timeline

aborrero triaged this task as High priority.
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

For the record, before the operation:

aborrero@labstore1004:~ $ sudo drbd-overview 
 1:test/0   Connected Primary/Secondary UpToDate/UpToDate /srv/test  ext4 9.8G 535M 8.7G 6%  
 3:misc/0   Connected Primary/Secondary UpToDate/UpToDate /srv/misc  ext4 5.0T 1.8T 3.0T 38% 
 4:tools/0  Connected Primary/Secondary UpToDate/UpToDate /srv/tools ext4 8.0T 5.7T 1.9T 76% 
aborrero@labstore1005:~ $ sudo drbd-overview 
 1:test/0   Connected Secondary/Primary UpToDate/UpToDate 
 3:misc/0   Connected Secondary/Primary UpToDate/UpToDate 
 4:tools/0  Connected Secondary/Primary UpToDate/UpToDate 

I'll leave this test running during the operation:

aborrero@tools-sgebastion-07:~$ while true ; do sleep 2 ; timeout 2 nc -z nfs-tools-project.svc.eqiad.wmnet 2049 && echo -n . || echo -n x ; done
.........................
aborrero@tools-sgebastion-08:~$ while true ; do sleep 2 ; timeout 2 nc -z nfs-tools-project.svc.eqiad.wmnet 2049 && echo -n . || echo -n x ; done
...................................

Checking every 2 seconds if the NFS port is reachable from the main toolforge bastions.

Done. No issues detected with the connection test.

I can write/read to NFS from Toolforge as usual.

For completeness, drbd after the operation:

aborrero@labstore1004:~ $ sudo drbd-overview 
 1:test/0   Connected Primary/Secondary UpToDate/UpToDate /srv/test  ext4 9.8G 535M 8.7G 6%  
 3:misc/0   Connected Primary/Secondary UpToDate/UpToDate /srv/misc  ext4 5.0T 1.8T 3.0T 38% 
 4:tools/0  Connected Primary/Secondary UpToDate/UpToDate /srv/tools ext4 8.0T 5.7T 1.9T 76% 
aborrero@labstore1005:~ $ sudo drbd-overview 
 1:test/0   Connected Secondary/Primary UpToDate/UpToDate 
 3:misc/0   Connected Secondary/Primary UpToDate/UpToDate 
 4:tools/0  Connected Secondary/Primary UpToDate/UpToDate