make sure that after reboot, rpc.statd starts on dumpsdata1002
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ArielGlenn
	Nov 28 2019, 11:44 AM

Description

Buster doesn't start statd automatically, in contrast to jessie (see https://launchpad.net/debian/+source/nfs-utils/+changelog and search for "don't start statd by default"). I have modified the defaults for /etc/default/nfs-common (see https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/553462/ ) but I can't restart the daemon right now because there are misc cron dumps in progress. So I've started statd manually for now via systemctl.

In order to be sure that statd does indeed run automatically, I should reboot when the wikidata weeklies are done, probably on Saturday.

Details

	Subject	Repo	Branch	Lines +/-
	explicitly start rpc.statd service on dumpsdata servers	operations/puppet	production	+9 -0

Customize query in gerrit

Event Timeline

ArielGlenn triaged this task as Medium priority.Nov 28 2019, 11:44 AM

ArielGlenn created this task.

For the record and for my future self, this issue manifested as failure to get locks over nfs from a client:

fcntl.lockf(fhandle, fcntl.LOCK_EX | fcntl.LOCK_NB)
OSError: [Errno 37] No locks available

Both server and client are using nfs v3.

I started rpc.statd manually on the other two dumpsdata servers as well.

ArielGlenn moved this task from Backlog to Up Next on the Dumps-Generation board.Nov 29 2019, 6:37 AM

Reboot done and rcp.statd did not start, so I have again restarted it manually. I'll leave things as they are for the weekend and see what's needed on Monday, since it's not urgent. Probably I will have to explicitly enable and start the service in puppet.

ArielGlenn renamed this task from reboot dumpsdata1002 to makre sure that after reboot, rpc.statd starts on dumpsdata1002.Dec 2 2019, 10:35 AM

ArielGlenn moved this task from Up Next to Active on the Dumps-Generation board.

https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1428486 shows that setting NEED_STATD=yes is guaranteed not to work. Explicit enable needed, patch coming.

Change 554066 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] explicitly start rpc.statd service on dumpsdata servers

https://gerrit.wikimedia.org/r/554066

gerritbot added a project: Patch-For-Review.Dec 2 2019, 1:43 PM

Change 554066 merged by ArielGlenn:
[operations/puppet@production] explicitly start rpc.statd service on dumpsdata servers

https://gerrit.wikimedia.org/r/554066

The above is now live. Need to do another reboot test when the misc crons are done, so that will be Saturday again. If it pans out, I'll add the locking back in to the adds-changes dumps then too.

Maintenance_bot removed a project: Patch-For-Review.Dec 2 2019, 3:10 PM

Rebooted and all is well. Sometime on Sunday I'll enable locking again on the adds-changes dumps and check Monday that they still run properly.

New adds-changes dumps are being produced after this patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/555732/ was deployed so I can close this now.

ArielGlenn moved this task from Active to Done on the Dumps-Generation board.Dec 20 2019, 11:43 AM

make sure that after reboot, rpc.statd starts on dumpsdata1002Closed, ResolvedPublicActions

Description

Details

Event Timeline

make sure that after reboot, rpc.statd starts on dumpsdata1002
Closed, ResolvedPublic
Actions