Page MenuHomePhabricator

Debianize Python's pid library to be able to use it from reportupdater
Closed, ResolvedPublic

Description

A community member has wrote a patch that substantially improves reportupdater handling of PID files.
See: T173604 and https://gerrit.wikimedia.org/r/c/analytics/reportupdater/+/623470/

Now, the code makes use of Python3's pid library, which is currently not available from an-launcher1002.
The pid lib needs to be debianized and added to the stat machines for the reportupdater improvement to work.

Some links that might help:

And some examples: https://gerrit.wikimedia.org/r/admin/repos/q/filter:operations%252Fdebs%252Fpython

Event Timeline

Hm, Perhaps @razzi can take this one? We have to get him permissions on the build server. It looks like we might need a new group for that, in lieu of making in an ops admin.

Change 628913 had a related patch set uploaded (by Razzi; owner: Razzi):
[operations/debs/python-pid@master] Initial debianization of python-pid

https://gerrit.wikimedia.org/r/628913

Change 628913 merged by Ottomata:
[operations/debs/python-pid@master] Initial debianization of python-pid

https://gerrit.wikimedia.org/r/628913

Change 628924 had a related patch set uploaded (by Razzi; owner: Razzi):
[operations/debs/python-pid@debian] Initial debianization of python-pid

https://gerrit.wikimedia.org/r/628924

Change 628924 merged by Ottomata:
[operations/debs/python-pid@debian] Initial debianization of python-pid

https://gerrit.wikimedia.org/r/628924

Change 628929 had a related patch set uploaded (by Razzi; owner: Razzi):
[operations/puppet@production] Add python3-pid debian package to reportupdater

https://gerrit.wikimedia.org/r/628929

Change 628929 merged by Razzi:
[operations/puppet@production] Add python3-pid debian package to reportupdater

https://gerrit.wikimedia.org/r/628929

Can @mforns confirm that reportupdater can use this package as its?

Can @mforns confirm that reportupdater can use this package as its?

It works great! (tested the new code in an-launcher1002)

Now, one behavior changed with the new use of the pid library: When reportupdater is executed and there's already another instance of the same job running, the new execution crashes with the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pid/base.py", line 198, in create
    self._flock(self.fh.fileno())
  File "/usr/lib/python3/dist-packages/pid/posix.py", line 22, in _flock
    fcntl.flock(self.fh.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
BlockingIOError: [Errno 11] Resource temporarily unavailable

Before, reportupdater would just log and exit. This situation can happen, if reportupdater is backfilling, and should theoretically not crash.
Will comment on that in the patch.