The transferpy is a package used for database backup and recovery. It has the capability to automatically detect a free port for netcat to listen. If we start two instances of transferpy simultaneously for the same destination machine, there is a possibility that both instances detect the same port as the free port (race condition). This need to be solved.
This ticket also needs to solve the transferpy checksum filename concurrency issue.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | jcrespo | T246435 Create or improve a tool for monitoring or automating tasks for Wikimedia databases | |||
| Resolved | Privacybatm | T248256 GSoC 2020 Proposal: Improve the framework to transfer files over the LAN | |||
| Resolved | Privacybatm | T256450 Solve transferpy concurrency issue with auto port detection and checksum file names |
This race condition can be solved/reduced by making a directory (mkdir) in temp as soon as we see a free port.
Change 608274 had a related patch set uploaded (by Privacybatm; owner: Privacybatm):
[operations/software/transferpy@master] Firewall.py: Solve auto port detection concurrency issue
https://gerrit.wikimedia.org/r/c/operations/software/transferpy/ /608274
Change 608274 merged by jenkins-bot:
[operations/software/transferpy@master] Firewall.py: Solve auto port detection concurrency issue
The shared checksum path on parallel execution issue should get top priority, as this was already observed problematic when running to independent executions of transfer.py to the same host as target. While they had no issues coordinating which port to use, they were reusing the same file for checksums.
Change 613633 had a related patch set uploaded (by Privacybatm; owner: Privacybatm):
[operations/software/transferpy@master] Transferer.py: Resolve concurrency issue with checksum file names
Change 614686 had a related patch set uploaded (by Privacybatm; owner: Privacybatm):
[operations/software/transferpy@master] Transferer.py: Add proper cleanup
Change 613633 merged by jenkins-bot:
[operations/software/transferpy@master] Transferer.py: Resolve concurrency issue with checksum file names
Change 614686 merged by jenkins-bot:
[operations/software/transferpy@master] Transferer.py: Add proper cleanup