The transfer.py file which is used for database backup and recovery needs a port to be passed for netcat to listen. Finding a free port by the user may not be convenient. Automating this would ease the use.
Description
Details
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | jcrespo | T246435 Create or improve a tool for monitoring or automating tasks for Wikimedia databases | |||
Resolved | Privacybatm | T248256 GSoC 2020 Proposal: Improve the framework to transfer files over the LAN | |||
Resolved | Privacybatm | T252171 Automate the detection of netcat listen port in transfer.py | |||
Resolved | Privacybatm | T252950 kill_job function in remote execution module of transfer framework does not close the ports instantly |
Event Timeline
Remember to send a patch, even if not finished, sooner than later- it is easier to talk about patches than empty code :-D.
Change 595516 had a related patch set uploaded (by Privacybatm; owner: Privacybatm):
[operations/software/wmfmariadbpy@master] transfer.py: Add the ability to auto-detect free port for netcat to listen
Regarding the ss issue: I was able to reproduce this:
# lsmod | wc -l 90 # netstat -tlpn > /dev/null # lsmod | wc -l 90 # ss -tlpn > /dev/null # lsmod | wc -l 92
So unless netstat is not available, I think I would stick to it or things will get more complicated, security-wise (remember what we talked about always trying to minimize side effects).
If we want more flexibility, we can make this a method of Firewall.py so we know were we can alter it later (external commands in a separate class).
Change 597158 had a related patch set uploaded (by Privacybatm; owner: Privacybatm):
[operations/software/wmfmariadbpy@master] Add comments to Firewall, MariaDB and transfer modules
Change 597158 merged by Jcrespo:
[operations/software/wmfmariadbpy@master] Add comments to Firewall, MariaDB and transfer modules
Change 595516 merged by Jcrespo:
[operations/software/wmfmariadbpy@master] transfer.py: Add the ability to auto-detect free port for netcat to listen
Aside from solving the issues I mention on the patch, the other thing we should not forget to update is the documentation. This is what it says now:
--port PORT Port used for netcat listening on the source. By default, 4444, but it must be changed if more than 1 transfer to the same host happen at the same time, or the second copy will fail top open the socket again. This port has its firewall disabled during transfer automatically with an extra iptables rule.
This is not true anymore, default right now is 4400 and it is smarter, but we can wait until we have a final implementation for the documentation as the last step.
This is now resolved, default port it now 4400, but will open a higher port if the port is in use.
--port still works, but it will fail if the port is not available.
In the future we may be able to configure the range to check.
Good work.