Page MenuHomePhabricator

Automate the detection of netcat listen port in transfer.py
Closed, ResolvedPublic

Description

The transfer.py file which is used for database backup and recovery needs a port to be passed for netcat to listen. Finding a free port by the user may not be convenient. Automating this would ease the use.

Event Timeline

jcrespo triaged this task as Medium priority.May 8 2020, 5:53 AM

Remember to send a patch, even if not finished, sooner than later- it is easier to talk about patches than empty code :-D.

Change 595516 had a related patch set uploaded (by Privacybatm; owner: Privacybatm):
[operations/software/wmfmariadbpy@master] transfer.py: Add the ability to auto-detect free port for netcat to listen

https://gerrit.wikimedia.org/r/595516

Regarding the ss issue: I was able to reproduce this:

# lsmod | wc -l
90
# netstat -tlpn > /dev/null
# lsmod | wc -l
90
# ss -tlpn > /dev/null
# lsmod | wc -l
92

So unless netstat is not available, I think I would stick to it or things will get more complicated, security-wise (remember what we talked about always trying to minimize side effects).

If we want more flexibility, we can make this a method of Firewall.py so we know were we can alter it later (external commands in a separate class).

Oh okay, Thank you for the update!

Change 597158 had a related patch set uploaded (by Privacybatm; owner: Privacybatm):
[operations/software/wmfmariadbpy@master] Add comments to Firewall, MariaDB and transfer modules

https://gerrit.wikimedia.org/r/597158

Change 597158 merged by Jcrespo:
[operations/software/wmfmariadbpy@master] Add comments to Firewall, MariaDB and transfer modules

https://gerrit.wikimedia.org/r/597158

Change 595516 merged by Jcrespo:
[operations/software/wmfmariadbpy@master] transfer.py: Add the ability to auto-detect free port for netcat to listen

https://gerrit.wikimedia.org/r/595516

Aside from solving the issues I mention on the patch, the other thing we should not forget to update is the documentation. This is what it says now:

--port PORT           Port used for netcat listening on the source. By default, 4444, but it must be changed if more 
                      than 1 transfer to the same host happen at the same time, or the second copy will fail top open
                      the socket again. This port has its firewall disabled during transfer automatically with an extra
                      iptables rule.

This is not true anymore, default right now is 4400 and it is smarter, but we can wait until we have a final implementation for the documentation as the last step.

This is now resolved, default port it now 4400, but will open a higher port if the port is in use.

--port still works, but it will fail if the port is not available.

In the future we may be able to configure the range to check.

Good work.