Page MenuHomePhabricator

transfer.py fails to run 2 commands
Closed, ResolvedPublicBUG REPORT

Assigned To
Authored By
Privacybatm
May 7 2020, 10:42 PM
Referenced Files
F31808438: output4
May 8 2020, 9:01 PM
F31807188: config.yaml
May 7 2020, 10:46 PM
F31807187: output3
May 7 2020, 10:42 PM
F31807178: config.yaml
May 7 2020, 10:42 PM
F31807181: output1
May 7 2020, 10:42 PM

Description

This ticket is about transfer.py which is used for database backup and recovery

Steps to Reproduce:

  1. Clone the repository operations/software/wmfmariadbpy and tun tox -e unit(worked successfully)
  2. Made passwordless ssh connection to 192.168.43.43
  3. Created config.yaml in /etc/cumin

then run: python transfer.py --no-checksum --no-compress --no-encrypt --type=file 192.168.43.43:/home/gem5lab/sender/test 192.168.43.43:/home/gem5lab/receiver

Actual Results:
Error :


I commented out open/close_firewall section and ran again gave me 2 failures: (file copied successfully)

Expected Results:
File copies successfully without any failures.

Event Timeline

The error you are getting is probably because you are using ubuntu, and I think that uses its own userpace solution "ufw" (uncomplicated firewall). We at Wikimedia support only Debian and we use iptables. Another possibility is that your user may not have permissions to handle the firewall.

What I recommend strongly is to consider setting up a virtual machine with Debian and iptables, so you have the exact same environment than we do. We support Debian but not Ubuntu. While this should not be a huge difference, small differences may make development more complicated. Remember that if it works on your laptop, but doesn't work on WMF servers, it won't be useful for us!

If you wanted to make this work, we could abstract the firewall opening method, and either add an option to skip the port opening or to support alternative methods.

The other "error" you are getting, is not an error, cumin reports failure if the command returns other than 0, but some of the commands will actually return different than 0, and that is ok. In particular,

bash -c [-d X]

checks if a directory X exists, and return different than 0 if it doesn't. This is something that we want (0 would be a undesired state), and even if cumin considers it like a command execution failure, it is 100% what we want (a directory not to exist as so it will not be overwritten).

What we care is if python has an exception, does something we don't want or returns an error itself. We expect cumin to return a non 0 result. Sadly, at the moment there is not way to hide cumin output- we will want to hide it if possible to not alert users of things like this.

Thank you for the clarification. I have

The error you are getting is probably because you are using ubuntu, and I think that uses its own userpace solution "ufw" (uncomplicated firewall). We at Wikimedia support only Debian and we use iptables. Another possibility is that your user may not have permissions to handle the firewall.

What I recommend strongly is to consider setting up a virtual machine with Debian and iptables, so you have the exact same environment than we do. We support Debian but not Ubuntu.

I have installed Debian to have the same environment as production. Still, I am facing issue with iptables.

I have couple of doubts:

  1. Should I install some specific version of iptables? (I'm using Debian10 with iptables v1.8.2)
  2. Should I do any configuration related to cumin in the sender/receiver machine?

So transfer.py is trying to run the following command:

/sbin/iptables -A INPUT -p tcp -s <source_host> --dport <dest_port> -j ACCEPT

Should I install some specific version of iptables? (I'm using Debian10 with iptables v1.8.2):

  • I doubt it is a version issue, iptables syntax shouldn't change from version to version.

What do you get if you try to run the above manually, on the machine to copy to (not the cumin one)? That will most likely give us an explanation of why it fails. transfer.py generally assumes you have a firewall setup, and it needs to open a "hole" to write to it. You can run iptables -L to understand your current configuration. Maybe it is not installed/setup on the target machine.

Which user is cumin sending commands as in the target machine? Maybe the user doesn't have enough permissions. Normally, transfer.py assumes it has root-like privileges in order to handle the firewall.

It is very likely it is one of the 2. For example, on my machine with Buster:

jynus@sangai:/media/home/jynus/puppet$ /sbin/iptables -L
iptables: Permission denied (you must be root).
jynus@sangai:/media/home/jynus/puppet$ sudo /sbin/iptables -L
[sudo] password for jynus: 
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination 

jynus@sangai:/media/home/jynus/puppet$ sudo /sbin/iptables -A INPUT -p tcp -s 192.168.1.244 --dport 4444 -j ACCEPT
jynus@sangai:/media/home/jynus/puppet$ sudo /sbin/iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     tcp  --  192.168.1.244        anywhere             tcp dpt:4444

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Don't worry, "setting up your development environement" is precisely part of this month's work.

In any case, it shouldn't return an exception, failing to open the firewall should just provide a useful error- you have found something you can fix there.

Thank you for your help.
Now it is running fine. There were 2 issues.

  1. Permission issue, Now running as root and everything is working fine. (I searched a lot to find a way to run as a non-root user (who has passwordless sudo access), but I couldn't find any!)
  2. Debian10-iptables uses nftables, changing it to the iptables-legacy mode resolved the --dport error.

@jcrespo Should we consider adding 'sudo' to these commands in the code?

Now running as root and everything is working fine.

Yeah, cumin is thought as a root/sysadmin orchestrator. I would consider this a bug, but the fix would be to update documentation (transfer.py) to add such requirement (hint hint 0:-D).

Debian10-iptables uses nftables, changing it to the iptables-legacy mode resolved the --dport error.

I wasn't aware of that, sorry. We use a mix of Debian 9 and Debian 10, and we are using exclusively an iptables workflow at WMF at the moment. This reinforces the idea on T252172#6118180, to separate the firewall handling to a separate, maintainable file.

Should we consider adding 'sudo' to these commands in the code?

I don't think so by default, as cumin requires root or sudo privileges, as documented: https://wikitech.wikimedia.org/wiki/Cumin#Production_infrastructure
I wouldn't be against it as a non-default option, but I don't think that should be a priority right now, as it could add complexity. If any, the change should be on cumin, not on transfer.py (and maybe that options is possible already).

I would focus on fixing documentation "how to install/requirement" and the refactoring first.

PS: not against increasing security by applying minimal privileges, but transfer does more things than running nc and handling iptables- it also connects to mysql and has access to private files, so it may increase complexity. Something to review, but for later stages.

I would focus on fixing documentation "how to install/requirement" and the refactoring first.

Thank you for your suggestion.

This reinforces the idea on T252172#6118180, to separate the firewall handling to a separate, maintainable file.

Yeah, it makes perfect sense.