delbu9c1 wrote:

As a short note, it is my understanding that this extension works on a list of known tor exit nodes. How often is this list updated? If it is not updated frequently enough, then it's likely that the more dynamic nodes will not be blocked.

• Shirik

djgwiki wrote:

As the checkuser that's spent all day chasing this person around (and the person who has subsequently found his request for a bot to automate what he's doing got tied up in needless red tape), an urgent fix would be appreciate. This isn't a theoretical vulnerability, it's a practical one that has absorbed a rather large amount of my time today.

-Deskana

Despite having just completed about 40 hours of travel, I isolated this problem to a failure to load the exit node list, caused by a regression in HttpFunctions resulting from its recent rewrite and the invalid SSL certificate held by the maintainers of the tor server.

I've fixed the issue preventing updates, and forced an immediate update. Tor nodes should again be blocked with immediate effect.

I'd recommend rolling back the manual blocks placed on them, as tor exit nodes have an average lifespan on the order of days or weeks, not five years.

djgwiki wrote:

I'll roll back the blocks tomorrow when I'm awake.

Thanks for the quick fix.

delbu9c1 wrote:

Based on recent CU evidence, it appears that there are still tor nodes that aren't being blocked:

http://en.wikipedia.org/w/index.php?diff=358434527&oldid=358430355

You're not giving me anything to work with, here. Mail me nodes that should be blocked that aren't, or edits through tor nodes that should have been disallowed, or *something*.

mike.lifeguard+bugs wrote:

(In reply to comment #6)

You're not giving me anything to work with, here. Mail me nodes that should be
blocked that aren't, or edits through tor nodes that should have been
disallowed, or *something*.

They weren't being blocked at all for that time period. At all. Do you still want a list of IPs? The GlobalBlocking log on Meta has some, I can pick out the relevant section if you need me to.

(In reply to comment #7)

They weren't being blocked at all for that time period. At all. Do you still
want a list of IPs? The GlobalBlocking log on Meta has some, I can pick out the
relevant section if you need me to.

I'm referring to the reopening of the bug.

mike.lifeguard+bugs wrote:

Here's a list of IPs I blocked during the time period from when this bug was re-opened. They were almost certainly all tor nodes at the time & used for vandalism.

130.217.76.77
24.94.65.2
77.109.139.87
80.193.114.10
83.78.60.241
92.62.52.188
95.170.245.53

avarab wrote:

(In reply to comment #9)

Here's a list of IPs I blocked during the time period from when this bug was
re-opened. They were almost certainly all tor nodes at the time & used for
vandalism.

130.217..76.77
24.94.65.2
77.109.139.87
80.193.114.10
83.78.60.241
92.62.52.188
95.170.245.53

This is almost certainly just a race condition. Here's a check against
the Tor exit node list (from
https://check.torproject.org/cgi-bin/TorBulkExitList.py):

$ for i in 130.217.76.77 24.94.65.2 77.109.139.87 80.193.114.10 83.78.60.241 92.62.52.188 95.170.245.53; do ack -c $i tor.txt; done
tor.txt:0
tor.txt:1
tor.txt:1
tor.txt:1
tor.txt:0
tor.txt:1
tor.txt:1

5/7 were exit nodes when I fetched the list, 2 no longer were (if they ever were in the past, I'll take Mike's word for it).

Tor exit nodes are spinning up and going down all the time. You can configure Tor so that it only runs at full throughput for a few minutes every day. I don't think it would be part of the exported *current* exit node list if you happened to fetch it a few hours later.

I think the best workaround for this would be to regularly poll the list of exit nodes and ban anyone that either is a Tor exit now, or has been so for the past 2 weeks or so.

I suspect it'll always be possible to sneak through. I could spin up a (new) Tor exit now that would be part of the Tor network within minutes. A dedicated attacker could just keep trying to vandalize Wikipedia hoping to get some edits in through Tor in that window.

avarab wrote:

(In reply to comment #10)

(In reply to comment #9)

Here's a list of IPs I blocked during the time period from when this bug was
re-opened. They were almost certainly all tor nodes at the time & used for
vandalism.

130.217..76.77
24.94.65.2
77.109.139.87
80.193.114.10
83.78.60.241
92.62.52.188
95.170.245.53

This is almost certainly just a race condition. Here's a check against
the Tor exit node list (from
https://check.torproject.org/cgi-bin/TorBulkExitList.py):

After looking at the source it looks like it is. Exit nodes are stored for 30 minutes but new ones are spinning up all the time.

See this graph for reference for how they vary over the day:

http://munin.nix.is/nix.is/v.nix.is/tor_routers.html

I suggest storing the fetch time along with the IP in memcache and updating it as appropriate, dropping old entries after some configured amount of time.

There's another race condition.
When the time comes to reload the tor cache[1], the cache is first marked as being updated, the list is fetched, then stored.
A request for the exit nodes while it is being reloaded will result in an empty set and thus ips not blocked.

1- I assume that loadExitNodes.php is run every 25 minutes by a cron job on wmf servers, but would happen even if it was directly loaded by the servers.

There's another race condition.

r66275 should fix that. Needs a sysadmin to apply it.

No intention to live-merge this revision, as it affects <30 seconds every half hour, and no evidence has been elicited that there is significant vandalism occurring in these refresh periods.

avarab wrote:

The vandalism isn't occurring in the 30 second window, but the 1800 second window that the tor exit list is cached for. In that half an hour a lot of new exit nodes will spin up, as the graph I showed demonstrated.

I.e. Platonides's patch doesn't solve the problem.

zzuuzz wrote:

I've reopened this bug, having followed it since it was first raised I can tell you that this extension has just been getting worse and is currently next to useless on enwiki. I've been consistently blocking several Tor nodes on most days since this was first raised. Now the vandals are really catching on and causing problems.

I can't determine any pattern - it affects long term permanent nodes as well as more dynamic ones. All the nodes have been able to exit to Wikipedia according to any Tor checker. There's a serious failure happening somewhere, so perhaps someone could take another look.

Can you provide specific instances?

The possible holes it occurs to me are the refetching time, some error, or new nodes which were added since the last update (the node list is cached for 30 minutes).

I'll try to take a look at this in the next couple of weeks, but I need specifics.

zzuuzz wrote:

Here's a sample from the last 24 hours. I've included the uptime reported by Tor as it was a few minutes ago, so it's not accurate but should give some idea. Of course many of the longer term nodes have been manually blocked by admins, but they pop up from time to time. This one for instance: 174.36.199.203 was editing recently.

108.7.61.129 uptime 468929
66.8.152.121 uptime 148865
206.248.138.177 uptime 3283382
82.95.152.30 uptime 1274475
91.200.79.122 uptime 8601
67.182.56.220 uptime 130268
92.243.189.2 uptime 549
88.161.177.191 uptime 915
46.4.208.206 uptime 207156
66.58.182.38 uptime 714921
99.251.211.225 uptime 12627
88.166.161.41 uptime 742
109.91.187.162 uptime 7138
88.160.214.26 uptime 507
109.191.8.168 uptime 183
66.74.163.57 uptime 915
98.126.1.234 uptime 426215

zzuuzz wrote:

Here are some more going back to the start of October. These are just the ones I've blocked. I'm aware there have been others. Again I've included the uptime as reported a few minutes ago. All were confirmed at the time I blocked them.

115.84.182.227 uptime 324953
194.0.229.54 uptime 1713811
178.63.198.71 now defunct
117.18.75.235 uptime 325500
89.16.175.194 uptime 55151
85.217.65.155 uptime 788037
77.109.139.87 uptime 1604624
78.229.64.71 uptime 4392
194.154.227.109 uptime 8252751
62.141.53.224 uptime 1158035
78.129.227.207 now defunct
109.174.52.21 now defunct
137.56.163.46 uptime 874168
128.6.224.107 uptime 7717424
178.18.16.119 now defunct
61.32.46.9 now defunct
137.56.163.64 uptime 794079
74.208.213.82 uptime 270456
91.213.50.235 uptime 3088635
71.80.220.255 uptime 324747
96.232.187.242 now defunct
78.46.39.228 uptime 894958
193.34.144.124 uptime 2466579
91.203.170.121 uptime 27816
69.175.122.198 now defunct
216.194.67.53 now defunct

I confirm that tor blocking is not working.

The list of tor nodes is empty (but it's there!) so the cron loader is running but not able to retrieve it.

My guess is that the server running loadExitNodes.php (used to be hume) can't connect to check.torproject.org:443 Maybe it can't even resolve the dns.

I found the problem: loadExitNodes.php (which does indeed run on hume) is run with php -n . Running it without the -n flag unbreaks it.

I've run the script manually so the exit node list is good now, but it'll be blanked by the broken script on its next run (it runs every 20 mins). I'll ask a root to remove the -n flag in the crontab.

Ariel did this, and we now have a full exit node list again, yay!