Page MenuHomePhabricator

IRC: Too many user connections (global)
Closed, ResolvedPublic

Description

Author: Beetstra.wiki

Description:
My bots run into problems connecting to IRC:

LINKREPORTER 000009 9: returned error: Closing Link: 208.80.155.255 (Too many user connections (global))
LINKREPORTER 000009 9: IRC: Reporter 9 got disconnect call from irc.freenode.net, attempt to reconnect @ ##

(part of LiWa3)

Can tools please have unlimited IRC connections. I also see sometimes one of my other bots disappear and only return after quite some time - suggesting it has the same issue until one line comes free.


Version: unspecified
Severity: normal

Details

Reference
bz64802

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:11 AM
bzimport added a project: Toolforge.
bzimport set Reference to bz64802.
bzimport created this task.May 3 2014, 5:34 PM

Only freenode can change this limit. Have you tried it with ipv6?

Info on requestion exemption:

20:29 < Chiyo> which, if that is the case, you can ask for an iline
20:29 < Chiyo> please e-mail ilines@freenode.net with the IP address range in question, the reason for the limit increase, the duration if applicable, how many simultaneous connections are needed, and reliable contact information :)

However, I was under the impression this had already been handled for labs. Maybe this is no longer the case due to the move to eqiad?

Beetstra.wiki wrote:

(In reply to Steinsplitter from comment #1)

Only freenode can change this limit. Have you tried it with ipv6?

I am not sure how IPV6 would solve this (except that Freenode probably sees it as another IP, so assigning again # connections to it) - it is still a limit. Not really a solution.

Freenode would indeed be the place to change the limit.

By the way, I think that severity should be higher than 'normal' .. this affects all users who have IRC-connecting bots, if one is lucky theirs connects but that means that someone else is unlucky. Basically, it seems that there will always be tools down (next time it may be morebots who is not able to connect).

Is this dependent on which tools-exec-instance is running the bot?

Also, from which host is this?

tools-login is
tools-dev is 208.80.155.132
tools-exec-0y is 208.80.155.14(y-1)

which is probably why I haven't had any issues with IRC bots -- there's not *that* many bots on exec nodes.

Beetstra.wiki wrote:

Note: the instance giving problems at the moment is tools-exec-10, I don't know however if the observed cases with my other bots were also while that particular bot was running on tools-exec-10.

Okay, I should have tested further than -01..-04:

tools-exec-01 is 208.80.155.140
tools-exec-02 is 208.80.155.141
tools-exec-03 is 208.80.155.142
tools-exec-04 is 208.80.155.143
tools-exec-05 is 208.80.155.144
tools-exec-06 is 208.80.155.145
tools-exec-07 is 208.80.155.146
tools-exec-08 is 208.80.155.147
tools-exec-09 is 208.80.155.255
tools-exec-10 is 208.80.155.255

So probably exec-09 and exec-10 are using a centralized labs NAT IP? Could we get exempts for those, or should the exec hosts just get their own IPs?

Even then, we might want to get exempts as multiple bots could get submitted to the same host.

scfc added a comment.May 3 2014, 7:30 PM

(In reply to Merlijn van Deen from comment #6)

[...]

So probably exec-09 and exec-10 are using a centralized labs NAT IP? [...]

Just to confirm: (Fixed) public IPs have been assigned via [[wikitech:Special:NovaAddress]] for:

  • tools-dev
  • tools-exec-01
  • tools-exec-02
  • tools-exec-03
  • tools-exec-04
  • tools-exec-05
  • tools-exec-06
  • tools-exec-07
  • tools-exec-08
  • tools-login
  • tools-mail
  • tools-proxy-test
  • tools-webproxy

So tools-exec-09 and tools-exec-10 are indeed using a NAT IP.

I, I have the same problem, my last instance was named intelirc, and it die when the server was disconnected :/

coren triaged this task as High priority.Nov 25 2014, 4:13 PM
coren set Security to None.
coren moved this task from Triage to Backlog on the Toolforge board.
coren moved this task from Backlog to In Progress on the Toolforge board.Dec 23 2014, 7:33 PM

Is this still an issue?

coren added a comment.Feb 10 2015, 6:13 PM

It is still an issue insofar as currently, not all exec nodes where bots run have distinct IPs so that identd works (this is a Freenode requirement to avoid all bots being "counted" as the same).

I'll raise the quota and add the IPs shortly, this will fix the issue.

coren closed this task as Resolved.Feb 10 2015, 6:26 PM

All exec nodes now have public IPs and should not run into per-IP limits on Freenode.

Restricted Application added a project: Cloud-Services. · View Herald TranscriptMay 9 2017, 8:01 PM