Page MenuHomePhabricator

irc.beta.wmflabs.org is unresponsive
Open, Needs TriagePublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

  • Connect to irc.beta.wmflabs.org port 6667 in any IRC client, or simply via telnet as a basic health check.

What happens?:

Connection times out.

$ telnet irc.beta.wmflabs.org 6667
Trying 172.16.6.8...

What should have happened instead?:

$ telnet irc.wikimedia.org 6667
Trying 2620:0:861:3:208:80:154:72...
Connected to irc1003.wikimedia.org.

Other information:

I was looking at this subdomain because I noticed it in DNS on Horizon as part of T289318: Move *.beta.wmflabs.org to *.beta.wmcloud.org.

What I've checked so far:

Security Groups for deployment-ircd03 in Horizon:
group irc
ALLOW IPv4 6667/tcp from 0.0.0.0/0
ALLOW IPv4 to 0.0.0.0/0
ALLOW IPv6 to ::/0
ALLOW IPv6 9390/udp from 2a02:ec80:a000::/56
ALLOW IPv4 9390/udp from 172.16.0.0/17
		'irc' => [ 'irc.svc.deployment-prep.eqiad1.wikimedia.cloud' ],


'wmgRC2UDPPort' => [
	'default' => 9390,
],

foreach ( $wmgLocalServices['irc'] as $i => $address ) {
		$wgRCFeeds["irc$i"] = [
			'formatter' => 'IRCColourfulRCFeedFormatter',
			'uri' => "udp://$address:$wmgRC2UDPPort/$wmgRC2UDPPrefix",
			'add_interwiki_prefix' => false,
  • I can connect to the host, with ssh deployment-ircd03.deployment-prep.eqiad1.wikimedia.cloud
  • When I run tcpdump there, and then make an edit or visit a to-me new wiki, I see messages being received from MediaWiki to the IRC server:
krinkle@deployment-ircd03:~$ sudo tcpdump -X dst port 9390

01:51:57.356469 IP deployment-mediawiki14.deployment-prep.eqiad1.wikimedia.cloud.43860 > deployment-ircd03.deployment-prep.eqiad1.wikimedia.cloud.9390: UDP, length 154
     … #central [[User:Krinkle]]@crhwiki https://crh.wikipedia.beta.wmflabs.org/wiki/Qullan%C4%B1c%C4%B1:Krinkle Krinkle *

Event Timeline

I think this whole service needs to be rebuilt. When I did an audit of running services on 2025-02-06 I found that deployment-ircd was still running the software stack from https://wikitech.wikimedia.org/wiki/Obsolete:IRCD rather than the expected https://wikitech.wikimedia.org/wiki/Ircstream service that has been used in production for a year or so.

Is this service actually in use? I wonder if we could get away with decomissioning it instead of figuring out how to port it to the new setup.

Mentioned in SAL (#wikimedia-releng) [2025-07-08T17:40:39Z] <Krinkle> T289318, T396088: Add DNS for irc.wikimedia.beta.wmcloud.org pointing to deployment-ircd03, same as irc.beta.wmflabs.org (while at it, addig missing "wikimedia" part to the new new wmcloud domain to match prod).