ircservserv not detecting message sender when running behind ZNC v1.8.2
Open, Needs TriagePublicBUG REPORT
Actions

Assigned To

None

Authored By

	bd808
	Nov 16 2024, 2:26 AM

Project Tags

Referenced Files

None

Subscribers

Aklapper

bd808

dcaro

Description

I just put a ZNC bouncer from https://gitlab.wikimedia.org/toolforge-repos/containers-bnc between ircservserv and libera.chat. This is intended to help with keeping the bot connected to the network.

Unfortunately the presence of the bouncer seems to be interfering with the bot's ability to identify the command sender:

[01:48]  ircservserv-wm (~ircservse@wikimedia/bot/ircservserv-wm) joined the channel
[01:48] ChanServ sets mode +v ircservserv-wm
[02:10]  <    bd808> !issync
[02:10]  <ircservserv-wm> Error: You don't have permission to update channel settings
[02:11]  <    bd808> interesting

$ kubectl logs --all-containers=true --ignore-errors --max-log-requests 10 --prefix=true --since=10m --follow --selector 'app.kubernetes.io/name in (bnc, ircservserv)'

[pod/bnc-8b9488f95-r627q/job] [2024-11-16 02:10:54.932417] (ircservserv-wm/libera) IRC -> ZNC [@time=2024-11-16T02:10:54.929Z :bd808!bd808@wikimedia/BDavis-WMF PRIVMSG #wikimedia-cloud :!issync]
[pod/bnc-8b9488f95-r627q/job] [2024-11-16 02:10:54.932683] (ircservserv-wm/libera) ZNC -> CLI [:bd808!bd808@wikimedia/BDavis-WMF PRIVMSG #wikimedia-cloud :!issync]
[pod/bnc-8b9488f95-r627q/job] [2024-11-16 02:10:54.935953] (ircservserv-wm/libera) CLI -> ZNC [PRIVMSG #wikimedia-cloud :Error: You don't have permission to update channel settings]
[pod/ircservserv-6759b74786-rg49j/job] [2024-11-16T02:10:54Z DEBUG ircservserv] Received !issync for #wikimedia-cloud from unknown
[pod/bnc-8b9488f95-r627q/job] [2024-11-16 02:10:54.936156] (ircservserv-wm/libera) ZNC -> IRC [PRIVMSG #wikimedia-cloud :Error: You don't have permission to update channel settings]

Details

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Upgrade to ZNC 1.9.0	toolforge-repos/containers-bnc!1	bd808	work/bd808/use-latest-versions	main

Customize query in GitLab

Related Objects

Mentioned In: T394466: [build-service] remove legacy fagiani/apt 0.2.5 builder from `--use-latest-versions` stack
T363027: [builds-builder] Support adding repositories for Apt buildpack
T380124: ircservserv being kicked for flooding
T380118: ircservserv should use SASL to authenticate to libera.chat
Mentioned Here: T380127: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble
T363027: [builds-builder] Support adding repositories for Apt buildpack
T363033: [builds-builder] Support using custom buildpacks

Event Timeline

bd808 created this task.Nov 16 2024, 2:26 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2024, 2:26 AM

The Received !issync for #wikimedia-cloud from unknown log comes from https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/blob/main/src/main.rs#L112-117

debug!(
    "Received !issync for {} from {}",
    message.response_target().unwrap_or("unknown"),
    extract_account(&message)
        .unwrap_or_else(|| "unknown".to_string())
);

What happens in extract_account(&message)?
https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/blob/main/src/lib.rs#L122-136

/// Given a message, extract the account of the sender
/// using the "account-tags" IRCv3 capability
pub fn extract_account(message: &Message) -> Option<String> {
    if let Some(tags) = &message.tags {
        for tag in tags {
            if tag.0 == "account" {
                if let Some(name) = &tag.1 {
                    return Some(name.to_string());
                }
            }
        }
    }

    None
}

I assume the problem is that ZNC is not sending the account-tag annotations that the implementation expects.

The logs for the bouncer seem to confirm that:

[pod/bnc-8b9488f95-r627q/job] [2024-11-16 02:10:54.932683] (ircservserv-wm/libera) ZNC -> CLI [:bd808!bd808@wikimedia/BDavis-WMF PRIVMSG #wikimedia-cloud :!issync]

But possibly only because ZNC did not register to receive those annotations from the upstream server? It at least seems to be also missing from the inbound message. Note that there is a server-time annotation on the inbound message:

[pod/bnc-8b9488f95-r627q/job] [2024-11-16 02:10:54.932417] (ircservserv-wm/libera) IRC -> ZNC [@time=2024-11-16T02:10:54.929Z :bd808!bd808@wikimedia/BDavis-WMF PRIVMSG #wikimedia-cloud :!issync]

Going back further in ZNC's logs we can see the bouncer tell the client it doesn't know about an account-tag capability:

[2024-11-16 01:48:18.118935] _LISTENER == ConnectionFrom(192.168.61.62, 38694) [Allowed]
[2024-11-16 01:48:18.119178] There are [0] clients from [192.168.61.62]
[2024-11-16 01:48:18.119345] (192.168.61.62) CLI -> ZNC [CAP REQ :multi-prefix account-tag]
[2024-11-16 01:48:18.119495] (192.168.61.62) ZNC -> CLI [:irc.znc.in CAP unknown-nick NAK :multi-prefix account-tag]
[2024-11-16 01:48:18.119718] (192.168.61.62) CLI -> ZNC [CAP END]

I connected to the bouncer using the irssi client and requested the denied capabilities with /msg *send_raw Server ircservserv-wm libera CAP REQ :multi-prefix account-tag. I first needed to load the send_raw module with /msg *status LoadMod send_raw.

[pod/bnc-8b9488f95-r627q/job] [2024-11-16 03:07:11.119718] (ircservserv-wm/libera) CLI -> ZNC [PRIVMSG *send_raw :Server ircservserv-wm libera CAP REQ :multi-prefix account-tag]
[pod/bnc-8b9488f95-r627q/job] [2024-11-16 03:07:11.120077] (ircservserv-wm/libera) ZNC -> IRC [CAP REQ :multi-prefix account-tag]
[pod/bnc-8b9488f95-r627q/job] [2024-11-16 03:07:11.133221] (ircservserv-wm/libera) ZNC -> CLI [:*send_raw!znc@znc.in PRIVMSG ircservserv-wm :Sent [CAP REQ :multi-prefix account-tag] to IRC server of ircservserv-wm/libera]
[pod/bnc-8b9488f95-r627q/job] [2024-11-16 03:07:11.133458] (ircservserv-wm/libera) IRC -> ZNC [@time=2024-11-16T03:07:11.120Z :silver.libera.chat CAP ircservserv-wm ACK :multi-prefix account-tag]

After that the messages from upstream carry the account annotation as hoped:

[pod/bnc-8b9488f95-r627q/job] [2024-11-16 03:09:01.110618] (ircservserv-wm/libera) IRC -> ZNC [@time=2024-11-16T03:09:01.108Z;account=bd808 :bd808!bd808@wikimedia/BDavis-WMF PRIVMSG ##stashbot :ircservserv-wm: hello]

The messages relayed to ircservserv still don't carry the annotation however:

[pod/bnc-8b9488f95-r627q/job] [2024-11-16 03:21:17.098013] (ircservserv-wm/libera) IRC -> ZNC [@time=2024-11-16T03:21:17.096Z;account=bd808 :bd808!bd808@wikimedia/BDavis-WMF PRIVMSG #wikimedia-cloud :!issync]
[pod/bnc-8b9488f95-r627q/job] [2024-11-16 03:21:17.098246] (ircservserv-wm/libera) ZNC -> CLI [:bd808!bd808@wikimedia/BDavis-WMF PRIVMSG #wikimedia-cloud :!issync]
[pod/ircservserv-6759b74786-rg49j/job] [2024-11-16T03:21:17Z DEBUG ircservserv] Received !issync for #wikimedia-cloud from unknown

https://wiki.znc.in/ChangeLog/1.9.0 seems to have the magic we want:

Added support for account-tag capability, also in corecaps module (#1746)

Unfortunately:

03:45 <ircservserv-wm> version
03:45 <*status> ZNC 1.8.2+deb2build5 - https://znc.in
03:45 <*status> IPv6: yes, SSL: yes, DNS: threads, charset: yes, i18n: yes,
                build: cmake

The container gets ZNC from apt.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

We would need a Ubuntu 24.04 noble base image to get 1.9.0.

Stashing the manual changes to https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config that got me this far here. This is the config changes needed to auth to the znc instance and the job to launch znc itself.

diff --git i/config.toml w/config.toml
index 41d1ba9..f72398d 100644
--- i/config.toml
+++ w/config.toml
@@ -49,10 +49,10 @@ channels = [
 owners = []
 nickname = "ircservserv-wm"
 alt_nicks = ["ircservserv-wm_", "ircservserv-wm__"]
-username = "wm-ircservserv"
-realname = "Hmmmmm"
-server = "irc.libera.chat"
-port = 6697
-use_tls = true
+username = "ircservserv-wm"
+realname = "https://meta.wikimedia.org/wiki/IRC/Bots/ircservserv"
+server = "bnc"
+port = 6667
+use_tls = false
 encoding = "UTF-8"
 user_info = "Bots go beep beep."
diff --git i/toolforge/jobs.yaml w/toolforge/jobs.yaml
index d7be8a1..c8c0183 100644
--- i/toolforge/jobs.yaml
+++ w/toolforge/jobs.yaml
@@ -5,3 +5,9 @@
   continuous: true
   emails: onfailure
   mount: all
+- name: bnc
+  command: bouncer
+  image: tool-containers/bnc:latest
+  continuous: true
+  emails: none
+  port: 6667

I also setup some envvars to configure znc:

$ toolforge envvars create BNC_USER ircservserv-wm
$ toolforge envvars create BNC_NICK ircservserv-wm
$ toolforge envvars create BNC_REALNAME https://meta.wikimedia.org/wiki/IRC/Bots/ircservserv
$ toolforge envvars create BNC_PASSWORD

The last change was to make a $HOME/password config file that does not prefix the password with the IRC account name.

bd808 renamed this task from ircservserv not detecting message sender when running behind ZNC bouncer to ircservserv not detecting message sender when running behind ZNC v1.8.2.Nov 16 2024, 4:45 AM

bd808 mentioned this in T380118: ircservserv should use SASL to authenticate to libera.chat.Nov 16 2024, 5:38 PM

bd808 mentioned this in T380124: ircservserv being kicked for flooding.Nov 16 2024, 8:57 PM

In T380108#10328834, @bd808 wrote:

We would need a Ubuntu 24.04 noble base image to get 1.9.0.

I created T380127: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble to see if this might be possible.

Other ways we could get a newer ZNC package:

With T363027: [builds-builder] Support adding repositories for Apt buildpack we could either find or create a PPA with a newer version.
With T363033: [builds-builder] Support using custom buildpacks we could build any version of ZNC we wanted from source.

Cursed idea: I think I could abuse the python build system to perform arbitrary source builds. Modern python has pluggable build backends which introduces the possibility of writing a completely custom backend. That is an extremely cursed idea.

A merely highly cursed idea would be to use hatch + hatch-build-scripts and then add a script step that does the needful of compiling ZNC. A helper for that could be invoke which is shell-oriented subprocess management system that can be used to write Rake-like tasks. I found an example of this idea in the wild with the CapyMOA project which uses hatch + hatch-build-scripts + invoke to download a JAR file and run stubgenj to generate Python stubs during the package build.

bd808 mentioned this in T363027: [builds-builder] Support adding repositories for Apt buildpack.Jan 22 2025, 6:12 PM

In T380108#10366159, @bd808 wrote:

Cursed idea: I think I could abuse the python build system to perform arbitrary source builds. Modern python has pluggable build backends which introduces the possibility of writing a completely custom backend. That is an extremely cursed idea.

A merely highly cursed idea would be to use hatch + hatch-build-scripts and then add a script step that does the needful of compiling ZNC. A helper for that could be invoke which is shell-oriented subprocess management system that can be used to write Rake-like tasks. I found an example of this idea in the wild with the CapyMOA project which uses hatch + hatch-build-scripts + invoke to download a JAR file and run stubgenj to generate Python stubs during the package build.

Another maybe not-so-cursed idea might be to create a python package that bundles a build of znc

T380127: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble is at least partially implemented at this point with a new --use-latest-versions argument to toolforge build start. The new buildpacks however don't like current https://gitlab.wikimedia.org/toolforge-repos/containers-bnc repo.

Building with the legacy buildpacks:

tools.bd808-test@tools-bastion-12:~$ toolforge build start --image-name bnc https://gitlab.wikimedia.org/toolforge-repos/containers-bnc
Waiting for the logs... if the build just started this might take a minute
[prepare] 2025-05-13T00:11:46.680339656Z 2025/05/13 00:11:46 Entrypoint initialization
...
[step-detect] 2025-05-13T00:12:03.850215627Z 3 of 4 buildpacks participating
[step-detect] 2025-05-13T00:12:03.850305249Z fagiani/apt     0.2.5
[step-detect] 2025-05-13T00:12:03.850320884Z heroku/python   0.8.0
[step-detect] 2025-05-13T00:12:03.850334684Z heroku/procfile 2.0.2
...
[step-results] 2025-05-13T00:12:51.671770019Z Built image tools-harbor.wmcloud.org/tool-bd808-test/bnc:latest@sha256:e5adb794379651c819002f123d99d3cf98cc0cafa7b84006b26089a940c1d3b4

Building with the new stack:

tools.bd808-test@tools-bastion-12:~$ toolforge build start --use-latest-versions --image-name bnc https://gitlab.wikimedia.org/toolforge-repos/containers-bnc
Waiting for the logs... if the build just started this might take a minute
[prepare] 2025-05-13T00:19:45.157490308Z 2025/05/13 00:19:45 Entrypoint initialization
...
[step-detect] 2025-05-13T00:20:03.675228280Z target distro name/version labels not found, reading /etc/os-release file
[step-detect] 2025-05-13T00:20:03.846777101Z 4 of 5 buildpacks participating
[step-detect] 2025-05-13T00:20:03.846849355Z fagiani/apt         0.2.5
[step-detect] 2025-05-13T00:20:03.846961265Z heroku/deb-packages 0.1.3
[step-detect] 2025-05-13T00:20:03.847132129Z heroku/python       0.26.1
[step-detect] 2025-05-13T00:20:03.847155742Z heroku/procfile     4.2.1
...
[step-build] 2025-05-13T00:20:04.508189328Z -----> Detected Aptfile changes, flushing cache
[step-build] 2025-05-13T00:20:04.528869392Z -----> Updating apt caches
[step-build] 2025-05-13T00:20:04.568831321Z        Reading package lists...
[step-build] 2025-05-13T00:20:04.580120949Z -----> Fetching .debs for znc
[step-build] 2025-05-13T00:20:04.696581994Z ERROR: failed to build: exit status 1
[step-fix-nested-procfile-launcher] 2025-05-13T00:20:05.681342457Z 2025/05/13 00:20:05 Skipping step because a previous step failed
[step-fix-imagemagick-config] 2025-05-13T00:20:05.890304373Z 2025/05/13 00:20:05 Skipping step because a previous step failed
[step-fix-permissions] 2025-05-13T00:20:06.400446359Z 2025/05/13 00:20:06 Skipping step because a previous step failed
[step-export] 2025-05-13T00:19:51.479838567Z 2025/05/13 00:19:51 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-export] 2025-05-13T00:20:06.524138441Z 2025/05/13 00:20:06 Skipping step because a previous step failed
[step-results] 2025-05-13T00:20:06.668298182Z 2025/05/13 00:20:06 Skipping step because a previous step failed

We now have heroku/deb-packages 0.1.3 in the stack along with the prior fagiani/apt 0.2.5. It is unclear which builder is failing, but I think it has to be fagiani/apt because the docs for heroku/deb-packages indicate that it detects on Aptfile but will stop with error messages about how to migrate to a different config format in project.toml.

bd808 opened https://gitlab.wikimedia.org/toolforge-repos/containers-bnc/-/merge_requests/1

Upgrade to ZNC 1.9.0

bd808 mentioned this in T394466: [build-service] remove legacy fagiani/apt 0.2.5 builder from `--use-latest-versions` stack.May 15 2025, 11:50 PM

bd808 merged https://gitlab.wikimedia.org/toolforge-repos/containers-bnc/-/merge_requests/1