I think this could be an interesting Wikimedia-Hackathon-2019 project, ideally with guidance from someone from cloud-services-team. Let’s add it to the board and see if anything comes out of it :)

https://github.com/wikimedia/puppet/blob/production/modules/scap/files/dologmsg is basically what you're after (if you want to save some wheel reinventing) - you can use it directly on the command line, rather than just via scap...

Just needs an ircecho instance running somewhere as per https://github.com/wikimedia/puppet/blob/production/modules/scap/files/manpages/man/dologmsg.1 that the hosts could access

Thanks, that helps. (I looked for scripts starting with log* in the $PATH, but didn’t think to search for ones with log anywhere in the name, so I didn’t find dologmsg.)

I think it would be good to run the ircecho on some VM, either in the Tools project or in a dedicated one – that way, we get a stable hostname (not some random grid/k8s worker) but aren’t open to the whole internet (should only be reachable from within *.eqiad.wmflabs IMO).

By the way, does ircecho authenticate itself on Freenode? I didn’t see anything like that skimming its source code.

In T222244#5149275, @LucasWerkmeister wrote:

By the way, does ircecho authenticate itself on Freenode? I didn’t see anything like that skimming its source code.

https://github.com/wikimedia/operations-debs-ircecho/blob/master/ircecho

Doesn't look to.... But logmsgbot is cloaked

[13:44:18] * [Whois] logmsgbot is logmsgbot!~logmsgbot@wikimedia/bot/logmsgbot (logmsgbot)
[13:44:18] * [Whois] logmsgbot has voice on channels: #wikimedia-operations
[13:44:18] * [Whois] logmsgbot is online via tepper.freenode.net (US)
[13:44:18] * [Whois] logmsgbot is using a secure connection
[13:44:18] * [Whois] logmsgbot is authed as logmsgbot
[13:44:18] * [Whois] End of /WHOIS list

Something in SingleServerIRCBot?

In T222244#5149275, @LucasWerkmeister wrote:

By the way, does ircecho authenticate itself on Freenode? I didn’t see anything like that skimming its source code.

It looks like we don't use that deb anymore. But instead it is:

In modules/profile/manifests/icinga/ircbot.pp there is $password_file = '/etc/icinga/.irc_secret and content => secret('icinga/icinga-wm_irc.secret'), which sets the password string.

Then this profile uses class { '::ircecho': and in there file { '/usr/local/bin/ircecho': with source => 'puppet:///modules/ircecho/ircecho.py',. So that is just a file in puppet.

Wait, I’m confused now – do we need an ircecho or a tcpircbot? modules/profile/manifests/icinga/ircbot.pp seems to be for the Icinga alerts, logmsgbot is in modules/profile/manifests/tcpircbot.pp as far as I can tell (with a separate Python script, modules/tcpircbot/files/tcpircbot.py, and getting its password from a different source as well, the passwords::logmsgbot class in private.git/modules/passwords/manifests/init.pp).

A side question regarding the following:

In T222244#5149275, @LucasWerkmeister wrote:

I think it would be good to run the ircecho on some VM, either in the Tools project or in a dedicated one – that way, we get a stable hostname (not some random grid/k8s worker) but aren’t open to the whole internet (should only be reachable from within *.eqiad.wmflabs IMO).

A dedicated project might take a while to set up (according to Cloud-VPS (Project-requests)) – is that process accelerated during the Hackathon or should I perhaps kick it off early? Or we can do it in deployment-prep – I don’t have admin rights there, but @Lucas_Werkmeister_WMDE does, and I guess the distinction between us will just become a bit blurry ;) I’m an idiot – the right project would be tools, not deployment-prep, and I don’t have rights there with any account.

People from the cloud team will be in Prague, for sure

I don't know if it would be a good idea to run the normal ircecho from prod inside tools like that, IIRC it doesn't do any authentication of the originator of the message?

wm-bot's relay functionality works in #wikmedia-cloud. Currently no token is needed. @Legoktm has some automation scripts for some of his projects that use this to send !log messages.

Mentioned in SAL (#wikimedia-cloud) [2019-05-15T20:24:57Z] <wm-bot> Toolforge dologmsg test (T222244)

@bd808 nice, thanks! I used the following script to generate the above message:

#!/bin/bash

groups=($(groups))

if ((${#groups[@]} > 1)); then
    printf >&2 '%s: more than one group found: %s\n' "$0" "${groups[*]}"
    exit 1
elif ((${#groups[@]} < 1)); then
    printf >&2 '%s: no group recognized\n' "$0"
    exit 1
fi
group=${groups[0]}

labsproject=$(</etc/wmflabs-project)
if ! [[ $group == "$labsproject"* ]]; then
    printf >&2 '%s: group does not start with "%s": %s\n' "$0" "$labsproject" "$group"
    exit 1
fi

message=$*
if [[ $message == "" ]]; then
    printf >&2 '%s: no message\n' "$0"
    exit 1
fi

printf '#wikimedia-cloud !log %s %s\n' "$group" "$*" > /dev/tcp/wm-bot2.wm-bot.eqiad.wmflabs/64834

dologmsg 'Toolforge dologmsg test (T222244)' # quoted because () are metacharacters, but the script should support multiple arguments (bare words) too

So I guess (assuming wm-bot’s relay functionality remains functional) we would just need to get that script reviewed and added in Toolforge’s $PATH? (The /etc/wmflabs-project bit is stolen from _become, the shell completion function for the become command.)

@Krenair I’m not sure what exactly you’re concerned about, to be honest – as far as I’m aware, the whole !log system isn’t very locked down at any point…?

(For bonus points, we could include the $SUDO_USER in the message – though that’s not trusted information, of course.)

Change 510750 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/puppet@production] Add script to log to SAL from Toolforge servers

https://gerrit.wikimedia.org/r/510750

Change 510750 merged by Bstorm:
[operations/puppet@production] Add script to log to SAL from Toolforge servers

https://gerrit.wikimedia.org/r/510750

Change 510999 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/puppet@production] Extract variables from Toolforge dologmsg

https://gerrit.wikimedia.org/r/510999

Change 511043 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/puppet@production] Add -h/--help option to Toolforge dologmsg command

https://gerrit.wikimedia.org/r/511043

Change 513759 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/puppet@production] dologmsg: add manpage

https://gerrit.wikimedia.org/r/513759

Change 511043 merged by Bstorm:
[operations/puppet@production] dologmsg: add -h/--help option

https://gerrit.wikimedia.org/r/511043

Change 510999 merged by Bstorm:
[operations/puppet@production] dologmsg: extract variables from Toolforge dologmsg

https://gerrit.wikimedia.org/r/510999

Change 513759 merged by Bstorm:
[operations/puppet@production] dologmsg: add manpage

https://gerrit.wikimedia.org/r/513759

Change 540188 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] dologmsg: ensure the directory exists before trying to add the man page

https://gerrit.wikimedia.org/r/540188

Change 540188 merged by Bstorm:
[operations/puppet@production] dologmsg: ensure the directory exists before trying to add the man page

https://gerrit.wikimedia.org/r/540188