Not entirely sure what's going on. The bot is running but there do not seem to be any emails coming in:

Thanks! It seems that it's an issue with some of the k8s hosts, but it happens relatively rarely. Combined with the overall migration to Gitlab (which will likely require a complete rethink of the bot) it's not a high priority to fix for me (but happy to give access to anyone who is interested to dig in!)

Cronjob disabled (one-off project which already finished a long time ago, but had forgotten to turn off the cronjob before...)

I think someone just restarted everything. Looking at the logs, there doesn't seem to be anything wrong with the Phab integration, rather the IRC bot had trouble connecting.

@Marostegui which command(s) did you run, exactly?

And it works if I first kill the webservice. Probably a quota issue then? A limit of 3 continuous jobs seemed pretty limited when I read it the first time, and when the webservice also counts against it...

I get the same exception if I try to run the job by hand:

tools.wikibugs@tools-sgebastion-10:~$ toolforge-jobs run redis2irc --command "libera/wrapper.sh redis2irc" --image tf-python39 --continuous --emails all
ERROR: unable to create job: "HTTP 403: likely an internal bug: 403 Client Error: Forbidden for url: https://k8s.tools.eqiad1.wikimedia.cloud:6443/apis/apps/v1/namespaces/tool-wikibugs/deployments. k8s JSON: {\"kind\": \"Deployment\", \"apiVersion\": \"apps/v1\", \"metadata\": {\"name\": \"redis2irc\", \"namespace\": \"tool-wikibugs\", \"labels\": {\"toolforge\": \"tool\", \"app.kubernetes.io/version\": \"1\", \"app.kubernetes.io/managed-by\": \"toolforge-jobs-framework\", \"app.kubernetes.io/created-by\": \"wikibugs\", \"app.kubernetes.io/component\": \"deployments\", \"app.kubernetes.io/name\": \"redis2irc\", \"jobs.toolforge.org/filelog\": \"yes\", \"jobs.toolforge.org/emails\": \"all\"}}, \"spec\": {\"template\": {\"metadata\": {\"labels\": {\"toolforge\": \"tool\", \"app.kubernetes.io/version\": \"1\", \"app.kubernetes.io/managed-by\": \"toolforge-jobs-framework\", \"app.kubernetes.io/created-by\": \"wikibugs\", \"app.kubernetes.io/component\": \"deployments\", \"app.kubernetes.io/name\": \"redis2irc\", \"jobs.toolforge.org/filelog\": \"yes\", \"jobs.toolforge.org/emails\": \"all\"}}, \"spec\": {\"restartPolicy\": \"Always\", \"containers\": [{\"name\": \"redis2irc\", \"image\": \"docker-registry.tools.wmflabs.org/toolforge-python39-sssd-base:latest\", \"workingDir\": \"/data/project/wikibugs\", \"command\": [\"/bin/sh\", \"-c\", \"--\", \"libera/wrapper.sh redis2irc 1>>redis2irc.out 2>>redis2irc.err\"], \"resources\": {}, \"env\": [{\"name\": \"HOME\", \"value\": \"/data/project/wikibugs\"}], \"volumeMounts\": [{\"mountPath\": \"/data/project\", \"name\": \"home\"}]}], \"volumes\": [{\"name\": \"home\", \"hostPath\": {\"path\": \"/data/project\", \"type\": \"Directory\"}}]}}, \"replicas\": 1, \"selector\": {\"matchLabels\": {\"toolforge\": \"tool\", \"app.kubernetes.io/version\": \"1\", \"app.kubernetes.io/managed-by\": \"toolforge-jobs-framework\", \"app.kubernetes.io/created-by\": \"wikibugs\", \"app.kubernetes.io/component\": \"deployments\", \"app.kubernetes.io/name\": \"redis2irc\", \"jobs.toolforge.org/filelog\": \"yes\", \"jobs.toolforge.org/emails\": \"all\"}}}}"

fdsfadsdfsa

bump²

fdsa

Test

See https://www.mediawiki.org/wiki/Wikibugs#Configuring_channels for how to do this.
An admin has to autovoice Wikibugs in the channel before the channel can be added to the configuration.

From the log file:

The main bit of code seems to be in https://github.com/pywikibot/Pywikibot-nightly-creator

I think I know what's going on -- it's an issue I fixed earlier for ReleaseTaggerBot (which uses the same email based system), but forgot to also apply here: T284587: https://github.com/wikimedia/labs-tools-forrestbot/commit/76a76db66a2fa6a54493b402bd9c68376f0d70e0

https://gerrit-reviewer-bot.toolforge.org/ contains the last 50-or-so log lines; if you have access to tool labs I'm fairly certain you can just read the .out and .err logs in the home dir. The logging is not super extensive but should give a decent idea of what's happening.

In T288381#7315659, @Legoktm wrote:

:S

Do you have a patch or an idea how much changes are needed for adding the integration? That does seem like the best way forward. cc'ing @brennen since I'm not really sure what our capacity for local patches is .

by "building" do you mean patching the GitLab code?

@Legoktm mentioned Debian's gitlab instance (Salsa) has some irc integration as well. Seems to be the built-in Irker plus something based on webhooks & https://salsa.debian.org/kgb-team/kgb/-/blob/master/script/kgb-bot. This is something we could also deploy in principle (with wikibugs just handling phab, and kgb handling merge requests etc).

With a little bit of hacking, I was able to make the Mattermost integration act like a regular webhook:

A few further notes after fiddling with a fresh Gitlab instance.

Some thoughts on architecture when using a webhook

https://gitlab.wikimedia.org/api/v4/events?target_type=merge_request seems to be pretty much what we need, except it can only filter by after={date} and not by date and time... so it's not particularly convenient for polling. We may have to go for the inverse route (either emails or webhooks).

Oooh, yes, that's going to be fun. A few thoughts.

Would be happy to be on the front lines with Wikibugs2, as it needs to be adapted for GitLab (T288381). Eat your own dog food, etc. (cc @Legoktm)

Note that I may have missed changes before 6 jun (which is the oldest log file).

Decided to use a script after realizing the number of clicks required for the manual approach. Should all be sorted now :-)

List seems to be the following:

Deployed and re-enabled bot. Looking into the logs now.

For (b): found a source email with the same issue -- the email has mixed CRLF and LF newlines which explains why things got confused. Created test + fix (see above). Will deploy this first, and will then investigate what I can find on other situations where the \r may have caused issues.

Hmm, that doesn't look good:

forrestbot.log.2021-06-05:2021-06-05 15:00:13,036 - forrestbot - DEBUG - <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/698040>: skipping (SkipMailException('Project mediawiki/core\r is not being watched',))

It's always great when you fix a bug twice, merge once, then come back yet another year later to merge the first one.

Ooof. That was a long time ago :-)

The main issue seems solved when replacing connect_ by connect; however the tests then still fail, most likely because the font settings etc. in Windows 10 have changed significantly over where they were in Windows 7. Given that the code is largely 'battle-tested' and the tests cannot be run automatically indeed removing the tests seems like a reasonable approach.

I'm not entirely convinced this should be solved by Wikibugs -- isn't this better solved at the client side by ignoring bots when it comes to alerts?

That would be nice! Unfortunately the display name is not part of the stream-events data format:

Thanks, @Legoktm!

@Nikerabbit this is a bit confusing to me. My understanding of the linked news post is that

We use a server password to login, cf. https://freenode.net/kb/answer/registration#logging-in. The login sequence looks like this:

Yeah, I just deleted the .err file, so there was probably some cache confused. Now that the bot has run again the .err file is there again (albeit empty).

You're right, the .err is indeed not truncated automatically. The first messages date back to about January this year, and given the 1.5MB log file size, I think truncating by hand is probably the best approach. I just did so :-)

They are rotated nightly (see https://github.com/wikimedia/labs-tools-forrestbot/blob/master/wblogging.py). We do sometimes generate a lot of logs in one day, but that's just because we have a development community that does a lot of work ;-)

The suggested replacement is https://quarry.wmflabs.org/ . For example, https://quarry.wmflabs.org/query/46608 is a converted version of brokenimages for nlwiki. https://quarry.wmflabs.org/query/46611 is an alternative version that is more convenient to export as wikitable (under 'download data').

Thanks for fixing this so quickly, @Legoktm!

See discussion in T237109:

Apologies for the mess; the bot was blocked on adding a task to T249730. When it crashes, it reprocesses the entire queue -- which is supposed to be idempotent as it checks whether the requested tags are already present...

....except that I forgot to update the dependencies! Did that and restarted wb2-phab... I think that should now really fix it.

Jobs restarted so this should be working now

Jobs restarted so this should be working now

I assume this should be handled equivalently to jenkins-bot and pipelinebot, i.e.: only report changes towards negative votes?