Page MenuHomePhabricator

ReleaseTaggerBot is down for three weeks
Open, Needs TriagePublicBUG REPORT

Description

Hi. Looks like the bot is down from December 13, 1:00 UTC. My rough calculation says it missed 977 merge messages (including 5 mine) and counting, and looks like the changelog for the train in three days will be empty*. All this is just as it looks to me, I do not understand in bots at all, so check this, please. Yesterday evening I payed attention that it does not work, and thought people will be back from holidays to launch it. But now I checked this deeply and believe something is broken. Thank you.

https://phabricator.wikimedia.org/p/ReleaseTaggerBot/
https://forrestbot.toolforge.org/forrestbot.log.txt
https://forrestbot.toolforge.org/forrestbot.err.txt
Hope this remains useful: https://phabricator.wikimedia.org/maniphest/query/pwyWySjUoiiG/#R

  • UPD: appears this part worked anyway.

Event Timeline

I don't know how to check that, but curious if some of the first missed merges were still from 1.46.0-wmf.7.

Per the error log, someone's malformed commit message has broken the bot, it seems. I'll have a poke.

Edit: Well, the queue is now empty. Whatever the problem was, it's unrecoverable.

Per the error log, someone's malformed commit message has broken the bot, it seems. I'll have a poke.

Edit: Well, the queue is now empty. Whatever the problem was, it's unrecoverable.

Can't you just try and check the next change or two that were ready at that moment chronologically, and run the bot on them again in a sandbox?

I don't know how to check that, but curious if some of the first missed merges were still from 1.46.0-wmf.7.

Found a way. Looks like there were less than 5 merges in 1.46.0-wmf.7 that were not tagged.

@Jdforrester-WMF Just to ask, is there anything that other folks can do to help with getting ReleaseTaggerBot back up & running?

@Jdforrester-WMF Just to ask, is there anything that other folks can do to help with getting ReleaseTaggerBot back up & running?

I don't think I can fix it, no. Main access is not mine, of course.

Unfortunately I'm currently overhelmed by other things in life so I don't have the capacity to take a detailed look. But more than happy to give people any access they need (should be largely documented at https://wikitech.wikimedia.org/wiki/Tool:ReleaseTaggerBot)

If I read the log correctly there are no emails in the inbox. Not entirely sure if the other messages really mean anything or are just warnings on the requests made to phabricator.

I think maybe the wikimedia-commits subscription got lost? I've re-added it at https://lists.wikimedia.org/postorius/lists/mediawiki-commits.lists.wikimedia.org/:

afbeelding.png (212×1 px, 25 KB)

In mailman, the email account is (unfortunately) linked to my primary email account - I'll need to split that up at some point in the future.

Okay, I think it's now broken because of the recent dmarc changes - the sender is no longer gerrit@wikimedia.org (https://github.com/wikimedia/labs-tools-forrestbot/blob/master/pop3bot.py#L61). I've commented this out on toolforge to see if that brings things back to life.

It works, thank you. Is there a chance to close the gap, or it's too late?

It works, thank you. Is there a chance to close the gap, or it's too late?

No, the old e-mails to mediawiki-commits@lists.wikimedia.org aren't archived so can't be re-processed, so it's not possible.

Is there a task for the DMARC changes that should get pinged for this breakage?

It works, thank you. Is there a chance to close the gap, or it's too late?

No, the old e-mails to mediawiki-commits@lists.wikimedia.org aren't archived so can't be re-processed, so it's not possible.

A pity. I hope somebody wants to write a one-time script that retrieves the data from the pages like https://www.mediawiki.org/wiki/MediaWiki_1.46/wmf.10/Changelog with checking the Gerrit, or just looks over the jenkins-bot log, to add all the missing project tags. But looks like it's not going to happen.

Is there a task for the DMARC changes that should get pinged for this breakage?

Potentially T404884: DMARC improvements?


Also -- thank you for your work on this @valhallasw, and apologies if I came across as pushy in T413701#11549320 :)
(I think I had assumed that @Jdforrester-WMF was the main maintainer/operator of ReleaseTaggerBot, and I'd therefore intended to ask "is there anything that folks who aren't maintainers of ReleaseTaggerBot can do to help". However, I worry that it might have instead come across as something like "are there other bot-maintainers that should be doing something about this issue", which wasn't my intention)

Not at all - I read it as an offer for help which is very welcome! But of course it has always been a bit of a hack project so documentation is a bit lacking :-)

I've just pushed this update and have pushed a (somewhat LLM-generated but as far as I can see correct) README.md to the repository to help future maintainers.

With regards to processing old changesets: this is tricky partially because of what James mentioned (the queue is gone) but also because the processing depends on "current" state (critically: the _current latest_ release branch -> merges into master will go into that branch plus one.

I think indeed a separate script to go through https://www.mediawiki.org/wiki/MediaWiki_1.46/wmf.10/Changelog and friends is probably the way to go. This could potentially still (re)use the repository/project filter in releasetaggerbot plus the logic to figure out which branches to add.