l10n-bot self-force-merging sometimes breaks mediawiki/core master. Example from today: https://gerrit.wikimedia.org/r/#/c/194725/
Yes, it sucks that our tests depend on the values of localisation messages in random languages, but that's the situation. This has happened in the past and will happen again in the future, and it's very annoying every time.
Please make the bot just vote C+2, without V+2, and let the tests run. If a commit fails the tests, anyone can amend it and fix up.
Change 194992 had a related patch set uploaded (by Legoktm):
Don't self-merge l10n-bot commits, only CR 2
I fully agree we should make it run the tests.
I'm concerned about the non-trivial stress this might put on the continuous integration system, though. As they typically sync these overnight all at once. That's a lot of jobs run at once and may clog the queue for several hours, if not more.
I'd recommend we first gather statistics of how many Gerrit repos this currently, roughly, affects.
For reference, in https://gerrit.wikimedia.org/r/#/c/102636/ @hashar blacklisted l10n-bot.
As for how many repos this affects, the following is how many merged commits l10n-bot uploads per day since mid-November:
It might be easier to just revert 02364f56df90ca9e64a3d63fd81c52643e4f9e97 : Raimond used to act on test failures IIRC, as L10n-bot is always run and checked manually by Raimond when he has some spare time during the day (thanks Raimond!).
L10n-bot was blacklisted because at some point some devs working in PST found that every day around lunch time (?) all merging and testing would be delayed. Rather than the daily count, perhaps the hourly peak is more interesting. Adding more capacity, or a dedicated queue, would be nice.
Would it help to change the time when I do the export from translatewiki.net? Currently I export and submit nearly every day between 19:00 and 22:00 UTC. I could change this to the European morning, this means between 8:00 and 12:00 UTC. In this case I suggest the time of the Localization cron job should be changed to 13:00 UTC. This way the jobs do not conflict with working time in California. But maybe more with the Wikidata team in Berlin?
Hm.. If it's automated and could be any time of day, then perhaps between 03:00 and 06:00 UTC. That would match 04:00 - 07:00 CET (UTC+1; Amsterdam, Berlin), and 17:00 - 20:00 PST (UTC -8, San Francisco).
Or between 16:00 and 19:00 UTC. That would match 17:00 - 20:00 CET and 06:00 - 09:00 PST.
I haven't looked back at the l10n-bot configuration in Zuul, but what could be done is:
This way when patches are sent they do not trigger any tests, on +2 they get the test and are magically merged by Zuul if the tests pass.
Note that the extra time spent in CI (~ 5 minutes) is enough to occasionally cause race conditions where, after the l10n patch is pushed to Gerrit, another commit already approved and being tested gets merged first. If that patch also changes i18n files, I wonder what the process is from there for @Raymond. Does it require manual work? Or would that update just fail and the next nightly import/export covers the gap?
Because there is manual work to do 03:00-06:00 UTC is impossible, 16:00-19:00 UTC would be possible for most of the days (depends on my personal schedule/working level).
Such conflict happens from time to time now too. When I sync the exports back with translatewiki production repo after the merges I identify such conflicts and if necessary I revert the export patch set.
I am fine having the l10n update changes be pushed between 19:00 and 22:00 UTC, though I am not sure why they need to occur daily :-D
As I wrote above we should skip running tests from check* / test Zuul pipelines, have l10n-bot only vote CR+2 and let gate-and-submit run the jobs and merge the change for us. @Legoktm change https://gerrit.wikimedia.org/r/#/c/194990/ is on this track.
@Raymond can you prepare a patch that makes l10n-bot only vote CR+2 instead of CR+2/V+2/submit ? We can them pair together one of the european evening to update both tools at the same time and test the behavior together. Any day would work for me after 9pm CET.
Change 194990 merged by jenkins-bot:
Don't ignore l10n-bot in gate-and-submit pipeline
@Raymond it seems the wednesday run used CR+2 and got changes merged via successful tests. Is there anything else to accomplish?
Related to this task: Since the change I get emails from watched gerrit projects about submitted l10n changes. This wasn't before (except changes manually approved).
Is there a way to ignore L10n-bot again (particularly changes owned by l10n and submitted by jenkins)?
With https://gerrit.wikimedia.org/r/#/c/198184/, every extension and skin should now have jenkins running basic phplint and merging changes.
Update: l10n-bot is currently being ignored again because jenkins machines were running out of disk space when extensions that hadn't had tests ever triggered before were suddenly cloning mediawiki/core a bunch of times. Adding relevant blockers which we need to resolve first.
I think the test confirmed my hypothesis. L10n-bot should still C+2 and V+2, but tests can be run alongside (capacity permitting, perhaps in a special queue) to file bugs in the process.
I am kind of lost with past week events. It seems the current situation is l10n-bot is now being ignored completely because the mediawiki extensions jobs fill the disk with copy of MediaWiki core.
I am thus marking this bug as being blocked by T93703: reduce copies of mediawiki/core in workspaces.
Yes, it sucks that our tests depend on the values of localisation messages in random languages, but that's the situation.
How hard would it be for those tests to force lang qqx which has a stable output?
It probably wouldn't be hard, but it would be tedious, since you'd probably have to update the expected values in each test from English text to the message keys.
We can just fix them as we find, I guess. Eventually it will get rare enough to not to bother people. I.e. when was the last time this happened?
I am not advocating that we skip tests for l10n updates... however that would risk patches getting stuck in gerrit and perhaps nobody even noticing. It should not be the responsibility of translatewiki.net people to fix those things.
If someone can find automatically tests which rely on translations (btw. we do not export English) we can make those as good first task and/or GCI tasks.
Change 462460 had a related patch set uploaded (by Hashar; owner: Reedy):
[translatewiki@master] Disable automatic V+2/submit for l10n-bot
Change 462460 merged by jenkins-bot:
[translatewiki@master] Disable automatic V+2/submit for l10n-bot
Surely this is not fixed, because just yesterday, L10n-bot broke the build with https://gerrit.wikimedia.org/r/c/mediawiki/core/+/465210 (I fixed it in https://gerrit.wikimedia.org/r/c/mediawiki/core/+/465299).
L10n-bot is only triggering the special i18n checker thing right now, we still need to run the full test suite against it.
I was thinking _why_ dummy job was used in all repositories. I guess this is something that is managed by integration/config.
Suggesting to decline because:
Hmm, fine, with 2 occurrences in 8 years this is probably not worth spending time to fix. I must have thought that the problems would be more common when reporting this. Humans break the build way more often than that.
L10n-bot just broke the build today D: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/969820
This scenario is supposed to be impossible: https://translatewiki.net/wiki/Repository_management#Message_group_synchronization_lock_(Strong_synchronization. We'll try investigate.