Steps to reproduce? LU was recently broken but has been just fixed to my knowledge.

It's https://he.wikipedia.org/wiki/%D7%9E%D7%93%D7%99%D7%94_%D7%95%D7%99%D7%A7%D7%99:Helppage-top-gethelp , and the current value is the English word "Help".

It's supposed to be the Hebrew word "עזרה".

Possibly related: T92721.

Another example of a message that doesn't seem to be updated: https://translatewiki.net/wiki/MediaWiki:Modifiedarticleprotection/he .

The translatewiki value since March 5 is:
שינה את רמת ההגנה של הדף "[[$1]]"

The version in the Hebrew Wikipedia on March 17 is:
שינה את רמת ההגנה של "[[$1]]"

(In case you can't see it, they are different :) )

So because these messages were exported correctly, the cause is likely in LocalisationUpdate.

Something happened about a month ago:

l10nupdate.log-20150205.gz:Saved 6998 new translations
l10nupdate.log-20150205.gz:Saved 5839 new translations
l10nupdate.log-20150207.gz:Saved 6038 new translations
l10nupdate.log-20150207.gz:Found no new translations
l10nupdate.log-20150208.gz:Saved 6331 new translations
l10nupdate.log-20150208.gz:Found no new translations
l10nupdate.log-20150209.gz:Saved 6481 new translations
l10nupdate.log-20150209.gz:Found no new translations
l10nupdate.log-20150210.gz:Saved 6909 new translations
l10nupdate.log-20150210.gz:Found no new translations
l10nupdate.log-20150211.gz:Saved 6971 new translations
l10nupdate.log-20150211.gz:Found no new translations
l10nupdate.log-20150212.gz:Found no new translations
l10nupdate.log-20150212.gz:Found no new translations

You can clearly see that for a few days other half still worked while the new branch was broken, and finally it stopped working when branches were cycled next time.

That time frame matches 1.21wmf16 and contains a change to LocalisationUpdate: https://www.mediawiki.org/wiki/MediaWiki_1.25/wmf16#LocalisationUpdate

It is highly likely that that change has the unwanted side effect breaking LU.

As for noticing these issues earlier, would be nice if the "Found no new/Saved X translations" would be in SAL or similar.

(Mid air collision) Better wait next run, there were further changes this morning: https://gerrit.wikimedia.org/r/#/c/197262/8

Change 197278 had a related patch set uploaded (by Nikerabbit):
Fix singular-plural typo causing extension and skin i18n files to be ignored

https://gerrit.wikimedia.org/r/197278

Change 197279 had a related patch set uploaded (by Nikerabbit):
Add code to handle core i18n locations

https://gerrit.wikimedia.org/r/197279

Change 197278 merged by jenkins-bot:
Fix singular-plural typo causing extension and skin i18n files to be ignored

https://gerrit.wikimedia.org/r/197278

Change 197279 merged by jenkins-bot:
Add code to handle core i18n locations

https://gerrit.wikimedia.org/r/197279

So is this one resolved now? The patch has been submitted and today's deployment should have synced everything...

In T92823#1130296, @mmodell wrote:

So is this one resolved now? The patch has been submitted and today's deployment should have synced everything...

It should be fixed for the 1.25wmf22 branch. We can either backport or just live with it for another week until the patch hits all wikis.

Change 197766 had a related patch set uploaded (by BryanDavis):
Fix singular-plural typo causing extension and skin i18n files to be ignored

https://gerrit.wikimedia.org/r/197766

Change 197767 had a related patch set uploaded (by BryanDavis):
Add code to handle core i18n locations

https://gerrit.wikimedia.org/r/197767

Change 197766 merged by jenkins-bot:
Fix singular-plural typo causing extension and skin i18n files to be ignored

https://gerrit.wikimedia.org/r/197766

Change 197767 merged by jenkins-bot:
Add code to handle core i18n locations

https://gerrit.wikimedia.org/r/197767

Change 197771 had a related patch set uploaded (by BryanDavis):
Backport LocalisationUpdate fixes from 1.25wmf22

https://gerrit.wikimedia.org/r/197771

Backport to 1.25wmf21 is queued up for the 2015-03-18T23:00Z SWAT window.

Change 197771 merged by jenkins-bot:
Backport LocalisationUpdate fixes from 1.25wmf22

https://gerrit.wikimedia.org/r/197771

The message is fixed, though I dunno why. I still see many errors in the log:

Warning: file_put_contents(/var/lib/l10nupdate/cache-1.25wmf21/l10nupdate-sco.json): failed to open stream: No such file or directory in /srv/mediawiki-staging/php-1.25wmf21/extensions/LocalisationUpdate/update.php on line 76
(repeats for each language, for wmf21 only)

At the end there is:

This is nc from the netcat-openbsd package. An alternative nc is available
in the netcat-traditional package.
usage: nc [-46DdhklnrStUuvzC] [-i interval] [-P proxy_username] [-p source_port]
          [-s source_ip_address] [-T ToS] [-w timeout] [-X proxy_protocol]
          [-x proxy_address[:port]] [hostname] [port[s]]
This is nc from the netcat-openbsd package. An alternative nc is available
in the netcat-traditional package.
usage: nc [-46DdhklnrStUuvzC] [-i interval] [-P proxy_username] [-p source_port]
          [-s source_ip_address] [-T ToS] [-w timeout] [-X proxy_protocol]
          [-x proxy_address[:port]] [hostname] [port[s]]
This is nc from the netcat-openbsd package. An alternative nc is available
in the netcat-traditional package.
usage: nc [-46DdhklnrStUuvzC] [-i interval] [-P proxy_username] [-p source_port]
          [-s source_ip_address] [-T ToS] [-w timeout] [-X proxy_protocol]
          [-x proxy_address[:port]] [hostname] [port[s]]
Refreshing ResourceLoader caches
DB connection error: Can't connect to MySQL server on 'xxx.yyy.zzz.ååå' (4) (xxx.yyy.zzz.ååå)

It also says that it updated 0 CDB files on both branches. That could just be because wmf22 was a brand new branch and wmf21 failed to missing directory above.

In T92823#1131096, @Nikerabbit wrote:

At the end there is:

This is nc from the netcat-openbsd package. An alternative nc is available
in the netcat-traditional package.
usage: nc [-46DdhklnrStUuvzC] [-i interval] [-P proxy_username] [-p source_port]
          [-s source_ip_address] [-T ToS] [-w timeout] [-X proxy_protocol]
          [-x proxy_address[:port]] [hostname] [port[s]]

nc usage printing is a known issue

In T92823#1131096, @Nikerabbit wrote:

The message is fixed, though I dunno why. I still see many errors in the log:

Warning: file_put_contents(/var/lib/l10nupdate/cache-1.25wmf21/l10nupdate-sco.json): failed to open stream: No such file or directory in /srv/mediawiki-staging/php-1.25wmf21/extensions/LocalisationUpdate/update.php on line 76
(repeats for each language, for wmf21 only)

Looking on tin it looks like this is still more blow back from changes that were made to reduce the possibility of privilege escalation by maintenance scripts. CommonSettings still says $wgLocalisationUpdateDirectory = "/var/lib/l10nupdate/cache-$wmfVersionNumber"; but the directory structure on disk is now /var/lib/l10nupdate/caches/cache-$wmfVersionNumber";. There are new directories in that location so some process has been updated to use the new location but apparently not all as the CommonSettings value shows.

At the end there is:

This is nc from the netcat-openbsd package. An alternative nc is available
in the netcat-traditional package.

This is T1387: /usr/local/bin/deploy2graphite broken on tin due to nc command syntax which should be fixed but isn't halting for anything in the process. All that is lost is Graphite data.

Refreshing ResourceLoader caches
DB connection error: Can't connect to MySQL server on 'xxx.yyy.zzz.ååå' (4) (xxx.yyy.zzz.ååå)

The real IP in the logs is the new wikitech server which is not fully reachable from the rest of the cluster. I'm not exactly sure why the ResourceLoader purge step was trying to talk to it. It's probably a configuration issue that should be spun off as a separate ticket.

It also says that it updated 0 CDB files on both branches. That could just be because wmf22 was a brand new branch and wmf21 failed to missing directory above.

There was a scap in the SWAT window ~4 hours before the l10nupdate run so this may have taken care of the wmf22 branch. The wmf21 branch is explained by the configuration problem noted above.

Change 197937 had a related patch set uploaded (by BryanDavis):
Update wgLocalisationUpdateDirectory to match l10nupdate-1

https://gerrit.wikimedia.org/r/197937

Change 197937 merged by jenkins-bot:
Update wgLocalisationUpdateDirectory to match l10nupdate-1

https://gerrit.wikimedia.org/r/197937

In T92823#1134621, @Nikerabbit wrote:

According to the l10nupdate log, in last night's update all went okay and I also did a spot check for one message that it really was updated. Which issues from the logs should be created as new tickets?

We have a ticket for the deploy2graphite errors and the DB connection error did not reappear. I think we are good for now. Thanks for helping get this working again @Nikerabbit!

@Nikerabbit -- I'm still seeing untranslated messages on Telugu:

https://te.wikipedia.org/wiki/%E0%B0%B5%E0%B0%BF%E0%B0%95%E0%B1%80%E0%B0%AA%E0%B1%80%E0%B0%A1%E0%B0%BF%E0%B0%AF%E0%B0%BE_%E0%B0%9A%E0%B0%B0%E0%B1%8D%E0%B0%9A:%E0%B0%AB%E0%B1%8D%E0%B0%B2%E0%B1%8B

Do we need to wait for a job to run or cache to clear?

I don't know what I should be looking at. Can you name the message? The message linked in the merged task is updated.

Oh, you're right -- I'm sorry. I thought that all of the visible messages on the page were translated, but it looks like there are some that weren't. The translated messages are showing up correctly. Thanks!