It's inefficient, with the script having to be pushed every time.

No; that's what EVALSHA is for.

Fellow large shops using Redis consider that "LUA is not production ready in Redis today"

The context for the quote makes it clear that they are talking about exposing Lua scripting as a generic facility on Redis clusters that have multiple applications as tenant clients. In that kind of setup, Lua scripting is a non-starter because of the ease with which a single developer error can cause a failure to cascade to multiple services. Our setup is different: we use Lua only on the job queue redises, and discourage the use of Lua elsewhere.

I'm not a fan of our use of Lua either, but I think the incident documentation demonizes it in a naive and unrigorous fashion.

I think the job queue needs to be overhauled pretty substantially. It has some major problems, like the lack of a fair and efficient job scheduling mechanism. But it's a lot of work, it's pretty remote from performance, and the value to users would be small. In my opinion, a better bet would be to improve monitoring but leave the current architecture in tact until we are ready to commit to this problem-domain the engineering resources a proper solution would require.

We should try to investigate the outage better to see what was the exact cause, then. Is this the right machine to look at for job queue redis? http://ganglia.wikimedia.org/latest/?r=custom&cs=05%2F01%2F2015+00%3A00&ce=05%2F04%2F2015+00%3A00&m=cpu_report&c=Redis+eqiad&h=rdb1001.eqiad.wmnet&tab=m&vn=&hide-hf=false&mc=2&z=small&metric_group=ALLGROUPS