Incident metadata

• Google Doc

The way I see it this was a case of "failure in depth".

• We are still using restbase as the default fallback for requests to /api/rest_v1; we shouldn't do that anymore imho
• There is no CI check for validity of lua scripts loaded in trafficserver; some have functional tests but a syntax error should have never passed CI, even in the absence of tests
• Lua scripts get automatically reloaded by trafficserver. We should have some alerting that tells us when ATS is logging so many lua syntax errors
• The script that restarts/reloads ATS config when a yaml file is changed should first check there's no error in the configuration, including lua possibly, and refuse restarting ATS unless it's safe to do so.

Tagging the task for the traffic team to create followups.

In T421203#11747752, @Joe wrote:

The way I see it this was a case of "failure in depth".

• We are still using restbase as the default fallback for requests to /api/rest_v1; we shouldn't do that anymore imho

We are aware, changing that is part of the Q4 cleanup for 5.2.2c

There is no CI check for validity of lua scripts loaded in trafficserver; some have functional tests but a syntax error should have never passed CI, even in the absence of tests

this is not accurate, the current tests fail to validate the configuration file, that in fact is a piece of lua code, I'll work on this today

Change #1260667 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] trafficserver: Validate .lua.conf files

https://gerrit.wikimedia.org/r/1260667

Change #1260667 merged by Vgutierrez:

[operations/puppet@production] trafficserver: Validate .lua.conf files

https://gerrit.wikimedia.org/r/1260667

Change #1262152 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] trafficserver: Extend validation for .lua.conf files

https://gerrit.wikimedia.org/r/1262152

Change #1262152 merged by Scott French:

[operations/puppet@production] trafficserver: Extend validation for .lua.conf files

https://gerrit.wikimedia.org/r/1262152