Page MenuHomePhabricator

traffic_server crash upon Lua reload: attempt to concatenate a table value
Open, MediumPublic

Description

Automatic reloads of trafficserver.service fleet-wide due to an innocuous remap.config change caused a crash on cp3063. The very same change was applied succesfully to all other hosts. From journalctl -u trafficserver.service:

Jan 15 18:51:37 cp3063 systemd[1]: Reloading Apache Traffic Server is a fast, scalable and extensible caching proxy server..
Jan 15 18:51:37 cp3063 traffic_manager[222215]: [Jan 15 18:51:37.693] {0x7fd5c5bea700} NOTE: User has changed config file remap.config
Jan 15 18:51:37 cp3063 systemd[1]: Reloaded Apache Traffic Server is a fast, scalable and extensible caching proxy server..
Jan 15 18:51:44 cp3063 traffic_manager[222215]: PANIC: unprotected error in call to Lua API (attempt to concatenate a table value)

The issue is likely subtle and due to some sort of race, given that we reload Lua on both ats-tls and ats-be fleet-wide multiple times a day without problems.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ema triaged this task as Medium priority.Jan 16 2020, 8:53 AM
ema moved this task from Triage to Caching on the Traffic board.

Mentioned in SAL (#wikimedia-operations) [2020-01-16T08:55:58Z] <ema> cp3063: ats-backend-restart to clear things up after traffic_server crash T242952

This just happened on cp1087:

Feb 05 15:14:05 cp1087 systemd[1]: Reloaded Apache Traffic Server is a fast, scalable and extensible caching proxy server..
Feb 05 15:14:12 cp1087 traffic_manager[229947]: PANIC: unprotected error in call to Lua API (attempt to concatenate a table value)
ema added a subscriber: Vgutierrez.

T247232 is another example of this bug, with a different Lua error (string length overflow).

The issue occurred yesterday on cp2023 and cp1081:

Mar 30 13:55:02 cp2023 traffic_manager[17786]: PANIC: unprotected error in call to Lua API (attempt to concatenate a thread value)
Mar 30 13:41:54 cp1081 traffic_manager[6497]: PANIC: unprotected error in call to Lua API (attempt to index a thread value)
BBlack added a subscriber: BBlack.

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!