traffic_server crash upon Lua reload: attempt to concatenate a table value
Closed, InvalidPublic
Actions

Assigned To

None

Authored By

	• ema
	Jan 16 2020, 8:53 AM

Description

Automatic reloads of trafficserver.service fleet-wide due to an innocuous remap.config change caused a crash on cp3063. The very same change was applied succesfully to all other hosts. From journalctl -u trafficserver.service:

Jan 15 18:51:37 cp3063 systemd[1]: Reloading Apache Traffic Server is a fast, scalable and extensible caching proxy server..
Jan 15 18:51:37 cp3063 traffic_manager[222215]: [Jan 15 18:51:37.693] {0x7fd5c5bea700} NOTE: User has changed config file remap.config
Jan 15 18:51:37 cp3063 systemd[1]: Reloaded Apache Traffic Server is a fast, scalable and extensible caching proxy server..
Jan 15 18:51:44 cp3063 traffic_manager[222215]: PANIC: unprotected error in call to Lua API (attempt to concatenate a table value)

The issue is likely subtle and due to some sort of race, given that we reload Lua on both ats-tls and ats-be fleet-wide multiple times a day without problems.

Related Objects

Mentioned In: T248938: ATS ts_lua coredumps on config reload
Mentioned Here: T247232: lua related crash on ats-be @ cp4032

Event Timeline

• ema created this task.Jan 16 2020, 8:53 AM

Restricted Application added a project: SRE. · View Herald TranscriptJan 16 2020, 8:53 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

• ema triaged this task as Medium priority.Jan 16 2020, 8:53 AM

• ema moved this task from Backlog to Caching on the Traffic board.

Mentioned in SAL (#wikimedia-operations) [2020-01-16T08:55:58Z] <ema> cp3063: ats-backend-restart to clear things up after traffic_server crash T242952

• ema updated the task description. (Show Details)Jan 16 2020, 9:02 AM

This just happened on cp1087:

Feb 05 15:14:05 cp1087 systemd[1]: Reloaded Apache Traffic Server is a fast, scalable and extensible caching proxy server..
Feb 05 15:14:12 cp1087 traffic_manager[229947]: PANIC: unprotected error in call to Lua API (attempt to concatenate a table value)

T247232 is another example of this bug, with a different Lua error (string length overflow).

• ema mentioned this in T248938: ATS ts_lua coredumps on config reload.Mar 31 2020, 8:01 AM

The issue occurred yesterday on cp2023 and cp1081:

Mar 30 13:55:02 cp2023 traffic_manager[17786]: PANIC: unprotected error in call to Lua API (attempt to concatenate a thread value)

Mar 30 13:41:54 cp1081 traffic_manager[6497]: PANIC: unprotected error in call to Lua API (attempt to index a thread value)

https://github.com/apache/trafficserver/pull/6571 could be handy to tune some TS lua aspects

CDanis subscribed.Mar 31 2020, 2:25 PM

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

@Vgutierrez Can you recall whether this has happened anytime since the few years this was reported?

Closing due to age and lack of response. If this is still occurring let's reopen.

traffic_server crash upon Lua reload: attempt to concatenate a table valueClosed, InvalidPublicActions

Description

Related Objects

Event Timeline

traffic_server crash upon Lua reload: attempt to concatenate a table value
Closed, InvalidPublic
Actions