Eventlogging's forwarder/zmq-legacy leaks memory over time
Closed, ResolvedPublic5 Estimated Story Points
Actions

Assigned To

Authored By

	elukey
	Feb 5 2018, 11:26 AM

Description

Today puppet wasn't able to run on eventlog1001 due to Could not evaluate: Cannot allocate memory - fork(2), and from top the main memory consumer was EL's forwarder/zmq-legacy. After stop/restart of that daemon the memory usage on eventlog1001 went from ~32G to 2G.

Some interesting graphs:

Screen Shot 2018-02-05 at 12.22.59 PM.png (1×2 px, 194 KB)

Screen Shot 2018-02-05 at 12.23.10 PM.png (1×2 px, 297 KB)

Not sure if the issue is known but it is serious enough for EL's availability to prioritize it :)

Event Timeline

elukey triaged this task as High priority.Feb 5 2018, 11:26 AM

elukey created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 5 2018, 11:26 AM

elukey added a subscriber: Ottomata.Feb 5 2018, 11:26 AM

@Krinkle :D how soon can we do https://gerrit.wikimedia.org/r/#/c/403560/ ?

• Nuria edited projects, added Analytics-Kanban; removed Analytics.Feb 5 2018, 5:21 PM

• Nuria set the point value for this task to 5.

• Nuria moved this task from Next Up to In Progress on the Analytics-Kanban board.

elukey moved this task from Backlog to Stalled on the User-Elukey board.Feb 5 2018, 5:37 PM

• Imarlier subscribed.Feb 9 2018, 9:43 PM

closing since we already have a tracking task/code review

	F13028400: Screen Shot 2018-02-05 at 12.22.59 PM.png
	Feb 5 2018, 11:26 AM

	F13028403: Screen Shot 2018-02-05 at 12.23.10 PM.png
	Feb 5 2018, 11:26 AM

Eventlogging's forwarder/zmq-legacy leaks memory over timeClosed, ResolvedPublic5 Estimated Story PointsActions

Description

Event Timeline

Eventlogging's forwarder/zmq-legacy leaks memory over time
Closed, ResolvedPublic5 Estimated Story Points
Actions