Page MenuHomePhabricator

Fix EL mysql consumer's deque push/pop {oryx} [3 pts]
Closed, ResolvedPublic

Description

In EventLogging mysql consumer, the events_batch deque is the structure that enables asynchronous communication between the process that reads from Kafka and the process that inserts into MySQL.

However, both processes are pushing/popping events to/from the same end of that deque, which is obviously wrong. They should push to one end, and pop from the other.

The related code is in handlers.py:250-300 and jrm.py:230-250
Some tests need to be updated, too (I think).

Event Timeline

mforns created this task.Dec 3 2015, 12:11 PM
mforns raised the priority of this task from to Needs Triage.
mforns updated the task description. (Show Details)
mforns added a project: Analytics-Backlog.
mforns added a subscriber: mforns.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptDec 3 2015, 12:11 PM
mforns renamed this task from Make EL mysql consumer's worker thread consume from the left side of the queue to Fix EL mysql consumer's deque push/pop usage.Dec 3 2015, 12:11 PM
mforns renamed this task from Fix EL mysql consumer's deque push/pop usage to Fix EL mysql consumer's deque push/pop usage {oryx}.
mforns set Security to None.

Change 258217 had a related patch set uploaded (by Mforns):
Fix bad push/pop from event batch deque

https://gerrit.wikimedia.org/r/258217

mforns renamed this task from Fix EL mysql consumer's deque push/pop usage {oryx} to Fix EL mysql consumer's deque push/pop and size {oryx}.Dec 10 2015, 8:47 PM
mforns claimed this task.
mforns edited projects, added Analytics-Kanban; removed Analytics-Backlog.
mforns moved this task from Next Up to In Progress on the Analytics-Kanban board.
mforns moved this task from In Progress to In Code Review on the Analytics-Kanban board.

Change 258217 merged by Nuria:
Fix bad push/pop from event batch deque

https://gerrit.wikimedia.org/r/258217

Nuria renamed this task from Fix EL mysql consumer's deque push/pop and size {oryx} to Fix EL mysql consumer's deque push/pop and size {oryx} [3].Dec 10 2015, 9:54 PM
mforns renamed this task from Fix EL mysql consumer's deque push/pop and size {oryx} [3] to Fix EL mysql consumer's deque push/pop and size {oryx} [3 pts].Dec 10 2015, 9:59 PM

Change 258384 had a related patch set uploaded (by Nuria):
Restore batch sizes as insertion rates are too slow

https://gerrit.wikimedia.org/r/258384

Change 258384 merged by Madhuvishy:
Restore batch sizes as insertion rates are too slow

https://gerrit.wikimedia.org/r/258384

mforns moved this task from Done to In Progress on the Analytics-Kanban board.Dec 11 2015, 5:02 PM
mforns moved this task from In Progress to Done on the Analytics-Kanban board.Dec 11 2015, 6:00 PM
mforns updated the task description. (Show Details)
mforns renamed this task from Fix EL mysql consumer's deque push/pop and size {oryx} [3 pts] to Fix EL mysql consumer's deque push/pop {oryx} [3 pts].Dec 14 2015, 4:34 PM
Nuria closed this task as Resolved.Dec 21 2015, 5:04 PM