Page MenuHomePhabricator

Update VideoCutTool architecture, to support huge traffic
Closed, ResolvedPublic


Note: Not to be confused with T338308. This is not a repeated task

Earlier we stabilised the tool with connecting the right socket wires with the respective client.
But the server might break when a huge traffic requests resources from it, (especially during Wikimedia Hackathons).

Let's try to stabilise it further, by not overwhelming the server on huge traffic


TitleReferenceAuthorSource BranchDest Branch
Update backend architecture for huge trafficcloudvps-repos/videocuttool/VideoCutTool!11varun-s22bullmq-patchmaster
Update backend architecture for huge trafficcloudvps-repos/videocuttool/VideoCutTool!8varun-s22mastermaster
Customize query in GitLab

Event Timeline

Lets go with a Redis queue, using BullMq, and try to initialize say 5 workers with no work. When every worker is busy, we keep the next request in the queue, and as soon as one worker frees up, the task gets assigned to that (from the queue)

Note: The number 5 is not decided yet, We are yet to figure out the upper limit, (mostly via robust testing).

Change 942508 had a related patch set uploaded (by Reputation22; author: Reputation22):

[labs/tools/VideoCutTool@master] Add BullMq Queue implementation to the tool

Change 942508 abandoned by Reputation22:

[labs/tools/VideoCutTool@master] Update backend architecture for huge traffic


merged already