Note: Not to be confused with T338308. This is not a repeated task
Earlier we stabilised the tool with connecting the right socket wires with the respective client.
But the server might break when a huge traffic requests resources from it, (especially during Wikimedia Hackathons).
Let's try to stabilise it further, by not overwhelming the server on huge traffic
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Update backend architecture for huge traffic | labs/tools/VideoCutTool | master | +953 -242 |
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Update backend architecture for huge traffic | cloudvps-repos/videocuttool/VideoCutTool!11 | varun-s22 | bullmq-patch | master | |
| Update backend architecture for huge traffic | cloudvps-repos/videocuttool/VideoCutTool!8 | varun-s22 | master | master |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Gopavasanth | T327582 Improve the functionality of VideoCutTool | |||
| Resolved | Reputation22 | T333466 [GSOC 2023 Proposal] Improve the functionality of VideoCutTool | |||
| Resolved | Reputation22 | T342810 Update VideoCutTool architecture, to support huge traffic |
Lets go with a Redis queue, using BullMq, and try to initialize say 5 workers with no work. When every worker is busy, we keep the next request in the queue, and as soon as one worker frees up, the task gets assigned to that (from the queue)
Note: The number 5 is not decided yet, We are yet to figure out the upper limit, (mostly via robust testing).
Change 942508 had a related patch set uploaded (by Reputation22; author: Reputation22):
[labs/tools/VideoCutTool@master] Add BullMq Queue implementation to the tool
varun-s22 opened https://gitlab.wikimedia.org/cloudvps-repos/videocuttool/VideoCutTool/-/merge_requests/6
Update backend architecture and previous 7 commits
varun-s22 closed https://gitlab.wikimedia.org/cloudvps-repos/videocuttool/VideoCutTool/-/merge_requests/6
Update backend architecture and previous 7 commits
varun-s22 opened https://gitlab.wikimedia.org/cloudvps-repos/videocuttool/VideoCutTool/-/merge_requests/8
Update backend architecture for huge traffic
varun-s22 closed https://gitlab.wikimedia.org/cloudvps-repos/videocuttool/VideoCutTool/-/merge_requests/8
Update backend architecture for huge traffic
varun-s22 opened https://gitlab.wikimedia.org/cloudvps-repos/videocuttool/VideoCutTool/-/merge_requests/11
Update backend architecture for huge traffic
soda merged https://gitlab.wikimedia.org/cloudvps-repos/videocuttool/VideoCutTool/-/merge_requests/11
Update backend architecture for huge traffic
Change 942508 abandoned by Reputation22:
[labs/tools/VideoCutTool@master] Update backend architecture for huge traffic
Reason:
merged already