Page MenuHomePhabricator

Wikitext events need to be sampled {lion}
Closed, ResolvedPublic1 Story Points

Description

The inflow of Wikitext events into eventlogging is to high for the system to the point that yesterday vanadium stopped processing events due to the VE inflow. Intake of events is 200 events per second about 150 of those events (per second) are invalid.

This can be fixing by sampling events, which is done for pretty much any incoming schema. Since VE needs all events for a session the sampling should happen around token, via bucketing. For example, if session token is between 1 and 10 we can send events it such if session token is <= 5, that would be a sampling of 50%.

Event Timeline

Nuria created this task.Mar 19 2015, 2:11 PM
Nuria assigned this task to Milimetric.
Nuria raised the priority of this task from to Needs Triage.
Nuria updated the task description. (Show Details)
Nuria added subscribers: Nuria, Milimetric, Jdforrester-WMF and 2 others.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 19 2015, 2:11 PM
Nuria set Security to None.
kevinator renamed this task from VE events need to be sampled to VE events need to be sampled {lion}.Mar 19 2015, 2:27 PM
kevinator triaged this task as High priority.
kevinator renamed this task from VE events need to be sampled {lion} to Wikitext events need to be sampled {lion}.Mar 27 2015, 11:02 PM
kevinator updated the task description. (Show Details)
Milimetric closed this task as Resolved.Apr 8 2015, 5:01 PM

Change 199132 had a related patch set uploaded (by Alex Monk):
Collect only 25% of WikiEditor events

https://gerrit.wikimedia.org/r/199132

Krenair reopened this task as Open.Apr 8 2015, 5:28 PM

Change 204156 had a related patch set uploaded (by Jforrester):
Collect only 25% of WikiEditor events

https://gerrit.wikimedia.org/r/204156

Change 204157 had a related patch set uploaded (by Jforrester):
Collect only 25% of WikiEditor events

https://gerrit.wikimedia.org/r/204157

Change 199132 merged by jenkins-bot:
Collect only 25% of WikiEditor events

https://gerrit.wikimedia.org/r/199132

Change 204157 merged by jenkins-bot:
Collect only 25% of WikiEditor events

https://gerrit.wikimedia.org/r/204157

Change 204156 merged by jenkins-bot:
Collect only 25% of WikiEditor events

https://gerrit.wikimedia.org/r/204156

Jdforrester-WMF closed this task as Resolved.Apr 14 2015, 11:52 PM
Jdforrester-WMF reassigned this task from Milimetric to Krenair.
Jdforrester-WMF lowered the priority of this task from High to Normal.
Jdforrester-WMF moved this task from Bug Fixes to Q4 on the VisualEditor board.
Jdforrester-WMF edited a custom field.
GOIII moved this task from Backlog to Closed on the WikiEditor board.Apr 3 2016, 9:08 AM