Page MenuHomePhabricator

Move Statsd buffer send from pre-send to post-send
Closed, ResolvedPublic

Description

Splitting this out from T288702: Reduce complexity and time spent in WANObjectCache.

Specifically, to eliminate the wall-time spent in Statsd-related code from the backend response times.

Note that for the past 5 years, it has already been the case that stats messages are buffered in-proces and sent to the Statsd server at the end of the request.

Todo:

  • Move any processing, filtering and formatting logic from individual timing() and increment()calls to the send phase.
  • Confirm the send phase happens in a post-send step.

Event Timeline

The first step of this was done, which consolidated all the processing logic into a single batch during the send phase:

Change 719146 merged by jenkins-bot:
stats: Defer BufferingStatsdDataFactory::normalizeMetricKey to send phase%%%
https://gerrit.wikimedia.org/r/719146

And app servers currently spend slight more than 0.1% of the backend latency of index.php and api.php requests in that step.

[…]

Change 725152 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] Deprecate wfLogProfilingData(), improve statsd/profiling docs

https://gerrit.wikimedia.org/r/725152

Change 725152 merged by jenkins-bot:

[mediawiki/core@master] Deprecate wfLogProfilingData(), improve statsd/profiling docs

https://gerrit.wikimedia.org/r/725152

Change 733135 had a related patch set uploaded (by Krinkle; author: Zabe):

[mediawiki/core@master] Add deprecation warnings to wfLogProfilingData()

https://gerrit.wikimedia.org/r/733135

Krinkle triaged this task as Medium priority.
Krinkle moved this task from Untriaged to Misc bookmarks on the Wikimedia-Performance-publish board.

Confirmed in latest flame graphs.

occupies about 0.2% of a daily index.php wall-clock flame graph, post-send from restInPeace.
https://performance.wikimedia.org/arclamp/svgs/daily/2021-10-24.excimer-wall.index.svgz?s=emit&x=170.8&y=1413

Krinkle updated the task description. (Show Details)

Change 733135 merged by jenkins-bot:

[mediawiki/core@master] Add deprecation warnings to wfLogProfilingData()

https://gerrit.wikimedia.org/r/733135