@MusikAnimal @kaldari — Please review this ticket and please edit as you see fit!

Quick update: After a brief, discussion, we decided that 'AbuseFilter performance' actually means two things:

1. The actual performance impact about AF on production wikis.
   • Runtime is measured per-filter in core AbuseFilter but it has been disabled for ENWP — T101648
   • I will reach out to @aaron & @He7d3r to discuss what we can do to fix the (ironically) sluggish performance measurement.
2. A reproducible testing framework which is consistent for testing our code changes.
   • We may use selenium (or the like) to create an environment for testing changes we make to AbuseFilter, including new filters that we may want to introduce
   • Depending on what Aaron and Helder share, we may be able to use Arc Lamp.

During a meeting with Max, Leon, Dayllan, David, Caroline and Trevor:

• At this stage, we don't need to measure each filter's performance, just the entire runtime.
• Use stats-d
• Depending on how much data we're putting in, we may need to talk to Analytics team to understand if their infrastructure can handle this.
• We'll measure on all wikis with AF enabled, we'll need to split the data by wiki

Things to measure:

• Runtime
• Number of conditions executed
• Number of filters executed
• Number of times a filter becomes disabled (rare, so stretch goal)

Also: meet or talk with Analytics team to make sure they are prepared to store this data

@kaldari What team or who can I speak to about stats-d? Mostly to ask if they will have the capacity for this and on getting some help to get it running locally

@dmaza: You'll want to ask the Performance team, probably @aaron. See also https://wikitech.wikimedia.org/wiki/Statsd. Also would be good to ask @MaxSem about it.

I have a question,

Runtime

I assume we only want the filter matching runtime. Do we care about the time it takes to create the abuselog or to execute the actions as part of it?

Also, Do we want a config flag to enable/disable this?

In T161059#3541448, @dmaza wrote:

I assume we only want the filter matching runtime. Do we care about the time it takes to create the abuselog or to execute the actions as part of it?

On a per-filter basis, the "matching" time sounds good, but for this task I was hoping to get total run time, start to finish, for an edit after it goes through all filters on a wiki. In other words, the period of time the user has to wait before their edit is saved. This may or may not include things like writing to the abuselog, not sure.

Also, Do we want a config flag to enable/disable this?

I think that makes sense, yes. It's good to be able to turn it off via a SWAT deploy should things go haywire.

@MusikAnimal

I was hoping to get total run time, start to finish, for an edit after it goes through all filters on a wiki. In other words, the period of time the user has to wait before their edit is saved. This may or may not include things like writing to the abuselog, not sure.

That's what I'm doing. My question was if I should include the time it takes to write to abuselog as well as the actions triggered.

I think we should.

Change 373561 had a related patch set uploaded (by Dmaza; owner: Dmaza):
[mediawiki/extensions/AbuseFilter@master] Add runtime metrics to statsd

https://gerrit.wikimedia.org/r/373561

Change 373561 merged by jenkins-bot:
[mediawiki/extensions/AbuseFilter@master] Add runtime metrics to statsd

https://gerrit.wikimedia.org/r/373561

Change 375072 had a related patch set uploaded (by Dmaza; owner: Dmaza):
[operations/mediawiki-config@master] Enable AbuseFilter runtime profile

https://gerrit.wikimedia.org/r/375072

Initially, the following wikis will be used to start measuring performace:

'commonswiki',
'enwiki', 
'mediawikiwiki', 
'metawiki, 
'testwiki', 
'wikidatawiki'

Change 375072 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable AbuseFilter runtime profile

https://gerrit.wikimedia.org/r/375072

Mentioned in SAL (#wikimedia-operations) [2017-09-05T23:39:49Z] <reedy@tin> Synchronized wmf-config/InitialiseSettings.php: Enable AbuseFilter runtime profile T161059 (duration: 00m 49s)