Right, im starting to see some things that may link together.
SFS (StopForumSpam) makes uses of the MediaWiki cache, to store some bad IP addresses, if it is not cached, it will fetch (curl) from a remote URL.
3:34 PM

The cache it will use right now is redis, as that is the main cache for MW, (so redis isn't only sessions).
As that is designed for minimal caching right now the SFS data probably leaves the cache quite often resulting in new fetches
Tobias Andersson
3:34 PM

that sounds awesome
Adam Shorland
3:35 PM

so thats probably 1 thing to look at (I didnt test this hypothosis yet)
3:35 PM

now about to look at addCidr specifically
3:37 PM

I dont see addCidr in SFS extension at all
3:38 PM

but addCidr is used in IPSet which is used in the DenyListManager
3:39 PM

So the 11 seconds is in https://github.com/wikimedia/mediawiki-extensions-StopForumSpam/blob/950d636ebde9f7846648967aade0ee8db6a80ab0/includes/DenyListManager.php#L181-L187
And will add the whole deny list to this IPSet object
GitHub
mediawiki-extensions-StopForumSpam/DenyListManager.php at 950d636ebde9f7846648967aade0ee8db6a80ab0 · wikimedia/mediawiki-extensions-StopForumSpam
Github mirror of MediaWiki extension StopForumSpam - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for contributing - mediawiki-extensions-StopF...
3:40 PM

Its around 72k IP addresses that it adds to this IPSet
3:40 PM

and it'll do this once per request where this SFS code patch is called, 
even if cached), and maybe that takes some time to do this, possibly made worse by CPU restrictions 
3:43 PM

Side note, there is also work moving towards deploying this on WMF production :slightly_smiling_face: https://phabricator.wikimedia.org/T273220
3:46 PM

Without doing any more advanced investigation, I suspect that

0) Could add a simple log message that outputs in wikibase.cloud production timing how long generating this IPSet takes
1) We would want to make sure that there is enough room in redis so that this global cache key (shared between all wikis) of the list if IPs doesnt get booted out too often.
2) It might be possible to change the extension to allow the cache to be configurable rather than just using the mw main cache
3) We may want to bump CPU limit for mediawiki so that it can do this bit of processing faster ( :warning: easiest way to see a performance gain I bet )
4) its possible the extension itself could have some optimization done

name                                                                                                                 wallTime   wallTime%   wallTime1   ownTime    ownTime%   ownTime1   count
Wikimedia\IPSet::addCidr                                                                                             11.34s     73.71       42.6us      8.04s      52.25      30.2us     266K 
curl_exec                                                                                                            1.58s      10.28       790.72ms    1.58s      10.27      790.45ms   2    
MediaWiki\StopForumSpam\Hooks::onGetUserPermissionsErrorsExpensive                                                   12.87s     83.67       2.15s       887.02ms   5.77       147.84ms   6    
Wikimedia\AtEase\AtEase::suppressWarnings                                                                            945.15ms   6.14        1.8us       801.14ms   5.21       1.5us      533K 
array_map                                                                                                            860.74ms   5.59        3.2us       641.9ms    4.17       2.4us      266K 
str_split                                                                                                            481.19ms   3.13        1.8us       481.19ms   3.13       1.8us      266K 
Wikimedia\AtEase\AtEase::quietCall                                                                                   1.73s      11.23       6.5us       466.32ms   3.03       1.8us      266K 
mysqli::query                                                                                                        461.3ms    3           2.71ms      461.3ms    3          2.71ms     170  
Wikimedia\IPSet::__construct                                                                                         11.73s     76.21       2.93s       384.57ms   2.5        96.14ms    4    
Wikimedia\AtEase\AtEase::restoreWarnings                                                                             811.65ms   5.28        3us         254.87ms   1.66       1us        266K 
explode