@bd808, is that data on toolviews.toolforge.org delayed or downsampled at all? My usage alone of whois-referral.toolforge.org exceeds what I'm seeing at https://toolviews.toolforge.org/api/v1/tool/whois-referral/daily/2020-07-01/2020-10-19
The database behind the toolviews tool should be updated each time the nginx front proxy access logs are rotated. That happens somewhere between 1 and 4 times a day depending on the total request volume sent to *.toolforge.org. But, I will admit that nobody is really watching the script that parses the nginx log after each rotation, so it may be miscounting things. Just eyeballing the result from https://toolviews.toolforge.org/api/v1/day/2020-10-01 I think total numbers look reasonable, but the fourohfour tool may be getting more than its rightful share of hits. The parsing script adds to the count for the fourohfour tool when it thinks the request was for an unknown tool, so this could indicate something going wrong with the data collection. I will start a new task to look into that.
|operations/puppet||production||+1 -1||toolviews: Fix logic bug affecting toolforge.org vhosts|
A quick check of the parsing script on tools-proxy-06 shows that the overcounting of the fourohfour tool is related to some failure in collecting the list of all known tools to validate against. Running head -10000 /var/log/nginx/access.log.1 | /usr/local/bin/toolviews.py --dry-run -v shows many 'Unknown tool "<toolname>"' messages for things that are valid tools. My current guess is that something has changed in the Python LDAP library that is used to create the list of all tools.