In T265784#6562223, @bd808 wrote:In T265784#6561226, @ST47 wrote:@bd808, is that data on toolviews.toolforge.org delayed or downsampled at all? My usage alone of whois-referral.toolforge.org exceeds what I'm seeing at https://toolviews.toolforge.org/api/v1/tool/whois-referral/daily/2020-07-01/2020-10-19
The database behind the toolviews tool should be updated each time the nginx front proxy access logs are rotated. That happens somewhere between 1 and 4 times a day depending on the total request volume sent to *.toolforge.org. But, I will admit that nobody is really watching the script that parses the nginx log after each rotation, so it may be miscounting things. Just eyeballing the result from https://toolviews.toolforge.org/api/v1/day/2020-10-01 I think total numbers look reasonable, but the fourohfour tool may be getting more than its rightful share of hits. The parsing script adds to the count for the fourohfour tool when it thinks the request was for an unknown tool, so this could indicate something going wrong with the data collection. I will start a new task to look into that.
Description
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| toolviews: Fix logic bug affecting toolforge.org vhosts | operations/puppet | production | +1 -1 |
Related Objects
Event Timeline
A quick check of the parsing script on tools-proxy-06 shows that the overcounting of the fourohfour tool is related to some failure in collecting the list of all known tools to validate against. Running head -10000 /var/log/nginx/access.log.1 | /usr/local/bin/toolviews.py --dry-run -v shows many 'Unknown tool "<toolname>"' messages for things that are valid tools. My current guess is that something has changed in the Python LDAP library that is used to create the list of all tools.
Change 635084 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] toolviews: Fix logic bug affecting toolforge.org vhosts
Change 635084 merged by Bstorm:
[operations/puppet@production] toolviews: Fix logic bug affecting toolforge.org vhosts
Thanks for spotting the missing data @ST47. Unfortunately we do not keep the old access.log files around so we will not be able to backfill the corrected data, but *.toolforge.org tools should be counted properly going forward.