Page MenuHomePhabricator

Investigate toolviews data collection
Closed, ResolvedPublic


@bd808, is that data on delayed or downsampled at all? My usage alone of exceeds what I'm seeing at

The database behind the toolviews tool should be updated each time the nginx front proxy access logs are rotated. That happens somewhere between 1 and 4 times a day depending on the total request volume sent to * But, I will admit that nobody is really watching the script that parses the nginx log after each rotation, so it may be miscounting things. Just eyeballing the result from I think total numbers look reasonable, but the fourohfour tool may be getting more than its rightful share of hits. The parsing script adds to the count for the fourohfour tool when it thinks the request was for an unknown tool, so this could indicate something going wrong with the data collection. I will start a new task to look into that.

Event Timeline

bd808 created this task.Oct 19 2020, 9:04 PM
Restricted Application added a project: User-bd808. · View Herald TranscriptOct 19 2020, 9:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
bd808 added a comment.Oct 19 2020, 9:08 PM

A quick check of the parsing script on tools-proxy-06 shows that the overcounting of the fourohfour tool is related to some failure in collecting the list of all known tools to validate against. Running head -10000 /var/log/nginx/access.log.1 | /usr/local/bin/ --dry-run -v shows many 'Unknown tool "<toolname>"' messages for things that are valid tools. My current guess is that something has changed in the Python LDAP library that is used to create the list of all tools.

Change 635084 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] toolviews: Fix logic bug affecting vhosts

bd808 triaged this task as High priority.Oct 19 2020, 9:29 PM
bd808 moved this task from To Do to Needs Review/Feedback on the User-bd808 board.
bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

Change 635084 merged by Bstorm:
[operations/puppet@production] toolviews: Fix logic bug affecting vhosts

bd808 closed this task as Resolved.Oct 19 2020, 11:13 PM

Thanks for spotting the missing data @ST47. Unfortunately we do not keep the old access.log files around so we will not be able to backfill the corrected data, but * tools should be counted properly going forward.