Results look correct to me with that query:

SELECT
    agent_type,
    count(1) as count
FROM
  webrequest
WHERE
  year = 2016
  AND month = 5
  AND day = 10
  AND uri_host LIKE "%wikidata.org"
  AND is_pageview
  AND pageview_info['page_title'] = "Special:RecentChangesLinked"
GROUP BY agent_type
ORDER BY count
LIMIT 999999;
----------------------------------------------
agent_type	count
spider	72
user	91328

What has changed:

• include wikidata.org as well as www.wikidata.org
• filter for pageview only (instead of all requests)
• filter pageview title instead of uri (there are multiple ways to query the same page with mediawiki).

Okay, so the first queries we used while looking also caught pages such as Special:RecentChangesLinked/Q13215 which are tracked as views to different pages!

Now all of the numbers look like the match up.

It looks like the UA throwing the data off here is not included in the spiders category but should be!

"mozilla [en] egranary digital library system"

That appears to be what caused the flood that was noticed.

In T135164#2290869, @Addshore wrote:

Okay, so the first queries we used while looking also caught pages such as Special:RecentChangesLinked/Q13215 which are tracked as views to different pages!

"%" is the wildcard character for LIKE in HQL and SQL. So uri_path LIKE "/wiki/Special:RecentChangesLinked%" would be expected to catch these, yes.

BTW, was there a particular reason why you were querying the webrequest table and not the much smaller pageview_hourly table? It gives you the same numbers with much less computational effort:

SELECT
    agent_type,
    SUM(view_count) AS count
FROM
  wmf.pageview_hourly
WHERE
  year = 2016
  AND month = 5
  AND day = 10
  AND project = "wikidata"  
  AND page_title = "Special:RecentChangesLinked"
GROUP BY agent_type
ORDER BY count
LIMIT 999999;
...
agent_type	count
spider	72
user	91328
2 rows selected (108.284 seconds)

In T135164#2291517, @Tbayer wrote:

In T135164#2290869, @Addshore wrote:

Okay, so the first queries we used while looking also caught pages such as Special:RecentChangesLinked/Q13215 which are tracked as views to different pages!

My issue here is I didn't realise that these would be tracked as separate page views! I would have thought sub pages of special pages would result in views of the main page.

As for webrequest vs pageview_hourly I was unaware of pageview_hourly until yesterday.

@Tbayer : I suggested @Addshore to request webrequest on a specific hour for detailed user_agent analysis.
For this check @Addshore, I would really have gone for ONE HOUR of data, making the volume of data to work real smaller (data is partitionned up to hour).
As for using the pageview_hourly for double checking number, this is the first thing I did, and I have finally double checked everything was ok on webrequest.

In T135164#2292179, @JAllemandou wrote:

@Tbayer : I suggested @Addshore to request webrequest on a specific hour for detailed user_agent analysis.
For this check @Addshore, I would really have gone for ONE HOUR of data, making the volume of data to work real smaller (data is partitionned up to hour).
As for using the pageview_hourly for double checking number, this is the first thing I did, and I have finally double checked everything was ok on webrequest.

Of course my queries when initially looking where for a single hour in the day, but not knowing about webrequest_daily of course means the only way to get the numbers for the whole day is to query the whole day on webrequest.

.Please try to notify owner of UA policy. If they add the word "bot" to UA this would automatically be marked as spider.

In T135164#2297815, @Nuria wrote:

.Please try to notify owner of UA policy. If they add the word "bot" to UA this would automatically be marked as spider.

Who is responsible for this?

The policy doesn't have an specific owner, if that is what you are asking. Here is is: https://meta.wikimedia.org/wiki/User-Agent_policy

This is again another instance of bot traffic that slips by, this UA might not be causing trouble now but there will be others. Merging into parent task