note that scholia queries generally start with the comment:

# tool: scholia

Thank you, David!

Task is refined and I'm starting work on it now. I'm assuming that event.wdqs_external_sparql_query is what I'd use for this, and thus we'd be getting aggregate/percent values within a 90 day period given the retention policy :)

Let me know if there's anything else that should be included in this!

Quick note on this:

There are two ways that need to be factored in to deriving if a query is from Scholia. Some queries do start with #tool: scholia as @dcausse suggested, but I checked for user agents and also found that the string "Scholia" is also used as a user agent. Big thing is that some of the queries have the comment and some have the user agent, but in no cases do we have both.

Here are some initial results for consideration. Using the following query over the full dataset from event.wdqs_external_sparql_query (last 90 days):

SELECT
    count(*) AS total_scholia_queries

FROM
    event.wdqs_external_sparql_query

WHERE
    query LIKE '%# tool: scholia%'
    OR http.request_headers['user-agent'] LIKE '%Scholia%'

Aggregate queries over the time period (count(*) with no WHERE clause): 869,239,193
Scholia queries over the time period: 27,639,881
Percent Scholia queries over the time period: 3.18%

In T353453#9524925, @AndrewTavis_WMDE wrote:

Quick note on this:

There are two ways that need to be factored in to deriving if a query is from Scholia. Some queries do start with #tool: scholia as @dcausse suggested, but I checked for user agents and also found that the string "Scholia" is also used as a user agent. Big thing is that some of the queries have the comment and some have the user agent, but in no cases do we have both.

Indeed I saw these two as well, I'm not sure how to interpret this yet but it could be that some are coming from web browsers browsing https://scholia.toolforge.org/ (#tool: scholia in the query) and the "Scholia" user-agent might be from some automated tooling used by scholia that we have yet to discover? Looking at the queries might help.
Regarding #tool: scholia something I noted is a non negligible portion of the traffic is coming from automated web crawlers, this might be interesting to identify and distinguish.

Results from the following query to check automate traffic via isSpiderUDF is that 91.36% of the #tool: scholia queries are automated:

WITH automate_or_not AS (
    SELECT
        is_spider(http['request_headers']['user-agent']) AS is_spider

    FROM
        event.wdqs_external_sparql_query

    WHERE
        query LIKE '%# tool: scholia%'
)

SELECT
    is_spider AS is_spider,
    count(*) AS total_queries
    
FROM
    automate_or_not
    
GROUP BY
    is_spider

@dcausse and I found the aforementioned UDF for this. Note for reporting: the UDF is based on user agents, so a similar comparison for queries that have the user agent "Scholia" will not work as they'd either all be automate or none of them would be.

Shifting now to inspecting queries in the following comparisons:

• #tool: scholia queries vs. user agent is "Scholia"
• For #tool: scholia queries, those that are spiders and those that aren't

Quick counts as in the sampling task to check uniqueness of queries and HTTP statuses (I don't think that other measures like variance over weeks, duration or number of characters would add much). Note that percentages below are for the sub-groups, not for all Scholia queries. Period for the following is all queries 90 days to the date of posting.

All queries with the #tool: scholia comment

Spider Comment Queries

Non-spider Comment Queries

All queries with the "Scholia" user agent

Discussion

Nothing jumps out per say from the above. Big thing is that we have a higher percentage of unique queries coming from those identified via a comment than those that are identified via a user agent. I don't think that we can say that one is for traffic/API and another is for general tooling because:

• Higher unique queries from those with a comment would possibly indicate unique user searches
• But then the commented queries also seem more "routine" where we have a higher percentage of queries being ran 11, 10 and 12 times in the period
  • The distribution order of the user agent identified queries is more in line with user behavior

As far as HTTP status, 99+% 200s across commented and user agent queries is great :)

Final thing from what's been discussed so far is looking at some examples from the above breakdowns.

Having derived quick samples (DISTRIBUTE BY rand() to mix it up, but nothing more), what I'm seeing is that the comment queries look to be very similar to one another regardless of if they're spiders or non-spiders. Could be that what we're thinking of as a non-spider just isn't being picked up by the UDF. Each of them has a PREFIX target: <http://www.wikidata.org/entity/QID_TARGET> at the top that then is assigned further on down the query. A final check for this could be to see how our counts above would change if we took the part of the query after this assignment is made such that we just have the template query. My expectation is that the whole distribution will shift to the left such that we have dramatically more unique queries at all levels in that the distinct part for most of these seems to be the QID in question. The queries themselves are varied based on all manner of things that could be found out about researchers: student-supervisor relationships, number of publications, etc, with default views like maps, graphs, bar charts and others being assigned at the top between the #tool: scholia comment and the prefix assignment.

The user agent based queries are totally different and normally extremely small. Templates for these are:

select ?class where { wd:QID_TARGET wdt:P279+ ?class }

SELECT ?class { wd:QID_TARGET wdt:P31 ?class }

SELECT ?doi { wd:QID_TARGET wdt:P356 ?doi }

So it seems that the assumption that these are for automated tooling was correct :) Minor helper queries to check what things are.

Let me know if a check of the queries after removing the the prefix assignment would be helpful. We could also check the user agent queries with just Q rather than the full QID to get an idea of their variance. At the very least this would give us an idea of the total number of query templates they have.

Hi Andrew, good idea to investigate the types of queries per source! The results seem highly relevant: Could you please add the % of user agent based queries to the results summary?

Credit on checking the queries goes to @dcausse :) Added the percent that are identified via a user agent to the results summary just now: 55.29%.

A follow up request from @Manuel on this was for the total IPs that are accessing Scholia. The following query was run for this:

SELECT
    count(
        DISTINCT CASE 
            WHEN query LIKE '%# tool: scholia%' THEN http.client_ip 
        END
    ) AS total_comment_ips,
    count(
        DISTINCT CASE 
            WHEN http.request_headers['user-agent'] LIKE '%Scholia%' THEN http.client_ip 
        END
    ) AS total_user_agent_ips

FROM
    event.wdqs_external_sparql_query

WHERE
    query LIKE '%# tool: scholia%'
    OR http.request_headers['user-agent'] LIKE '%Scholia%'

Results are:

• Total IPs is: 28,918
• total_comment_ips: 28,740 (99.4%)
• total_user_agent_ips: 178 (0.6%)

Updated the above comment and description with a second run and also ran a query for the total IPs for the given period, with the result being 2,115,166. Percent Scholia queries for the period is thus 28918 / 2115166 * 100, or 1.37%.