The use is for correlation of request patterns in the data lake and those originating from tools.

I don't have any objections to @dr0ptp4kt becoming a Toolforge admin, but I am curious about his stated intentions. Adam, is there a task that gives more information on what you are hoping to do here? On the surface I'm not sure what information having the access logs to the Toolforge front proxy will give you that relates to the production traffic recorded in the data lake.

The immediate term thing I'm checking is query density for WDQS, in particular for scholarly article oriented queries as part of the WDQS graph split.

For example, I'm trying to align page load scoped-200s contained in these:

curl -X GET "https://toolviews.toolforge.org/api/v1/tool/scholia/day/2024-05-05" -H "accept: application/json"

and query.wikidata.org SPARQL query counts like those in https://superset.wikimedia.org/superset/explore/p/N9kQ7ZanEZD/ which uses this query:

SELECT "http_method" AS "http_method",
       "content_type" AS "content_type",
       "http_status" AS "http_status",
       "cache_status" AS "cache_status",
       "access_method" AS "access_method",
       "agent_type" AS "agent_type",
       "referer_class" AS "referer_class",
       COUNT(*) AS "count"
FROM "wmf"."webrequest"
WHERE "year" = 2024
  AND "month" = 5
  AND "day" = 5
  AND "uri_host" = 'query.wikidata.org'
  AND "referer" LIKE 'https://scholia.toolforge.org/%'
GROUP BY "http_method",
         "content_type",
         "http_status",
         "cache_status",
         "access_method",
         "agent_type",
         "referer_class"
ORDER BY "count" DESC
LIMIT 10000;

In this tool's case there can be multiple queries per page load, which is most likely to scale with JS-capable clients making such requests.

+1 from me

As defined in the application process I have sent a message to the cloud-admin mailing list: https://lists.wikimedia.org/hyperkitty/list/cloud-admin@lists.wikimedia.org/thread/NKIDWUGLPZOQMQ4EI3WXYU47SKWDG5WF/

In https://lists.wikimedia.org/hyperkitty/list/cloud-admin@lists.wikimedia.org/thread/NKIDWUGLPZOQMQ4EI3WXYU47SKWDG5WF/, @fnegri wrote:

This is a subset of the "Tool root" permissions [1] that are usually assigned to users who need to do administrative work in Toolforge. Given Adam's needs are more limited, I don't think we need to add any other permission other than the membership of the "admin" tool.

I am fairly certain he will also need to be added to the "roots" sudoers group via Horizon to be able to read the nginx logs from the front proxies that he is interested in data mining. https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#What_makes_a_root/Giving_root_access

In T364761#9801439, @bd808 wrote:

In https://lists.wikimedia.org/hyperkitty/list/cloud-admin@lists.wikimedia.org/thread/NKIDWUGLPZOQMQ4EI3WXYU47SKWDG5WF/, @fnegri wrote:

This is a subset of the "Tool root" permissions [1] that are usually assigned to users who need to do administrative work in Toolforge. Given Adam's needs are more limited, I don't think we need to add any other permission other than the membership of the "admin" tool.

I am fairly certain he will also need to be added to the "roots" sudoers group via Horizon to be able to read the nginx logs from the front proxies that he is interested in data mining. https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#What_makes_a_root/Giving_root_access

I think this is fine. We will add the account to both, the admin tool and the root sudoers group.

• add @dr0ptp4kt to admin tool
• add @dr0ptp4kt to the root sudoers group

Apologies about this taking so long since the last comment on this task – seems it got a bit stuck in workboard limbo. Let us know if you need any further help.

Thanks, the nginx logs are accessible on tools-proxy-7 and tools-proxy-8 for me now.