We would like to know how many filters in each wiki are using the rmspecials() function so that we can decide on how to modify this function so that it would not treat white space as a special character.
This would require running a queries on the production servers hence tagging Wikimedia-Site-requests.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Resolved | Huji | T263024 Make rmspecials preserve whitespace | |||
| Resolved | Urbanecm | T262047 Determine the usage of rmspecials on WMF wikis |
@Daimona did we ever document the process for how to engage DBAs to run queries for Wikimedia-abusefilter-global-maintainers tasks? Is "site-requests" the right tag? Or is it DBA?
Based on information I scraped from last year, rmspecials is used 137 times in 28 wikis. I can rerun the query today if you want. Note that I don't have an access to production servers. Just regular API queries with abusefilter helper right.
Right, right. T191978#5704225 is what I vaguely remembered. I'll make a task for writing that shell script.
Would you mind running an API-based query again? It'll take us a while to address T262052 and there is no reason to wait for it.
Additionally, would you mind sharing the program you use to run these queries? Perhaps we can create a nimble JS gadget instead of, or in addition to, the shell script described in T262052. That way, abusefiler helpers and abusefilter maintainers can run queries too, not just those with shell access.
I can run a query tonight (PT). What information do you need precisely? A list of sites and filter ids?
Sorry, the code is written in Racket. I don't think it's gonna be helpful to you. It also has a bunch of other stuff as a part of my research project that I want to keep private for now.
@Nullzero the reason I asked is because to the best of my knowledge, AbuseFilter does not expose an API end point for its filter search functionality. So I am curious how you are searching the filters. If you provide some guidance, I should be able to write the JS Gadget myself.
Oh, I simply downloaded everything. Here's an excerpt of a request I sent:
(define resp (query `([action . "query"]
[list . "abusefilters"]
[abfstartid . ,start-id]
[abflimit . "500"]
[abfshow . "!deleted|enabled"]
[abfprop . "id|description|pattern|private"])))Then, you can either grep or parse+traverse the AST for a more advanced search.
It would make sense to have a tool at toolforge that cron this request regularly (every day?) for a more responsive search. But also note it needs some authentication since private filter details should not be viewable in public.
Got it. That should be doable. All I need to figure out is the best way to get a current list of all wikis' base URLs. The rest is a bunch of API calls.
That said, I'm not going to start working on it right away. I have a few reservations, which I will describe on T262052
What about https://meta.wikimedia.org/w/api.php?action=sitematrix&format=json&smtype=language, @Huji?
That said, I'm not going to start working on it right away. I have a few reservations, which I will describe on T262052
Thanks. This is helpful. I currently extract sites from https://meta.wikimedia.org/wiki/List_of_Wikimedia_projects_by_size which I think might not be up-to-date.
That is great!
I started working on the script, and got the first part down: fetch sitematrix and loop through the wiki URLs. But querying the APIs of each wiki failed, because of CORS restrictions. @Nullzero how did you get around CORS?
@Huji Could you please share an example of your code (minimum failing example please) and the error message? I'll be happy to help!
The problem with CORS as I understand, occurs because you are running it from a site. I scraped information from a script running from my computer directly, so there's no problem.
As stated above, i was trying to make it a Gadget. So yes, I was running it off of a site. First I tried to run it via Meta; that way the action=sitematrix part worked, but the calls to the individual wikis failed. Then I tried to run it off of my own computer (a file on my drive, opened in the browser); this time, even the call to meta failed. I guess I can run it from the command line with node or something, but I'm trying to make a tool that is easy for people without shell access to use, and most people know how to run a script from the browser, but not necessarily from the command line.
See P12512. For now, I'd rather keep it private.
That would be caused by T210790: Allow cross-origin requests by default in the Action API, AFAICS, it is currently not possible to do an cross-origin request for an API. That's going to change, however - see the task linked :-). It should be already possible to make cross-origin requests to the rest API (T262425). It seems this can't be a gadget rn :/.
Now that T262052 has been fixed, I am going to assign this to Urbanecm who I think will be able to fulfill this task in ~2 weeks, when the maintenance script reaches WMF production servers.
@Huji Done, but I had to change the script locally to ignore labtestwiki, because wikiadmin user didn't have user rights to that database. That wiki is for tests only, and we can ignore that for now.
Results:
@Urbanecm because it took me so long to review these, @Billinghurst has asked that we rerun the query above.
If it is possible to search by "contains rmspecials AND does not contain remwhitespace" that would be ideal. Otherwise, you can rerun the exact search above, and I can review them by hand. I have a few days of increased availability this and next week and can do it quickly this time.
@Huji I've repeated the search above:
Unfortunately, there isn't an option to exclude filters matching a given pattern. It should be possible to do a set difference with the results above, or run the query manually, but I guess it could have false positives due to filters that use rmwhitespace but not together with rmspecials.
Daimona beated me, was just running it :). You can also use https://search-filters.toolforge.org/ to self-service those searches (and if you have spare time, improvements of that tool to be more flexible with search queries would be appreciated).
That is a cool tool, @Daimona. I have added a review of it to my TODO list. For one thing, I already found a typo worth fixing.
Closing this task as complete again.