| • aborrero | |
| Oct 31 2024, 9:49 AM |
| F57683510: image.png | |
| Nov 5 2024, 3:58 PM |
There is a report in T378461: Investigate Wikipedia bot/userscript usage of citoid impacting www.pro-football-reference.com that indicates that the refill tool may be misbehaving regarding its usage of the citoid production service.
Please, review it as soon as possible.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T378461 Investigate Wikipedia bot/userscript usage of citoid impacting www.pro-football-reference.com | |||
| Open | Curb_Safe_Charmer | T378686 refill: review citoid usage |
While it is possible for someone to call the reFill API directly, and therefore to misuse the API to bombard a website with requests, that would be reflected in the uwsgi.log file for the service. The log file is huge, having not been trimmed since 2019, and contains millions of rows. I have looked at the log entries for the period in question and seemingly all the requests are from internal IPs, suggesting that those requests have been generated through the front end, rather than by direct calls to the API. The log file indicates the Wikipedia page that the user has requested citoid to expand references for, and I can't see any pattern that would suggest that someone is automating calls to the front end to generate thousands of requests to a page used in a reference in a particular article. The log implies normal usage by interactive users, using reFill manually to improve articles.
Note that the uwsgi.log records the Wikipedia page, and language code for the version of Wikipedia, that the user has requested citoid to expand references for, rather than the URL of those references, so the log file makes no reference to www.pro-football-reference.com.
If reFill was being misused I would expect to see the same page appearing in the log tens of thousands of times, with that page probably just containing one URL which would be www.pro-football-reference.com as that would result in citoid hitting that page that number of times. While I have looked at the log using normal tools rather than anything approaching forensic, it doesn't look like reFill is what is causing citoid to max out that page.
Looking at English Wikipedia, there are over 5,000 articles that reference pro-football-reference.com and that were edited between 19 and 31 October 2024. It seems to me to be not beyond the imagination that a website referenced so many times in so many pages could be the subject of a great many citoid searches.... but not 30,000 times in 12 hours.
There is certainly nothing in the service log that indicates that reFill was called 30,000 times in any 12 hours.
Thanks for looking into this!
That's from october 10-11 - it's showing 391 hits for that url, and that's sampled so it's 391*128 = around 50,000 hits. The user agent says reFill but of course the user agent could be fake - is the IP listed there correct? If not it's someone else perhaps hosting the same tool elsewhere... but that's the only ip using a refill user agent... is that your correct user agent?
@akosiaris ideas?
I see in the other thread you don't have Turnilo access, sorry. The user-agent is 'reFill/2 (http://en.wikipedia.org/wiki/User:Zhaofeng_Li/reFill)'
I'm not sure what the privacy policy is re:IPs but https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Turnilo, you need nda access
The IPs listed in turnilo are internal toolforge IPs. Specifically 172.16.5.177 (for reFill), which resolves to a toolforge kubernetes worker, tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud. This isn't PII, we can share it publicly on this task. I also don't think we need any kind of more access given we are talking toolforge IPs. The IPs are part of RFC 1918, which specified which IP addresses are in the private ranges. Those addresses will never be routeable or routeable in the public internet, we can be pretty sure that they are correct.
The refill front end has been running on tools-k8s-worker-nfs-9 since 2024-10-17T20:36:11Z per https://k8s-status.toolforge.org/namespaces/tool-refill/. It could have been on a different worker before that.
The refill API is running on various workers. See https://k8s-status.toolforge.org/namespaces/tool-refill-api/.
pods currently on tools-k8s-worker-nfs-9: https://k8s-status.toolforge.org/nodes/tools-k8s-worker-nfs-9/
pods currently on tools-k8s-worker-nfs-11: https://k8s-status.toolforge.org/nodes/tools-k8s-worker-nfs-11/
Peak traffic for refill was on Oct 10 (graph pasted below) so what happened after that isn't particularly important. And refill is apparently already running in another node.
Do these definitely resolve to refill?
We can be pretty reasonably sure that they did at that point in time.
I've had another look at the refill-api log and I can't see anything unusual for October 10. I have worked out what's being going on. Usually, Wikipedia editors run reFill over a page, but there's also a little-used feature to paste in raw wiki markup by going to https://refill.toolforge.org/ng/ and clicking "Use Custom Wikicode". Starting at 0400, someone repeatedly pasted in, or (far more likely) used the API to submit, variations on the following:
{'page': 'Wikipedia:Sandbox', 'fam': 'wikipedia', 'code': 'en', 'wikicode': '==R==\n{{colbegin|colwidth=20em}}\n*[[Buster Raborn]]<ref>https://www.pro-football-reference.com/players/R/RaboBu20.htm</ref>\n*[[Kevin Rader (American football)|Kevin Rader]]<ref>https://www.pro-football-reference.com/players/R/RadeKe00.htm</ref>\n*[[Alex Rado]]<ref>https://www.pro-football-reference.com/players/R/RadoAl20.htm</ref>\n*[[George Rado]]<ref>https://www.pro-football-reference.com/players/R/RadoGe20.htm</ref>\n*[[Vincent Ragunas]]<ref>https://www.pro-football-reference.com/players/R/RaguVi20.htm</ref>\n*[[Chris Rainey]]<ref>https://www.pro-football-reference.com/players/R/RainCh00.htm</ref>\n*[[Peter Rajkovich]]<ref>https://www.pro-football-reference.com/players/R/RajkPe20.htm</ref>\n*[[Antwaan Randle El]]<ref>https://www.pro-football-reference.com/players/R/RandAn00.htm</ref>\n*[[Walter Rasby]]<ref>https://www.pro-football-reference.com/players/R/RasbWa00.htm</ref>\n*[[Leo Raskowski]]<ref>https://www.pro-football-reference.com/players/R/RaskLe20.htm</ref>\n*[[Randy Rasmussen (American football b. 1960)|Randy Rasmussen]] (born 1960)<ref>https://www.pro-football-reference.com/players/R/RasmRa21.htm</ref>\n*[[Keiwan Ratliff]]<ref>https://www.pro-football-reference.com/players/R/RatlKe20.htm</ref>\n*[[Eric Ravotti]]<ref>https://www.pro-football-reference.com/players/R/RavoEr20.htm</ref>\n*[[Israel Raybon]]<ref>https://www.pro-football-reference.com/players/R/RaybIs20.htm</ref>\n*[[Dave Reavis]]<ref>https://www.pro-football-reference.com/players/R/ReavDa20.htm</ref>\n*[[Bert Rechichar]]<ref>https://www.pro-football-reference.com/players/R/RechBe00.htm</ref>\n*[[Isaac Redman]]<ref>https://www.pro-football-reference.com/players/R/RedmIs20.htm</ref>\n*[[Jeff Reed (American football)|Jeff Reed]]<ref>https://www.pro-football-reference.com/players/R/reedxjef01.htm</ref>\n*[[Malik Reed]]<ref>https://www.pro-football-reference.com/players/R/ReedMa01.htm</ref>\n*[[Dan Reeder]]<ref>https://www.pro-football-reference.com/players/R/ReedDa20.htm</ref>\n*[[Jerry Reese (defensive lineman)|Jerry Reese]] (born 1964)<ref>https://www.pro-football-reference.com/players/R/ReesJe20.htm</ref>\n*[[John Reger]]<ref>https://www.pro-football-reference.com/players/R/RegeJo00.htm</ref>\n*[[Willie Reid (American football)|Willie Reid]]<ref>https://www.pro-football-reference.com/players/R/ReidWi00.htm</ref>\n*[[Will Renfro]]<ref>https://www.pro-football-reference.com/players/R/RenfWi20.htm</ref>\n*[[Joe Repko]]<ref>https://www.pro-football-reference.com/players/R/RepkJo20.htm</ref>\n*[[Jared Retkofsky]]<ref>https://www.pro-football-reference.com/players/R/RetkJa00.htm</ref>\n*[[Randy Reutershan]]<ref>https://www.pro-football-reference.com/players/R/ReutRa20.htm</ref>\n*[[Billy Reynolds (American football)|Billy Reynolds]]<ref>https://www.pro-football-reference.com/players/R/ReynBi00.htm</ref>\n*[[Jim Reynolds (defensive back)|Jim Reynolds]]<ref>https://www.pro-football-reference.com/players/R/ReynJi21.htm</ref>\n*[[Don Rhodes]]<ref>https://www.pro-football-reference.com/players/R/RhodDo20.htm</ref>\n*[[Dave Ribble]]<ref>https://www.pro-football-reference.com/players/R/RibbDa20.htm</ref>\n*[[Perry Richards]]<ref>https://www.pro-football-reference.com/players/R/RichPe00.htm</ref>\n*[[Daryl Richardson]]<ref>https://www.pro-football-reference.com/players/R/RichDa00.htm</ref>\n*[[Huey Richardson]]<ref>https://www.pro-football-reference.com/players/R/RichHu20.htm</ref>\n*[[Terry Richardson (American football)|Terry Richardson]]<ref>https://www.pro-football-reference.com/players/R/RichTe21.htm</ref>\n*[[Rock Richmond]]<ref>https://www.pro-football-reference.com/players/R/RichRo20.htm</ref>\n*[[Tom Ricketts (American football)|Tom Ricketts]]<ref>https://www.pro-football-reference.com/players/R/RickTo20.htm</ref>\n*[[Stevan Ridley]]<ref>https://www.pro-football-reference.com/players/R/RidlSt00.htm</ref>\n*[[Jay Riemersma]]<ref>https://www.pro-football-reference.com/players/R/RiemJa00.htm</ref>\n*[[John Rienstra]]<ref>https://www.pro-football-reference.com/players/R/RienJo20.htm</ref>\n*[[Dick Riffle]]<ref>https://www.pro-football-reference.com/players/R/RiffDi20.htm</ref>\n*[[Avon Riley]]<ref>https://www.pro-football-reference.com/players/R/RileAv20.htm</ref>\n*[[Cameron Riley]]<ref>https://www.pro-football-reference.com/players/R/RileCa20.htm</ref>\n*[[Elijah Riley]]<ref>https://www.pro-football-reference.com/players/R/RileEl01.htm</ref>\n*[[Gabe Rivera]]<ref>https://www.pro-football-reference.com/players/R/RiveGa20.htm</ref>\n*[[Elandon Roberts]]<ref>https://www.pro-football-reference.com/players/R/RobeEl00.htm</ref>\n*[[Jack Roberts (American football)|Jack Roberts]]<ref>https://www.pro-football-reference.com/players/R/RobeJa21.htm</ref>\n*[[Adrian Robinson]]<ref>https://www.pro-football-reference.com/players/R/RobiAd01.htm</ref>\n*[[Allen Robinson]]<ref>https://www.pro-football-reference.com/players/R/RobiAl02.htm</ref>\n*[[Ed Robinson (linebacker)|Ed Robinson]] (born 1970)<ref>https://www.pro-football-reference.com/players/R/RobiEd21.htm</ref>\n*[[Gil Robinson]]<ref>https://www.pro-football-reference.com/players/R/RobiGi20.htm</ref>\n*[[Jack Robinson (tackle)|Jack Robinson]]<ref>https://www.pro-football-reference.com/players/R/RobiJa20.htm</ref>\n*[[Mark Robinson (linebacker)|Mark Robinson]] (born 1999)<ref>https://www.pro-football-reference.com/players/R/RobiMa03.htm</ref>\n*[[Mike Rodak]]<ref>https://www.pro-football-reference.com/players/R/RodaMi20.htm</ref>\n*[[Mark Rodenhauser]]<ref>https://www.pro-football-reference.com/players/R/RodeMa20.htm</ref>\n*[[John Rodgers (American football)|John Rodgers]]<ref>https://www.pro-football-reference.com/players/R/RodgJo20.htm</ref>\n*[[Ben Roethlisberger]]<ref>https://www.pro-football-reference.com/players/R/RoetBe00.htm</ref>\n*[[Fran Rogel]]<ref>https://www.pro-football-reference.com/players/R/RogeFr00.htm</ref>\n*[[Cullen Rogers]]<ref>https://www.pro-football-reference.com/players/R/RogeCu20.htm</ref>\n*[[Eli Rogers]]<ref>https://www.pro-football-reference.com/players/R/RogeEl00.htm</ref>\n*[[Jim Rorison]]<ref>https://www.pro-football-reference.com/players/R/RoriJi20.htm</ref>\n*[[Oliver Ross (offensive lineman)|Oliver Ross]] (born 1974)<ref>https://www.pro-football-reference.com/players/R/RossOl20.htm</ref>\n*[[Allen Rossum]]<ref>https://www.pro-football-reference.com/players/R/RossAl99.htm</ref>\n*[[Pete Rostosky]]<ref>https://www.pro-football-reference.com/players/R/RostPe20.htm</ref>\n*[[Tom Rouen]]<ref>https://www.pro-football-reference.com/players/R/rouentom01.htm</ref>\n*[[Eric Rowe]]<ref>https://www.pro-football-reference.com/players/R/RoweEr00.htm</ref>\n*[[Bob Rowley]]<ref>https://www.pro-football-reference.com/players/R/RowlBo20.htm</ref>\n*[[John Rowser]]<ref>https://www.pro-football-reference.com/players/R/RowsJo20.htm</ref>\n*[[Mark Royals]]<ref>https://www.pro-football-reference.com/players/R/RoyaMa20.htm</ref>\n*[[Orpheus Roye]]<ref>https://www.pro-football-reference.com/players/R/RoyeOr20.htm</ref>\n*[[Aubrey Rozzell]]<ref>https://www.pro-football-reference.com/players/R/RozzAu20.htm</ref>\n*[[Mason Rudolph (American football)|Mason Rudolph]]<ref>https://www.pro-football-reference.com/players/R/RudoMa00.htm</ref>\n*[[Guy Ruff]]<ref>https://www.pro-football-reference.com/players/R/RuffGu20.htm</ref>\n*[[Ernie Ruple]]<ref>https://www.pro-football-reference.com/players/R/RuplEr20.htm</ref>\n*[[Darius Rush]]<ref>https://www.pro-football-reference.com/players/R/RushDa00.htm</ref>\n*[[Andy Russell (American football)|Andy Russell]]<ref>https://www.pro-football-reference.com/players/R/RussAn00.htm</ref>\n*[[Gary Russell (American football)|Gary Russell]]<ref>https://www.pro-football-reference.com/players/R/RussGa00.htm</ref>\n*[[Ed Ryan (American football player)|Ed Ryan]]<ref>https://www.pro-football-reference.com/players/R/RyanEd20.htm</ref>\n{{colend}}\n', 'preferences': {'dateFormat': {'en': 'mdy'}, 'addAccessDates': False}}
They varied the request to use different first letters of surnames.
They ran this (if I count correctly) 629 times until 1809 on October 14.
I think the markup they were using came from pages like https://en.wikipedia.org/wiki/List_of_NFL_players_(R)
The IP addresses in the log are all 192.168.*.*
Hi @Curb_Safe_Charmer, thanks for the update, the IPs are a bit weird, we 'll have to dig more into that. Did the logged requests have a user-agent by any chance?
@akosiaris no, unfortunately not. Here's a snippet of the log:
[pid: 13|app: 0|req: 7940/19276] 192.168.36.110 () {60 vars in 1075 bytes} [Thu Oct 10 04:01:32 2024] OPTIONS /fixWikipage => generated 0 bytes in 3 msecs (HTTP/1.1 200) 7 headers in 293 bytes (0 switches on core 0)
{'page': 'Wikipedia:Sandbox', 'fam': 'wikipedia', 'code': 'en', 'wikicode': "==T==\n{{colbegin|colwidth=20em}}\n*[[Kelly Talavou]]<ref>https://www.pro-football-reference.com/players/T/TalaKe00.htm</ref>\n*[[Lorenzo Taliaferro]]<ref>https://www.pro-football-reference.com/players/T/TaliLo00.htm</ref>\n*[[T.J. Tampa]]<ref>https://www.pro-football-reference.com/players/T/TampTJ00.htm</ref>\n*[[Robert Tate (American football)|Robert Tate]]<ref>https://www.pro-football-reference.com/players/T/TateRo00.htm</ref>\n*[[Chester Taylor]]<ref>https://www.pro-football-reference.com/players/T/TaylCh01.htm</ref>\n*[[Leland Taylor (American football)|Leland Taylor]]<ref>https://www.pro-football-reference.com/players/T/TaylLe20.htm</ref>\n*[[Shannon Taylor (American football)|Shannon Taylor]]<ref>https://www.pro-football-reference.com/players/T/TaylSh20.htm</ref>\n*[[Travis Taylor (American football)|Travis Taylor]]<ref>https://www.pro-football-reference.com/players/T/TaylTr01.htm</ref>\n*[[Tyrod Taylor]]<ref>https://www.pro-football-reference.com/players/T/TaylTy00.htm</ref>\n*[[Adam Terry]]<ref>https://www.pro-football-reference.com/players/T/TerrAd20.htm</ref>\n*[[Vinny Testaverde]]<ref>https://www.pro-football-reference.com/players/T/TestVi00.htm</ref>\n*[[Adalius Thomas]]<ref>https://www.pro-football-reference.com/players/T/ThomAd00.htm</ref>\n*[[De'Anthony Thomas]]<ref>https://www.pro-football-reference.com/players/T/ThomDe05.htm</ref>\n*[[Earl Thomas (defensive back)|Earl Thomas]] (born 1989)<ref>https://www.pro-football-reference.com/players/T/ThomEa99.htm</ref>\n*[[Jason Thomas (American football)|Jason Thomas]]<ref>https://www.pro-football-reference.com/players/T/ThomJa20.htm</ref>\n*[[Joe Thomas (linebacker)|Joe Thomas]] (born 1991)<ref>https://www.pro-football-reference.com/players/T/ThomJo04.htm</ref>\n*[[Bennie Thompson (American football)|Bennie Thompson]]<ref>https://www.pro-football-reference.com/players/T/ThomBe00.htm</ref>\n*[[Christian Thompson (American football)|Christian Thompson]]<ref>https://www.pro-football-reference.com/players/T/ThomCh01.htm</ref>\n*[[Deonte Thompson]]<ref>https://www.pro-football-reference.com/players/T/ThomDe04.htm</ref>\n*[[Eric Tomlinson]]<ref>https://www.pro-football-reference.com/players/T/TomlEr01.htm</ref>\n*[[Fitzgerald Toussaint]]<ref>https://www.pro-football-reference.com/players/T/TousFi00.htm</ref>\n*[[Johnny Townsend (American football)|Johnny Townsend]]<ref>https://www.pro-football-reference.com/players/T/TownJo01.htm</ref>\n*[[James Trapp]]<ref>https://www.pro-football-reference.com/players/T/TrapJa20.htm</ref>\n*[[Brynden Trawick]]<ref>https://www.pro-football-reference.com/players/T/TrawBr00.htm</ref>\n*[[Laquon Treadwell]]<ref>https://www.pro-football-reference.com/players/T/TreaLa00.htm</ref>\n*[[Justin Tucker]]<ref>https://www.pro-football-reference.com/players/T/TuckJu00.htm</ref>\n*[[De'Lance Turner]]<ref>https://www.pro-football-reference.com/players/T/TurnDe01.htm</ref>\n*[[Eric Turner (American football)|Eric Turner]]<ref>https://www.pro-football-reference.com/players/T/TurnEr00.htm</ref>\n*[[Floyd Turner]]<ref>https://www.pro-football-reference.com/players/T/TurnFl00.htm</ref>\n*[[David Tyree]]<ref>https://www.pro-football-reference.com/players/T/TyreDa00.htm</ref>\n*[[Deangelo Tyson]]<ref>https://www.pro-football-reference.com/players/T/TysoDe00.htm</ref>\n{{colend}}\n", 'preferences': {'dateFormat': {'en': 'mdy'}, 'addAccessDates': False}}
[pid: 12|app: 0|req: 5413/19277] 192.168.167.105 () {64 vars in 1159 bytes} [Thu Oct 10 04:01:33 2024] POST /fixWikipage => generated 236 bytes in 12 msecs (HTTP/1.1 202) 4 headers in 151 bytes (1 switches on core 0)
[pid: 13|app: 0|req: 7941/19278] 192.168.166.32 () {40 vars in 711 bytes} [Thu Oct 10 04:01:42 2024] GET / => generated 3842 bytes in 3 msecs (HTTP/1.1 200) 3 headers in 113 bytes (1 switches on core 0)
[pid: 12|app: 0|req: 5414/19279] 192.168.36.110 () {60 vars in 1209 bytes} [Thu Oct 10 04:01:33 2024] GET /statusStream/fixWikipage/7b2c9b71-47d2-4df1-8dd9-268a519d0f58 => generated 272794 bytes in 19151 msecs (HTTP/1.1 200) 4 headers in 147 bytes (22 switches on core 0)
[pid: 12|app: 0|req: 5415/19280] 192.168.36.110 () {40 vars in 715 bytes} [Thu Oct 10 04:02:13 2024] GET / => generated 3842 bytes in 3 msecs (HTTP/1.1 200) 3 headers in 113 bytes (1 switches on core 0)
[pid: 13|app: 0|req: 7942/19281] 192.168.166.32 () {60 vars in 1071 bytes} [Thu Oct 10 04:02:23 2024] OPTIONS /fixWikipage => generated 0 bytes in 2 msecs (HTTP/1.1 200) 7 headers in 293 bytes (0 switches on core 0)