Still getting a 504 when trying to access https://refill.toolforge.org/

Started working again after running

Following process at https://en.wikipedia.org/wiki/Wikipedia:Refill/restart

I didn't do anything; problem seems to have gone away of its own accord.

Hi @Referentis - are you still experiencing this?

I agree with Pppery. If I browse to https://collections.ram.ac.uk/IMU/#/details/ecatalogue/893 and view the source, the title element is as follows:

I used the insert citation button in Visual Editor to test whether this is specific to reFill or affects Citoid more generally. I can confirm that the same behaviour happens when using Visual Editor to insert the citation.

That edit had an edit summary of "Filled in 13 bare reference(s) with reFill 2, Script-assisted style fixes and per CS1" and included a bunch of manual of style changes that reFill is incapable of. I took a copy of the page prior to the edit and copied to may sandbox, then ran reFill over it to see which changes were made by reFill.

This would need to be implemented in Citoid (https://phabricator.wikimedia.org/project/profile/62/) rather than reFill.

The language parameter is not set by reFill. As you suspect, it is set by citoid, or more accurately, by Zotero. There are some open tasks relating to language which you can check to see if the issue is already logged:

Agreed, please go ahead

I can confirm that is working now, thanks TNT.

I can reproduce this in Chrome, but not in incognito mode. The code the error occurs in is part of the LastPass browser extension. It is a known issue.

Seems to have resolved itself.

I've tried restarting the webservice but that hasn't helped. Sammy, are you around to take a look?

This time, seems to have come back to life after simply running

webservice restart

on refill-api.

@akosiaris no, unfortunately not. Here's a snippet of the log:

I've had another look at the refill-api log and I can't see anything unusual for October 10. I have worked out what's being going on. Usually, Wikipedia editors run reFill over a page, but there's also a little-used feature to paste in raw wiki markup by going to https://refill.toolforge.org/ng/ and clicking "Use Custom Wikicode". Starting at 0400, someone repeatedly pasted in, or (far more likely) used the API to submit, variations on the following:

Please see my update to T378686.

Do these definitely resolve to refill?

I couldn't access the https://turnilo.wikimedia.org link - I get "Service access denied due to missing privileges." What access do we need?

I've added @AManWithNoPlan as a subscriber. They are one of the maintainers of CitationBot.

I have completed my review of usage of the reFill service during the dates in question and have documented my findings in T378686. To summarise, there is no indication that reFill is being misused or generated this traffic.

There is certainly nothing in the service log that indicates that reFill was called 30,000 times in any 12 hours.

Looking at English Wikipedia, there are over 5,000 articles that reference pro-football-reference.com and that were edited between 19 and 31 October 2024. It seems to me to be not beyond the imagination that a website referenced so many times in so many pages could be the subject of a great many citoid searches.... but not 30,000 times in 12 hours.

Note that the uwsgi.log records the Wikipedia page, and language code for the version of Wikipedia, that the user has requested citoid to expand references for, rather than the URL of those references, so the log file makes no reference to www.pro-football-reference.com.

While it is possible for someone to call the reFill API directly, and therefore to misuse the API to bombard a website with requests, that would be reflected in the uwsgi.log file for the service. The log file is huge, having not been trimmed since 2019, and contains millions of rows. I have looked at the log entries for the period in question and seemingly all the requests are from internal IPs, suggesting that those requests have been generated through the front end, rather than by direct calls to the API. The log file indicates the Wikipedia page that the user has requested citoid to expand references for, and I can't see any pattern that would suggest that someone is automating calls to the front end to generate thousands of requests to a page used in a reference in a particular article. The log implies normal usage by interactive users, using reFill manually to improve articles.

I claimed the wrong task - this is the correct one

I probably should have claimed T378686 rather than this one.

In T369439#10239449, @Pppery wrote:

https://refill.toolforge.org/ still shouldn't be a 403. It should at the very least redirect to https://refill.toolforge.org/ng if you want to declare the old version dead

I'm closing this, as the old version of reFill is now considered dead. Long live the /ng version.

Appears to be working again now, but no idea why.

Are you able to take a look TNT?

running webservice restart made no difference

tools.refill-api@tools-bastion-13:~$ kubectl get pod refill-api-6c65bdbf7f-9zd7s --output=yaml

I have set to low priority as there's a viable workaround.

Hoping Sammy might take a look

Copy of #wikimedia-cloud IRC chat:

Backend appears to have been working throughout. Seems to be a frontend issue. Restarted webservice and https://refill.toolforge.org/ now giving a 403. Workaround is to use https://refill.toolforge.org/ng/ instead.

Working fine right now.

It is working again now - no idea why.

Assigning to Sammy as not sure what else to try.

@Keith_D it looks fine to me - I know the WMF were doing some maintenance earlier which might have caused a blip. Please confirm it is now working so we can close.

@dcaro it has started working now! Not sure why.

Used kubectl scale to terminate all pods. No change - still 'Waiting for an available worker.'

tools.refill-api@tools-sgebastion-10:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
refill-api-79cc6f87b4-92p9k 1/1 Running 0 23m
refill-api-scheduler-79fcfc546d-jn98n 1/1 Running 0 19d
refill-api-worker-f7d869fdf-bnhcq 1/1 Running 5 (4d2h ago) 19d

Following steps that @TheresNoTime left in T359159 to identify problem with pod.

Tried restarting refill-api web service, but no change.

Brilliant, @TheresNoTime, thank you!

Any ideas Sammy?

Seems to already be in the Citoid backlog as T157152.

@Maxim_Masiutin I have re-tagged this as an issue with Citoid. reFill is just a wrapper for Citoid, which it uses to actually generate the expanded reference. Citoid is maintained by the WMF.

@DarklitShadow I cannot reproduce the problem. Is it still happening for you?