pywikibot Site.upload repeats "Invalid CSRF token" and "Sleeping for" infinitely without exiting
Open, HighPublicBUG REPORT
Actions

Assigned To

None

Authored By

	Hoi
	Aug 16 2023, 9:27 AM

Project Tags

Referenced Files

None

Subscribers

Description

Steps to replicate the issue (include links if applicable):

Not sure how to reproduce. I encounter the issue when uploading files using pywikibot. Meanwhile, there are two other upload tasks running smoothly with the same codebase on the same VPS.

Notably, I toggled on asynchronous=True. The following code (simplified) is used:

site.upload(
    source_filename=filename,
    filepage=FilePage(site, pagename),
    text=text,
    comment=comment,
    asynchronous=size > 256 * 1024 * 1024,
    chunk_size=32 * 1024 * 1024,
    ignore_warnings=["was-deleted"],
)

What happens?:
pywikibot keeps logging the following lines without throwing an error or returning:

WARNING: API error badtoken: Invalid CSRF token.
Sleeping for 9.9 seconds, 2023-08-16 09:14:44

What should have happened instead?:
Either:
a. It refreshes the CSRF token automatically and retry.
b. It throws an exceptions.
c. It returns False to indicate a failure.

Software version (skip for WMF-hosted wikis like Wikipedia):
pywikibot/8.3.0
Wikimedia Commons

Other information (browser name/version, screenshots, etc.):

Logs:

INFO:__main__:PDF for REDACTED_A.pdf (298436465 B)           
INFO:__main__:Uploading File:REDACTED_A.pdf                                                                                                                   
Sleeping for 9.9 seconds, 2023-08-16 08:22:32                                                                                                                                                 
Sleeping for 10.0 seconds, 2023-08-16 08:22:42                                                                                                                                                
Sleeping for 9.9 seconds, 2023-08-16 08:22:52                                                                                                                                                 
Sleeping for 9.8 seconds, 2023-08-16 08:23:03                                                                                                                                                 
Sleeping for 10.0 seconds, 2023-08-16 08:23:12                                                                                                                                                
INFO:__main__:PDF for REDACTED_B.pdf (303047029 B)                                                                                              
INFO:__main__:Uploading REDACTED_B.pdf                                                                                                                 
Sleeping for 9.9 seconds, 2023-08-16 08:30:34                                                  
Sleeping for 9.9 seconds, 2023-08-16 08:30:44                                                                                                                                                 
Sleeping for 9.9 seconds, 2023-08-16 08:30:54                                                                                                                                                 
Sleeping for 9.9 seconds, 2023-08-16 08:31:04                                                                                                                                                 
Sleeping for 9.7 seconds, 2023-08-16 08:31:14                                                                                                                                                 
ERROR: User assertion failed. Forcing re-login.                                                
Logging in to commons:commons as REDACTED_ACCOUNT_NAME                                                                                                                                             
WARNING: API warning (main): Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/> for notice of A
PI deprecations and breaking changes. Use [[Special:ApiFeatureUsage]] to see usage of deprecated features by your application.                                                                
WARNING: API warning (login): Fetching a token via "action=login" is deprecated. Use "action=query&meta=tokens&type=login" instead.                                                           
ERROR: Received incorrect login token. Forcing re-login.                                                                                                                                      
Sleeping for 9.1 seconds, 2023-08-16 08:31:25                                                                                                                                                 
WARNING: API error badtoken: Invalid CSRF token.                                               
Sleeping for 9.9 seconds, 2023-08-16 08:31:34                 
...
WARNING: API error badtoken: Invalid CSRF token.
Sleeping for 9.8 seconds, 2023-08-16 09:26:54
WARNING: API error badtoken: Invalid CSRF token.
Sleeping for 9.9 seconds, 2023-08-16 09:27:04
WARNING: API error badtoken: Invalid CSRF token.
Sleeping for 9.9 seconds, 2023-08-16 09:27:14
...
WARNING: API error badtoken: Invalid CSRF token.
Sleeping for 9.9 seconds, 2023-08-16 10:21:04
WARNING: API error badtoken: Invalid CSRF token.
Sleeping for 9.9 seconds, 2023-08-16 10:21:14
WARNING: API error badtoken: Invalid CSRF token.
Sleeping for 9.9 seconds, 2023-08-16 10:21:24
...
WARNING: API error badtoken: Invalid CSRF token.
Sleeping for 9.9 seconds, 2023-08-16 14:35:45
...

Event Timeline

Hoi created this task.Aug 16 2023, 9:27 AM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptAug 16 2023, 9:28 AM

Hoi updated the task description. (Show Details)Aug 16 2023, 10:21 AM

Xqt triaged this task as High priority.Aug 16 2023, 12:20 PM

Hoi updated the task description. (Show Details)Aug 16 2023, 2:37 PM

@Hoi: Do you have "@" either in the bot account name or in the account's password?

@Xqt Yes.

user-config.py:

mylang = 'commons'
family = 'commons'
usernames['commons']['commons'] = 'SomeBot'

password_file = "user-password.py"

console_encoding = 'utf-8'

user-password.py:

("SomeBot", BotPassword('appname', 'password'))

In the last a few days, I experienced infinite sleep several times. No invalid CSRF token, just Sleeping for ... kept repeating for at least several hours until I terminated it.

In T344327#9819408, @Hoi wrote:

In the last a few days, I experienced infinite sleep several times. No invalid CSRF token, just Sleeping for ... kept repeating for at least several hours until I terminated it.

Was ist for 15, 30, 45, 60, 120, 180, 240, 300 seconds or sth else?

In T344327#9819572, @Xqt wrote:

In T344327#9819408, @Hoi wrote:

In the last a few days, I experienced infinite sleep several times. No invalid CSRF token, just Sleeping for ... kept repeating for at least several hours until I terminated it.

Was ist for 15, 30, 45, 60, 120, 180, 240, 300 seconds or sth else?

No. It was Sleeping for 9.9 seconds....

My upload script just downloads a file from the upstream (with my custom proxy) and uploads it to Commons directly one by one.

Here is the recent log:
https://hoiscript.toolforge.org/temp/upload-zjlib.err.txt
(search the keyword Last processed: backwards)

And the code:
https://hoiscript.toolforge.org/temp/upload1.py.txt

pywikibot Site.upload repeats "Invalid CSRF token" and "Sleeping for" infinitely without exitingOpen, HighPublicBUG REPORTActions

Description

Event Timeline

pywikibot Site.upload repeats "Invalid CSRF token" and "Sleeping for" infinitely without exiting
Open, HighPublicBUG REPORT
Actions