| LucasWerkmeister | |
| Sep 2 2025, 7:10 PM |
| F65949756: T403519.webm | |
| Sep 2 2025, 7:44 PM |
| F65949753: T403519.mp4 | |
| Sep 2 2025, 7:44 PM |
Several Toolforge tools using the mwapi library are suddenly failing to make edits. (It might not actually be related to the library – remains to be seen.) For some reason, they get errors about invalid CSRF tokens.
Affected tools include:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | JTweed-WMF | T398814 WE5.1.1 Session storage protection | |||
| Resolved | JTweed-WMF | T400365 Support multiple session stores | |||
| Resolved | matmarex | T362324 Disable PHPSessionHandler in Wikimedia production | |||
| Resolved | matmarex | T403519 Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. |
Some QuickCategories logs (apparently truncated):
tools.quickcategories@tools-bastion-13:~$ toolforge jobs logs --last=1000 background-runner 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise APIError.from_doc(doc['error']) 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] mwapi.errors.APIError: badtoken: Invalid CSRF token. -- None 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise APIError.from_doc(doc['error']) 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] mwapi.errors.APIError: badtoken: Invalid CSRF token. -- None 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise APIError.from_doc(doc['error']) 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] mwapi.errors.APIError: badtoken: Invalid CSRF token. -- None 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise APIError.from_doc(doc['error']) 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] mwapi.errors.APIError: badtoken: Invalid CSRF token. -- None 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise APIError.from_doc(doc['error']) 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] mwapi.errors.APIError: badtoken: Invalid CSRF token. -- None 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise APIError.from_doc(doc['error']) 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] mwapi.errors.APIError: badtoken: Invalid CSRF token. -- None 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise APIError.from_doc(doc['error']) 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] mwapi.errors.APIError: badtoken: Invalid CSRF token. -- None 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise APIError.from_doc(doc['error']) 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] mwapi.errors.APIError: badtoken: Invalid CSRF token. -- None 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise APIError.from_doc(doc['error']) 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] mwapi.errors.APIError: badtoken: Invalid CSRF token. -- None 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise APIError.from_doc(doc['error']) 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] mwapi.errors.APIError: badtoken: Invalid CSRF token. -- None 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise APIError.from_doc(doc['error']) 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] mwapi.errors.APIError: badtoken: Invalid CSRF token. -- None 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] Traceback (most recent call last): 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/./background_runner.py", line 83, in <module> 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] command_finish = runner.run_command(command_pending) 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 236, in run_command 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise e 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 210, in run_command 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] response = self.session.post(**params) 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 341, in post 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self.request('POST', params=params, auth=auth, 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] query_continue=query_continue, files=files, 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] continuation=continuation) 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 171, in request 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self._request(method, params=normal_params, auth=auth, 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] files=files) 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^ 2025-09-02T18:14:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 127, in _request 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] Traceback (most recent call last): 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/./background_runner.py", line 83, in <module> 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] command_finish = runner.run_command(command_pending) 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 236, in run_command 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise e 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 210, in run_command 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] response = self.session.post(**params) 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 341, in post 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self.request('POST', params=params, auth=auth, 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] query_continue=query_continue, files=files, 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] continuation=continuation) 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 171, in request 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self._request(method, params=normal_params, auth=auth, 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] files=files) 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^ 2025-09-02T18:19:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 127, in _request 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] Traceback (most recent call last): 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/./background_runner.py", line 83, in <module> 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] command_finish = runner.run_command(command_pending) 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 236, in run_command 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise e 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 210, in run_command 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] response = self.session.post(**params) 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 341, in post 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self.request('POST', params=params, auth=auth, 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] query_continue=query_continue, files=files, 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] continuation=continuation) 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 171, in request 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self._request(method, params=normal_params, auth=auth, 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] files=files) 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^ 2025-09-02T18:24:46+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 127, in _request 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] Traceback (most recent call last): 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/./background_runner.py", line 83, in <module> 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] command_finish = runner.run_command(command_pending) 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 236, in run_command 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise e 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 210, in run_command 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] response = self.session.post(**params) 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 341, in post 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self.request('POST', params=params, auth=auth, 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] query_continue=query_continue, files=files, 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] continuation=continuation) 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 171, in request 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self._request(method, params=normal_params, auth=auth, 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] files=files) 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^ 2025-09-02T18:30:00+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 127, in _request 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] Traceback (most recent call last): 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/./background_runner.py", line 83, in <module> 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] command_finish = runner.run_command(command_pending) 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 236, in run_command 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise e 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 210, in run_command 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] response = self.session.post(**params) 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 341, in post 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self.request('POST', params=params, auth=auth, 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] query_continue=query_continue, files=files, 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] continuation=continuation) 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 171, in request 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self._request(method, params=normal_params, auth=auth, 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] files=files) 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^ 2025-09-02T18:35:12+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 127, in _request 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] Traceback (most recent call last): 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/./background_runner.py", line 83, in <module> 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] command_finish = runner.run_command(command_pending) 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 236, in run_command 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise e 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 210, in run_command 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] response = self.session.post(**params) 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 341, in post 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self.request('POST', params=params, auth=auth, 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] query_continue=query_continue, files=files, 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] continuation=continuation) 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 171, in request 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self._request(method, params=normal_params, auth=auth, 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] files=files) 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^ 2025-09-02T18:40:27+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 127, in _request 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] Traceback (most recent call last): 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/./background_runner.py", line 83, in <module> 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] command_finish = runner.run_command(command_pending) 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 236, in run_command 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise e 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 210, in run_command 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] response = self.session.post(**params) 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 341, in post 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self.request('POST', params=params, auth=auth, 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] query_continue=query_continue, files=files, 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] continuation=continuation) 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 171, in request 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self._request(method, params=normal_params, auth=auth, 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] files=files) 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^ 2025-09-02T18:45:41+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 127, in _request 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] Traceback (most recent call last): 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/./background_runner.py", line 83, in <module> 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] command_finish = runner.run_command(command_pending) 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 236, in run_command 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise e 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 210, in run_command 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] response = self.session.post(**params) 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 341, in post 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self.request('POST', params=params, auth=auth, 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] query_continue=query_continue, files=files, 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] continuation=continuation) 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 171, in request 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self._request(method, params=normal_params, auth=auth, 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] files=files) 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^ 2025-09-02T18:50:49+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 127, in _request 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] Traceback (most recent call last): 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/./background_runner.py", line 83, in <module> 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] command_finish = runner.run_command(command_pending) 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 236, in run_command 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise e 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 210, in run_command 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] response = self.session.post(**params) 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 341, in post 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self.request('POST', params=params, auth=auth, 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] query_continue=query_continue, files=files, 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] continuation=continuation) 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 171, in request 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self._request(method, params=normal_params, auth=auth, 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] files=files) 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^ 2025-09-02T18:55:55+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 127, in _request 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] Traceback (most recent call last): 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/./background_runner.py", line 83, in <module> 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] command_finish = runner.run_command(command_pending) 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 236, in run_command 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise e 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 210, in run_command 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] response = self.session.post(**params) 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 341, in post 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self.request('POST', params=params, auth=auth, 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] query_continue=query_continue, files=files, 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] continuation=continuation) 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 171, in request 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self._request(method, params=normal_params, auth=auth, 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] files=files) 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^ 2025-09-02T19:01:03+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 127, in _request 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] Traceback (most recent call last): 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/./background_runner.py", line 83, in <module> 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] command_finish = runner.run_command(command_pending) 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 236, in run_command 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] raise e 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/workspace/runner.py", line 210, in run_command 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] response = self.session.post(**params) 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 341, in post 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self.request('POST', params=params, auth=auth, 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] query_continue=query_continue, files=files, 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] continuation=continuation) 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 171, in request 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] return self._request(method, params=normal_params, auth=auth, 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] files=files) 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] ^^^^^^^^^^^^ 2025-09-02T19:06:18+00:00 [background-runner-6d4b45f48-h4xdd] [job] File "/layers/heroku_python/venv/lib/python3.13/site-packages/mwapi/session.py", line 127, in _request
And a traceback from Mishramilan at 18:01 UTC:
Traceback (most recent call last): File "/data/project/mishramilan/www/python/venv/lib/python3.11/site-packages/flask/app.py", line 1463, in wsgi_app response = self.full_dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/project/mishramilan/www/python/venv/lib/python3.11/site-packages/flask/app.py", line 872, in full_dispatch_request rv = self.handle_user_exception(e) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/project/mishramilan/www/python/venv/lib/python3.11/site-packages/flask/app.py", line 870, in full_dispatch_request rv = self.dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^ File "/data/project/mishramilan/www/python/venv/lib/python3.11/site-packages/flask/app.py", line 855, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "app.py", line 277, in index response = submit_change() ^^^^^^^^^^^^^^^ File "app.py", line 244, in submit_change add_extid(form_data) File "app.py", line 174, in add_extid session.post(**arguments) File "/data/project/mishramilan/www/python/venv/lib/python3.11/site-packages/mwapi/session.py", line 341, in post return self.request('POST', params=params, auth=auth, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/project/mishramilan/www/python/venv/lib/python3.11/site-packages/mwapi/session.py", line 171, in request return self._request(method, params=normal_params, auth=auth, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/project/mishramilan/www/python/venv/lib/python3.11/site-packages/mwapi/session.py", line 127, in _request raise APIError.from_doc(doc['error']) mwapi.errors.APIError: badtoken: Invalid CSRF token. -- See https://www.wikidata.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce>
I added some debug code to QuickCategories and I seem to be getting different CSRF tokens at different times:
Runner.__post__init query+tokens response: {'batchcomplete': '', 'query': {'tokens': {'csrftoken': '3e[redacted]5c68b74322+\\'}}}
Runner.run_command query+tokens+userinfo response: {'batchcomplete': '', 'query': {'tokens': {'csrftoken': 'c0[redacted]4868b74322+\\'}, 'userinfo': {'id': 37627, 'name': 'Lucas Werkmeister'}}}
Runner.run_command query+tokens+userinfo response 2: {'batchcomplete': '', 'query': {'tokens': {'csrftoken': 'c0[redacted]4868b74322+\\'}, 'userinfo': {'id': 37627, 'name': 'Lucas Werkmeister'}}}I redacted part of the CSRF token, but that’s a suspiciously long identical string at the end there. Here it is differently aligned so you can see it more easily:
Runner.__post__init query+tokens response: {'batchcomplete': '', 'query': {'tokens': {'csrftoken': '3e[redacted]5c68b74322+\\'}}}
Runner.run_command query+tokens+userinfo response: {'batchcomplete': '', 'query': {'tokens': {'csrftoken': 'c0[redacted]4868b74322+\\'}, 'userinfo': {'id': 37627, 'name': 'Lucas Werkmeister'}}}
Runner.run_command query+tokens+userinfo response 2: {'batchcomplete': '', 'query': {'tokens': {'csrftoken': 'c0[redacted]4868b74322+\\'}, 'userinfo': {'id': 37627, 'name': 'Lucas Werkmeister'}}}That’s four full identical bytes – quite unlikely if these are meant to be random.
Sorry, my debug code was accidentally printing the same API response twice. It actually looks more like this:
Runner.__post__init query+tokens response: {'batchcomplete': '', 'query': {'tokens': {'csrftoken': '05[redacted]5368b7442c+\\'}}}
Runner.run_command query+tokens+userinfo response 1: {'batchcomplete': '', 'query': {'tokens': {'csrftoken': 'c3[redacted]4368b7442c+\\'}, 'userinfo': {'id': 37627, 'name': 'Lucas Werkmeister'}}}
Runner.run_command query+tokens+userinfo response 2: {'batchcomplete': '', 'query': {'tokens': {'csrftoken': 'ba[redacted]d068b7442d+\\'}, 'userinfo': {'id': 37627, 'name': 'Lucas Werkmeister'}}}The last two lines are two immediately consecutive calls, on the same mwapi Session instance, giving me different CSRF tokens – but still with the last four bytes… almost matching because I just noticed one of them ends in d instead of c. (Ignoring the final +\ which is just there to test encoding handling.)
What’s going on here? o_O
Runner.__post__init query+tokens+userinfo response: {'batchcomplete': '', 'query': {'tokens': {'csrftoken': 'ee[redacted]af68b744da+\\'}, 'userinfo': {'id': 37627, 'name': 'Lucas Werkmeister'}}}
Runner.run_command query+tokens+userinfo response 1: {'batchcomplete': '', 'query': {'tokens': {'csrftoken': '19[redacted]a468b744da+\\'}, 'userinfo': {'id': 37627, 'name': 'Lucas Werkmeister'}}}
Runner.run_command query+tokens+userinfo response 2: {'batchcomplete': '', 'query': {'tokens': {'csrftoken': '05[redacted]9668b744da+\\'}, 'userinfo': {'id': 37627, 'name': 'Lucas Werkmeister'}}}Is that a timestamp at the end of the token? (0xda = 218 is somewhat later than 0x2d = 45. It could be in seconds.)
Okay, yeah, apparently tokens end with a hexadecimal timestamp. TIL. So that explains the similar string at the end; and beyond that, it suggests that these sessions, for whatever reason, are not getting their token persisted at all, so the API is regenerating it on each call.
A similar problem (the same one?) was reported as https://github.com/maxlath/wikibase-cli/issues/192 .
Pinging some people from two recent-ish deploys that look related to authentication in some way, disable $wgPHPSessionHandling on group1 and SUL3: Use metawiki as central wiki: @matmarex @Hokwelum @DAlangi_WMF @Tgr – any chance this could be related to T362324 or T402527?
It happens even in the API sandbox (i.e. in a cookie session rather than an OAuth session): every request is giving me a new CSRF token.
These are supposed to be stable, right?
But on the other hand, I’m still able to edit Wikidata manually, despite getting a different mw.user.tokens.get('csrfToken') on every page load. I’m so confused right now.
Some helpful comments from @Krinkle in #mediawiki-core:
Lucas_WMDE: I suspect you may be mixing the hmac-encoded timestamp for the underlying cookie and secret in session data
It's normal that each time you ask for a token it looks different. view-source is sufficient to observe that in the user.tokens.set call.
It's not a new cookie, or new session, or new secret.
csrf token = hmac(secret, rand, timestamp)
validate = decode() && time-timestamp< maxage
The minted token isn't stored anywhere (not a nonce).
And okay, I can now see that the timestamp is not just appended to the token but also goes into the HMAC:
protected function toStringAtTimestamp( $timestamp ) { return hash_hmac( 'md5', $timestamp . $this->salt, $this->secret, false ) . dechex( $timestamp ) . self::SUFFIX; }
But still – if the timestamp is exactly the same, as it was in e.g. T403519#11140935, then we should expect the whole token to be the same, right?
FWIW, I just tested QuickCategories on English Wikipedia and it still works there: https://en.wikipedia.org/w/index.php?oldid=1251769135&diff=1309218088
So this might indeed be a group1/group2 difference like that $wgPHPSessionHandling change…
Change #1184161 had a related patch set uploaded (by Lucas Werkmeister; author: Lucas Werkmeister):
[operations/mediawiki-config@master] Revert "Set $wgPHPSessionHandling to 'disable' on group1 wikis"
We've talked on IRC and decided to try reverting that config change, since it's safe to do, and see if it helps.
I can't speak to the wider issue, but regarding the different tokens you see via the API Sandbox (and on articles via view-source), that's perfectly normal and not new!
CSRF tokens by their nature appear different each time you look at them. So you may have the same session, the same session data, the same secret, and the same session cookies; but get a slightly different token each time.
This allows applications like MediaWiki to expire CSRF tokens without actually storing a lot of session data, or churning cookies.
It works by defining the public token as an encrypted message (HMAC if I recall correctly), which includes a timestamp. This means the message appears to be constantly changing but they're based on the same underlying secret. These messages are stateless and don't need to be stored to work, only the secret.
Change #1184161 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "Set $wgPHPSessionHandling to 'disable' on group1 wikis"
Mentioned in SAL (#wikimedia-operations) [2025-09-02T21:01:36Z] <lucaswerkmeister-wmde@deploy1003> Started scap sync-world: Backport for [[gerrit:1184161|Revert "Set $wgPHPSessionHandling to 'disable' on group1 wikis" (T362324 T403519)]]
Mentioned in SAL (#wikimedia-operations) [2025-09-02T21:06:20Z] <lucaswerkmeister-wmde@deploy1003> lucaswerkmeister-wmde, lucaswerkmeister: Backport for [[gerrit:1184161|Revert "Set $wgPHPSessionHandling to 'disable' on group1 wikis" (T362324 T403519)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
Mentioned in SAL (#wikimedia-operations) [2025-09-02T21:15:03Z] <lucaswerkmeister-wmde@deploy1003> Finished scap sync-world: Backport for [[gerrit:1184161|Revert "Set $wgPHPSessionHandling to 'disable' on group1 wikis" (T362324 T403519)]] (duration: 13m 27s)
With the above deployment, QuickCategories seemingly started working again. (Though I still get the CSRF error locally, which is very confusing.) Yay…?
That would be because my local test targets testwiki (group0), where the revert didn’t apply. I guess we also want to revert Set $wgPHPSessionHandling to 'disable' on group0 wikis to unbreak tools on those wikis.
Change #1184166 had a related patch set uploaded (by Lucas Werkmeister; author: Lucas Werkmeister):
[operations/mediawiki-config@master] Revert "Set $wgPHPSessionHandling to 'disable' on group0 wikis"
Once this is deployed, it should still be possible to reproduce the bug on Test Wikipedia (testwiki). You can use QuickCategories to create a new batch for that wiki, with commands like
User:Lucas Werkmeister/sandbox|+Category:Testing T403519
(or another test page of your choice), and optionally a title (edit summary) like testing [[phabricator:T403519]]. If you do, please run the resulting batch using the “Run these commands” button. Do not use the “Run whole batch in background” button (the more prominent one), because that would break the background runner for all users of the tool. If the bug is present, you should see some sort of error (not sure what it’ll look like – I’ve only seen it on localhost where Flask’s debug mode changes the appearance). If the bug is not present, an edit will be made and you’ll see it on the batch page.
Change #1184166 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "Set $wgPHPSessionHandling to 'disable' on group0 wikis"
Mentioned in SAL (#wikimedia-operations) [2025-09-02T21:55:54Z] <lucaswerkmeister-wmde@deploy1003> Started scap sync-world: Backport for [[gerrit:1184166|Revert "Set $wgPHPSessionHandling to 'disable' on group0 wikis" (T362324 T403519)]]
Mentioned in SAL (#wikimedia-operations) [2025-09-02T22:00:16Z] <lucaswerkmeister-wmde@deploy1003> lucaswerkmeister, lucaswerkmeister-wmde: Backport for [[gerrit:1184166|Revert "Set $wgPHPSessionHandling to 'disable' on group0 wikis" (T362324 T403519)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
It looks like it – this is what the debug code looked like with the revert applied:
Runner.__post__init query+tokens+userinfo response: {'batchcomplete': '', 'servedby': 'mw-debug.eqiad.pinkunicorn-6cdd68c648-cbxrk', 'query': {'tokens': {'csrftoken': 'b6[redacted]c968b768f4+\\'}, 'userinfo': {'id': 3127, 'name': 'Lucas Werkmeister'}}}
Runner.run_command query+tokens+userinfo response 1: {'batchcomplete': '', 'servedby': 'mw-debug.eqiad.pinkunicorn-6cdd68c648-cbxrk', 'query': {'tokens': {'csrftoken': '4c[redacted]9a68b768f5+\\'}, 'userinfo': {'id': 3127, 'name': 'Lucas Werkmeister'}}}
Runner.run_command query+tokens+userinfo response 2: {'batchcomplete': '', 'servedby': 'mw-debug.eqiad.pinkunicorn-6cdd68c648-cbxrk', 'query': {'tokens': {'csrftoken': '4c[redacted]9a68b768f5+\\'}, 'userinfo': {'id': 3127, 'name': 'Lucas Werkmeister'}}}The first two were a second apart and got different tokens, the last two ran in the same second and got the same token. (You’ll have to take my word for it that the redacted part is also identical ;P)
Mentioned in SAL (#wikimedia-operations) [2025-09-02T22:05:57Z] <lucaswerkmeister-wmde@deploy1003> Finished scap sync-world: Backport for [[gerrit:1184166|Revert "Set $wgPHPSessionHandling to 'disable' on group0 wikis" (T362324 T403519)]] (duration: 10m 03s)
Should be resolved on all wikis except testwiki at the moment. Leaving the task open for further investigation by people who know the auth stack better than I do. (So far it seems like mwapi is probably not relevant – it’s just a popular library for Python tools – but OAuth might be, perhaps specifically OAuth 1.)
Would be nice to get T126257: The API should not require CSRF tokens for an OAuth request done.
Are these all about edit tokens? ApiQueryTokens::getToken() persists the session for other kinds of tokens but not for edit (csrf) tokens, which seems incorrect when the edit token isn't anonymous.
I haven't verified it but this is my guess of what's happening:
Or rather, it does that when the session data has changed (as it's trying to catch session data changes on the PHP side, but probably can't actually differentiate them from proper SessionManager-based data changes).
As far as I’m aware yes, though I didn’t test any other token types (e.g. rollback) while the bug was live on group0+1.
I set up the OAuth Hello World! app (https://oauth-hello-world.toolforge.org/) locally, pointed it to my local wiki, and I could reproduce this problem when using the "Post to your talk page" function.
I enabled debug logging for the 'session' and 'session-sampled' channels, here are the logs for the requests that the OAuth Hello World! app makes – failing to edit with PHPSessionHandling: 'disable', and succeeding with PHPSessionHandling: 'enable':
Diff:
diff --git a/disable.txt b/enable.txt index 87c1fc9..1be38f4 100644 --- a/disable.txt +++ b/enable.txt @@ -3,30 +3,21 @@ session.DEBUG: OAuth request for consumer e0a2ef4f94c61f4c216adf498493fdb6 by us session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" is unsaved, marking dirty in constructor {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t"} session.DEBUG: SessionBackend "require/require_once/MediaWiki\Session\Session->renew/MediaWiki\Session\SessionBackend->renew" metadata dirty for renew(): require/require_once/MediaWiki\Session\Session->renew/MediaWiki\Session\SessionBackend->renew {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","callers":"require/require_once/MediaWiki\\Session\\Session->renew/MediaWiki\\Session\\SessionBackend->renew"} session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" save: dataDirty=1 metaDirty=1 forcePersist=0 {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","dataDirty":1,"metaDirty":1,"forcePersist":0} +session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" force-persist due to persist() {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t"} +session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" save: dataDirty=0 metaDirty=1 forcePersist=1 {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","dataDirty":0,"metaDirty":1,"forcePersist":1} +session-sampled.INFO: Session store: write for manual {"remember":true,"metaDirty":true,"dataDirty":false,"forcePersist":true,"action":"write","reason":"manual","id":"moagfq0ju8k3qmeasut3608e6s4eki4t","provider":"MediaWiki\\Extension\\OAuth\\SessionProvider","user":"Matma Rex","clientip":"10.0.2.2","userAgent":"oauth-hello-world"} session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" data dirty due to dirty(): MediaWiki\User\User->getName/MediaWiki\User\User->load/MediaWiki\User\User->loadFromSession/MediaWiki\Session\Session->set/MediaWiki\Session\SessionBackend->dirty {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","callers":"MediaWiki\\User\\User->getName/MediaWiki\\User\\User->load/MediaWiki\\User\\User->loadFromSession/MediaWiki\\Session\\Session->set/MediaWiki\\Session\\SessionBackend->dirty"} session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" data dirty due to dirty(): MediaWiki\User\User->getName/MediaWiki\User\User->load/MediaWiki\User\User->loadFromSession/MediaWiki\Session\Session->set/MediaWiki\Session\SessionBackend->dirty {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","callers":"MediaWiki\\User\\User->getName/MediaWiki\\User\\User->load/MediaWiki\\User\\User->loadFromSession/MediaWiki\\Session\\Session->set/MediaWiki\\Session\\SessionBackend->dirty"} session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" data dirty due to dirty(): MediaWiki\User\User->getName/MediaWiki\User\User->load/MediaWiki\User\User->loadFromSession/MediaWiki\Session\Session->set/MediaWiki\Session\SessionBackend->dirty {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","callers":"MediaWiki\\User\\User->getName/MediaWiki\\User\\User->load/MediaWiki\\User\\User->loadFromSession/MediaWiki\\Session\\Session->set/MediaWiki\\Session\\SessionBackend->dirty"} session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" save: dataDirty=1 metaDirty=0 forcePersist=0 {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","dataDirty":1,"metaDirty":0,"forcePersist":0} +session-sampled.INFO: Session store: write for other {"remember":true,"metaDirty":false,"dataDirty":true,"forcePersist":false,"action":"write","reason":"other","id":"moagfq0ju8k3qmeasut3608e6s4eki4t","provider":"MediaWiki\\Extension\\OAuth\\SessionProvider","user":"Matma Rex","clientip":"10.0.2.2","userAgent":"oauth-hello-world"} session.DEBUG: Saving all sessions on shutdown session.DEBUG: SessionManager using store Wikimedia\ObjectCache\CachedBagOStuff session.DEBUG: OAuth request for consumer e0a2ef4f94c61f4c216adf498493fdb6 by user Matma Rex {"clientip":"10.0.2.2","user":"Matma Rex","consumer":"e0a2ef4f94c61f4c216adf498493fdb6","result":"success"} -session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" is unsaved, marking dirty in constructor {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t"} -session.DEBUG: SessionBackend "require/require_once/MediaWiki\Session\Session->renew/MediaWiki\Session\SessionBackend->renew" metadata dirty for renew(): require/require_once/MediaWiki\Session\Session->renew/MediaWiki\Session\SessionBackend->renew {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","callers":"require/require_once/MediaWiki\\Session\\Session->renew/MediaWiki\\Session\\SessionBackend->renew"} -session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" save: dataDirty=1 metaDirty=1 forcePersist=0 {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","dataDirty":1,"metaDirty":1,"forcePersist":0} -session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" data dirty due to dirty(): MediaWiki\User\User->getName/MediaWiki\User\User->load/MediaWiki\User\User->loadFromSession/MediaWiki\Session\Session->set/MediaWiki\Session\SessionBackend->dirty {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","callers":"MediaWiki\\User\\User->getName/MediaWiki\\User\\User->load/MediaWiki\\User\\User->loadFromSession/MediaWiki\\Session\\Session->set/MediaWiki\\Session\\SessionBackend->dirty"} -session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" data dirty due to dirty(): MediaWiki\User\User->getName/MediaWiki\User\User->load/MediaWiki\User\User->loadFromSession/MediaWiki\Session\Session->set/MediaWiki\Session\SessionBackend->dirty {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","callers":"MediaWiki\\User\\User->getName/MediaWiki\\User\\User->load/MediaWiki\\User\\User->loadFromSession/MediaWiki\\Session\\Session->set/MediaWiki\\Session\\SessionBackend->dirty"} -session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" data dirty due to dirty(): MediaWiki\User\User->getName/MediaWiki\User\User->load/MediaWiki\User\User->loadFromSession/MediaWiki\Session\Session->set/MediaWiki\Session\SessionBackend->dirty {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","callers":"MediaWiki\\User\\User->getName/MediaWiki\\User\\User->load/MediaWiki\\User\\User->loadFromSession/MediaWiki\\Session\\Session->set/MediaWiki\\Session\\SessionBackend->dirty"} session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" data dirty due to dirty(): MediaWiki\Api\ApiQueryTokens::getToken/MediaWiki\User\User->getEditTokenObject/MediaWiki\Session\Session->getToken/MediaWiki\Session\Session->set/MediaWiki\Session\SessionBackend->dirty {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","callers":"MediaWiki\\Api\\ApiQueryTokens::getToken/MediaWiki\\User\\User->getEditTokenObject/MediaWiki\\Session\\Session->getToken/MediaWiki\\Session\\Session->set/MediaWiki\\Session\\SessionBackend->dirty","context.api_module_name":"query","context.api_client_useragent":"oauth-hello-world","context.api_query_module_name":"tokens"} session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" save: dataDirty=1 metaDirty=0 forcePersist=0 {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","dataDirty":1,"metaDirty":0,"forcePersist":0} +session-sampled.INFO: Session store: write for other {"remember":true,"metaDirty":false,"dataDirty":true,"forcePersist":false,"action":"write","reason":"other","id":"moagfq0ju8k3qmeasut3608e6s4eki4t","provider":"MediaWiki\\Extension\\OAuth\\SessionProvider","user":"Matma Rex","clientip":"10.0.2.2","userAgent":"oauth-hello-world"} session.DEBUG: Saving all sessions on shutdown session.DEBUG: SessionManager using store Wikimedia\ObjectCache\CachedBagOStuff session.DEBUG: OAuth request for consumer e0a2ef4f94c61f4c216adf498493fdb6 by user Matma Rex {"clientip":"10.0.2.2","user":"Matma Rex","consumer":"e0a2ef4f94c61f4c216adf498493fdb6","result":"success"} -session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" is unsaved, marking dirty in constructor {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t"} -session.DEBUG: SessionBackend "require/require_once/MediaWiki\Session\Session->renew/MediaWiki\Session\SessionBackend->renew" metadata dirty for renew(): require/require_once/MediaWiki\Session\Session->renew/MediaWiki\Session\SessionBackend->renew {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","callers":"require/require_once/MediaWiki\\Session\\Session->renew/MediaWiki\\Session\\SessionBackend->renew"} -session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" save: dataDirty=1 metaDirty=1 forcePersist=0 {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","dataDirty":1,"metaDirty":1,"forcePersist":0} -session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" data dirty due to dirty(): MediaWiki\User\User->getId/MediaWiki\User\User->load/MediaWiki\User\User->loadFromSession/MediaWiki\Session\Session->set/MediaWiki\Session\SessionBackend->dirty {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","callers":"MediaWiki\\User\\User->getId/MediaWiki\\User\\User->load/MediaWiki\\User\\User->loadFromSession/MediaWiki\\Session\\Session->set/MediaWiki\\Session\\SessionBackend->dirty"} -session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" data dirty due to dirty(): MediaWiki\User\User->getId/MediaWiki\User\User->load/MediaWiki\User\User->loadFromSession/MediaWiki\Session\Session->set/MediaWiki\Session\SessionBackend->dirty {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","callers":"MediaWiki\\User\\User->getId/MediaWiki\\User\\User->load/MediaWiki\\User\\User->loadFromSession/MediaWiki\\Session\\Session->set/MediaWiki\\Session\\SessionBackend->dirty"} -session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" data dirty due to dirty(): MediaWiki\User\User->getId/MediaWiki\User\User->load/MediaWiki\User\User->loadFromSession/MediaWiki\Session\Session->set/MediaWiki\Session\SessionBackend->dirty {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","callers":"MediaWiki\\User\\User->getId/MediaWiki\\User\\User->load/MediaWiki\\User\\User->loadFromSession/MediaWiki\\Session\\Session->set/MediaWiki\\Session\\SessionBackend->dirty"} -session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" data dirty due to dirty(): MediaWiki\Api\ApiQueryTokens::getToken/MediaWiki\User\User->getEditTokenObject/MediaWiki\Session\Session->getToken/MediaWiki\Session\Session->set/MediaWiki\Session\SessionBackend->dirty {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","callers":"MediaWiki\\Api\\ApiQueryTokens::getToken/MediaWiki\\User\\User->getEditTokenObject/MediaWiki\\Session\\Session->getToken/MediaWiki\\Session\\Session->set/MediaWiki\\Session\\SessionBackend->dirty"} -session.DEBUG: SessionBackend "moagfq0ju8k3qmeasut3608e6s4eki4t" save: dataDirty=1 metaDirty=0 forcePersist=0 {"session":"moagfq0ju8k3qmeasut3608e6s4eki4t","dataDirty":1,"metaDirty":0,"forcePersist":0} session.DEBUG: Saving all sessions on shutdown
It looks like @Tgr's guess was correct. In the failing case the session data is never written.
Change #1186595 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):
[mediawiki/core@master] ApiQueryTokens: Persist any new token, instead of depending on the type
Change #1186595 merged by jenkins-bot:
[mediawiki/core@master] ApiQueryTokens: Persist any new token, instead of depending on the type
Change #1187062 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):
[mediawiki/core@wmf/1.45.0-wmf.17] ApiQueryTokens: Persist any new token, instead of depending on the type
Change #1187063 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):
[mediawiki/core@wmf/1.45.0-wmf.18] ApiQueryTokens: Persist any new token, instead of depending on the type
Change #1187062 merged by jenkins-bot:
[mediawiki/core@wmf/1.45.0-wmf.17] ApiQueryTokens: Persist any new token, instead of depending on the type
Change #1187063 merged by jenkins-bot:
[mediawiki/core@wmf/1.45.0-wmf.18] ApiQueryTokens: Persist any new token, instead of depending on the type
Mentioned in SAL (#wikimedia-operations) [2025-09-10T20:27:38Z] <reedy@deploy1003> Started scap sync-world: Backport for [[gerrit:1187062|ApiQueryTokens: Persist any new token, instead of depending on the type (T403519)]], [[gerrit:1187063|ApiQueryTokens: Persist any new token, instead of depending on the type (T403519)]], [[gerrit:1187066|Revert^2 "Set $wgPHPSessionHandling to 'disable' on group1 wikis" (T362324)]], [[gerrit:1187065|Revert^2 "Set $wgPHPSessionHandling to 'disable'
Mentioned in SAL (#wikimedia-operations) [2025-09-10T20:32:07Z] <reedy@deploy1003> reedy, matmarex: Backport for [[gerrit:1187062|ApiQueryTokens: Persist any new token, instead of depending on the type (T403519)]], [[gerrit:1187063|ApiQueryTokens: Persist any new token, instead of depending on the type (T403519)]], [[gerrit:1187066|Revert^2 "Set $wgPHPSessionHandling to 'disable' on group1 wikis" (T362324)]], [[gerrit:1187065|Revert^2 "Set $wgPHPSessionHandling to 'disable' on grou
Mentioned in SAL (#wikimedia-operations) [2025-09-10T20:41:47Z] <reedy@deploy1003> Finished scap sync-world: Backport for [[gerrit:1187062|ApiQueryTokens: Persist any new token, instead of depending on the type (T403519)]], [[gerrit:1187063|ApiQueryTokens: Persist any new token, instead of depending on the type (T403519)]], [[gerrit:1187066|Revert^2 "Set $wgPHPSessionHandling to 'disable' on group1 wikis" (T362324)]], [[gerrit:1187065|Revert^2 "Set $wgPHPSessionHandling to 'disable
@LucasWerkmeister and I verified that the tools work correctly now, after re-deploying the config.