Searched For: ERROR:PleasereportthisissuetotheToolforgeadmins
```lang=shell-session
tools.libtell@tools-bastion-12:~$ toolforge components config create config.y …
ScannerError: while scanning for the next token
found character '@' th … "config.yaml", line 8, column 17
Please report this issue to the Toolforge admins if it persists: https://w.wiki/6Z … alid YAML file should not trigger the "Please report this issue to the Toolforge admins if it persists" line.
**Description**:
When navigating to the Special:CommunityConfiguration/Mentorship page and attempting to save the configuration form, an error message is displayed with an unparsed link. The expected behavior is if the form has errors, any links in error messages to be correctly parsed.
**Steps to Reproduce**:
1. Ensure your language is set to `es`
1. Go to Special:CommunityConfiguration/Me … ogged out or not have permissions to save a configuration form for …
This will allow us to get rid of a bunch of code that tries to handle all the different versions the job spec has been through, and he … ffected tools:
{P84274}
=== How to recreate your jobs ===
```
me@mylaptop$ ssh login.toolforge.org
<myuser>@tools-bastion-15: … ools.<mytool>@tools-bastion-15:~$ toolforge jobs dump recreating_jobs.yaml
t … correctly configured
## Now in the order you want
tools.<mytool>@tools-bastion-15:~$ toolforge jobs delete <myjob>
tools.<mytool>@tools-bastion-15:~$ …
tools.multichill@tools-bastion-12:~$ toolforge jobs load jobs.yml --job coord-fr … ng job 'coord-from-exif-mysql'...
ERROR: TjfCliError: Unknown error (403 Client Error: Forbidden for url: https://k8s.t … l-multichill/cronjobs?dryRun=All)
ERROR: Please report this issue to the Toolforge admins if it persists: https://w.wiki/6Zuu
**Steps to replicate the issue** (include links if applicable):
* Go to: https://author-disambiguator.toolforge.org/work_item_oauth.php
* If it w … dozen times until it fails
Note - this is an php app running on kubernet … ambiguator etc.
**What happens?**:
Fatal error: Uncaught mysqli_sql_exception: p … at should have happened instead?**:
You should have seen the default page for the application (after OAuth login)
… r WMF-hosted wikis like Wikipedia)…
Earwig's Copyvio Detector was not working for the last few days. When an article name is pasted on the "Page title" field, the return message is "An error occurred while using the search engine (Google Error: HTTP Error 403: Forbidden). Note: there is a daily limit on the number of search queries the tool is allowed to make. You may repeat the check without using the search engine." See here [https:/ … use_links=1&turnitin=0]. Not sure the problem is on Wikimedia or search … site. Kindly assist and thank you.…
should just exit, but now it shows:
```
local.tf-test2@toolslocal:~$ toolforge jobs logs testlogs1 | head
2026-0 … ] Tue Jul 28 02:52:50 PM UTC 2026
ERROR: An internal error occurred while executing this command.
Traceback (most recent call last):
File "/usr/lib/python3/dist-pac … PipeError: [Errno 32] Broken pipe
ERROR: Please report this issue to the Toolforge admins if it persists: https://w.wiki/6Zuu
```
**Steps to replicate the issue** (include links if applicable):
* Be in a tool with old build se … uilds
* Get an old build ID with `toolforge build list`
* Try to get its logs with `toolforge build logs`
**What happens?**:
```lang=shell-session,lines=1
tools.wdactle@tools-bastion-15:~$ toolforge build list
build_id … 30Z https://gitlab.wikimedia.org/toolforge-repos/wdactle N/A N/A … 31Z https://gitlab.wikimedia.org/toolforge…
**Steps to replicate the issue** (include links if applicable):
* Build spacemedia tool using Toolforge build service (see build.sh below)
* Wait for build to complete
**What happens?**:
* toolforge logs -f reports an error:
```
2024-05-08T12:24:31+00:00 [ … [build-ptcx2] [INFO] Recompiling the module because of changed source … les with javac [debug release 17] to target/classes
ERROR: An internal error occured while executing this command.
Traceback (most recent call last)…
```lang=shell-session
$ toolforge envvars create GITLAB_EVENTS_URL … cal:8000/sse/
EnvvarsClientError: Error contacting k8s cluster.
Please report this issue to the Toolforge admins if it persists: https://w.wiki/6Z … vices.nodeports 0 0
```
If you follow job logs with `toolforge jobs logs -f`, you get a read tim … t after 30 seconds of inactivity. This seems overly short. Especially considering the fact that job logs are deleted about a minute after the job finishes -- it's not like I c … nd see what happened.
```
$ date;toolforge jobs logs -l10 -f pano-2789e5c30c … i 02 Feb 2024 01:08:39 PM UTC
...
ERROR: An internal error occured while executing this command.
Traceback (most recent call last)…
The initial build out of the new Toolforge jobs framework has been deployed.
We move into the BETA phase, where we expect to:
* collect feedback from beta tes … ers
* identify & correct flaws in the design
* identify & correct gener … features that may be interesting to the community
See live documentation … /wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework
How to operate with the framework in this beta stage:
```lang=shell-session
$ ssh dev-buster.…
pending quite a lot of time in investigating and finding workarounds to fix inconsistency issues brought up by the Swift backend of Docker Distribut … with Alexandros, we came up with the following high level plan as proposal to move things forward in Q3:
1) Do … Ceph as backend (see T394476 for the Data Persistence Part). We'd need to pick it up, and complete it (reas … us image sizes, etc..). We'd need to pay attention to bottlenecks with the new storage infrastructure, and (hopefully few irrelevant) bugs in …
I was running this ill-formed command today:
```
too … a.test@toolsbeta-sgebastion-05:~$ toolforge jobs run --command "sh -c 'env; e … est/tool-test:latest test
```
and the cli only shows:
```
ERROR: An internal error occured while executing this command.
Traceback (most recent call last):
File "/usr/lib/python3/dist-pac … .exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: https://api.svc.toolsbet … ia.cloud:30003/jobs/api/v1/jobs/
…
**Steps to replicate the issue** (include links if applicable):
What I did:
Become the `bridgebot` tool; try to kick off a build (`toolforge build start https://gitlab.wikimedia.org/toolforge-repos/bridgebot`); get a quota error (“DENIED: adding 199.7 MiB of s … rage resource, which when updated to current usage of 938.6 MiB will exceed the configured upper limit of 1.0 GiB.”). Inspect the builds:
```lang=shell-session
tools.bridgebot@tools-bastion-13:~$ …
Kindly migrate your tool(https://grid-deprecation.toolforge.org/t/heritage) from Toolforge GridEngine to Toolforge Kubernetes.
Toolforge GridEngine is getting deprecated. … techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/
Please note that a volunteer may perform this migration if this has not been done after some time.
If you have already migrated this tool, kindly mark this as resolved.
If you would rather shut down this tool, kindly do so and mark …
Recently @Hawkeye7 and @bd808 reported that the job builder is failing.
```lang= … 4] < Guest85> BuildClientError: The build service seems to be down – please retry in a few minutes.
[22:44] < Guest85> Please report this issue to the Toolforge admins if it persists: https://w.wiki/6 … [22:46] < bd808> Guest85: did the error persist?
[22:47] < Guest85> Bee … awkeye7 logged on as a guest
```
The cause seems to be harbor, which has some flappin … hose 'restarting' services switch …
**Author:** `otto`
**Description:**
It'd be really helpful if admins for parent projects had permissions to run gerrit create-project. I'm and admin for the Analytics project, and I can crea … positories. Would it be possible to make this happen?
Thanks!
--------------------------
**Version**: unspecified
**Severity**: enhancement
Kindly migrate your tool(https://grid-deprecation.toolforge.org/t/phetools) from Toolforge GridEngine to Toolforge Kubernetes.
Toolforge GridEngine is getting deprecated. … techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/
Please note that a volunteer may perform this migration if this has not been done after some time.
If you have already migrated this tool, kindly mark this as resolved.
If you would rather shut down this tool, kindly do so and mark …
On April 19, 2023 it was identified that the Graph extension, which uses the older Vega 1 & Vega 2 libraries, … of security vulnerabilities.
In the interest of the security of our users, the Graph extension was disabled on W … ams are working quickly on a plan to respond to these vulnerabilities.
//**We re … at any other third party users of the Graph extension should disable the use of that extension on their wi … onfiguration change will suppress …
**Steps to replicate the issue** (include links if applicable):
* `toolforge jobs delete setup-historical-data-`
**What happens?**:
The client returns a 400 error back from kubernetes:
```lang=she … ebotng-review@tools-bastion-15:~$ toolforge jobs delete setup-historical-data-
ERROR: TjfCliError: Unknown error (400 Client Error: Bad Request for url: https://k8s … uebotng-review/jobs?labelSelector=toolforge%3Dtool%2Capp.kubernetes.io%2Fmana … %2Fname%3Dsetup-historical-data-)
…
ed (T121979#3923914).~~
**2024 update: Those changes that motivated this task were reverted, but there are … uration changes.**
Someone needs to run `cleanupTitles.php` to fix them. But please **do a dry run first** (`php main … upTitles.php --dry-run`) and post the results here. As far as I know the script is not run regularly, so t … e might be other broken titles in the database already (I think it's likely we have some broken due to Unicode normalization changes aft … d ICU recently). It would be good …
**Steps to replicate the issue** (include links if applicable):
Run `toolforge jobs logs -f bot` and wait for an … nded character.
**What happens?**:
While testing ClueBot NG running as a pack and monitoring `toolforge jobs logs -f bot` and exception w … uebotng.INFO: Processing: [[OkrÄ
ERROR: An internal error occurred while executing this command.
Traceback (most recent call last):
File "/usr/lib/python3/dist-pac … base 16: b''
During handling of …
Kindly migrate your tool(https://grid-deprecation.toolforge.org/t/multichill) from Toolforge GridEngine to Toolforge Kubernetes.
Toolforge GridEngine is getting deprecated. … techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/
Please note that a volunteer may perform this migration if this has not been done after some time.
If you have already migrated this tool, kindly mark this as resolved.
If you would rather shut down this tool, kindly do so and mark …
Currently people commit to the private repo on puppetmaster1001. We should move this to puppetserver1001.
Cergen certs c … ut most use cases have been moved to cfssl already (T357750), but that doesn't need to block the switch of the private repo; in the unlikely case that we still need to update a cergen-issued cert, we can still run the commands on puppetmaster1001 and then copy them to puppetserver1001 for commiting.
Kindly migrate your tool(https://grid-deprecation.toolforge.org/t/mbh) from Toolforge GridEngine to Toolforge Kubernetes.
Toolforge GridEngine is getting deprecated. … techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/
Please note that a volunteer may perform this migration if this has not been done after some time.
If you have already migrated this tool, kindly mark this as resolved.
If you would rather shut down this tool, kindly do so and mark …
Kindly migrate your tool(https://grid-deprecation.toolforge.org/t/srwiki) from Toolforge GridEngine to Toolforge Kubernetes.
Toolforge GridEngine is getting deprecated. … techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/
Please note that a volunteer may perform this migration if this has not been done after some time.
If you have already migrated this tool, kindly mark this as resolved.
If you would rather shut down this tool, kindly do so and mark …
We increased the overall timeout from the client to 10m, but there's a timeout when waiting for the pod initialization that's way lower, we probably want to increase that one too.
The current timeout while waiting for the pod to start seems to be ~1m:
```
toolsbeta.tf-test@lima-bookworm:~$ time toolforge build start https://gitlab.wikimedia.org/toolforge-repos/wm-lol --ref with_everything
BuildClientError: Error getting the logs for tf-test-buildpacks-pipel … timed out waiting for pipelinerun …
or disabling ContentTranslation for non-Autoconfirmed completely due to:
1: CX cannot display a warning from the abusive filter. (This should’ve been fixed by {T12391 … kipedia)
2: Cannot stop new users to publish pure machine translation. … not allow non-Autoconfirmed users to use the tool at all) is not feasible sinc … rewrite of BetaFeature extension to allow user group restriction.
Instead, we should address the second concern by adding zhwiki …
When trying to restart StewardBot's k8s pod, I received the following internal server error:
```lines=15
tools.stewardbots@t … tart
Restarting StewardBot pod...
ERROR: An internal error occured while executing this command.
Traceback (most recent call last):
File "/usr/lib/python3/dist-pac … .exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: https://api.svc.tools.eq … rt/stewardbot
During handling of the above exception, another exceptio …
Traceback (most recent call last)…
I suggest overloading the `run` subcommand to allow triggering a one-off cronjo … o` is a registered cronjob, then `toolforge-jobs run foo` should run the job.
This is useful for testing purposes, invoking manual runs from the command line, invoking from another job (along with T315729), etc.
https://tools.wmflabs.org/dplbot/
The webservice for the project 'dplbot' on Tool Labs has … for several days. If you attempt to access any of the project's pages, you get the "No Webservice" page. But if I log in to the project, "qstat" shows the webservice is running, and the error logs don't indicate any problem. … ) "webservice restart" works, and the webpages then become available, but this typically only lasts a few minutes and then they go down again.
…
The Analytics team would like to have a dedicated Airflow instance to start experimenting with it, like the Discovery team has been doing so far. The goal would be to start playing with Airflow with different workloads to see requirements, things missing, etc.. not to have a final RFC about how multip … w. That will come in a later step :)
It would be nice to re-use all the work done, generalizing it a little bit to avoid the Discovery/Search specific bits. …
I'm getting spammed pretty badly by Mailer-Daemon@tools.wmflabs.org. The messages seem to be from promethus@tools.wmflabs.org so this is probably just tools admins getting the flood. A quick search didn't turn … s wrong, but my inbox is hurting. Please fix this if you can. Thank you!!
The ultimate problem seems to be `OSError: [Errno 28] No space … tools-prometheus-05 (and perhaps the other in the pair), but the email alias/list or something like that clearly has some kind of …
Performance schema statistics were not deployed to public prometheus/grafana because … kis with low load, we were told). The alternative was going to be to deploy and aggregate that to tendril or to a private prometheus instance, bu … all deployers did not have access to that (only to wiki dbs/those with root access). Please check performance schema/sys util … mething exposed that shouldn't be to deployers. It is available on the production, but not pooled with a … s correct, we will provide access …
an-master100[1,2] need to be upgraded to Buster, one at the time preserving data in the partitions.
There are a couple of tasks that we might want to do before:
- T276906
- T265126
Then we should create the partman recipe to preserve the partitions/volumes that we care about, and proceed with the reimage!
batm
Web Profile: https://github.com/ajupazhamayil
Location (country): India
Typical working hours: 05:00 PM to 01:00 AM - GMT +05:30
===Synopsis===
The transfer method in transfer.py is … le for copying data from one host to another. It is a highly used func … it runs serially. It mainly uses the cumin library and nc command to fulfil the requirement. It can open firewall … erform compression and encryption to the input. My project aims to improve …
Until Thursday, May 24, 2018, the syntax of the input for mathematical formulae w … in a position that we can change the input syntax. Before doing so, we … ld agree on (realistic) goals how to change the syntax.
Concerning the input syntax it has been suggested to
* remove all modifications done b … nguage/Extension:Math/Roadmap for the current proposed update strategy.
Concerning the output it has been suggested to
* get a WMF staff member to join …
The main thing #product-analytics needs to adapt to {T172410} is a tool that lets us … ular wiki replicas without having to think about which shard they are … sed on a discussion with @elukey, the best way to accomplish this would probably be a library (owne … y Analytics) that handles routing the query to the specific shard, although we're open to any other method.
This library would have a signature so … ery, wiki)`. It //wouldn't// need to provide for cross-wiki joins or a … e from all shards; we are willing …
Earlier today, I tried to reload jobs for the `urbanecmbot` tool on Toolforge. I used `toolforge jobs load jobs.yaml` for that. Ho … mbot@tools-bastion-13 ~/11bots
$ toolforge jobs load jobs.yaml
INFO: loading job 'afd-announcer'...
ERROR: TjfCliError: Unknown error (404 Client Error: Not Found for url: https://k8s.t … mbot/jobs/nnc-announcer-28885815)
ERROR: Please report this issue to the Toolforge admins if it persists: https://w.wiki/6Z … obs, sometimes, it fails right on …
wikibugs quit IRC, so I did a `toolforge jobs restart irc` and watched the logs via `toolforge jobs logs irc -f`. It rejoined and appears to be working, however:
```lang=pyc … 13:16Z irc3.wikibugs INFO: Trying to join #wikimedia-cloud
ERROR: An internal error occured while executing this command.
Traceback (most recent call last):
File "/usr/lib/python3/dist-pac … WantReadError
During handling of the above exception, another exceptio …
Traceback (most recent call last)…
We currently don't have any way to track the number of times our videos are viewed. This was done by https://tools.wmflabs … ars that tool is malfunctioning. This is a child task of the greater effort to collect and report these metrics from mediawiki. Once that is underway, this task can use those metrics to produce a standard dataset and maybe expose it via the pageview api.
Phabricator user: `Kavaljeet_Singh`
- Wikimedia Global Account (SUL): `Kavaljeet Singh`
IDM shows that … kimedia account is already linked to my Developer account.
However, when I go to https://admin.toolforge.org and click "Join now" to request Toolforge membership, I get:
"Wikimedia ac … aljeet Singh' is already attached to another Developer account."
Peop … nvestigated and suggested opening this task. It appears that Striker may … ng an IntegrityError while saving …
```lang=shell-session
tools.bd808-test@tools-bastion-14:~$ toolforge jobs logs -f rproxy
2026-02-11T18 … xy-55589cd764-jcf4z] [job] {"time":"2026-02-11T18:54:07.171760704Z","level":"INFO","msg":"Compiled route","route":{"Path":"/musicbrainz(/|$)(.*)","Upstream":"https://musicbrainz.org","Template":"/$2","PathRegexp":"/musicbrainz(/|$)(.*)","UpstreamURL":{"Scheme":"https","Opaque":"","User":null,"
Host":"musicbrainz.org","Path":"","RawPath":"","OmitHost":false,"ForceQuery":fal
se,"RawQuery"…
mediawiki/php-1.40.0-wmf.21/includes/upload/UploadFromChunks.php(248): UploadFromChunks->outputChunk(str … 21/includes/api/ApiUpload.php(278): UploadFromChunks->addChunk(string … 21/includes/api/ApiUpload.php(157): ApiUpload->getChunkResult(array)
… 21/includes/api/ApiUpload.php(128): ApiUpload->getContextResult()
#4 … .21/includes/api/ApiMain.php(1901): ApiUpload->execute()
#5 /srv/medi … f.21/includes/api/ApiMain.php(878): ApiMain->executeAction()
#6 /srv/ … f.21/includes/api/ApiMain.php(849)…
To reproduce, make a typo in extension.json. The following will appear in logs:
> … client 127.0.0.1:40328] PHP Fatal error: Uncaught Exception: /var/www/ht … ient 127.0.0.1:40328] **PHP Fatal error: Uncaught Error: Class 'WebRequest' not found in … ack trace:\n#0 [internal function]: MediaWiki\\HeaderCallback::callba … /HeaderCallback.php on line 63**
This is possibly harmless (because invalid extension.json is fatal error anyway), but the cause of …
**Steps to replicate the issue** (include links if applicable):
This happens very frequently when I wanted to work with webservice in toolforge.
* `become campwiz`
* `toolforge webservice stop`
*
Also my previous encounter `toolforge webservice buildservice start --m … t all` >> P90317
**What happens?**:
```
Traceback (most recent call last):
File "/usr/lib/python3/dist-pac … (_ssl.c:1029)
During handling of the above exception, another exceptio …
Traceback (most recent call last)…
## Create job with filelog set to true. Listing job works:
```
local.tf-test@lima-kilo:~$ toolforge jobs flush
local.tf-test@lima-kil … test@lima-kilo:~$ PAYLOAD='{"name": "test-job", "imagename": "bookworm", "cmd": "sleep 3600", "continuous": true, "filelog": true}'
local.tf-test@lima-kilo:~$ … /json" -d "$PAYLOAD" "$URL"
{"job":{"name":"test-job","cmd":"sleep 3600","imagename":"bookworm","filelog":true,"filelog_stdout":"/data/project/tf-test/test-job.out","filelog_stderr"…
Hello,
ListeriaBot returns "Last line: ERROR: Login failed" for (some/all?) qu … sieren" (Update list manually) in the right upper corner I get the message "Last line: ERROR: Login failed" and the page is not updated.
https://listeria.toolforge.org/index.php?action=update&lang= … _verstorbener_Personen
Could you please have a look a this?
Thanks a lot!
Also see
https://listeria.toolforge.org/botstatus.php
> This task is part of the [[ https://meta.wikimedia.org/wik … ey Results) work. It is connected to the [[ https://www.mediawiki.org/wiki … e Incubation ]], a new experiment to implement and test recommendations to support language incubation. This specific task is related to [[ https://www.mediawiki.org/wiki … atement:** If we move addwiki.php to MediaWiki Core and customize it to Wikimedia, we will improve code q … ension table SQL files which need …
Emails from The Wikipedia Library are largely bro … y issues, but it now appears that this problem is affecting at least all … We will also display a notice in the library.
**Solution**
We are wor … emails up and running again, but this solution involves a substantial technical overhead in the long run.
We are currently investigating our options for the future, which may involve integrations some of the content of emails into the library directly (e.g. access …
IMPORTANT: Make sure to read the [Outreachy participant instructio … ) thoroughly before commenting on this task. This space is for project-specific que … ===Approved license
I assert that this Outreachy internship project will … nse ]] that is also identified by the FSF as a [[ https://www.gnu.org/l … pproved for free cultural works ]]:
* Yes
=== No proprietary software:
I assert that this Outreachy internship project will forward the interests of free and open source … ?
* 1-2 people
===Brief summary
…
**Steps to replicate the issue** (include links if applicable):
Run `toolforge jobs logs -f cluebot3` (under `tool.cluebot3`)
**What happens?**:
After deploying (image build + restart) `cluebot3` the `jobs logs` command failed to `--follow`.
When executing `toolforge jobs logs -f cluebot3` (under `tool.cluebot3`) the command printed the logs then exited.
After re-runni … ools.cluebot3@tools-bastion-12:~$ toolforge jobs logs -f cluebot3
ERROR…
rently, screen reader users who navigate by headings constantly hear the word "edit" at the end of each heading because section edit links are part of the <h#> elements. It should be changed so that the <h#> elements contain only the heading text, with the section edit links separated out.
This was originally a bug for the link coming before the heading text, which has since been changed. The original description is preserved … textbrowsers) it takes quite long …
```lang=shell-session
tools.mbh@tools-bastion-15:~$ toolforge jobs run emptalks --command "mono … tied_talks.exe" --image tf-mono68
ERROR: TjfCliError: <html>
<head><title … /1.21.0</center>
</body>
</html>
ERROR: Please report this issue to the Toolforge admins if it persists: https://w.wiki/6Zuu
```
**Steps to replicate the issue** (include links if applicable):
* Try and create a variable via `toolforge envvars create`
**What happens?**:
A generic error is thrown
```lang=shell-session
t … ols.cluebotng@tools-bastion-15:~$ toolforge -d envvars create IRC_RELAY_SENDE … _SLIDING_WINDOW_CONFIG '[{"window": 2, "limit": 3}, {"window": 30, "limit": 100}]'
DEBUG:toolforge_weld.config:Unable to find config file /etc/toolforge/toolforge.yaml, skipping
DEBUG:toolforge_we … .config:Updating config from /etc/…
**Steps to replicate the issue** (include links if applicable):
I have no idea if this bug is reproducible, since I just got an error message telling me to report it. Here's what I did, though:
* `become usage-bot`
* `toolforge jobs run --image tool-usage-bot/t … t -e ${PWD}test.err test-2330`
* `toolforge jobs logs test-2330`
**What happens?**:
I got this pile of Python stack traces:
```
ERROR: An internal error occured while executing this…
nterexample,lang=shell-session
tools.stewardbots@tools-bastion-12:~$ toolforge jobs --debug dump
DEBUG:toolforge_weld.config:Unable to find config file /etc/toolforge/toolforge.yaml, skipping
DEBUG:toolforge_we … .config:Updating config from /etc/toolforge/common.yaml
DEBUG:toolforge_weld.config:Unable to find config file /data/project/stewardbots/.toolforge.yaml, skipping
DEBUG:toolforge_weld.config:Unable to find config file /data/project/stewardbots/.config/…
While debugging an unrelated issue with the disk utilization, we noticed this in the `dmesg` output:
```
[Tue Apr 29 … ff encoding: Clause 74 BaseR
```
This is further confirmed by the drop in traffic (https://grafana. … a.org/goto/OWs50UbNR?orgId=1) and the `getsel` output for `lvs3009`:
```
Record: 33
Date/Time: … Critical
Description: A fatal error was detected on a component at bu … BCM5720 Gigabit Ethernet PCIe
```
doc1001.eqiad.wmnet is the backend for https://doc.wikimedia … created as well.
time frame: by the end of Q3 2020
I get 503 error when trying to edit large size pages in Ukrainia … indows 11, Firefox 134.0.2. I try to do same actions after relogin, re … a One 116.0.5366.71 produces same error. I also tried to do so via api — I can edit. I a … y smartphone in Samsung Browser.
Error details examples:
Request served … 070 cp3070, Varnish XID 664829451
Error: 503, Backend fetch failed at Sun … 070 cp3070, Varnish XID 711411132
Error: 503, Backend fetch failed at Sun … 070 cp3070, Varnish XID 422504693
…
There are 3 permanently working tools in "rv" Toolforge tool. I stopped and started again one of them and it fails to start, "specified command fails to run". I flushed jobs list and sta … t start, "specified command fails to run". Two other tools did not cha … evelopers are doing some works on Toolforge servers?
4f: DENIED: adding 108.0 MiB of storage resource, which when updated to current usage of 1000.2 MiB will exceed the configured upper limit of 1.0 GiB … t] 2025-03-05T04:17:03.423622017Z ERROR: failed to export: failed to write image to the following tags: [tools-harbor.wmc … rage resource, which when updated to current usage of 1000.2 MiB will exceed the configured upper limit of 1.0 GiB … 2025-03-05T04:17:03.436520911Z | The above error is likely because …
Sometimes, builds-api gives up waiting for the logs when it's waiting for the pipelinerun to start:
```
{"time":"2024-01-10T12:35:59.919354032Z","id":"","remote_ip":"127.0.0.1","host":"127.0.0.1:8000","method":"GET","uri":"/v1/build/tf-test-buildpacks-pipe … vk5/logs?follow=True","user_agent":"toolforge_builds_cli toolforge_we … on-requests/2.28.1","status":500,"error":"","latency":60390174565,"latency_human":"1m0.390174565s","bytes_in":0,"bytes_out":122}
{"level"…
```lang=shell-session
$ time toolforge jobs logs --follow bouncer
2024-0 … messages returned immediately]...
ERROR: An internal error occured while executing this command.
Traceback (most recent call last):
File "/usr/lib/python3/dist-pac … WantReadError
During handling of the above exception, another exceptio …
Traceback (most recent call last):
File "/usr/lib/python3/dist-pac … , in recv_into
raise timeout('The read operation timed out')
socket.timeout: …
ing down 4 specific instances, I found bad text table entries, where the text was presumably deflated but … is as well.
Note that according to the stack trace, the failure to retrieve a sha1 from the revision results in a lookup of the content. We are dumping metadata … race:
```
PHP Warning: gzinflate(): data error
#0 [internal function]: MWExceptionHandler::handleError(i … udes/Storage/SqlBlobStore.php(593): gzinflate(string)
#2 /srv/mediawi … udes/Storage/SqlBlobStore.php(520)…
ToolWatch. The purpose of this tool is to check the health status of tools deployed on ToolForge and present the information in a visually appeali … rd.
Description:
After exploring the list of tools on https://toolhub. … that many tools on https://admin.toolforge.org/tools are inactive, I believe it would be valuable to have a centralized dashboard that displays the health status of all deployed tools on ToolForge.
ToolWatch would perform regular checks on …
```
toolsbeta.test@toolsbeta-bastion-6:~$ toolforge-deploy/utils/run_functional_tests … nvvars-api smoke
(in test file toolforge-deploy/functional-tests/envvars-a … s-smoke-test.bats, line 20)
`toolforge envvars create "TEST_ENVVAR_1" "t … s"' failed
EnvvarsClientError: Error contacting k8s cluster.
Please report this issue to the Toolforge admins if it persists: https://w.wiki/6Z … sbeta.test@toolsbeta-bastion-6:~$ toolforge envvars quota
quota used … sbeta.test@toolsbeta-bastion-6:~$ …
**Steps to replicate the issue** (include links if applicable):
* Create a yaml file that's an o … ple:
```lang=yaml
foo: bar
```
* `toolforge jobs load test.yaml`
**What happens?**:
```lang=python,counterexample
ERROR: An internal error occured while executing this command.
Traceback (most recent call last):
File "/usr/lib/python3/dist-pac … dices must be integers, not 'str'
ERROR: Please report this issue to the Toolforge admins: https://w.wiki/6Zuu
```
**What should have happened instead?**…
**Steps to replicate the issue** (include links if applicable):
* Go to https://commons.wikimedia.org/wik … k on ZoomViewer
**What happens?**:
```No webservice
The URL you have requested, https://zoomviewer.toolforge.org/index.php?f=Himalaya,_Indian_ … at should have happened instead?**:
Image should be displayed.
**Oth … r name/version, screenshots, etc.):
Chrome on Windows 10.
Also on ot … bnail is produced for these files):
* https://commons.wikimedia.org/w … ct_Darrang,_Assam_(10620000).jpg
…
i> noice
I tried something:
tools.hitaden@tools-sgebastion-10:~/www$ toolforge build delete *
I'm going to delete 1 builds, continue? [y/N]: y
BuildClientError: user hitaden not allowed to act on build js
Please report this issue to the Toolforge admins if it persists: https://w.wiki/6Zuu
Deleted 0 builds
```
Add a quota command to display information about used Ha … f triggers, concurrent jobs, etc.)
e requested: 10 continuous jobs, 3 one time jobs, 8+24Gi
Reason: Due to the need to traverse the entire history of the article and do Longest Common Sub … e 2GiB of memory. As for checking the archive as a one-off task, for la … emory requirements work fine with the old Grid Engine.
===The Ultimate Calendar===
The project includes technical and non-technical innovations to make event communication more dir … -efficient and user-friendly. For this purpose, an **event database system **is to be created and established. It is … m**, **data maintenance according to the Wiki principle** and **diverse da … ps, Stakeholders and Functions of the System in their interest:====
=====1 Event Organizers=====
The project focuses on non-commercial … izers. For them, two functions of …
From a toolforge `webservice shell` console:
```
… 254.243").get_whois('ripencc')
'% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http: … support/db-terms-conditions.pdf
%ERROR:201: access denied for 185.15.56. … db/faq/faq-db/why-did-you-receive-the-error-201-access-denied
% This query was served by the RIPE Database Query Service versi … e-use-policy/why-did-i-receive-an-…
**List of steps to reproduce** (step by step, including full links if applicable):
* Go to https://video2commons.toolforge.org/
* Click "Login on Wikimedia Commons"
**What happens?**:
```lang=pytb
Traceback (most recent call last):
File "/data/project/video2common … in initiate headers={'User-Agent': user_agent})
File "/data/project … /video2commons/service.manifest
# This file is used by Toolforge infrastructure.
# Please do not edit manually at …
https://github.com/stjohann/DiscordWikiBot | DiscordWikiBot ]] with the help of a C# library called [[ ht … /EvtSource | EvtSource ]]. Around the end of November 2019 it started to report an exception every 15 minutes (counted from the start of the connection to stream.wikimedia.org) from some u … . I haven’t changed anything in the code around that time and nothing … dia.org since July 2019.
I tried to look what might’ve gone wrong, … myself, so I thought I could ask …
hosts have complained about replication lag in our monitoring systems
Per T242602 and T252382#6125029 and IRC discussion,
the install_servers (install*) should … erver like apt* servers and serve the tftp environment.
Then we should … eate an install server in each of the edge POPs.
---
install3001.wiki … anymore; and OS install is working
This task is for two separate wikitext … hat occurred when adding comments to pages using DiscussionTools. They were both triggered by the deployment of the replying API (T252558), but had d … hey are unrelated, and now all of the comments are on this task, so let's keep it as is.
=== Issue 1
(occurred on: 6-7 August; affe … 0-100, see T260393#6384098)
When the replying API was serializing the modified document to HTML, it encoded various characte … t recognizing unmodified parts of …
The spacemedia tool seems to regularly (weekly?) hits some URL … cannot reach and starts crashing. This fills the disk with docker logs on the host that it is running on.
So first, I wanted to let you know that it is crashing … ng you), and second, can you send error logs to the tools NFS homedir on /data/project/spacemedia? I'm taking the action item to tighten up log rotation to prevent this from crashing cluster nodes.
Thanks!
Before any migration to the eqiad1 deployment, the outbout email path for of Toolforge seems to be (for @wikimedia.org recipients):
1) a toolforge server (10.0.0.x) sends the email, using relay `tools-mail-02.eqiad.wmflabs`
2) the tools relay server (10.0.0.x) recvs the email and forwards it to `mx1001.wikimedia.org`, i.e, prod relays
3) the prod (public addr) relay handles the email correctly (follow up steps are not of interest right now)
…
ng with puppet-compiler, we found that CloudVPS instances don't seem to get labs roles/classes applied in the generated catalog.
At first, we suspected of the `environment` variable:
```
"environment": "production"
```
But @Joe confirmed via IRC that the issue is somewhere else, probably in the `puppet-compiler` codebase itself … ec-1442.tools.eqiad.wmflabs.pson) :
```lines=15
{
"document_type": "Catalog",
"data": {
"tags": ["settings","default","standard", … "sudo","node","class"],
"name"…
<span class="n">ERROR</span><span class="p">:</span> <span class="n">An</span> … ">internal</span> <span class="n">error</span> <span class="n">occurred</ … >executing</span> <span class="n">this</span> <span class="n">command</s … s="n">last</span><span class="p">):</span>
<span class="n">File</sp … >ValueError</span><span class="p">:</span> <span class="n">invalid</s … ass="mi">16</span><span class="p">:</span> <span class="n">b</span><s … ass="n">of</span> <span class="n">…
tools.jhstools@tools-sgebastion-10:~$ toolforge jobs list --output long
/usr/bin/toolforge-jobs:15: DeprecationWarning: pkg_ … resources import load_entry_point
ERROR: An internal error occured while executing this command.
Traceback (most recent call last):
File "/usr/lib/python3/dis … .exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: https://api.svc.tools.eq … ia.cloud:30003/jobs/api/v1/jobs/
The above exception was the…
## Summary
Before:
* On the server, MediaWiki calls the Mathoid service which uses MathJax (in Node.js) to parse LaTeX plain text and genera … e MathML output and SVG output.
* The MathML render is hidden by default with CSS.
* The SVG output is stored as a file an … ered via an `<img>`.
After:
* On the server, MediaWiki directly parses … lain text and generates MathML.
* The MathML render is visible by defau … (client-side JavaScript) improves the MathML rendering by using `<svg>`.
## Plan
Background: …
Umbrella task to track the work required towards upgrading our Kubernetes clusters to Kubernetes 1.31.
We're currently … there are some bigger requirement to be dealt with before moving to a newer version:
[x] Migrate away … iner runtime: T269684
[x] We need to migrate away from PodSecurityPolicies: T273507
Together with the Kubernetes update, we need to **update the following other components**:
Core components (more or less on all clusters):
* ✅ **calico** (currenty 3.23.3 … orts compression on its protocol; …
-right}
== Goal
Following T128115 which was resolved in April 2017, the next logical step would be to drop support for ES5 when the proportion of browsers in use tha … _for_IE11 | we've already set out to have special treatment ]] for new … ES6 ]]
- https://caniuse.com/es6
Alright I finally got everything sorted out here!
The username of your existing account … e (Specify which service you need to get access to, e.g. Icinga, Grafana, Superset etc): Matomo
The specific LDAP group that you want to be added to (optional): wmf
For contractors only:
C … act contact person: Nazneen Nawaz
etherpad database (`etherpadlite`) lives in m1.
The current design (I don't know if this has changed in different versions … bin
1 row in set (0.031 sec)
```
This table is huge on disk:
```
root@d … 233G Jan 22 06:13 store.ibd
```
This is an approximate number of rows (it is probably higher):
```
cumin2024@db2160.codfw.wmnet[ … (0.031 sec)
```
While right now this is not causing any inmediate issu … slower) - we are concerned about this model, scalability and future. I … options and approaches if we want …
burg, Germany Time Zone: UTC+1 (CET)
Typical working hours: 3:00 PM to 9:00 PM CET (flexible)
Synopsis
N … Opus 4.6 was extensively used for the creation of this proposal, however all the responsibility of what was said h … cked and edited by me, and I bear the responsibility for all things said here.
Wikidata is one of the largest knowledge graphs on the web, and it has many inconsistenc … s in how entities are classified. The distinction between individuals, … ly as instances and subclasses of …
60604-222120-85mxn8qwb5
- 20260604-222612-b8x95li2oj
All failed due to an internal api issue, example
```
tools.cluebotng@tools-bastion-15:~$ toolforge components deployment show 202606 … Got exception: Some builds failed to start: backup-database(error:HTTPSConnectionPool(host='api.svc … iad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=20)) core(error:HTTPSConnectionPool(host='api.svc … iad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=20)) grafana-alloy(…
Per parent ticket, we need to migrate the ttmserver indices from the production CirrusSearch cluster to OpenSearch on k8s.
Here is the tentative maintenance plan, subject to approval by stakeholders
- Stand … in k8s (DPE SRE)
- Verify access to new clusters (@Nikerabbit )
- TBD: Load the ttmserver indices into K8s. This will be a joint effort between DP … y with OpenSearch on one hand and the application on the other. I'm guessing it will probably be similar …
**Steps to replicate the issue** (include links if applicable):
* Access logs for `celery-worker … uebotng-review`
**What happens?**:
Error is returned
```
tools.cluebotng-review@tools-bastion-15:~$ toolforge jobs logs -f celery-worker
ERROR: TjfCliError: Internal Server Error
ERROR: Please report this issue to the Toolforge admins if it persists: https://w.wiki/6Z … ebotng-review@tools-bastion-15:~$ toolforge jobs logs -f celery-worker
ERROR: TjfCliError: Internal Server …
kartotherian is being moved to wikikube. When that is complete, the server backend will be moved to Bookworm.
We'll be reusing six … ormer ganeti nodes a test cluster to test the new stack (setup at T380144) until we eventually reimage the maps* nodes.
The high level plan is:
[x] install a … kworm node and fix all issues
[x] the remaing replicas
[x] test the OSM import on the new cluster
[x] point wikikube/staging to use the bookworm cluster
[x] if …
== Problem
Templates and Modules need to be shared between wikis. It was … equested ages ago, but mostly due to technical limitations (cache inva … t can be done without any changes to the server-side code. This way users can start experimenting … useful templates, and figure out the social aspect of this feature and its limitations. And … ctionality is actually needed for this to work well, and have an easy migra … that functionality is in place. …
05d31cf0092c12a0d6/includes/Pages/UnarchivePage.php) do not check if the request is POSTed or has an edit … er can convince an election admin to archive or unarchive a page witho … ion.
2 - Additionally, if one of the options for a single transferable … would get XSSed if they voted for the malicious option. This problem is caused by [line 432 of … yPage.php#1228) does not check if the request is POSTed or has a valid edit token, thus also leading to…
**Project title:** //Rewriting PendingChangesBot from PHP to Python //
**Description of proj … ** // Wikimedia Finland rewriting the PendingChangesBot (ie. [[ https: … g edits in Finnish Wikipedia, and the target is that we could deprecate the old PHP version in 2026.
**Back … wiki, huwiki, fiwiki, ruwiki; see the [[ https://meta.wikimedia.org/wik … Revisions ]] for tracking changes to articles. There are two different modes. In the first mode, edits need …
The `stewards*` production virtual machines were created (T344164) to facilitate onboarding and offboar … ng stewards from/their roles, see the parent task. One of the major bottlenecks when (on/off)bo … s is provisioning/revoking access to a number of mailing lists, such a … stewards-usergroup`
I would like to wire membership management for those lists to the tool that I'm working on. However … easily accessible API. According to a discussion with @Ladsgroup I had about Mailman's API, …
Now that we have moved all traffic to HAProxy, we should replace the current pipeline
* Varnish logs -> Varnishkafka -> Kafka
to
* HAProxy -> RSyslog -> Kafka
The format of the messages sent to Kafka should remain the same, so for the content, to avoid breaking existing pipelines and tools.
Roughly, the actions needed are:
* Investigate whether the log format can be transformed to match the VarnishKafka one (checkout the VarnishKafka configuration file f … information)
* Configure RSyslog …