We usually create a project for that. https://phabricator.wikimedia.org/project/profile/8517/ was just created

Can you do a backup, zip it and put it somewhere (Toolforge accessible?) so at least data is recoverable? Yeah sure, risk of it getting lost was always there. But here you're actively deleting it.

The error in the log:

tools.heritage@tools-bastion-14:~/bin$ tail -5 ~/logs/harvesting.err
WARNING: 20 primkey(s) missing on List_of_historic_places_in_British_Columbia_Coast (monuments_ca-muni_(en))
Page [[commons:Commons:Monuments database/Unknown fields/monuments ca-muni (en)]] saved
Page [[commons:Commons:Monuments database/Unknown fields/Statistics]] saved
curl: option --insecure --cert /data/project/heritage/.toolskube/client.crt --key /data/project/heritage/.toolskube/client.key: is unknown
curl: try 'curl --help' or 'curl --manual' for more information

Heritage is used for https://commons.wikimedia.org/wiki/Commons:Monuments_database and has gotten quite large, see https://commons.wikimedia.org/wiki/Commons:Monuments_database/Statistics . We're open to pull requests to improve performance, see https://gerrit.wikimedia.org/r/q/project:labs/tools/heritage

In T223051#10396990, @TheDJ wrote:

I'm assuming this was fixed with T240509. Uploading at https://test.wikipedia.org/wiki/File:2018-05-11_Joensuu_station_4.jpg shows that the metadata is now parsed.

However fixing it for the existing uploads requires force reparsing of the metadata of each of these files.

mwscript maintenance/refreshImageMetadata.php --wiki=commonswiki --start="Eckerlochstieg_19.jpg" --stop="Eckerlochstieg_19.jpg" --force --verbose
mwscript maintenance/refreshImageMetadata.php --wiki=commonswiki --start="Bodebruch_in_April_2019_17.jpg" --stop="Bodebruch_in_April_2019_17.jpg" --force --verbose
mwscript maintenance/refreshImageMetadata.php --wiki=commonswiki --start="2018-05-11_Joensuu_station_4.jpg" --stop="2018-05-11_Joensuu_station_4.jpg" --force --verbose
mwscript maintenance/refreshImageMetadata.php --wiki=commonswiki --start="20170910_xl_P1150222-windkraftanlagen--wka--im-windpark--werder-zinndorf--rehfelde-amt-maerkische-schweiz.jpg" --stop="20170910_xl_P1150222-windkraftanlagen--wka--im-windpark--werder-zinndorf--rehfelde-amt-maerkische-schweiz.jpg" --force --verbose

I'm sure there are many more, but unfortunately identifying them among the hundred million jpgs, is going to be a challenge

Any more work needed or is this done?

We have https://heritage.toolforge.org/api/api.php?action=search&srcountry=nl&srlang=nl&format=csv&limit=500&srcontinue= but that's the monuments_all table. I this still needed? I am working on Wikidata <-> list comparison.

That fixed it: https://commons.wikimedia.org/w/index.php?title=Commons:Monuments_database/Unknown_fields/monuments_ro_(ro)&diff=prev&oldid=1107965421

Did a fresh pywikibot install from git

WARNING: <string>:1: _PasswordFileWarning: The BotPassword entry should only include the suffix

The file contained:

('ErfgoedBot', BotPassword('ErfgoedBot@ErfgoedBot-Toolforge2025', 'xxx'))

In the log I see:

Login failed for normal ErfgoedBot@ErfgoedBot@ErfgoedBot-Toolforge2025 from ...

Changed it to:

('ErfgoedBot', BotPassword('ErfgoedBot-Toolforge2025', 'xxx'))

Now it works:

Logging in to wikipedia:nl as ErfgoedBot@ErfgoedBot-Toolforge2025
Logged in on wikipedia:nl as ErfgoedBot.

And in the log:

Login succeeded for normal ErfgoedBot from 172.16.2.61 - login (wikipedia:nl; User:ErfgoedBot) Pywikibot/10.7.0 (g1) requests/2.32.5 Python/3.13.5.final.0

Generated a new bot password ErfgoedBot@ErfgoedBot-Toolforge2025 . Fired up the bot and getting this error:

In T407159#11273538, @Mstyles wrote:

Are you concerned that someone logged into the bot account without your permission? You said you were able to log into the bot account

Tired of the email spam:

There have been 1,755 failed attempts to log in to your account since the last time you logged in. If it wasn't you, please make sure your account has a strong password.

• Find reqId in Logstash

Able to login as the bot and https://commons.wikimedia.org/wiki/Special:BotPasswords/ErfgoedBot-Toolforge is configured

@Ciell why did you mark this one as invalid?

Seems to work again? I didn't change anything.

In T388809#11025715, @BCornwall wrote:

Thanks for your patience. Hopefully we're done-done now. :)

Did just get this error, but second try gave normal output. Might be related?

tools.multichill@tools-bastion-12:~$ toolforge jobs list
ERROR: TjfCliError: Unknown error (404 Client Error: Not Found for url: https://k8s.tools.eqiad1.wikimedia.cloud:6443/apis/ba
tch/v1/namespaces/tool-multichill/jobs/sdoc-cc-by-sa-4.0-29228247)

Thanks! Looking good:

tools.multichill@tools-bastion-12:~$ toolforge jobs load jobs.yml --job coord-from-exif-mysql
INFO: loading job 'coord-from-exif-mysql'...
Job coord-from-exif-mysql created
INFO: 51 job(s) loaded successfully

https://pywikibot.org redirects to https://pywikibot.toolforge.org/ . https://pywikipedia.org currently redirects to https://www.wikimedia.org/ . This should be changed to https://pywikibot.toolforge.org/ .

Problem persists. Any update?

No, I'm not buying this. The name Pywikipedia is still used in several place like for example the mailing lists. This shouldn't give a certificate error. The funky dns seem to make this a bit harder:

pywikipedia.org.	3600	IN	NS	ns063.auroradns.info.
pywikipedia.org.	3600	IN	NS	ns061.auroradns.eu.
pywikipedia.org.	3600	IN	NS	ns062.auroradns.nl.

I'll ask whoever is doing this at Wikimedia Nederland these days to update the dns. Stalled on that.

In T395205#10885984, @Tgr wrote:

• Tech News
• wikitech-l email

We can merge the patch after the next branch cut, that will give people almost two weeks to fix their bots.

I put a warning on https://www.mediawiki.org/wiki/Manual:File_table linking to this ticket. Please remove when done.

I can't revert or upload any files, I just get "The file "mwstore://local-multiwrite/local-public/4/4e/Interieur_-_'s-Gravenhage_-_20089866_-_RCE.jpg" is in an inconsistent state within the internal storage backends"

In T381918#10395218, @Dzahn wrote:

Both T350793#10378435 and today T350793#10394688 happened and caused realtime debugging. This seems to be at odds with the claim that nobody uses it anyways.

@Gehel you created this task two months ago. It's clear what the community wants. What are you going to do?

Don't bother, nobody uses it anyway as long as it has authentication. That should be removed, see T297995. Not even clear what the future will be, see T376979

I just did a toolforge jobs list and had 3 jobs running.

The log didn't go back very far. I tried starting some jobs and ran into limits right away:

Burn it with fire preferably after it has been removed from the wiki's.

@dcaro if you merge a task, can you please copy over relevant information? In this case the fact someone already wrote the code in T376099 is quite relevant and that went missing.

Happy to hear it works now!

No, don't. SPARQL endpoints should be authenticated, especially the SPARQL endpoints by Wikimedia. T297995 should be resolved. This proposal is just a very bad work around to replace an even worse work around (https://commons.wikimedia.org/wiki/Commons:SPARQL_query_service/API_endpoint)

Just remove the damn authentication. T297995 has been open for three years. You have been running a quite stable, but also quite useless service for the past three years. It's extremely demotivating that I and other volunteers spend time on importing and adding structured data, but I can't properly query it. It's the primary reason why I haven't really added a lot of new things over the last years.

How are you going to handle planned work on underlying infrastructure? Will you send out alarms or will you correlate it to the planned work so people know what is going on?

Ipv6 vrrp is all link-local if I recall correctly. Did you configure it like that?

I'm pretty sure we have a very old task for this one, maybe even from Bugzilla times. I'll do a bit of touching to empty out that list of junk.

Currently using it for edits like https://commons.wikimedia.org/w/index.php?title=File:John_Meintz,_punished_during_World_War_I_-_NARA_-_283633_-_restored.jpg&diff=prev&oldid=909842445 where I add missing qualifiers to an existing claim. ( self.site.save_claim(claim, summary=summary, tags='BotSDC') )

Thanks for the replies related to translating. Dead end for this task.

In T343131#10054821, @Jarekt wrote:

With implementation of T359761 / T366623 and replacement of {{dir}} and {{bcp47}} templates with brand new parser functions, we should loose some more templatelinks (124M +5M to be precise). I also replaced many calls to {{Autotranslate}} templates in template namespace with the Lua calls directly to c:Module:Autotranslate which might reduce it even further.

Hitting the first "Unable to start, out of quota for memory, memory" errors for jobs.

@Raymond_Ndibe what is the status? I see you merged something. I just tested and the problem still exists.

In T174633#9828797, @JeanFred wrote:

Oh, geez, we never followed-up on that >_>. Adding at least MIT sounds good to me.

No activity for 5 years. I'm just going to go ahead and mark this as declined. If people want to use old symlinks, that's fine. No added value in breaking those.

In T192733#9774988, @Xqt wrote:

Not related to Pywikibot:
https://codesearch.wmcloud.org/pywikibot/?q=trunk%7Crewrite%7Ccompat%7Cpywikipedia&files=&excludeFiles=&repos=

Repo is still very active, just moved to github long time ago, see https://github.com/multichill/toollabs .

For the deletions: https://commons.wikimedia.org/w/index.php?title=Special:Log&page=&tagfilter=&type=&user=Multichill&wpFormIdentifier=logeventslist&wpdate=&wpfilters%5B0%5D=newusers&dir=prev&offset=20230817183454%7C342854881

Deleted:
https://commons.wikimedia.org/w/index.php?title=MediaWiki:Wm-license-cc-by-4.0-au-text&action=edit&redlink=1
https://commons.wikimedia.org/w/index.php?title=MediaWiki:Wm-license-cc-by-4.0-nz-text&action=edit&redlink=1
https://commons.wikimedia.org/w/index.php?title=MediaWiki:Wm-license-cc-by-sa-3.0-IGO-text&action=edit&redlink=1

Report is up. Also had the bot do the easy fixes.

My guess would be that the code at https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/blob/main/tjf_cli/loader.py?ref_type=heads#L23 is tripping up (or not used).