Page MenuHomePhabricator
Create Task
Maniphest T410842

Monuments database Statistics didn't update since 8 November 2025
Closed, ResolvedPublicBUG REPORT
Actions

Description

I noticed at https://commons.wikimedia.org/w/index.php?title=Commons:Monuments_database/Statistics&action=history that it didn't update for a while. https://commons.wikimedia.org/w/index.php?title=Commons:Monuments_database/Unknown_fields/Statistics&action=history did update today.

Errors recently introduced show up at https://commons.wikimedia.org/w/index.php?title=Commons%3AMonuments_database%2FUnknown_fields%2Fmonuments_nl_%28nl%29&diff=1120213424&oldid=1110494603

If I look at https://heritage.toolforge.org/api/api.php?action=search&srcountry=nl&srlang=nl&format=json&limit=500&srid=5249 , it points to https://nl.wikipedia.org/w/index.php?title=Lijst_van_rijksmonumenten_aan_de_Singel_(Amsterdam,_Noord)&oldid=67283668 so all the November 2025 edits are missing.

So looks like the harvesting completes, but the merging into the big table fails.

I believe @JeanFred did a bunch of upgrades in T409003 that might have triggered this. T409716 might be related too (not sure).

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Fix harvesting jobsMultichilllabs/tools/heritagemaster+1 -1
Customize query in gerrit

Related Objects

Event Timeline

Multichill created this task.Nov 23 2025, 2:55 PM
JeanFred added a comment.Dec 19 2025, 10:02 AM

Fairly sure this is due to e344ea80eb14

Antanana subscribed.Dec 19 2025, 10:09 AM
Multichill added a comment.Jan 5 2026, 5:56 PM

The error in the log:

tools.heritage@tools-bastion-14:~/bin$ tail -5 ~/logs/harvesting.err
WARNING: 20 primkey(s) missing on List_of_historic_places_in_British_Columbia_Coast (monuments_ca-muni_(en))
Page [[commons:Commons:Monuments database/Unknown fields/monuments ca-muni (en)]] saved
Page [[commons:Commons:Monuments database/Unknown fields/Statistics]] saved
curl: option --insecure --cert /data/project/heritage/.toolskube/client.crt --key /data/project/heritage/.toolskube/client.key: is unknown
curl: try 'curl --help' or 'curl --manual' for more information

Took me a bit to figure out that it's an escaping problem. This seems to fix it:

tools.heritage@tools-bastion-14:~/bin$ git diff defaults.sh
diff --git a/bin/defaults.sh b/bin/defaults.sh
index bc70cdf..a9ac624 100644
--- a/bin/defaults.sh
+++ b/bin/defaults.sh
@@ -5,7 +5,7 @@ echo_time() {
 create_toolforge_job() {
     local json_payload="$1"
     curl -X 'POST' \
-        "$TOOLFORGE_API_FLAGS" \
+        $TOOLFORGE_API_FLAGS \
         "$TOOLFORGE_API_JOBS_ENDPOINT" \
         -H 'accept: application/json' \
         -H 'Content-Type: application/json' \

That trigger a job:

tools.heritage@tools-bastion-14:~/bin$ toolforge jobs list
+--------------------------+----------------------+------------------------------------------+
|        Job name:         |      Job type:       |                 Status:                  |
+--------------------------+----------------------+------------------------------------------+
| fill-table-monuments-all |       one-off        |            Running for 2m49s             |
|     update-monuments     | schedule: 0 3  * * * | Last schedule time: 2026-01-05T03:00:00Z |
+--------------------------+----------------------+------------------------------------------+

And now https://heritage.toolforge.org/api/api.php?action=search&srcountry=nl&srlang=nl&format=html&limit=500&srid=531286 links to my recent edit

Also some other jobs popped up:

tools.heritage@tools-bastion-14:~/bin$ toolforge jobs list
+------------------------+----------------------+------------------------------------------+
|       Job name:        |      Job type:       |                 Status:                  |
+------------------------+----------------------+------------------------------------------+
|   categorize-images    |       one-off        |            Running for 2m10s             |
|  post-harvesting-php   |       one-off        |            Running for 2m11s             |
| post-harvesting-python |       one-off        |            Running for 2m10s             |
|    update-monuments    | schedule: 0 3  * * * | Last schedule time: 2026-01-05T03:00:00Z |
+------------------------+----------------------+------------------------------------------+

https://commons.wikimedia.org/w/index.php?title=Commons%3AMonuments_database%2FStatistics&diff=1142505837&oldid=1111233096 got updated too!

gerritbot added a comment.Feb 12 2026, 6:51 PM

Change #1239197 had a related patch set uploaded (by Multichill; author: Multichill):

[labs/tools/heritage@master] Fix harvesting jobs

https://gerrit.wikimedia.org/r/1239197

gerritbot added a project: Patch-For-Review.Feb 12 2026, 6:51 PM
Multichill added a project: Hackathon-Northwestern-Europe-2026.Feb 22 2026, 10:45 AM
gerritbot added a comment.Mar 13 2026, 11:29 AM

Change #1239197 merged by jenkins-bot:

[labs/tools/heritage@master] Fix harvesting jobs

https://gerrit.wikimedia.org/r/1239197

Maintenance_bot removed a project: Patch-For-Review.Mar 13 2026, 11:30 AM
Multichill moved this task from Proposed Projects to Tinkering in Progress on the Hackathon-Northwestern-Europe-2026 board.Mar 13 2026, 12:40 PM
Stashbot added a comment.Mar 13 2026, 2:27 PM

Mentioned in SAL (#wikimedia-cloud) [2026-03-13T14:27:54Z] <wmbot~jeanfred@tools-bastion-15> Deploy 84a13d7, ddda476, dbb689f, dd7ce20, 1d9f43f, 5dbba03, cd1600e (T410842)

Multichill closed this task as Resolved.Mar 13 2026, 2:30 PM
Multichill claimed this task.

All done!!!!!

Multichill moved this task from Tinkering in Progress to Ready for showcase on the Hackathon-Northwestern-Europe-2026 board.Mar 13 2026, 2:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits