Change 958564 had a related patch set uploaded (by Jean-Frédéric; author: Jean-Frédéric):

[labs/tools/heritage@master] Add known fields for ua_uk

https://gerrit.wikimedia.org/r/958564

Change 958564 merged by jenkins-bot:

[labs/tools/heritage@master] Add known fields for ua_uk

https://gerrit.wikimedia.org/r/958564

Mentioned in SAL (#wikimedia-cloud) [2023-10-15T20:27:07Z] <wm-bot> <jeanfred> Deploy 70616a4 (T345695), 37fe60b, 764e236 (T345695), b46cdbe, 6caa9be, c81c7f6, 60cd8c3, 7925f7a, 9a70390, 0a8c490 (T346681)

I do not know if this is connected to this patch, but the bot just "lost" 90k of Ukrainian Monuments from the database (so now only 6k are listed), so now the database is almost useless, as one cannot open the information about monuments from the template on commons on the pictures:

https://commons.wikimedia.org/w/index.php?title=Commons:Monuments_database/Statistics&diff=prev&oldid=812353141

and the report with unknown fields is still full: https://commons.wikimedia.org/wiki/Commons:Monuments_database/Unknown_fields/monuments_ua_(uk)

Checked the logs quickly:

ERROR: Unknown error occurred when processing country ua in lang uk
(1048, "Column 'wd_item' cannot be null")

So, yeah, this is definitely linked to 0a8c490 :-/

@Lokal_Profil What do you think ? Is there a proper way to map to wd_item and having it nullable somehow ? Or shall we just revert that mapping?

I quickly commented out the wd_item part

#        {
#            "dest": "wd_item",
#            "source": "wikidata",
#            "check": "checkWD"
#        },

so that hopefully next harvest does not crash

Apologies for not seeing this before.

So a missing valus should be harvested as an empty string. But then it is sent to the checkWD-check which returns None (not False) [in checkers.py:L115]. This converts the value from an empty string to None [in update_database.py:L256] which clashes with the sql format for the field wd_item NOT NULL DEFAULT '' [in monument_tables.py:L77].

Spontaneously I'd say we should only set field_value to None if run_check gives False, not on a None value. But sadly I think we make use of this feature to change empty Lat and Lon values to None values (lat/lon are allowed to be Null). So maybe rather add the following before monument_tables.py:L65?

elif column== wd_item:
    fields_sql.append(
        '`{}` {} DEFAULT NULL,'.format(column, default_type))

I think we however would also need to add the following under "sql_data" in the ua_uk config to bring the wikidata info along to monuments_all.

"wd_item": {
    "value": "wd_item",
    "type": "Field"
}