T206913 was an appropriate fix :)

This has not failed us in a while, closing as Resolved.

I’ll close this as Resolved since we tackled this with T153746: ErfgoedBot categorisation process should ignore hidden categories.

I actually (tried to) did that on purpose... except for the on top of other content ^__^

It is now grouped by source page. I believe this was done in ac7ab3688b (deployed in July 2018).

This looks like fixed in https://tools.wmflabs.org/heritage/api/api.php?action=statistics&stcountry=ua&format=html ?

In the initial discussions with @Effeietsanders and @m.hekmat the form was supposed to be year specific. Shall I rename the form accordingly to WLM 2018, or shall we leave it as is ?

Thanks for your guidance @hashar !

@Aklapper Would you be able to provide guidance on the proper process to either have permission to do this, or request to someone who does?

In T200339#4664500, @Multichill wrote:

Fixing the lists right now. Example edit: https://en.wikipedia.org/w/index.php?title=List_of_listed_buildings_in_Oyne%2C_Aberdeenshire&type=revision&diff=864025741&oldid=857873175

@hashar @Legoktm Would you be able to provide guidance on how to do this? I tried understanding by going through rCICF but failed :-(

Work in progress output : https://commons.wikimedia.org/wiki/User:Jean-Fr%C3%A9d%C3%A9ric/Monuments_database_Statistics_fr_fr

In T204036#4636400, @Lokal_Profil wrote:

Although looking in the logs I see <type 'exceptions.KeyboardInterrupt'> which suggests a similar problem to T203417: update_database crashing due to KeyboardInterrupt

If CommonsDelinker edits are problematic on Wikidata, can’t it just be blocked there until the bugs are fixed?

In T194996#4610857, @dbarratt wrote:

@JeanFred We actually found that most developers didn't want to remove the password from their ssh keys (which was required for running it in a container)

Latest report has some 17 empty sources :) https://commons.wikimedia.org/wiki/Commons:Monuments_database/Statistics

Confirmed too.

In T200339#4586737, @MichaelMaggs wrote:

The WP lists of UK sites are unmaintained by us, and have no connection whatsoever with the WLM-UK campaigns. They also omit the vast majority of grade II buildings. Of course the WP lists may receive odd edits from individual editors but those aren't reflected back to Wikidata.

Resolved this at the same time as T204427.

The country name is not consistently used across links, pages etc

Good catch.

I have been thinking about how to go about that for a while now (took me quite a while too as I was confused by all the wrapping that pywikibot does on top of logging...)

FYI, they recently revamped the Phabricator extension for Sentry (cf. https://github.com/getsentry/sentry-plugins/pull/373) − should make it to the next Sentry release (unscheduled yet).

We tried running harvesting just on gb-sct, and it crashed again (so it’s not due to some weird timing linked to full harvest).

We noticed a lot of warnings for missing prim keys.

Oh, of course, this does not make use of the virtual environment >_>

Can’t reproduce locally…

$ docker-compose run --rm bot python erfgoedbot/populate_image_table.py
Starting labs-tools-heritage_db_1 ... done
Working on all countrycodes
Found 96 countries with monument tracker templates to work on

Been thinking about this. If we have diagnosed this properly, the issue is that categorize_images locks the table for its entire run. Indeed the code is:

The workaround solution is now now working per T202281: no jstop command available in a GridEngine job

I would have hoped the tests would run faster after this, but a time tox -e py27 still yields ~30s :-(

In T202677#4561834, @Lokal_Profil wrote:

I'm sure we had some other mechanism for finding and highlighting these but cannot find it.

Confirmed in later categorisation job: https://commons.wikimedia.org/wiki/Special:Diff/319422336

In T203816#4568739, @MichaelMaggs wrote:

The numbers given for the UK at https://tools.wmflabs.org/wikiloves/monuments/2018/United%20Kingdom don't match the actual daily uploads which we categorise as they happen in Commons cats Category:WLM-UK 2018 unfiltered 09-xx where xx is the day of the month. You can see the numbers in those cats here: https://commons.wikimedia.org/wiki/Category:Images_from_Wiki_Loves_Monuments_2018_in_the_United_Kingdom.

I suspect the tool may be incorrectly working to UTC rather than the local midnight-to-midnight times.

Ok, smth must be off, because it’s fairly unlikely that on https://tools.wmflabs.org/wikiloves/monuments/2017/Germany, 39 images could be uploaded on the first day by 0 joiners :p

Daily data is now exposed at https://tools.wmflabs.org/wikiloves/monuments/2018/France

Yeah, I did not find how to integrate Shellcheck but bashate is good enough for now :) Thanks!

In T203428#4561818, @Lokal_Profil wrote:

Is the cause as simple as forgetting a return statement on categorizeImage() when categorisation is successful?