I think people are no longer aware of how these processes work. Step one is making people aware of this. For example, i would not be surprised that if the Iran community were better aware of what to do, they might actually do it. A clear instruction email with pointers could go a long way. This may require some help in designing category trees on commons.

I think also it would make sense to split the countries in two tiers: one tier where active maintenance takes place, which runs separate from the countries that do not maintain actively. Could we run the second tier weekly or monthly, perhaps? (for the overview pages - harvesting ideally still happens frequently)

finally, if the tool of dudemanfellabro could be improved a bit, and is maintained again, it could go a long way in decreasing the other lists. Right now the problem is in part that you can't 'blacklist' images - so they keep showing up. Also the category tree on commons is polluted because someone has been adding multiple identifiers to certain categories, where the identifier only really is relevant to a part of the category.

this won't solve the whole problem at once, but maybe a little bit?

I would support blacklisting but following that up with an e-mail to the local organisers of blacklisted countries. Positive response and action as a result of those e-mails would take the country of the blacklist. That way those countries are not blocking the others.

It might also be worth splitting the job up into a harvest and a categorise one.

In T174871#3576293, @Lokal_Profil wrote:

It might also be worth splitting the job up into a harvest and a categorise one.

I initially was not keen on doing that, as I am not sure how well the infrastructure would support parallel processing ; but on second thoughts it should be fine (especially as categorisation does not write to the DB).

Change 377483 had a related patch set uploaded (by Jean-Frédéric; owner: Jean-Frédéric):
[labs/tools/heritage@master] Split categorization out of daily update job

https://gerrit.wikimedia.org/r/377483

Change 377483 merged by jenkins-bot:
[labs/tools/heritage@master] Split categorization out of daily update job

https://gerrit.wikimedia.org/r/377483

Mentioned in SAL (#wikimedia-cloud) [2017-09-14T09:36:53Z] <JeanFred> Deploy latest from Git master: d2aa019, 0766491, d73eb9e (T174871), 5b00f0b, f8ff2a6 (T175839)

Change 378021 had a related patch set uploaded (by Jean-Frédéric; owner: Jean-Frédéric):
[labs/tools/heritage@master] Skip categorisation for some countries

https://gerrit.wikimedia.org/r/378021

Change 378021 merged by jenkins-bot:
[labs/tools/heritage@master] Skip categorisation for some countries

https://gerrit.wikimedia.org/r/378021

Before closing this task we should (possibly as a subtask) reach out to the local organisers of the skipped countries and explain what they need to do to get of the list and why it is important that they do (i.e. the way to find things in commons is through categories)

Mentioned in SAL (#wikimedia-cloud) [2017-09-17T18:28:43Z] <JeanFred> Deploy latest from Git master: b3c3ab0 (T174871)

Change 378641 had a related patch set uploaded (by Lokal Profil; owner: Lokal Profil):
[labs/tools/heritage@master] [WIP] Ensure skipped image categorizations are mentioned in stats

https://gerrit.wikimedia.org/r/378641

Change 378641 merged by jenkins-bot:
[labs/tools/heritage@master] Ensure skipped image categorizations are mentioned in stats

https://gerrit.wikimedia.org/r/378641

Mentioned in SAL (#wikimedia-cloud) [2017-09-21T19:35:40Z] <JeanFred> Deploy latest from Git master: 730d577 (T174871)

@JeanFred Can this be closed?

In T174871#4314133, @Effeietsanders wrote:

@JeanFred Can this be closed?

The actual work on this has been done. What is most likely missing is pinging the organisers whose countries are blacklisted at Commons:Monuments database/Categorization/Statistics and explaining what they need to do if they want to be included.

For good measure we should probably also look at the logs to make sure categorisation has not crashed since we last updated this.

Adding to 2018 since we are likely blacklisting more countries but still haven't pinged organisers about this explaining what they need to do to get of the blacklist.

Typo meant that https://gerrit.wikimedia.org/r/#/c/labs/tools/heritage/+/456758/ did not get picked up. It blacklists another 8 countries.

Mentioned in SAL (#wikimedia-cloud) [2018-09-01T20:23:06Z] <Lokal-Profil> Deploy latest from Git master: 17b27b3, c5122c7 (T200337), 6e85721, b48135e (T174871)

In T174871#4528302, @Lokal_Profil wrote:

Adding to 2018 since we are likely blacklisting more countries but still haven't pinged organisers about this explaining what they need to do to get of the blacklist.

Breaking this out as a separate task at T203338