Now that the bot is fixed, we need to make sure the update pipeline works, and that the Monuments database is updated
Description
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | JeanFred | T110498 Make sure the Monuments database update runs daily | |||
Resolved | JeanFred | T110497 Run a full update of the monuments database | |||
Resolved | JeanFred | T105818 Port ErfgoedBot to Pywikibot-core | |||
Resolved | JeanFred | T110420 Harvesting of several countries does not work because of Namespace filtering issue |
Event Timeline
Yes, update is still in progress. Various countries have been processed for project es, but the monuments_all table will only get updated at the end of the full process. Will keep you posted!
@Emijrp Hmmm, so on one hand reprocessing mysteriously stalled, on the other hand it should have been fine. Can you check on your end ?
Mainland Spain is still empty https://tools.wmflabs.org/wlm-maps/#7/37.182/-4.982
What es: lists are you parsing? Perhaps the titles or the template schema changed?
Config is in monuments_config.py
'project' : u'wikipedia', 'lang' : u'es', 'headerTemplate' : u'Cabecera BIC', 'rowTemplate' : u'Fila BIC', 'commonsTemplate' : u'BIC', 'namespaces' : [104],
I’m rerunning the full processing again. Once this is done, I will focus on (es, es) to investigate further.
Ok, mystery solved:
370244:Working on countrycode "es" in language "es" 370245-ERROR: u'Namespace identifier(s) not recognised: 104' 370246-Unknown error occurred when processing country es in lang es
Looks like the same underlying error than T110420… Will have to investigate further...
It fails with 102, 104... special namespaces. It seems that you have to add them into the family file in families directory?
MariaDB [s51138__heritage_p]> SELECT COUNT(*) from monuments_all WHERE country='es' AND lang='es'; +----------+ | COUNT(*) | +----------+ | 705 | +----------+ 1 row in set (0.00 sec)
So there *is* some data in the database for (es, es) (albeit not much). These should how up. Cursory looks show a lot of errors because no primary key is available.
Yay! Is the bot still running? I see some points in Spain map, but there are still many gaps. For example, this list is not parsed yet https://es.wikipedia.org/wiki/Anexo:Bienes_de_inter%C3%A9s_cultural_de_la_Comunidad_de_Madrid
Yes, the 705 are all that is parsed. I ran (es,es) manually and here is the dump : P1940.
MariaDB [s51138__heritage_p]> select count(*) from monuments_all where lang='es' and country='es'; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row in set (0.00 sec)
Full update, including all additional tasks (until picture categorisation) has passed successfully. Yay! Closing as Resolved.