Developer of Tool-inteGraality. Maintainer of Tool-wikiloves and Wiki-Loves-Monuments
User Details
User Details
- User Since
- Oct 6 2014, 10:01 PM (497 w, 2 d)
- Availability
- Available
- IRC Nick
- JeanFred
- LDAP User
- Jean-Frédéric
- MediaWiki User
- Jean-Frédéric [ Global Accounts ]
Tue, Mar 26
Tue, Mar 26
Mar 17 2024
Mar 17 2024
JeanFred committed R2566:3a93a59f2446: Run updates with toolforge-jobs in Ansible deployment playbook.
Run updates with toolforge-jobs in Ansible deployment playbook
Switch to Python3.9 as default interpreter
JeanFred committed R2566:bde571587721: Add pyproject.toml with pytest configuration to run tests outside of tox.
Add pyproject.toml with pytest configuration to run tests outside of tox
JeanFred reopened T319787: Migrate heritage from Toolforge GridEngine to Toolforge Kubernetes as "Open".
Mar 14 2024
Mar 14 2024
Closing as resolved from my perspective. Please reopen if need be. Thanks!
Please see https://wikitech.wikimedia.org/wiki/Tool:Wikiloves#Scope : this tool currently does not work for global events.
Feb 15 2024
Feb 15 2024
JeanFred committed R2566:3c23229df7cd: Change PropertyColumn key format and use it as column reference when rendering….
Change PropertyColumn key format and use it as column reference when rendering…
Feb 10 2024
Feb 10 2024
JeanFred closed T319778: Migrate governance-timeline from Toolforge GridEngine to Toolforge Kubernetes as Invalid.
This tool can be deleted actually. I just disabled it in the toolsadmin console.
JeanFred added a comment to T320160: Migrate wikiloves from Toolforge GridEngine to Toolforge Kubernetes.
JeanFred closed T320160: Migrate wikiloves from Toolforge GridEngine to Toolforge Kubernetes as Resolved.
Done in April 2023 via 1b084c487a90 & 442fa0d0fdd2.
Jan 29 2024
Jan 29 2024
Nov 21 2023
Nov 21 2023
Remove statsd integration as not available anymore in WMCS
Do not use Mocks for Coumns in test_queries
JeanFred committed R2566:1f41a79498a7: Move some more formatting logic out of queries Jinja template to Column objects.
Move some more formatting logic out of queries Jinja template to Column objects
This was due to T326266: Remove the WMCS statsd/Graphite service: as the cloudmetrics0003 host was removed, and pystatsd has the interesting behaviour of crashing out if the statsd host is unavailable (https://github.com/jsocol/pystatsd/issues/130)
Oct 29 2023
Oct 29 2023
JeanFred added a comment to T236590: Allow grouping by date properties.
💡 (thanks to “Fictional characters whose birth/death date is in the current decade” from the Query Service example page
BIND(YEAR(?date) as ?year). BIND(xsd:integer(?year/10) as ?decade). FILTER(?decade = 200).
JeanFred added a comment to T236590: Allow grouping by date properties.
This took me a long long time, but I think I’m mostly done.
JeanFred closed T312729: Support grouping by other Wikibase concepts than wdt:Pxyz, a subtask of T294889: Assess whether integraality works/makese sense for Lexemes, as Resolved.
JeanFred closed T312729: Support grouping by other Wikibase concepts than wdt:Pxyz, a subtask of T312728: Refactor integraality grouping configuration and logic, as Resolved.
3ba5e84 solves this.
Oct 22 2023
Oct 22 2023
JeanFred added a comment to T346681: Known fields for ua_uk.
I quickly commented out the wd_item part
# {
# "dest": "wd_item",
# "source": "wikidata",
# "check": "checkWD"
# },so that hopefully next harvest does not crash
JeanFred updated subscribers of T346681: Known fields for ua_uk.
@Lokal_Profil What do you think ? Is there a proper way to map to wd_item and having it nullable somehow ? Or shall we just revert that mapping?
JeanFred added a comment to T346681: Known fields for ua_uk.
Checked the logs quickly:
ERROR: Unknown error occurred when processing country ua in lang uk (1048, "Column 'wd_item' cannot be null")
So, yeah, this is definitely linked to 0a8c490 :-/
Oct 17 2023
Oct 17 2023
JeanFred committed R2566:7ee0cbbc01a6: Fix incorrect SPARQL syntax in Sitelink's `get_filter_for_negative_query`.
Fix incorrect SPARQL syntax in Sitelink's `get_filter_for_negative_query`
Fix relative path to `wikiprojects.json` static file
JeanFred committed R2566:82f90c49aaac: Add unit tests for `get_query_for_items_for_property_[posi|nega]tive` for….
Add unit tests for `get_query_for_items_for_property_[posi|nega]tive` for…
Support for sitelink name in queries Jinja template
JeanFred committed R2566:d7f09b390b7e: Add unit tests for positive/negative_query for all Line objects.
Add unit tests for positive/negative_query for all Line objects
Add support for sitelink column
Add Sitelink column to the PropertyStatisticsTest fixtures
Oct 12 2023
Oct 12 2023
Oct 5 2023
Oct 5 2023
JeanFred added a comment to T312726: Add integraality column for sitelinks.
Syntax-wise, I checked for inspiration the JSON rendering (example) − the sitelinks are keyed as “frwiki” or “bnwikivoyage“. That looks good enough ; but can I then use that in a SPARQL query? The SPARQL seems to use URLs, eg
?article schema:about ?item. ?article schema:isPartOf <https://en.wikipedia.org/>.
Oct 3 2023
Oct 3 2023
JeanFred committed R2566:3ba5e84e6da0: Move HTML formatting of predicates to the GroupingConfiguration objects.
Move HTML formatting of predicates to the GroupingConfiguration objects
JeanFred committed R2566:220d51e56eaa: Retire the use of `grouping property` for `grouping predicate`.
Retire the use of `grouping property` for `grouping predicate`
Explicitly support only Pxyz as grouping property
Add support for grouping by any Wikibase predicate
JeanFred committed R2566:da9982058cd0: Create grouping configuration based on the property datatype.
Create grouping configuration based on the property datatype
Oct 2 2023
Oct 2 2023
Closing as invalid, as there is nothing much I can do there from the service side.
JeanFred moved T347897: integraality column for EntitySchema compliance? from Backlog to Enhancements on the Tool-inteGraality board.
JeanFred committed R2566:3f9a3d5193c5: Do not crash on redirected items when formatting grouping links.
Do not crash on redirected items when formatting grouping links
Sep 27 2023
Sep 27 2023
JeanFred committed R2566:07ef2b53c351: Do not add higher grouping empty column when there is no higher grouping.
Do not add higher grouping empty column when there is no higher grouping
Sep 26 2023
Sep 26 2023
JeanFred committed R2566:ac64840b866c: Move SPARQL-formatting code from `get_query_for_items_for_property_negative` to….
Move SPARQL-formatting code from `get_query_for_items_for_property_negative` to…
Leverage GroupingConfiguration in Column.get_info_query
JeanFred committed R2566:06cb52d776e9: Move SPARQL-formatting code from `get_query_for_items_for_property_positive` to….
Move SPARQL-formatting code from `get_query_for_items_for_property_positive` to…
Remove redundant attributes from PropertyStatistics
JeanFred committed R2566:3e3f47fc9d08: Add GroupingConfiguration object as attribute to PropertyStatistics.
Add GroupingConfiguration object as attribute to PropertyStatistics
Move some SPARQL-related things to `sparql_utils.py`
JeanFred committed R2566:e575508be446: Move `get_grouping_information` logic to GroupingConfiguration.
Move `get_grouping_information` logic to GroupingConfiguration
Introduce GroupingConfiguration concept tree
Sep 18 2023
Sep 18 2023
JeanFred created T346681: Known fields for ua_uk.
JeanFred closed T231796: 502 Bad Gateway error in wikiloves images view when passing a user as Resolved.
Four years later, finally took the time to look into it properly :)
Sep 6 2023
Sep 6 2023
Add tox target to reformat Python files using Black
Aug 29 2023
Aug 29 2023
Aug 28 2023
Aug 28 2023
Harvesting has been stable for a few days now − closing as Resolved 🎉
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
Aug 27 2023
Aug 27 2023
JeanFred added a comment to T249678: Add OAuth login to mailman for accessing list memberships/archive viewing.
Aug 25 2023
Aug 25 2023
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
@Lokal_Profil Thanks! I have the two STRICT_TRANS_TABLES open patches manually applied on the server, so I’ll only be able to deploy your changes once they are merged.
Aug 24 2023
Aug 24 2023
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
Grepping through the logs for errors, only 2:
ERROR: Unknown error occurred when processing country de-he in lang de (1048, "Column 'wd_item' cannot be null")
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
Monuments Database is back to 1.7M monuments 🎉 https://commons.wikimedia.org/wiki/Commons:Monuments_database/Statistics
Aug 23 2023
Aug 23 2023
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
Harvesting ran today without issue − until the very last step:
2023-08-23_18:22:51 Update monuments_all table... ERROR 1292 (22007) at line 514: Truncated incorrect DECIMAL value: ''
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
Edited the code in place on Toolforge to add a sql_mode argument to the pymysql connection object. If that works out, I’ll submit a Gerrit patch.
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
Ah, so the SQL mode setting simply does not stick. Running again:
MariaDB [s51138__heritage_p]> SELECT @@SQL_MODE, @@GLOBAL.SQL_MODE; +-------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------+ | @@SQL_MODE | @@GLOBAL.SQL_MODE | +-------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------+ | STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION | STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION | +-------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------+
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
Hmmm, running SELECT @@SQL_MODE, @@GLOBAL.SQL_MODE; again in s51138__heritage_p I’m getting
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
Harvesting ran over night − still 84 errors :/
ERROR: Unknown error occurred when processing country ir in lang fa (1406, "Data too long for column 'image' at row 1") -- ERROR: Unknown error occurred when processing country se-arbetsl in lang sv (1406, "Data too long for column 'id' at row 1") -- ERROR: Unknown error occurred when processing country de-nrw-bm in lang de (1406, "Data too long for column 'beschreibung' at row 1") -- ERROR: Unknown error occurred when processing country es-ct in lang ca (1265, "Data truncated for column 'prot' at row 1") -- ERROR: Unknown error occurred when processing country ro in lang ro (1406, "Data too long for column 'adresa' at row 1") -- ERROR: Unknown error occurred when processing country be-vlg in lang fr (1406, "Data too long for column 'classement' at row 1") -- ERROR: Unknown error occurred when processing country ie in lang en (1265, "Data truncated for column 'number' at row 1") -- ERROR: Unknown error occurred when processing country hu in lang hu (1265, "Data truncated for column 'site' at row 1") -- ERROR: Unknown error occurred when processing country ch2 in lang de (1406, "Data too long for column 'fotobeschreibung' at row 1") -- ERROR: Unknown error occurred when processing country gb-eng in lang en (1406, "Data too long for column 'name' at row 1") -- ERROR: Unknown error occurred when processing country rs in lang sr (1265, "Data truncated for column 'site' at row 1") -- ERROR: Unknown error occurred when processing country be-wal in lang fr (1406, "Data too long for column 'nom_objet' at row 1") -- ERROR: Unknown error occurred when processing country uy in lang es (1406, "Data too long for column 'monumento' at row 1") -- ERROR: Unknown error occurred when processing country es in lang ca (1265, "Data truncated for column 'prot' at row 1") -- ERROR: Unknown error occurred when processing country gb-nir in lang en (1406, "Data too long for column 'hb' at row 1") -- ERROR: Unknown error occurred when processing country aq in lang en (1406, "Data too long for column 'description' at row 1") -- ERROR: Unknown error occurred when processing country ch-old in lang en (1265, "Data truncated for column 'kgs_nr' at row 1") -- ERROR: Unknown error occurred when processing country no in lang no (1265, "Data truncated for column 'id' at row 1") -- ERROR: Unknown error occurred when processing country us-ca in lang en (1265, "Data truncated for column 'refnum' at row 1") -- ERROR: Unknown error occurred when processing country fr in lang fr (1406, "Data too long for column 'notice' at row 1") -- ERROR: Unknown error occurred when processing country it-bz in lang de (1406, "Data too long for column 'beschreibung' at row 1") -- ERROR: Unknown error occurred when processing country ca-prov in lang en (1366, "Incorrect integer value: '––' for column `s51138__heritage_p`.`monuments_ca-prov_(en)`.`idm` at row 1") -- ERROR: Unknown error occurred when processing country th in lang th (1265, "Data truncated for column 'site' at row 1") -- ERROR: Unknown error occurred when processing country il in lang he (1406, "Data too long for column 'description' at row 1") -- ERROR: Unknown error occurred when processing country de-he in lang de (1406, "Data too long for column 'beschreibung' at row 1") -- ERROR: Unknown error occurred when processing country pt in lang pt (1406, "Data too long for column 'designacoes' at row 1") -- ERROR: Unknown error occurred when processing country fr-object in lang fr (1406, "Data too long for column 'description' at row 1") -- ERROR: Unknown error occurred when processing country mx in lang es (1406, "Data too long for column 'id' at row 1") -- ERROR: Unknown error occurred when processing country be-bru in lang nl (1406, "Data too long for column 'bouwdoor' at row 1") -- ERROR: Unknown error occurred when processing country au in lang en (1366, "Incorrect double value: '' for column `s51138__heritage_p`.`monuments_au_(en)`.`lon` at row 35") -- ERROR: Unknown error occurred when processing country be-wal in lang nl (1406, "Data too long for column 'descr_nl' at row 1") -- ERROR: Unknown error occurred when processing country gb-sct in lang en (1265, "Data truncated for column 'hb' at row 1") -- ERROR: Unknown error occurred when processing country es-vc in lang ca (1265, "Data truncated for column 'prot' at row 1") -- ERROR: Unknown error occurred when processing country hr in lang hr (1406, "Data too long for column 'arhitekt' at row 1") -- ERROR: Unknown error occurred when processing country jp-nhs in lang en (1406, "Data too long for column 'comments' at row 1") -- ERROR: Unknown error occurred when processing country za in lang en (1406, "Data too long for column 'description' at row 1") -- ERROR: Unknown error occurred when processing country pa in lang es (1406, "Data too long for column 'descripcion' at row 1") -- ERROR: Unknown error occurred when processing country ar in lang es (1406, "Data too long for column 'direccion' at row 1") -- ERROR: Unknown error occurred when processing country sr in lang commons (1366, "Incorrect double value: '' for column `s51138__heritage_p`.`monuments_sr_(nl)`.`lon` at row 2") -- ERROR: Unknown error occurred when processing country pl in lang pl (1406, "Data too long for column 'nazwa' at row 1") -- ERROR: Unknown error occurred when processing country in in lang en (1406, "Data too long for column 'description' at row 1") -- ERROR: Unknown error occurred when processing country es in lang es (1406, "Data too long for column 'lugar' at row 1") -- ERROR: Unknown error occurred when processing country wlpa-es-ct in lang ca (1406, "Data too long for column 'descripcio' at row 1") -- ERROR: Unknown error occurred when processing country be-wal in lang en (1406, "Data too long for column 'descr_nl' at row 1") -- ERROR: Unknown error occurred when processing country us in lang en (1406, "Data too long for column 'description' at row 1") -- ERROR: Unknown error occurred when processing country at in lang de (1406, "Data too long for column 'beschreibung' at row 1") -- ERROR: Unknown error occurred when processing country dk-bygning in lang da (1265, "Data truncated for column 'systemnrbyg' at row 1") -- ERROR: Unknown error occurred when processing country by in lang be-tarask (1406, "Data too long for column 'name' at row 1") -- ERROR: Unknown error occurred when processing country eg in lang ar (1406, "Data too long for column 'description' at row 1") -- ERROR: Unknown error occurred when processing country de-nrw-k in lang de (1406, "Data too long for column 'beschreibung' at row 1") -- ERROR: Unknown error occurred when processing country mt in lang de (1406, "Data too long for column 'beschreibung' at row 1") -- ERROR: Unknown error occurred when processing country ug in lang en (1406, "Data too long for column 'description' at row 1") -- ERROR: Unknown error occurred when processing country ca-fed in lang en (1406, "Data too long for column 'address' at row 1") -- ERROR: Unknown error occurred when processing country tn in lang fr (1406, "Data too long for column 'monument' at row 1") -- ERROR: Unknown error occurred when processing country il-npa in lang he (1406, "Data too long for column 'description' at row 1") -- ERROR: Unknown error occurred when processing country nl in lang nl (1265, "Data truncated for column 'type_obj' at row 1") -- ERROR: Unknown error occurred when processing country cn in lang en (1406, "Data too long for column 'designation' at row 1") -- ERROR: Unknown error occurred when processing country be-vlg in lang en (1406, "Data too long for column 'address' at row 1") -- ERROR: Unknown error occurred when processing country pe in lang es (1406, "Data too long for column 'direccion' at row 1") -- ERROR: Unknown error occurred when processing country ee in lang et (1406, "Data too long for column 'aadress' at row 1") -- ERROR: Unknown error occurred when processing country nl-gem in lang nl (1406, "Data too long for column 'objnr' at row 1") -- ERROR: Unknown error occurred when processing country ph in lang en (1406, "Data too long for column 'description' at row 1") -- ERROR: Unknown error occurred when processing country it in lang it (1265, "Data truncated for column 'site' at row 1") -- ERROR: Unknown error occurred when processing country gh in lang en (1406, "Data too long for column 'id' at row 1") -- ERROR: Unknown error occurred when processing country sk in lang de (1406, "Data too long for column 'beschreibung-de' at row 1") -- ERROR: Unknown error occurred when processing country be-vlg in lang nl (1406, "Data too long for column 'adres' at row 1") -- ERROR: Unknown error occurred when processing country de-by in lang de (1406, "Data too long for column 'beschreibung' at row 1") -- ERROR: Unknown error occurred when processing country iq in lang ar (1406, "Data too long for column 'description' at row 1") -- ERROR: Unknown error occurred when processing country es-gl in lang gl (1406, "Data too long for column 'notas' at row 1") -- ERROR: Unknown error occurred when processing country fr in lang ca (1265, "Data truncated for column 'prot' at row 1") -- ERROR: Unknown error occurred when processing country cz in lang cs (1406, "Data too long for column 'description' at row 1") -- ERROR: Unknown error occurred when processing country dk-fortids in lang da (1265, "Data truncated for column 'fredningsnummer' at row 1") -- ERROR: Unknown error occurred when processing country am in lang hy (1406, "Data too long for column 'id' at row 1") -- ERROR: Unknown error occurred when processing country gb-wls in lang en (1406, "Data too long for column 'notes' at row 1") -- ERROR: Unknown error occurred when processing country wlpa-at in lang de (1406, "Data too long for column 'Beschreibung' at row 1") -- ERROR: Unknown error occurred when processing country ru in lang ru (1406, "Data too long for column 'description' at row 1") -- ERROR: Unknown error occurred when processing country ua in lang uk (1265, "Data truncated for column 'site' at row 1") -- ERROR: Unknown error occurred when processing country cl in lang es (1366, "Incorrect integer value: 'S/N' for column `s51138__heritage_p`.`monuments_cl_(es)`.`id` at row 1") -- ERROR: Unknown error occurred when processing country ch in lang de (1406, "Data too long for column 'anzeige-adresse' at row 1") -- ERROR: Unknown error occurred when processing country pt-wd in lang pt (1366, "Incorrect double value: '' for column `s51138__heritage_p`.`monuments_pt-wd_(pt)`.`lon` at row 3") -- ERROR: Unknown error occurred when processing country co in lang es (1406, "Data too long for column 'id' at row 1") -- ERROR: Unknown error occurred when processing country ca-muni in lang en (1366, "Incorrect integer value: '––' for column `s51138__heritage_p`.`monuments_ca-muni_(en)`.`idm` at row 1") -- ERROR 1292 (22007) at line 514: Truncated incorrect DECIMAL value: '' 2023-08-23_04:30:04 Restart the categorization job... -- ERROR: Unknown error occurred when processing country in-com in lang commons Language 'commons' does not exist in family wikipedia
Aug 22 2023
Aug 22 2023
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
(The annoying thing is that I don’t have yet support to test such things in my local docker-compose setup)
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
Aug 21 2023
Aug 21 2023
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
It seems to me that "Data too long for column X" means bad data in the source tables. The harvesting used to ignore that, now it does not. The proper fix is to correct the source data, but we can’t do all that.
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
One example of
(1406, "Data too long for column 'image' at row 1")
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
One example of
(1366, "Incorrect double value: '' for column s51138__heritage_p.monuments_pk_(en).lon at row 1")
would be
REPLACE INTO monuments_pk_(en) (source, number, prov_iso, description, address, district, lon, monument_article, registrant_url) VALUES (//en.wikipedia.org/w/index.php?title=List_of_cultural_heritage_sites_in_Balochistan,_Pakistan&oldid=1139504719, BA-2, PK-BA, [[Nindo Damb]], Ornach Valley, Tehsil Wadh, [[Killa Abdullah District]], , Nindo_Damb, BA-2)
JeanFred added a comment to T341773: Update queries based on templatelinks migration to use linktaget.
@Bodhisattwa Could you take care of adding the necessary configuration for the remaining years ?
Jul 4 2023
Jul 4 2023
JeanFred added a comment to T332899: [EPIC] Migrate selected R-based Wikidata products .
Hey @AndrewTavis_WMDE & @Manuel , I only used the dashboard to get a nice visualization of the external ID galaxy − see this slide (the red bubble is the video-game related IDs)
Jun 20 2023
Jun 20 2023
JeanFred updated subscribers of T332899: [EPIC] Migrate selected R-based Wikidata products .
I was pointed to this ticket by @Lydia_Pintscher: I wanted to update a presentation slide that uses https://wikidata-analytics.wmcloud.org/app_direct/WD_ExternalIdentifiersDashboard (that’s the only dashboard I can remember using)
Jun 14 2023
Jun 14 2023
JeanFred added a comment to T338987: Multiple DB-related errors during harvesting.
Likely related to the MariaDB upgrade T301949
Jun 13 2023
Jun 13 2023
Looks like this is happening since April 7th
Jun 11 2023
Jun 11 2023
JeanFred committed R2566:9712666babc4: Guard against empty result in `populate_groupings` when getting the counts for….
Guard against empty result in `populate_groupings` when getting the counts for…
Extract method `populate_groupings` from `retrieve_data`
Include cells contents in Grouping equality operator
JeanFred added a comment to T338684: Hard crash when configured with a column (property) that is never used on any items.
Confirmed that https://www.wikidata.org/w/index.php?title=User%3AAmadalvarez%2Fdashboard%2Fmotociclisme&diff=1912981582&oldid=1912428855 was updated correctly.