User Details
- User Since
- Jun 9 2015, 9:03 AM (457 w, 6 d)
- Availability
- Available
- IRC Nick
- dcausse
- LDAP User
- DCausse
- MediaWiki User
- DCausse (WMF) [ Global Accounts ]
Fri, Mar 8
Thu, Mar 7
Discussed the issue today with @JAllemandou and the reason is that CirrusSearch in some circonstances might send these outdated events, we will fix the root cause (T359580) and in the meantime these alerts for this dataset can be ignored.
Tue, Mar 5
changed the layout of the query a bit by moving the logistic function introduced in T271799 to the top-level so that it wraps the new nearmatch clause
Mon, Mar 4
Compiled 10 real world examples at https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_graph_split/Federated_Queries_Examples
final report available at https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/WDQS_Graph_Split_Impact_Analysis
@Physikerwelt thanks for your feedback.
The new builder moved the result to #4 which is better but still not enough and it's beaten by 3 other images because other criteria:
- weighted_tags:image.linked.from.wikipedia.lead_image/Q458
- statement_keywords:p180=q458
Fri, Mar 1
Since the upgrade I believe that we are affected by https://github.com/ether/etherpad-lite/issues/5401. Wondering if a stale config.json file got kept with padOptions.userName & userColor set to false instead of null.
Thu, Feb 29
Mon, Feb 26
Tue, Feb 20
Feb 9 2024
Feb 8 2024
Feb 2 2024
@mfossati OK resumed the dag to process 2024-01-15, marked 2024-01-22 explicitly as failed while you figure out what's going on there.
@mfossati if 2024-01-22 is running it probably means that it's comparing against an index that do not have 2024-01-15 suggestions, should I skip 2024-01-15 and let the dag pick 2024-01-22?
WIP:
- included the new 100k queries sample named QUERY-Q4 from T349512 (random sample that is representative of the query length and runtime)
- the % of affected queries (deduplicated) per tool is (sample being the QUERY-Q4 sample mentionned above)
The above graph should be taken with a grain of salt as the number of queries per datapoints varies a lot (86 queries for Listeria vs 85k for random), these numbers are being reviewed so no conclusions should be drawn yet but it does not seem that we obtain the same numbers that were found originally in Wikidata_Subgraph_Query_Analysis where 2.5% of the total query count are being identified as requiring scholarly articles.
A more qualitative analysis is in progress:
- analyze of the user agents to understand what usecases are mainly affected, preliminary results show that for instance a single UA is the cause of 50% of the affected queries
- extract some SPARQL queries to start evaluating how federation could be applied/tested
@dr0ptp4kt thanks! is the difference in the number of successful queries only explained by the improvement in query time or are there some improvements in the number of queries that timeout as well?
Feb 1 2024
The airflow dag (search) image_suggestions_weekly has been paused today
Jan 31 2024
Jan 30 2024
Scanning dumps from 2024/01/21 we can find 1623 duplicated statement ids (full list here: https://people.wikimedia.org/~dcausse/T356161_sdc_duplicated_statement_ids.csv)
@Lucas_Werkmeister_WMDE thanks for all the context! I get that it only affects WikibaseMediaInfo. Can we exclude Wikibase as a culprit possibly affecting wikidata or should we run a quick investigation to find possible duplicated statement identifiers in the wikidata RDF dumps?
Jan 29 2024
Jan 26 2024
I worked with Paladox in the past on the gerrit codebase, I trust him to use his +2 rights wisely.
WIP: https://people.wikimedia.org/~dcausse/T355040_EARLY_DRAFT_wdqs_query_results_analysis.html (UA redacted for now)
Jan 25 2024
Jan 24 2024
Nice improvements seen on the young GC rate after deploying the change:
Jan 19 2024
Quick report on the progress being made:
- Our query logs do not only contain sparql queries and the sparql client used to collect the data has to be adapted to support these (ASK, CONSTRUCT, DESCRIBE) (https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/991622)
- Getting failures due to response size, bumped the limit to 16M but still getting problems, I might stop here and simply tag & ignore such massive queries moving forward
- Getting very bad numbers from Listeria and MixNMatch (34% and 17% identical respectively), avg result size is 1.6k and 8k so might explain partly why getting identical results is difficult, need more investigations to understand the cause...
- Getting pretty mediocre numbers for WikidataIntegrator at 88% with very small avg result size at 8, more investigation needed
- Pywikibot and SPARQLWrapper are good at 99.4% for both
Jan 18 2024
Jan 16 2024
tools.jar should only be in jdk8, I'm surprised that this problem did not occur while sonar was running java11
Jan 15 2024
Jan 9 2024
Selecting only namespace=6 does trigger the MediaSearch query profile which does not include the all_near_match field which is the one helping the most to rank almost perfect title matches to the top.
I believe that the fix would be to fix the MediaSearch query builder to include an optional clause on the all_near_match field.
@Cparle do you remember if not including all_near_match was done on purpose and if it would break any existing usecases to add it?
Jan 8 2024
Therefore, we should use mw-api-int-async (w/o retries) instead of mw-api-int-async-ro
Jan 5 2024
Jan 4 2024
Closed as a duplicate of T299290, quickly testing it seems that the 502 is triggered depending on the query size:
select * { service <https://lingualibre.org/sparql> { ?e <https://lingualibre.org/prop/direct/P3> <aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa> . } }
will work but adding another "a" will fail the query, this problem is reproduced at T299290#7667535 directly using the lingualibre endpoint without using federation.
Dec 19 2023
Numbers look correct:
Dec 18 2023
Dec 15 2023
Dec 14 2023
Bumping envoy resources did help a bit, cpu throttling is reduced (still a bit present tho):