Page MenuHomePhabricator
Create Task
Maniphest T382009

Fulltext search is broken on wikidata.beta.wmflabs.org
Closed, ResolvedPublic
Actions

Description

Fulltext search on Beta Wikidata is broken:
https://wikidata.beta.wmflabs.org/w/api.php?action=query&list=search&srsearch=test responds

{
    "error": {
        "code": "cirrussearch-parse-error",
        "info": "Query was not understood. Please make it simpler. The query was logged to improve the search system.",
        "*": "See https://wikidata.beta.wmflabs.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/> for notice of API deprecations and breaking changes."
    },
    "servedby": "deployment-mediawiki13"
}

Special:Search shows the same error. Prefix search via wbsearchentities works fine.

The logs for this request contain the following error message:

Search backend error during entity_full_text search for 'test' after 60: Parse error on Cannot search on field [descriptions.mul] since it is not indexed.

https://beta-logs.wmcloud.org/app/discover#/doc/5f0c9be0-0b6f-11ec-9cde-3f4490e09a26/logstash-mediawiki-1-7.0.0-1-2024.12.11?id=fy-CtpMBbBwOTv6eUzuZ (see https://wikitech.wikimedia.org/wiki/OpenSearch_Dashboards#Beta_Cluster_Logstash for credentials)
The descriptions.mul part of the error message is interesting, because the "mul" language code is only valid for labels and aliases, but not for descriptions.

Related Objects

Event Timeline

Jakob_WMDE created this task.Dec 11 2024, 4:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 11 2024, 4:57 PM
Jakob_WMDE updated the task description. (Show Details)Dec 11 2024, 4:58 PM
• Muhammad_Yasser_Jazirahly_WMDE subscribed.Dec 11 2024, 4:59 PM
Lucas_Werkmeister_WMDE subscribed.EditedDec 11 2024, 5:04 PM

Might have been caused by our EntitySchema search work (T375641 and subtasks), but I feel like the search team will know better how to investigate / debug this, to be honest 😅

Can also be reproduced on Special:Search: https://wikidata.beta.wmflabs.org/w/index.php?search=search – copying the cirrusDumpQuery in case it’s useful (cirrusDumpResult errors out):

{
    "__main__": {
        "description": "entity_full_text search for 'search'",
        "path": "wikidatawiki_content\/_search",
        "params": {
            "timeout": "10s"
        },
        "query": {
            "track_total_hits": true,
            "_source": [
                "namespace",
                "title",
                "namespace_text",
                "wiki",
                "timestamp",
                "sitelink_count",
                "statement_count",
                "labels.en",
                "descriptions.en",
                "labels.mul",
                "descriptions.mul"
            ],
            "fields": [],
            "query": {
                "bool": {
                    "should": [
                        {
                            "term": {
                                "title.keyword": "SEARCH"
                            }
                        },
                        {
                            "bool": {
                                "minimum_should_match": 0,
                                "filter": [
                                    {
                                        "bool": {
                                            "minimum_should_match": 1,
                                            "should": [
                                                {
                                                    "match": {
                                                        "all": {
                                                            "query": "search",
                                                            "operator": "AND"
                                                        }
                                                    }
                                                },
                                                {
                                                    "match": {
                                                        "all.plain": {
                                                            "query": "search",
                                                            "operator": "AND"
                                                        }
                                                    }
                                                },
                                                {
                                                    "match": {
                                                        "descriptions.en": {
                                                            "query": "search",
                                                            "operator": "AND"
                                                        }
                                                    }
                                                },
                                                {
                                                    "match": {
                                                        "labels.en": {
                                                            "query": "search",
                                                            "operator": "AND"
                                                        }
                                                    }
                                                },
                                                {
                                                    "match": {
                                                        "descriptions.mul": {
                                                            "query": "search",
                                                            "operator": "AND"
                                                        }
                                                    }
                                                },
                                                {
                                                    "match": {
                                                        "labels.mul": {
                                                            "query": "search",
                                                            "operator": "AND"
                                                        }
                                                    }
                                                }
                                            ]
                                        }
                                    }
                                ],
                                "should": [
                                    {
                                        "dis_max": {
                                            "tie_breaker": 0,
                                            "queries": [
                                                {
                                                    "constant_score": {
                                                        "filter": {
                                                            "match": {
                                                                "labels.en.near_match": "search"
                                                            }
                                                        },
                                                        "boost": 0.78
                                                    }
                                                },
                                                {
                                                    "constant_score": {
                                                        "filter": {
                                                            "match": {
                                                                "labels.en.near_match_folded": "search"
                                                            }
                                                        },
                                                        "boost": 0.01
                                                    }
                                                },
                                                {
                                                    "constant_score": {
                                                        "filter": {
                                                            "match": {
                                                                "labels.mul.near_match": "search"
                                                            }
                                                        },
                                                        "boost": 0.038000000000000006
                                                    }
                                                },
                                                {
                                                    "constant_score": {
                                                        "filter": {
                                                            "match": {
                                                                "labels.mul.near_match_folded": "search"
                                                            }
                                                        },
                                                        "boost": 0.0005
                                                    }
                                                }
                                            ]
                                        }
                                    },
                                    {
                                        "bool": {
                                            "minimum_should_match": 1,
                                            "should": [
                                                {
                                                    "match": {
                                                        "all": {
                                                            "query": "search",
                                                            "operator": "OR",
                                                            "boost": 0.04
                                                        }
                                                    }
                                                },
                                                {
                                                    "match": {
                                                        "all.plain": {
                                                            "query": "search",
                                                            "operator": "OR",
                                                            "boost": 0.04
                                                        }
                                                    }
                                                },
                                                {
                                                    "dis_max": {
                                                        "tie_breaker": 0.2,
                                                        "queries": [
                                                            {
                                                                "match": {
                                                                    "labels.en.plain": {
                                                                        "query": "search",
                                                                        "boost": 0.07
                                                                    }
                                                                }
                                                            },
                                                            {
                                                                "match": {
                                                                    "descriptions.en.plain": {
                                                                        "query": "search",
                                                                        "boost": 0.07
                                                                    }
                                                                }
                                                            },
                                                            {
                                                                "match": {
                                                                    "labels.en": {
                                                                        "query": "search",
                                                                        "boost": 0.07
                                                                    }
                                                                }
                                                            },
                                                            {
                                                                "match": {
                                                                    "descriptions.en": {
                                                                        "query": "search",
                                                                        "boost": 0.07
                                                                    }
                                                                }
                                                            },
                                                            {
                                                                "match": {
                                                                    "labels.mul.plain": {
                                                                        "query": "search",
                                                                        "boost": 0.003
                                                                    }
                                                                }
                                                            },
                                                            {
                                                                "match": {
                                                                    "descriptions.mul.plain": {
                                                                        "query": "search",
                                                                        "boost": 0.003
                                                                    }
                                                                }
                                                            },
                                                            {
                                                                "match": {
                                                                    "descriptions.mul": {
                                                                        "query": "search",
                                                                        "boost": 0.003
                                                                    }
                                                                }
                                                            },
                                                            {
                                                                "match": {
                                                                    "labels.mul": {
                                                                        "query": "search",
                                                                        "boost": 0.003
                                                                    }
                                                                }
                                                            }
                                                        ]
                                                    }
                                                }
                                            ]
                                        }
                                    }
                                ]
                            }
                        }
                    ],
                    "minimum_should_match": 1,
                    "filter": [
                        {
                            "terms": {
                                "namespace": [
                                    0,
                                    120
                                ]
                            }
                        }
                    ]
                }
            },
            "highlight": {
                "pre_tags": [
                    "\ue000"
                ],
                "post_tags": [
                    "\ue001"
                ],
                "fields": {
                    "title": {
                        "type": "experimental",
                        "fragmenter": "none",
                        "number_of_fragments": 0,
                        "matched_fields": [
                            "title.keyword"
                        ]
                    },
                    "labels.en.plain": {
                        "type": "experimental",
                        "fragmenter": "none",
                        "number_of_fragments": 0,
                        "options": {
                            "skip_if_last_matched": true,
                            "return_snippets_and_offsets": true
                        }
                    },
                    "descriptions.en.plain": {
                        "type": "experimental",
                        "fragmenter": "none",
                        "number_of_fragments": 0,
                        "options": {
                            "skip_if_last_matched": true
                        }
                    },
                    "labels.mul.plain": {
                        "type": "experimental",
                        "fragmenter": "none",
                        "number_of_fragments": 0,
                        "options": {
                            "skip_if_last_matched": true,
                            "return_snippets_and_offsets": true
                        }
                    },
                    "descriptions.mul.plain": {
                        "type": "experimental",
                        "fragmenter": "none",
                        "number_of_fragments": 0,
                        "options": {
                            "skip_if_last_matched": true
                        }
                    },
                    "labels.*.plain": {
                        "type": "experimental",
                        "fragmenter": "none",
                        "number_of_fragments": 0,
                        "options": {
                            "skip_if_last_matched": true,
                            "return_snippets_and_offsets": true
                        }
                    },
                    "descriptions.*.plain": {
                        "type": "experimental",
                        "fragmenter": "none",
                        "number_of_fragments": 0,
                        "options": {
                            "skip_if_last_matched": true
                        }
                    }
                }
            },
            "stats": [
                "entity_full_text",
                "simple_bag_of_words"
            ],
            "rescore": [
                {
                    "window_size": 512,
                    "query": {
                        "query_weight": 1,
                        "rescore_query_weight": 10,
                        "rescore_query": {
                            "token_count_router": {
                                "text": "search",
                                "fallback": {
                                    "match_none": {}
                                },
                                "field": "text",
                                "conditions": [
                                    {
                                        "gt": 1,
                                        "query": {
                                            "multi_match": {
                                                "type": "phrase",
                                                "slop": 0,
                                                "fields": [
                                                    "all^0.001",
                                                    "all.plain^0.01"
                                                ],
                                                "query": "search"
                                            }
                                        }
                                    },
                                    {
                                        "gt": 10,
                                        "query": {
                                            "match_none": {}
                                        }
                                    }
                                ]
                            }
                        }
                    }
                },
                {
                    "window_size": 8192,
                    "query": {
                        "query_weight": 1,
                        "rescore_query_weight": 2,
                        "score_mode": "total",
                        "rescore_query": {
                            "function_score": {
                                "score_mode": "sum",
                                "functions": [
                                    {
                                        "script_score": {
                                            "script": {
                                                "source": "pow(doc['incoming_links'].value , 1) \/ ( pow(doc['incoming_links'].value, 1) + pow(100,1))",
                                                "lang": "expression"
                                            }
                                        },
                                        "weight": 0.6
                                    },
                                    {
                                        "script_score": {
                                            "script": {
                                                "source": "pow(doc['sitelink_count'].value , 2) \/ ( pow(doc['sitelink_count'].value, 2) + pow(20,2))",
                                                "lang": "expression"
                                            }
                                        },
                                        "weight": 0.4
                                    }
                                ]
                            }
                        }
                    }
                }
            ],
            "size": 21
        },
        "options": {
            "timeout": "10s"
        }
    }
}
Lucas_Werkmeister_WMDE added a project: [Archived]Wikidata Dev Team.Dec 11 2024, 5:05 PM
Lucas_Werkmeister_WMDE added a project: wmde-wikidata-tech.
Lucas_Werkmeister_WMDE added a comment.Dec 11 2024, 5:07 PM
In T382009#10398041, @Lucas_Werkmeister_WMDE wrote:

Might have been caused by our EntitySchema search work (T375641 and subtasks), but I feel like the search team will know better how to investigate / debug this, to be honest 😅

Specifically, I have no idea how to tell if this is due to: newest code changes on master that are broken and will also break in prod once the train resumes next week; some indexing being missing on beta (T376252#10382852?); or some other reason.

Jakob_WMDE updated the task description. (Show Details)Dec 11 2024, 5:16 PM
dcausse added a comment.Dec 11 2024, 5:57 PM

I'm reindexing wikidata there, if it's caused by the mul language code this should hopefully fix the issue.

dcausse closed this task as Resolved.Dec 11 2024, 6:05 PM
dcausse claimed this task.
dcausse moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.

seems to be working OK now

Jakob_WMDE added a comment.Dec 12 2024, 8:54 AM

@dcausse that was quick, thanks!

Lucas_Werkmeister_WMDE added a comment.Dec 12 2024, 12:17 PM

Thank you!

Etonkovidova mentioned this in T389449: [commons-beta] Special:MediaSearch displays "Inavlid search. Query was not understood" for any search.Mar 20 2025, 1:18 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits