Page MenuHomePhabricator

illegal_argument_exception: Mapper for [heading] conflicts with existing mapping in other types
Closed, ResolvedPublic

Description

I'm running updateSearchIndexConfig.php to create new search index mappings (on wikibase client, so no special wikibase stuff) and the script fails:

illegal_argument_exception: Mapper for [heading] conflicts with existing mapping in other types:
[mapper [heading] has different [omit_norms] values, cannot change from disable to enabled]

Event Timeline

aude created this task.Jul 29 2016, 4:08 PM
Restricted Application added projects: Discovery, Discovery-Search. · View Herald TranscriptJul 29 2016, 4:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

@Smalyshev probably related to your patch. I'm trying to remember, but i think the change in norms here was intentional?

aude added a comment.Jul 29 2016, 4:13 PM

I don't think i had existing mappings for the wiki, so the error message may be misleading

Re reading the exception message, something is odd here. When it says conflicts with existing mapping in other types, that's referring to the two top level types we use: page and namespace. The namespace mapping only has two fields though, wiki and name. There is no heading to conflict with.

Might you be able to post the result of the following in a paste:

curl localhost:9200/<wikiid>/_mapping?pretty

aude added a comment.Jul 29 2016, 4:29 PM

okay, this happens when updating the mapping

aude added a comment.Jul 29 2016, 4:33 PM
{
"enwiki_content_first": {
"mappings": {
"page": {
"dynamic": "false",
"_all": {
"enabled": false
},
"properties": {
"all": {
"type": "string",
"term_vector": "with_positions_offsets",
"similarity": "default",
"fields": {
"plain": {
"type": "string",
"term_vector": "with_positions_offsets",
"similarity": "default",
"analyzer": "plain",
"search_analyzer": "plain_search",
"position_increment_gap": 10
}
},
"analyzer": "text",
"search_analyzer": "text_search",
"position_increment_gap": 10
},
"all_near_match": {
"type": "string",
"norms": {
"enabled": false
},
"index_options": "freqs",
"similarity": "default",
"fields": {
"asciifolding": {
"type": "string",
"norms": {
"enabled": false
},
"index_options": "freqs",
"similarity": "default",
"analyzer": "near_match_asciifolding",
"position_increment_gap": 10
}
},
"analyzer": "near_match",
"position_increment_gap": 10
},
"auxiliary_text": {
"type": "string",
"term_vector": "with_positions_offsets",
"similarity": "default",
"fields": {
"plain": {
"type": "string",
"term_vector": "with_positions_offsets",
"similarity": "default",
"analyzer": "plain",
"search_analyzer": "plain_search",
"position_increment_gap": 10
}
},
"copy_to": [
"all"
],
"analyzer": "text",
"search_analyzer": "text_search",
"position_increment_gap": 10
},
"category": {
"type": "string",
"term_vector": "with_positions_offsets",
"similarity": "default",
"fields": {
"lowercase_keyword": {
"type": "string",
"norms": {
"enabled": false
},
"index_options": "docs",
"similarity": "default",
"analyzer": "lowercase_keyword",
"position_increment_gap": 10,
"ignore_above": 5000
},
"plain": {
"type": "string",
"term_vector": "with_positions_offsets",
"similarity": "default",
"analyzer": "plain",
"search_analyzer": "plain_search",
"position_increment_gap": 10
}
},
"copy_to": [
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all"
],
"analyzer": "text",
"search_analyzer": "text_search",
"position_increment_gap": 10
},
"coordinates": {
"type": "nested",
"properties": {
"coord": {
"type": "geo_point",
"lat_lon": true
},
"country": {
"type": "string",
"index": "not_analyzed"
},
"dim": {
"type": "float"
},
"globe": {
"type": "string",
"index": "not_analyzed"
},
"name": {
"type": "string",
"index": "no"
},
"primary": {
"type": "boolean"
},
"region": {
"type": "string",
"index": "not_analyzed"
},
"type": {
"type": "string",
"index": "not_analyzed"
}
}
},
"external_link": {
"type": "string",
"norms": {
"enabled": false
},
"index_options": "docs",
"analyzer": "keyword",
"ignore_above": 5000
},
"file_text": {
"type": "string",
"term_vector": "with_positions_offsets",
"similarity": "default",
"fields": {
"plain": {
"type": "string",
"term_vector": "with_positions_offsets",
"similarity": "default",
"analyzer": "plain",
"search_analyzer": "plain_search",
"position_increment_gap": 10
}
},
"copy_to": [
"all"
],
"analyzer": "text",
"search_analyzer": "text_search",
"position_increment_gap": 10
},
"heading": {
"type": "string",
"term_vector": "with_positions_offsets",
"norms": {
"enabled": false
},
"similarity": "default",
"fields": {
"plain": {
"type": "string",
"term_vector": "with_positions_offsets",
"norms": {
"enabled": false
},
"similarity": "default",
"analyzer": "plain",
"search_analyzer": "plain_search",
"position_increment_gap": 10
}
},
"copy_to": [
"all",
"all",
"all",
"all",
"all"
],
"analyzer": "text",
"search_analyzer": "text_search",
"position_increment_gap": 10
},
"incoming_links": {
"type": "long"
},
"language": {
"type": "string",
"norms": {
"enabled": false
},
"index_options": "docs",
"analyzer": "keyword",
"ignore_above": 5000
},
"local_sites_with_dupe": {
"type": "string",
"norms": {
"enabled": false
},
"index_options": "docs",
"analyzer": "lowercase_keyword",
"ignore_above": 5000
},
"namespace": {
"type": "long"
},
"namespace_text": {
"type": "string",
"norms": {
"enabled": false
},
"index_options": "docs",
"analyzer": "keyword",
"ignore_above": 5000
},
"opening_text": {
"type": "string",
"term_vector": "with_positions_offsets",
"similarity": "default",
"fields": {
"plain": {
"type": "string",
"term_vector": "with_positions_offsets",
"similarity": "default",
"analyzer": "plain",
"search_analyzer": "plain_search",
"position_increment_gap": 10
}
},
"copy_to": [
"all",
"all",
"all"
],
"analyzer": "text",
"search_analyzer": "text_search",
"position_increment_gap": 10
},
"outgoing_link": {
"type": "string",
"norms": {
"enabled": false
},
"index_options": "docs",
"analyzer": "keyword",
"ignore_above": 5000
},
"redirect": {
"dynamic": "false",
"properties": {
"namespace": {
"type": "long"
},
"title": {
"type": "string",
"term_vector": "with_positions_offsets",
"similarity": "default",
"fields": {
"keyword": {
"type": "string",
"norms": {
"enabled": false
},
"index_options": "docs",
"similarity": "default",
"analyzer": "keyword",
"position_increment_gap": 10
},
"near_match": {
"type": "string",
"term_vector": "with_positions_offsets",
"norms": {
"enabled": false
},
"index_options": "docs",
"similarity": "default",
"analyzer": "near_match",
"position_increment_gap": 10
},
"near_match_asciifolding": {
"type": "string",
"term_vector": "with_positions_offsets",
"norms": {
"enabled": false
},
"index_options": "docs",
"similarity": "default",
"analyzer": "near_match_asciifolding",
"position_increment_gap": 10
},
"plain": {
"type": "string",
"term_vector": "with_positions_offsets",
"similarity": "default",
"analyzer": "plain",
"search_analyzer": "plain_search",
"position_increment_gap": 10
},
"prefix": {
"type": "string",
"term_vector": "with_positions_offsets",
"norms": {
"enabled": false
},
"index_options": "docs",
"similarity": "default",
"analyzer": "prefix",
"search_analyzer": "near_match",
"position_increment_gap": 10
},
"prefix_asciifolding": {
"type": "string",
"term_vector": "with_positions_offsets",
"norms": {
"enabled": false
},
"index_options": "docs",
"similarity": "default",
"analyzer": "prefix_asciifolding",
"search_analyzer": "near_match_asciifolding",
"position_increment_gap": 10
},
"suggest": {
"type": "string",
"similarity": "default",
"analyzer": "suggest",
"position_increment_gap": 10
}
},
"copy_to": [
"suggest",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match"
],
"analyzer": "text",
"search_analyzer": "text_search",
"position_increment_gap": 10
}
}
},
"source_text": {
"type": "string",
"term_vector": "with_positions_offsets",
"norms": {
"enabled": false
},
"similarity": "default",
"fields": {
"plain": {
"type": "string",
"term_vector": "with_positions_offsets",
"norms": {
"enabled": false
},
"similarity": "default",
"analyzer": "plain",
"search_analyzer": "plain_search",
"position_increment_gap": 10
},
"trigram": {
"type": "string",
"norms": {
"enabled": false
},
"index_options": "docs",
"similarity": "default",
"analyzer": "trigram",
"position_increment_gap": 10
}
},
"analyzer": "text",
"search_analyzer": "text_search",
"position_increment_gap": 10
},
"suggest": {
"type": "string",
"analyzer": "suggest"
},
"template": {
"type": "string",
"norms": {
"enabled": false
},
"index_options": "docs",
"analyzer": "lowercase_keyword",
"ignore_above": 5000
},
"text": {
"type": "string",
"term_vector": "with_positions_offsets",
"similarity": "default",
"fields": {
"plain": {
"type": "string",
"term_vector": "with_positions_offsets",
"similarity": "default",
"analyzer": "plain",
"search_analyzer": "plain_search",
"position_increment_gap": 10
},
"word_count": {
"type": "token_count",
"store": true,
"analyzer": "plain"
}
},
"copy_to": [
"all"
],
"analyzer": "text",
"search_analyzer": "text_search",
"position_increment_gap": 10
},
"text_bytes": {
"type": "long",
"index": "no"
},
"timestamp": {
"type": "date",
"format": "dateOptionalTime"
},
"title": {
"type": "string",
"term_vector": "with_positions_offsets",
"similarity": "default",
"fields": {
"keyword": {
"type": "string",
"norms": {
"enabled": false
},
"index_options": "docs",
"similarity": "default",
"analyzer": "keyword",
"position_increment_gap": 10
},
"near_match": {
"type": "string",
"term_vector": "with_positions_offsets",
"norms": {
"enabled": false
},
"index_options": "docs",
"similarity": "default",
"analyzer": "near_match",
"position_increment_gap": 10
},
"near_match_asciifolding": {
"type": "string",
"term_vector": "with_positions_offsets",
"norms": {
"enabled": false
},
"index_options": "docs",
"similarity": "default",
"analyzer": "near_match_asciifolding",
"position_increment_gap": 10
},
"plain": {
"type": "string",
"term_vector": "with_positions_offsets",
"similarity": "default",
"analyzer": "plain",
"search_analyzer": "plain_search",
"position_increment_gap": 10
},
"prefix": {
"type": "string",
"term_vector": "with_positions_offsets",
"norms": {
"enabled": false
},
"index_options": "docs",
"similarity": "default",
"analyzer": "prefix",
"search_analyzer": "near_match",
"position_increment_gap": 10
},
"prefix_asciifolding": {
"type": "string",
"term_vector": "with_positions_offsets",
"norms": {
"enabled": false
},
"index_options": "docs",
"similarity": "default",
"analyzer": "prefix_asciifolding",
"search_analyzer": "near_match_asciifolding",
"position_increment_gap": 10
},
"suggest": {
"type": "string",
"similarity": "default",
"analyzer": "suggest",
"position_increment_gap": 10
}
},
"copy_to": [
"suggest",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match",
"all_near_match"
],
"analyzer": "text",
"search_analyzer": "text_search",
"position_increment_gap": 10
},
"wiki": {
"type": "string",
"norms": {
"enabled": false
},
"index_options": "docs",
"analyzer": "keyword",
"ignore_above": 5000
},
"wikibase_item": {
"type": "string",
"norms": {
"enabled": false
},
"index_options": "docs",
"analyzer": "keyword",
"ignore_above": 5000
}
}
},
"namespace": {
"dynamic": "false",
"_all": {
"enabled": false
},
"properties": {
"name": {
"type": "string",
"norms": {
"enabled": false
},
"index_options": "docs",
"analyzer": "near_match_asciifolding",
"ignore_above": 5000
}
}
}
}
}
}

IIRC when we talked about norms we arrived to the conclusion that we do want norms on array fields, including heading. But it may be that you can't just enable them on existing index, you have to reindex. Not sure what's the correct approach here. @dcausse, what do you think?

Smalyshev triaged this task as High priority.

Yes, unfortunately analysis config and some mapping changes cannot be applied on an existing index.
On the other hand the error message printed by cirrus should make it clear that an inplace reindex is necessary.

debt moved this task from needs triage to Up Next on the Discovery-Search board.Aug 3 2016, 8:29 PM

Change 302816 had a related patch set uploaded (by Smalyshev):
Improve error message on update fail.

https://gerrit.wikimedia.org/r/302816

Smalyshev lowered the priority of this task from High to Normal.Aug 3 2016, 8:54 PM

Change 302816 merged by jenkins-bot:
Improve error message on update fail.

https://gerrit.wikimedia.org/r/302816

debt closed this task as Resolved.Aug 4 2016, 5:03 PM
debt removed a project: Discovery-Search.
debt added a subscriber: debt.

this was merged today, closing