Page MenuHomePhabricator
Create Task
Maniphest T136559

Elasticsearch: illegal longitude value [219.38] for coordinates.coord
Closed, ResolvedPublic
Actions

Description

After we upgraded codfw to elasticsearch 2.3.3 we started to see some exceptions during doc updates:

MapperParsingException[failed to parse]; nested: IllegalArgumentException[illegal longitude value [219.38] for coordinates.coord];
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:154)
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:309)
        at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:580)
        at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:559)
        at org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:212)
        at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:224)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardUpdateOperation(TransportShardBulkAction.java:389)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:191)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
        at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: illegal longitude value [219.38] for coordinates.coord
        at org.elasticsearch.index.mapper.geo.GeoPointFieldMapperLegacy.parse(GeoPointFieldMapperLegacy.java:334)
        at org.elasticsearch.index.mapper.geo.BaseGeoPointFieldMapper.parse(BaseGeoPointFieldMapper.java:484)
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:309)
        at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:326)
        at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:252)
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:306)
        at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:326)
        at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:252)
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:122)
        ... 19 more

It happened to [enwiki_content][page][14640471] which seems to be Mars.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Don't index non-Earth coordinatesmediawiki/extensions/GeoDatawmf/1.28.0-wmf.3+24 -6
Don't index non-Earth coordinatesmediawiki/extensions/GeoDatawmf/1.28.0-wmf.4+24 -6
Don't index non-Earth coordinatesmediawiki/extensions/GeoDatamaster+24 -6
Customize query in gerrit

Event Timeline

dcausse created this task.May 30 2016, 2:55 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptMay 30 2016, 2:55 PM
dcausse added a project: GeoData.May 30 2016, 2:56 PM
Danny_B added a project: Elasticsearch.May 30 2016, 3:03 PM
Restricted Application added projects: Discovery-ARCHIVED, Discovery-Search. · View Herald TranscriptMay 30 2016, 3:03 PM
gerritbot subscribed.May 31 2016, 8:56 PM

Change 292026 had a related patch set uploaded (by MaxSem):
Don't index non-Earth coordinates

https://gerrit.wikimedia.org/r/292026

gerritbot added a project: Patch-For-Review.May 31 2016, 8:56 PM
gerritbot added a comment.May 31 2016, 10:07 PM

Change 292026 merged by jenkins-bot:
Don't index non-Earth coordinates

https://gerrit.wikimedia.org/r/292026

EBernhardson assigned this task to MaxSem.May 31 2016, 10:11 PM
EBernhardson edited projects, added Discovery-Search (Current work); removed Discovery-Search.
EBernhardson moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.May 31 2016, 10:15 PM
gerritbot added a comment.May 31 2016, 10:27 PM

Change 292045 had a related patch set uploaded (by MaxSem):
Don't index non-Earth coordinates

https://gerrit.wikimedia.org/r/292045

gerritbot added a comment.May 31 2016, 10:28 PM

Change 292047 had a related patch set uploaded (by MaxSem):
Don't index non-Earth coordinates

https://gerrit.wikimedia.org/r/292047

gerritbot added a comment.May 31 2016, 10:29 PM

Change 292047 abandoned by MaxSem:
Don't index non-Earth coordinates

Reason:
8O

https://gerrit.wikimedia.org/r/292047

gerritbot added a comment.May 31 2016, 10:36 PM

Change 292047 restored by MaxSem:
Don't index non-Earth coordinates

https://gerrit.wikimedia.org/r/292047

gerritbot added a comment.May 31 2016, 10:37 PM

Change 292047 had a related patch set uploaded (by MaxSem):
Don't index non-Earth coordinates

https://gerrit.wikimedia.org/r/292047

gerritbot added a comment.May 31 2016, 11:19 PM

Change 292045 merged by jenkins-bot:
Don't index non-Earth coordinates

https://gerrit.wikimedia.org/r/292045

gerritbot added a comment.May 31 2016, 11:19 PM

Change 292047 merged by jenkins-bot:
Don't index non-Earth coordinates

https://gerrit.wikimedia.org/r/292047

Stashbot subscribed.May 31 2016, 11:47 PM

Mentioned in SAL [2016-05-31T23:47:19Z] <dereckson@tin> Synchronized php-1.28.0-wmf.4/extensions/GeoData/includes/Hooks.php: Don't index non-Earth coordinates (T136559) (duration: 00m 24s)

Stashbot added a comment.May 31 2016, 11:54 PM

Mentioned in SAL [2016-05-31T23:54:54Z] <dereckson@tin> Synchronized php-1.28.0-wmf.3/extensions/GeoData/includes/Hooks.php: Don't index non-Earth coordinates (T136559) (duration: 00m 23s)

ReleaseTaggerBot added projects: MW-1.28-release (WMF-deploy-2016-06-07_(1.28.0-wmf.5)), MW-1.28-release (WMF-deploy-2016-05-31_(1.28.0-wmf.4)), MW-1.28-release (WMF-deploy-2016-05-24_(1.28.0-wmf.3)).Jun 1 2016, 3:01 AM
dcausse added a subscriber: EBernhardson.Jun 1 2016, 9:41 AM

I can still see errors, I suppose that this happens when we update non-content data like incoming links, elastic should try to reparse the source from the index and then fail?

We could maybe try to reindex these docs from mysql using forceSearchIndex, unfortunately we have only --fromId/--toId options with this script so it may not be very convenient.
On the other hand there are only 2577 docs affected on enwiki according to this query:

{
        "fields":[],
        "query": {
                "nested" : {
                        "path":"coordinates",
                        "query":{
                                "bool":{
                                        "must_not":[
                                                {"term":{"coordinates.globe":"earth"}}
                                        ],
                                        "filter": {"exists":{"field": "coordinates.globe"}}
                                }
                        }
                }
        }
}

@EBernhardson what do you think? Should we add a new option to forceSearchIndex that accepts a list of ids or is it ok to go for a brute force approach and run forceSearchIndex for all ids?

EBernhardson added a comment.Jun 1 2016, 6:49 PM

forceSearchIndex for everything will probably take some time, although it is the easy solution. I'm kinda ambivalent. Would perhaps be nice to be able to take a list of ids but not strictly necessary. I suppose if we think it would only take an hour or two to write (probably) can put together a patch to take the id's to reindex as input. The script is a bit of a mess though so it might not be so easy, in which case a full reindex isn't the end of the world.

EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Jun 3 2016, 12:46 AM
MaxSem closed this task as Resolved.Jun 7 2016, 5:18 PM
Aklapper removed a subscriber: GeoData.May 16 2023, 12:06 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits