Page MenuHomePhabricator

Geocoding UDF should be more resilient
Closed, ResolvedPublic

Description

Records like the following trigger a NullPointerException:

{"hostname":null,"sequence":null,"dt":null,"time_firstbyte":null,"ip":"70.39.186.182","cache_status":null,"http_status":null,"response_size":null,"http_method":null,
"uri_host":null,"uri_path":null,"uri_query":null,"content_type":null,"referer":null,"x_forwarded_for":null,"user_agent":null,"accept_language":null,"x_analytics":null,"r
ange":null,"is_pageview":null,"webrequest_source":"upload","year":"2015","month":"1","day":"20","hour":"20"}

Repro:

CREATE TEMPORARY FUNCTION geocode as 'org.wikimedia.analytics.refinery.hive.GeocodedDataUDF';

select geocode(ip) from wmf.webrequest where year=2015 and month=1 and day=20 and hour=20 limit 10;

Event Timeline

Nuria created this task.Feb 11 2015, 2:26 AM
Nuria raised the priority of this task from to Needs Triage.
Nuria updated the task description. (Show Details)
Nuria added a subscriber: Nuria.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 11 2015, 2:26 AM
kevinator triaged this task as Normal priority.Feb 11 2015, 3:47 PM
kevinator set Security to None.
gerritbot added a subscriber: gerritbot.

Change 190092 had a related patch set uploaded (by QChris):
Fix NPE in GeocodedDataUDF for countries with iso code but no name

https://gerrit.wikimedia.org/r/190092

Patch-For-Review

Change 190093 had a related patch set uploaded (by QChris):
Fix NPE in GeocodedDataUDF for location without lon/lat

https://gerrit.wikimedia.org/r/190093

Patch-For-Review

Change 190094 had a related patch set uploaded (by QChris):
Fix potential NPE in Geocode's subdivision extraction

https://gerrit.wikimedia.org/r/190094

Patch-For-Review

Change 190095 had a related patch set uploaded (by QChris):
Fix potential NPE stopping to abuse getters in Geocode data extraction

https://gerrit.wikimedia.org/r/190095

Patch-For-Review

Change 190096 had a related patch set uploaded (by QChris):
Harden GeocodeDataUDF's extraction of values against NPEs

https://gerrit.wikimedia.org/r/190096

Patch-For-Review

Change 190092 merged by Ottomata:
Fix NPE in GeocodedDataUDF for countries with iso code but no name

https://gerrit.wikimedia.org/r/190092

Change 190093 merged by Ottomata:
Fix NPE in GeocodedDataUDF for location without lon/lat

https://gerrit.wikimedia.org/r/190093

Change 190094 merged by Ottomata:
Fix potential NPE in Geocode's subdivision extraction

https://gerrit.wikimedia.org/r/190094

Change 190095 merged by Ottomata:
Fix potential NPE stopping to abuse getters in Geocode data extraction

https://gerrit.wikimedia.org/r/190095

Change 190096 merged by Ottomata:
Harden GeocodeDataUDF's extraction of values against NPEs

https://gerrit.wikimedia.org/r/190096

Ottomata closed this task as Resolved.Feb 24 2015, 3:14 PM
kevinator moved this task from Next Up to Done on the Analytics-Kanban board.Feb 24 2015, 3:16 PM