Page MenuHomePhabricator

MaxMind IP files needs to be automatically updated and version controlled (Pakistan Mobile Dashboard Issues)
Closed, DeclinedPublic

Description

We depend on the MaxMind database file to automatically updated so our GeoIP translations are accurate. Currently this is not the case.

AKAIK there are at least 3 locations/uses of this file:

  1. stat1002 (/usr/share/GeoIP is automatically updated by puppet)
  2. HDFS (unknown location)
  3. WikiStats (unknown location)

We also need to be able to retain and use older versions of the file so as part of this defect, we'll need to come up with a scheme for retaining older versions of the db file.

This was first exposed due to a jump in Pakistan IPs in the Wikipedia Zero dashboards.


Version: unspecified
Severity: normal

Details

Reference
bz53762

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 1:50 AM
bzimport set Reference to bz53762.
bzimport added a subscriber: Unknown Object (MLST).

Prioritization and scheduling of this bug is tracked on Mingle card
https://mingle.corp.wikimedia.org/api/v2/projects/analytics/1132 (previous comment was generated by bingle containing a small bug, that has now been fixed).

otto wrote:

I think we should just have puppet keep the files historically and symlink to the current one. For HDFS, we can just sync the full /usr/share/GeoIP directory into HDFS, and load whichever file is needed.

Wikistats squid log processing is on stat1002.
Only use of ip->geo is in '/usr/local/bin/geoiplogtag'

Prioritization and scheduling of this bug is tracked on Mingle card
https://mingle.corp.wikimedia.org/projects/analytics/1132 (previous
comment was generated by bingle containing a small bug, that has now been
fixed). argh another bug :)