Switch Varnish's GeoIP code to libmaxminddb/GeoIP2
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	faidon
	May 15 2015, 9:32 AM

Description

We currently have GeoIP/libgeoipip code in our VCL for the GeoIP cookies & geoiplookup. We should replace it with code that uses libmaxminddb & the GeoIP2 databases. This would allow us proper IPv6 lookups as well and, depending on their accuracy and our accuracy requirements, might just let us fully decommission the separate geoiplookup.wikimedia.org domain.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
varnish: Improve GeoIP to use cookie domain similar to prod	operations/puppet	production	+31 -13
varnish: remove libgeoip from text VCL compilation	operations/puppet	production	+1 -1
GeoIP VCL: re-set old IPv6 no-data cookies	operations/puppet	production	+5 -0
varnish: switch from libGeoIP to libmaxminddb	operations/puppet	production	+271 -215
Prep varnishd for libmaxminddb-based VCL	operations/puppet	production	+4 -4

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	BBlack	T121925 [EPIC] GeoIP lookup for IPv6 connections delays banner loading
Open	None	T121937 [EPIC] Fix and improve geolocation, and ensure it's working properly for FR campaigns
Resolved	BBlack	T100902 Get rid of geoiplookup service
Resolved	BBlack	T143270 ULS GeoIP should not use meta.wm.o/geoiplookup
Resolved	• AndyRussG	T143271 CN: Stop using the geoiplookup HTTPS service (always use the Cookie)
Resolved	BBlack	T99226 Switch Varnish's GeoIP code to libmaxminddb/GeoIP2
Resolved	BBlack	T89688 Varnish GeoIP is broken for HTTPS+IPv6 traffic
Resolved	BBlack	T146638 Remove "GeoIP lookup" service from https://status.wikimedia.org

Event Timeline

faidon created this task.May 15 2015, 9:32 AM

faidon raised the priority of this task from to Needs Triage.

faidon updated the task description. (Show Details)

faidon added projects: acl*sre-team, Traffic.

faidon subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 15 2015, 9:32 AM

BBlack triaged this task as Medium priority.May 20 2015, 8:17 AM

BBlack added a parent task: T100902: Get rid of geoiplookup service.Jun 3 2015, 11:09 PM

faidon mentioned this in T59126: Provide a JavaScript API for geoip lookups.Jun 22 2015, 10:15 PM

Pcoombe subscribed.Sep 14 2015, 10:20 AM

Restricted Application added a subscriber: Matanya. · View Herald TranscriptSep 14 2015, 10:20 AM

Pcoombe mentioned this in T73267: Enable IPv6 on donate.wikimedia.org.Sep 14 2015, 11:21 AM

Krenair subscribed.Sep 14 2015, 11:24 AM

BBlack mentioned this in T89688: Varnish GeoIP is broken for HTTPS+IPv6 traffic.Nov 12 2015, 4:28 PM

Yurik mentioned this in T116678: Country mapping routine for proxied requests.Nov 13 2015, 6:53 AM

Change 253619 had a related patch set uploaded (by Faidon Liambotis):
varnish: switch from libGeoIP to libmaxminddb

https://gerrit.wikimedia.org/r/253619

gerritbot added a project: Patch-For-Review.Nov 17 2015, 3:49 PM

How to test accuracy? The original accuracy test for MaxMind was https://meta.wikimedia.org/wiki/MaxMindCityTesting

The vendor for both databases is the same, so the accuracy of the IPv4 data would be the same. The IPv6 switch means that we'd geolocate IPv6 directly now instead of falling back to IPv4; this would possibly be less accurate but I think we should just do it anyway (but not during the fundraiser period).

faidon mentioned this in T119801: Update GeoIP cookie to include netspeed info.Nov 29 2015, 2:57 AM

ori added a project: Fundraising-Backlog-Old.Dec 18 2015, 10:41 PM

ori set Security to None.

ori added a subscriber: Jgreen.

faidon mentioned this in T121922: Firefox SPDY-coalesces requests to geoiplookup over text-lb, causing GeoIP IPv6 failures.Dec 18 2015, 10:49 PM

Krinkle added a parent task: T121925: [EPIC] GeoIP lookup for IPv6 connections delays banner loading.Dec 18 2015, 10:52 PM

ori added a parent task: T121937: [EPIC] Fix and improve geolocation, and ensure it's working properly for FR campaigns.Dec 18 2015, 11:58 PM

awight added a project: Fundraising-Backlog.Dec 19 2015, 12:25 AM

• AndyRussG subscribed.Dec 19 2015, 2:59 AM

• DStrine moved this task from Triage to Next on the Fundraising-Backlog board.Dec 22 2015, 8:16 PM

• DStrine moved this task from Next to Blocked or not fr-tech on the Fundraising-Backlog board.

looks like this was waited for until the fundraising was over, can it be resumed now? also would the migration to varnish4 have any impact on it?

It's mostly been blocked on lack of anyone having time to work on it, too. At this point, it's probably wiser to delay this to post-varnish4 instead of doing work under varnish3 that will have to be re-evaluated/ported to varnish4 shortly after. Will update blockers.

BBlack added a subtask: T131503: Convert text cluster to Varnish 4.Apr 25 2016, 3:05 PM

• DStrine removed a project: Fundraising-Backlog-Old.Aug 5 2016, 8:41 PM

BBlack mentioned this in rOPUP0d024b052d41: varnish: switch from libGeoIP to libmaxminddb [WIP!].Aug 17 2016, 6:16 PM

BBlack mentioned this in rOPUPa6e76e41b7b0: varnish: switch from libGeoIP to libmaxminddb [WIP!].

BBlack mentioned this in rOPUP6a5ce8b297a6: varnish: switch from libGeoIP to libmaxminddb [WIP!].

BBlack removed a subtask: T131503: Convert text cluster to Varnish 4.Aug 17 2016, 7:29 PM

BBlack mentioned this in rOPUP5a5e4f23f3e4: varnish: switch from libGeoIP to libmaxminddb.Aug 17 2016, 7:32 PM

BBlack mentioned this in rOPUP371d7cc737d0: varnish: switch from libGeoIP to libmaxminddb.Aug 17 2016, 7:43 PM

BBlack added a parent task: T143270: ULS GeoIP should not use meta.wm.o/geoiplookup.Aug 17 2016, 10:06 PM

BBlack added a parent task: T143271: CN: Stop using the geoiplookup HTTPS service (always use the Cookie).Aug 17 2016, 10:09 PM

I'm planning to merge up the new version of the Varnish GeoIP code from rOPUP371d7cc737d0 in the next couple of days. If anyone has the time to give it some review for stupid mistakes, it would be appreciated!

Change 305419 had a related patch set uploaded (by BBlack):
GeoIP VCL: re-set old IPv6 no-data cookies

https://gerrit.wikimedia.org/r/305419

BBlack mentioned this in rOPUPba16782ef746: GeoIP VCL: re-set old IPv6 no-data cookies.Aug 17 2016, 11:22 PM

BBlack mentioned this in rOPUP9915203b86bb: varnish: switch from libGeoIP to libmaxminddb.

BBlack mentioned this in rOPUP2518bec4c795: GeoIP VCL: re-set old IPv6 no-data cookies.Aug 18 2016, 2:28 PM

BBlack mentioned this in rOPUPa7e48761944b: varnish: switch from libGeoIP to libmaxminddb.

BBlack mentioned this in rOPUPa395524ee4d6: GeoIP VCL: re-set old IPv6 no-data cookies.Aug 18 2016, 2:34 PM

BBlack mentioned this in rOPUPe6d05006e2ca: varnish: switch from libGeoIP to libmaxminddb.

BBlack mentioned this in rOPUPaf358507c753: varnish: switch from libGeoIP to libmaxminddb.Aug 18 2016, 3:55 PM

BBlack mentioned this in rOPUP8c9baa88f16b: GeoIP VCL: re-set old IPv6 no-data cookies.

BBlack mentioned this in rOPUP99eef684fe34: varnish: switch from libGeoIP to libmaxminddb.Aug 18 2016, 4:00 PM

BBlack mentioned this in rOPUPd2b67dd62189: GeoIP VCL: re-set old IPv6 no-data cookies.