Page MenuHomePhabricator

Disable GeoIP Legacy Download / Identify all users of legacy (v1) GeoIP datasets and inform them of the need to switch to GeoIP2 dataset
Open, MediumPublic

Description

The legacy GeoIP dataset is being deprecated in May 2022. This tasks is to:

  • Identify all users of the legacy GeoIP datasets and inform them of the need to switch to GeoIP2 dataset
  • Modify the puppet code to no longer download the databases from MaxMind and then propagate to other servers/destinations.

Event Timeline

odimitrijevic renamed this task from Deprecate GeoIP Legacy Download to Disable GeoIP Legacy Download.Mar 14 2022, 5:38 AM
jbond triaged this task as Medium priority.Mar 21 2022, 11:42 AM

Change 773648 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] geoip::data::maxmind: deactivate timer for downloading of legacy DBs

https://gerrit.wikimedia.org/r/773648

Change 773649 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] puppetmaster::geoip: stop using class for legacy maxmind downloads in prod

https://gerrit.wikimedia.org/r/773649

Change 773843 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] puppetmaster:geoip: stop trying to download GeoIP1 legacy databases

https://gerrit.wikimedia.org/r/773843

Change 773844 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] geoip::maxmind: remove code for absenting old resources

https://gerrit.wikimedia.org/r/773844

Change 773845 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] geoip::maxmind: rename the legacy timer to geoip2

https://gerrit.wikimedia.org/r/773845

Change 773648 abandoned by Dzahn:

[operations/puppet@production] geoip::data::maxmind: deactivate timer for downloading of legacy DBs

Reason:

I have uploaded 3 new patches instead.

https://gerrit.wikimedia.org/r/773648

Change 773649 abandoned by Dzahn:

[operations/puppet@production] puppetmaster::geoip: stop using class for legacy maxmind downloads in prod

Reason:

just remove the geoIP 1 databases from the list and keep download geoIP 2 databases

https://gerrit.wikimedia.org/r/773649

Change 773844 merged by Dzahn:

[operations/puppet@production] geoip::maxmind: remove code for absenting old resources

https://gerrit.wikimedia.org/r/773844

Change 773845 merged by Dzahn:

[operations/puppet@production] geoip::maxmind: rename the update timers, don't use 'legacy' term

https://gerrit.wikimedia.org/r/773845

Mentioned in SAL (#wikimedia-operations) [2022-04-04T21:14:17Z] <mutante> puppetmaster1001/puppetmaster2003 - geoip / maxmind database update timers renamed. 'geoip_update_legacy' became 'geoip_update_main', 'geoip_update' became 'geoip_update_ipinfo'. Not using the confusing 'legacy' term anymore as was suggested as part of (T303464)

Change 773843 merged by Dzahn:

[operations/puppet@production] puppetmaster:geoip: stop trying to download GeoIP1 legacy databases

https://gerrit.wikimedia.org/r/773843

Modify the puppet code to no longer download the databases from MaxMind and then propagate to other servers/destinations.

This is done.

puppet code has been changed to stop trying to download v1 databases. It is still downloading v2 databases.

The relevant change here is: https://gerrit.wikimedia.org/r/c/operations/puppet/+//2/modules/profile/manifests/httpbb.pp

This means the puppetmaster(s) will not try to fetch the v1 DBs anymore. This still worked as of today but as the ticket says it will expire for real in May. Now we stopped before it will cause errors.

It does NOT mean any files were removed, neither from the puppetmasters nor from the appserver that sync data from the puppetmasters. The v1 databases were NOT deleted.. they will simply not be updated anymore from now on.

Additionally, as suggested by others, we renamed the update jobs to be less confusing.

The job that used to be called 'geoip_update_legacy' and downloaded both v1 and v2 databases is now called "geoip_update_main" and downloads only v2 databases.

The job that was simply called "geoip_update" is now called "geoip_update_ipinfo" and does the same thing it did before, only downloads additional enterprise DBs for the IPInfo extension, unchanged besides the name.

At this part I am giving back the ticket to the pool/analytics/traffic because there is still the part:

"Identify all users of the legacy GeoIP datasets and inform them of the need to switch to GeoIP2 dataset"

Dzahn renamed this task from Disable GeoIP Legacy Download to Disable GeoIP Legacy Download / Identify all users of legacy (v1) GeoIP datasets and inform them of the need to switch to GeoIP2 dataset.Apr 5 2022, 9:57 PM
Dzahn removed Dzahn as the assignee of this task.
Dzahn subscribed.

@Dzahn I mentioned over email, but I thought I would add a note here as well, we are still receiving alerts, last one was on May 2, stating that we are downloading the legacy database, is that expected?

@jhathaway Yes and no. What I definitely disabled was that the puppetmasters try to fetch this to copy it to puppet volatile, which was from there used by mw appservers. And I would have said what John said on Slack, it's probably because I had not merged that change until April 4.

BUT.. looking at the IP address from their mail, that is install1003 and that was NOT expected. So I will check that out and take a look at install*.

Additionally we were never sure if anything else, besides appservers, like analytics, is using these but also saw no real way to check that besides seeing if something/someone complains.

@Dzahn that makes sense, so I assume it is okay that we also received a notice saying the invoice has lapsed, since these downloads are no longer needed?

The part that we don't (can't actually) renew the license for legacy (v1) database is known, yes.

But what isn't is that there seems to be a licensing issue with one of the v2 databases. But there is already a mail thread with sales.

I merged some duplicates into T302864. This might be another one.