In T288375 we are discussing how this extension would have access to the maxmind databases when we migrate MediaWiki to Kubernetes. It would be helpful to know how important the "freshness" of the database is, i.e. how quickly after maxmind publishes an update do we need that update in production? Maxmind updates the database mostly in a weekly rhythm, but for certain elements quicker (and slower). See: https://support.maxmind.com/geoip-faq/databases-and-database-updates/how-often-are-the-geoip2-and-geoip-legacy-databases-updated/. One of the options is too include the maxmind database in the release, which happens mostly daily, but in extreme cases (no deployment weeks for example, christmas holidays, etc) there can be intervals of up to 10 days without a release. Is that acceptable to the users of IP Info? The migration to Kubernetes is currently in the works, a first version can be accessed via the Wikimedia Debug extension for Firefox or Chrome and then selecting "k8s-experimental", we are projecting a full migration for early 2022.

My understanding is that the changes in the data are minimal from one version to the next; it is not like the ownership of hundreds of thousands of IPs changes each time. Maybe the location associated with some IPs within the same ASN changes based on latest data, but overall MaxMind (and other similar) data is pretty static.

If this assumption is correct, then a 10-day lag would be acceptable. If this assumption can be vetted, perhaps by diffing the consecutive versions of MaxMind, then the decision would be more thoughtfully made.

In T288844#7296789, @Huji wrote:

My understanding is that the changes in the data are minimal from one version to the next; it is not like the ownership of hundreds of thousands of IPs changes each time. Maybe the location associated with some IPs within the same ASN changes based on latest data, but overall MaxMind (and other similar) data is pretty static.

If this assumption is correct, then a 10-day lag would be acceptable. If this assumption can be vetted, perhaps by diffing the consecutive versions of MaxMind, then the decision would be more thoughtfully made.

Couldn't have put it better myself. Is there a way to assess how radical are the changes with every update?

The ideal scenario would of course be that the data be as fresh as possible at any given time. However if the changes are minimal, like @Huji suggests, then a 10-day delay should be acceptable.

@Niharika Based on my read, it also looks like the 10 day delay would only be when there were holidays too. What's the next step here assuming we're okay with that? It sounds like @sbassett is moving forward with looking into this.

In T288844#7321649, @mepps wrote:

It sounds like @sbassett is moving forward with looking into this.

Er, whoops, I'm actually not looking into the MaxMind dbs for IPInfo, at least not based upon my recent comment on Slack. I was just offering that as a means of comparison for where other, similar data feeds could possibly live, if data size and delivery issues could be accommodated. There are potentially some shared use-cases for certain data feeds which have been discussed on this private task: T265845.

In T288844#7321649, @mepps wrote:

@Niharika Based on my read, it also looks like the 10 day delay would only be when there were holidays too. What's the next step here assuming we're okay with that? It sounds like @sbassett is moving forward with looking into this.

Hmm, I am not sure who is responsible for the next steps here. @wkandek is that your team?

In T288844#7349692, @Niharika wrote:

In T288844#7321649, @mepps wrote:

@Niharika Based on my read, it also looks like the 10 day delay would only be when there were holidays too. What's the next step here assuming we're okay with that? It sounds like @sbassett is moving forward with looking into this.

Hmm, I am not sure who is responsible for the next steps here. @wkandek is that your team?

If the easiest path to getting the spur.us data feed product purchased and on the roadmap (even if long-term) for ext:IPInfo is a similar integration to how the Maxmind dbs are currently managed within production, then yes, this likely involves @wkandek's team, or at least some folks within SRE. I'll note that I did have a conversation with a few different individuals within the #wikimedia-cloud IRC channel (tried to determine if that was a suitable environment or not) and this was the suggestion that came from that conversation: T290917: New Service Request Security API Gateway.

@sbassett My bad -- sounds like there has been a mix-up of threads here. I was talking about the next steps for getting MaxMind license key, user ID and product IDs into production. This ticket is for that specifically.
Let's open another ticket for Spur integration, if there isn't one already.

In T288844#7350579, @Niharika wrote:

@sbassett My bad -- sounds like there has been a mix-up of threads here. I was talking about the next steps for getting MaxMind license key, user ID and product IDs into production. This ticket is for that specifically.

Ah, ok, sorry for the confusion.

Let's open another ticket for Spur integration, if there isn't one already.

I'll plan to create a subtask for that under the aforementioned T290917: New Service Request Security API Gateway.

Change 721595 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] geoip: replace maxmind update cron with system timer and config

https://gerrit.wikimedia.org/r/721595

fwiw: While looking at this I found we have the email alias maxmind@wikimedia and it forwards to fr-tech@wikimedia. So it might be interesting that Fundraising people are handling that special email address, not Analytics, Traffic or another team. Or there might be 3 licenses?

edit: I mailed maxmind/fr-tech about this and asked, will update this once I get a response

further update: identified the relevant files in the private repo that hold the userId and license key for production.

comments in there refer to the account with an individual @wikimedia.org address but the user in question does not work for WMF since years. We need to clean this up as well

@phuedx Hi, I would like to compare the UserId, LicenseKey and ProductIds between what I see in production, the ones possibly used by other teams and the one recently purchased by Anti-Harassment.

I would add the new license into the private puppet repo, in addition to keeping the old license as legacy for a transitional period. From there we can deploy it to the appservers.

Could you share the new license information with me in a secure way? I see you have shell access to deployment, so maybe could you put it into a home dir on a prod server, make sure only you can read it and I look at it using root? Would that work?

Also you say "includes the GeoIP2 Anonymous IP - Proxy Detection and Enterprise databases". Would you know the product IDs for those by any chance? I see on our side so far we have 106 (Country), 133 (City) and 115 (Region).

Thanks!

There is a mechanism that first downloads the database files centrally to the puppetmaster so that appservers can then fetch the files from a local master instead of all re-downloading them from MaxMind directly. (This slighlty complicates the puppet setup though.)

There is also a cron job to try that once daily where updates are supposed to actually happen about weekly. I'll convert that cron to a systemd timer as part of a wider effort to get rid of all legacy cron jobs.

The existing puppet class can only use exactly one license key but could be changed to allow multiple ones in parallel.

It already has 2 modes, one which downloads only free databases and another that downloads commercially paid-for databases, called fetch_private.

The product IDs listed there are:

32             product_ids    => [
33                 '106', # GeoIP.dat
34                 '115', # GeoIPRegion.dat
35                 '132', # GeoIPCity.dat
36                 '133', # GeoIPCity.dat
37                 '171', # GeoIPNetSpeed.dat
38                 '177', # GeoIPNetSpeedCell.dat
39                 'GeoIP2-City',
40                 'GeoIP2-Connection-Type',
41                 'GeoIP2-Country',
42                 'GeoIP2-ISP',
43                 ],

Change 723337 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] puppetmaster::geoip: temp. install a new MaxMind license in parallel

https://gerrit.wikimedia.org/r/723337

In T288844#7375732, @Dzahn wrote:

Could you share the new license information with me in a secure way? I see you have shell access to deployment, so maybe could you put it into a home dir on a prod server, make sure only you can read it and I look at it using root? Would that work?

If you have a GPG key, then we could exchange public keys and I could encrypt a plaintext file with yours. I'm happy to do whichever is easiest for you though :)

Also you say "includes the GeoIP2 Anonymous IP - Proxy Detection and Enterprise databases". Would you know the product IDs for those by any chance? I see on our side so far we have 106 (Country), 133 (City) and 115 (Region).

I believe that they are GeoIP2-Anonymous-IP and GeoIP2-Enterprise, respectively.

cc'ing @dom_walden and @imaigwilo, the QTEs that work with Anti-Harassment Tools.

Are the MaxMind databases deployed to the Beta Cluster or will they be? I ask because it would make testing the IPInfo extension easier.

Edit

My guess is that deploying the commercially-licensed databases to the Beta Cluster is a no go but are the others?

In T288844#7376870, @phuedx wrote:

If you have a GPG key, then we could exchange public keys and I could encrypt a plaintext file with yours. I'm happy to do whichever is easiest for you though :)

Sounds good! :)

key 37E9B5C6F5F6A067: "Daniel Zahn (WMF) <dzahn@wikimedia.org>" should be on keyservers, but since there is often trouble with them, here it is as well:

https://people.wikimedia.org/~dzahn/gpg.txt

In T288844#7376887, @phuedx wrote:

Are the MaxMind databases deployed to the Beta Cluster or will they be? I ask because it would make testing the IPInfo extension easier.
My guess is that deploying the commercially-licensed databases to the Beta Cluster is a no go but are the others?

I haven't checked what is actually deployed in beta but the puppet class puppetmaster::geoip has a parameter Boolean $fetch_private.

If that is set to true then "# Fetch the proprietary paid-for MaxMind database" which means we get the databases including product IDs 106,115,132,133,171,177. If it's set to false then we only get 506, 517, 533 which are called "legacy" and apparently don't need a license.

I received the new license info from @phuedx , encrypted with GPG. I decrypted it and added it to the private puppet repository into the passwords class, right next to the existing license.

It can be accessed as user_id => $passwords::geoip::user_id_ipinfo and license_key => $passwords::geoip::license_key_ipinfo from puppet now.

It can then be used with the class { '::geoip::data::maxmind':.

But still needs a change to replace the existing license on a single server or to install multiple licenses at once, for which the class would have to be converted to a defined type, because :geoip::data::maxmind is a singleton.

Change 724860 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] geoip: create transitional class geoip::data::maxmind::ipinfo

https://gerrit.wikimedia.org/r/724860

Change 723337 abandoned by Dzahn:

[operations/puppet@production] puppetmaster::geoip: refactor to allow installing maxmind databases for IP Info

Reason:

merged into https://gerrit.wikimedia.org/r/c/operations/puppet/+/724860

https://gerrit.wikimedia.org/r/723337

Change 724860 merged by Dzahn:

[operations/puppet@production] geoip: create transitional class geoip::data::maxmind::ipinfo

https://gerrit.wikimedia.org/r/724860

Mentioned in SAL (#wikimedia-operations) [2021-10-01T21:44:33Z] <mutante> puppetmasters - temp. disabling puppet one more time, now for a different deploy, to fetch an additional MaxMind database - T288844

Change 725379 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] geoip: fix syntax in erb template for new maxmind job

https://gerrit.wikimedia.org/r/725379

Change 725379 merged by Dzahn:

[operations/puppet@production] geoip: fix syntax in erb template for new maxmind job

https://gerrit.wikimedia.org/r/725379

Mentioned in SAL (#wikimedia-operations) [2021-10-01T22:15:42Z] <mutante> puppetmaster2001 - sudo /usr/local/bin/geoipupdate_job after adding new shell command and timer - succesfully downloaded enterprise database for T288844

After quite some necessary puppet changes (see above) we are now at a state where we could succesfully download new databases, including the Enterprise DB and using the new license key, to the puppetmasters.

There is still some ongoing work where we should stop doing pulling MaxMind databases (old or new) to all puppetmasters instead of juse one while at the same time puppet masters are syncing the volatile dir, where these files live, between each other but it doesn't block this.

Next step will be shortly that we pull the new files from puppetmasters to the application servers and then you will be able to use them for IP Info.

So the good news summarized:

@puppetmaster1001:/var/lib/puppet/volatile/GeoIPInfo# ls
GeoIP2-Anonymous-IP.mmdb  GeoIP2-Enterprise.mmdb

^ This works.

Change 726094 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] mediawiki/geoip: add option to also pull new MaxMind databases from master

https://gerrit.wikimedia.org/r/726094

Change 726102 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] puppetmaster::geoip: test if new license lets us download needed databases

https://gerrit.wikimedia.org/r/726102

Change 726102 merged by Dzahn:

[operations/puppet@production] puppetmaster::geoip: test if new license lets us download needed databases

https://gerrit.wikimedia.org/r/726102

Tested whether we can download all the existing databases PLUS the new databases using the same license.. and we can't.

When I tried that I got ""Invalid product ID or subscription expired"" for most of the existing ones.

Mentioned in SAL (#wikimedia-operations) [2021-10-05T20:06:46Z] <mutante> cumin 'puppetmaster*' "disable-puppet 'T288844 - T273673 - gerrit:721595 - ${USER}'"

Change 721595 merged by Dzahn:

[operations/puppet@production] geoip: replace maxmind update cron with system timer and config

https://gerrit.wikimedia.org/r/721595

Change 726684 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] geoip: remove absented cron code for maxmind update

https://gerrit.wikimedia.org/r/726684

Change 726684 merged by Dzahn:

[operations/puppet@production] geoip: remove absented cron code for maxmind update

https://gerrit.wikimedia.org/r/726684

Change 726696 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] puppetmaster/geoip: add puppet CA server name to geoip config pt.1

https://gerrit.wikimedia.org/r/726696

Change 726696 merged by Dzahn:

[operations/puppet@production] puppetmaster/geoip: add puppet CA server name to geoip config pt.1

https://gerrit.wikimedia.org/r/726696

Change 726699 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] geoip: skip pulling of maxmind files if not on the puppet CA server

https://gerrit.wikimedia.org/r/726699

Change 726699 merged by Dzahn:

[operations/puppet@production] geoip: skip pulling of maxmind files if not on the puppet CA server

https://gerrit.wikimedia.org/r/726699

Change 726703 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] geoip: use ISO-8601 date format in logs and fix log shell redirection

https://gerrit.wikimedia.org/r/726703

Change 726703 merged by Dzahn:

[operations/puppet@production] geoip: use ISO-8601 date format in logs and fix log shell redirection

https://gerrit.wikimedia.org/r/726703

Change 726718 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] geoip: also limit pulling of legacy databases to CA server

https://gerrit.wikimedia.org/r/726718

Change 726718 merged by Dzahn:

[operations/puppet@production] geoip: also limit pulling of legacy databases to CA server

https://gerrit.wikimedia.org/r/726718

Change 726720 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] puppetmaster/geoip: add puppet CA server name to geoip config pt.2

https://gerrit.wikimedia.org/r/726720

Change 726720 merged by Dzahn:

[operations/puppet@production] puppetmaster/geoip: add puppet CA server name to geoip config pt.2

https://gerrit.wikimedia.org/r/726720

Change 726094 merged by Dzahn:

[operations/puppet@production] mediawiki/geoip: add option to also pull new MaxMind databases from master

https://gerrit.wikimedia.org/r/726094

Change 726991 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] mediawiki: roll out maxmind dbs for ipinfo on canary appservers

https://gerrit.wikimedia.org/r/726991

Change 726991 merged by Dzahn:

[operations/puppet@production] mediawiki: roll out maxmind dbs for ipinfo on canary appservers

https://gerrit.wikimedia.org/r/726991

@phuedx The 2 new databases GeoIP2-Anonymous-IP.mmdb and GeoIP2-Enterprise.mmdb we got with the new license have now been deployed on all _canary_ appservers. (mwdebug* and selected mw* hosts).

The local path is /usr/share/GeoIPInfo (as opposed to previously existing /usr/share/GeoIP).

It would now be possible to test the IPInfo extension using that on a debug host. Hope that is good news :)

Deploying to the rest of the fleet is now just a simple switch in Hiera.

In T288844#7407062, @Dzahn wrote:

It would now be possible to test the IPInfo extension using that on a debug host. Hope that is good news :)

It is. Thank you very much for your work on this, @Dzahn!

@Joe So for the current/pre-k8s setup this is resolved, minus one Hiera flip to enable on all appservers what is enabled on canaries now.

I am planning to do that and then I would normally call this ticket resolved, but you also said to keep this topic open to talk about a solution for this in k8s. I'll suggest a subtask for that, ok?

Change 732099 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] mediawiki::appserver: fetch additional MaxMind databases on all appservers

https://gerrit.wikimedia.org/r/732099

Change 732099 merged by Dzahn:

[operations/puppet@production] mediawiki::appserver: fetch additional MaxMind databases on all appservers

https://gerrit.wikimedia.org/r/732099

Change 732440 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] mediawiki::appserver: fetch additional MaxMind databases on API servers

https://gerrit.wikimedia.org/r/732440

Change 732440 merged by Dzahn:

[operations/puppet@production] mediawiki::appserver: fetch additional MaxMind databases on API servers

https://gerrit.wikimedia.org/r/732440

The new database files are now rolled out to all production app and API servers (mediawiki::canary_appserver, mediawiki::appserver, mediawiki::appserver::canary_api, mediawiki::appserver::api) through the fetch_ipinfo_dbs key in Hiera.

Existing databases are also installed on mwmaint, mwlog and jobrunners via the mediawiki::common role but the new ones are not and I don't think they need to be.

I made a follow-up task to discuss how we will solve this with MediaWiki on Kubernetes: T293939

@phuedx I think for your purposes this should be solved now. On our side we have to discuss how to do this once we moved to Kubernetes soon'ish but you shouldn't have to worry about it for now and could start making tests on current prod infra.