Page MenuHomePhabricator

Migrate operations-dns-lint Jenkins job to labs instances
Closed, ResolvedPublic

Description

The job runs gdnsd and ends up depending on GeoIP as used in production. It is thus tied to productionSlaves label. When running it on a Precise labs instance we are missing /usr/share/GeoIP/GeoLiteCityv6.dat (and most probably other files):

Building remotely on integration-slave-precise-1013 (contintLabsSlave phpflavor-zend UbuntuPrecise)
  in workspace /mnt/jenkins-workspace/workspace/operations-dns-lint
Checkout:operations-dns-lint / /mnt/jenkins-workspace/workspace/operations-dns-lint - 
Last Built Revision: Revision 8063b2325e190ca65dc2c019ca404e501a5558e1 (detached)
Cloning the remote Git repository
Cloning repository git://zuul.eqiad.wmnet/operations/dns
Checking out Revision 8063b2325e190ca65dc2c019ca404e501a5558e1 (detached)
Cleaning workspace
Resetting working tree
[operations-dns-lint] $ /bin/bash -xe /tmp/hudson2459328528010707213.sh
+ mkdir -p /mnt/jenkins-workspace/workspace/operations-dns-lint/build
+ /usr/local/bin/authdns-lint /mnt/jenkins-workspace/workspace/operations-dns-lint /mnt/jenkins-workspace/workspace/operations-dns-lint/build
Using /mnt/jenkins-workspace/workspace/operations-dns-lint as the input working directory
Using /mnt/jenkins-workspace/workspace/operations-dns-lint/build as the output working directory (gdnsd chroot)
Generating zonefiles from zone templates
Generating gdnsd config
Copying GeoIP databases inside the chroot:
  /usr/share/GeoIP/GeoIPCity.dat
  /usr/share/GeoIP/GeoIP.dat
  /usr/share/GeoIP/GeoIPv6.dat
cp: cannot stat `/usr/share/GeoIP/GeoLiteCityv6.dat': No such file or directory
Finished: FAILURE

Details

Related Gerrit Patches:

Event Timeline

hashar created this task.May 11 2015, 1:36 PM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added a subscriber: hashar.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 11 2015, 1:36 PM
hashar triaged this task as Low priority.May 28 2015, 12:19 PM

The files are fetched from MaxMind on the puppetmaster under /volatile/ and are only fetched on production hosts.

faidon raised the priority of this task from Low to Medium.May 29 2015, 10:55 AM
faidon added a subscriber: faidon.

The files are fetched from MaxMind on the puppetmaster under /volatile/ and are only fetched on production hosts.

Tha'ts not exactly true. We do have a MaxMind subscription for production and indeed, we cannot share those files with Labs projects.

However, the freely-available GeoLite products can be installed in Labs and, in fact, we have puppet code to do exactly this. I'm not sure why GeoLiteCityv6.dat failed above. We can we try on a new Labs instance that runs jessie and has the geoip class included? I'll help troubleshoot that.

All that being said, it'd be ideal if Jenkins would actually check our configuration against the proper, production, non-free MaxMind databases (that cannot be installed in Labs in any way). Any ideas on how to accommodate this scenario?

However, the freely-available GeoLite products can be installed in Labs and, in fact, we have puppet code to do exactly this. I'm not sure why GeoLiteCityv6.dat failed above. We can we try on a new Labs instance that runs jessie and has the geoip class included? I'll help troubleshoot that.

The CI slaves include chain is:

contint::packages
authdns::lint
geoip
geoip::data::puppet

Thus the geoip package is not installed. We have geoip-database on gallium, probably because the geoip puppet class used to install it.


Seems on labs we want to include geoip::data::package which provides geoip-database, but that is not sufficient, we need geoip-database and geoip-database-contrib as well:

geoip-database files
/usr/share/GeoIP/GeoIP.dat
/usr/share/GeoIP/GeoIPv6.dat

geoip-database-extra files
/usr/share/GeoIP/GeoIPASNum.dat
/usr/share/GeoIP/GeoIPCity.dat

geoip-database-contrib files
/usr/share/GeoIP/GeoIP.dat
/usr/share/GeoIP/GeoIPASNum.dat
/usr/share/GeoIP/GeoIPASNumv6.dat
/usr/share/GeoIP/GeoIPv6.dat
/usr/share/GeoIP/GeoLiteCity.dat
/usr/share/GeoIP/GeoLiteCityv6.dat

That later package also provides update scripts /usr/sbin/geoip-database-contrib_update
/usr/sbin/update-geoip-database. Not sure what they do nor whether they are enabled by default.

Seems to me geoip::data::package on Jessie should also install geoip-database-extra and geoip-database-contrib.

That would be a first step.

@faidon wrote:

All that being said, it'd be ideal if Jenkins would actually check our configuration against the proper, production, non-free MaxMind databases (that cannot be installed in Labs in any way). Any ideas on how to accommodate this scenario?

If we had a Jessie server in prod realm (maybe on ganeti), we could attach it to the Jenkins master and then tie the job to run on that machine. The server would not run the rest of the CI job, so we would "just" need:

  • java7 / jenkins-slave user and its ssh public key (embedded in role::ci::slave)
  • authdns::lint puppet class

Then duplicate the current Jenkins job to run on that host which is trivial.

Change 217467 had a related patch set uploaded (by Hashar):
contint: authdns::lint on light Jessie slave

https://gerrit.wikimedia.org/r/217467

I have pooled a Jessie slave that is very minimal. https://integration.wikimedia.org/ci/computer/integration-lightslave-jessie-1002/ with labels DebianJessie and contintLabsSlave. It is not suitable for other CI jobs but has authdns::lint included and would let us migrate the Jenkins job to it.

Seems the blocker is now having the GeoIP file GeoLiteCityv6.dat so either:

@hashar wrote in T98737#1320553

Seems to me geoip::data::package on Jessie should also install geoip-database-extra and geoip-database-contrib.

Or @hashar wrote in T98737#1320570

If we had a Jessie server in prod realm (maybe on ganeti), we could attach it to the Jenkins master and then tie the job to run on that machine.

No, we don't rely on the packages for GeoIP databases. We use geoipupdate instead and GeoLiteCityv6.dat was a hack and isn't available there AFAIK (the database is essentially empty). The T98003 task is about gdnsd 2.2.0 which is the move to GeoIP2, so this is a very short-term problem.

I'll copy it manually on that slave (ugh, I know!) and remove it very soon once we migrate to 2.2.0.

Change 217486 had a related patch set uploaded (by Hashar):
Migrate operations-dns-lint to labs/Jessie

https://gerrit.wikimedia.org/r/217486

hashar claimed this task.Jun 11 2015, 12:50 PM

Change 217486 merged by jenkins-bot:
Migrate operations-dns-lint to labs/Jessie

https://gerrit.wikimedia.org/r/217486

Change 217501 had a related patch set uploaded (by Hashar):
contint: authdns::lint is only now only for Jessie labs

https://gerrit.wikimedia.org/r/217501

Change 217501 merged by Faidon Liambotis:
contint: authdns::lint is only now only for Jessie labs

https://gerrit.wikimedia.org/r/217501

hashar closed this task as Resolved.Jun 11 2015, 1:13 PM

All resolved once @faidon found a workaround for the GeoIP file \O/

Change 217467 merged by Faidon Liambotis:
contint: authdns::lint on light Jessie slave

https://gerrit.wikimedia.org/r/217467