Page MenuHomePhabricator
Create Task
Maniphest T98737

Migrate operations-dns-lint Jenkins job to labs instances
Closed, ResolvedPublic
Actions

Description

The job runs gdnsd and ends up depending on GeoIP as used in production. It is thus tied to productionSlaves label. When running it on a Precise labs instance we are missing /usr/share/GeoIP/GeoLiteCityv6.dat (and most probably other files):

Building remotely on integration-slave-precise-1013 (contintLabsSlave phpflavor-zend UbuntuPrecise)
  in workspace /mnt/jenkins-workspace/workspace/operations-dns-lint
Checkout:operations-dns-lint / /mnt/jenkins-workspace/workspace/operations-dns-lint - 
Last Built Revision: Revision 8063b2325e190ca65dc2c019ca404e501a5558e1 (detached)
Cloning the remote Git repository
Cloning repository git://zuul.eqiad.wmnet/operations/dns
Checking out Revision 8063b2325e190ca65dc2c019ca404e501a5558e1 (detached)
Cleaning workspace
Resetting working tree
[operations-dns-lint] $ /bin/bash -xe /tmp/hudson2459328528010707213.sh
+ mkdir -p /mnt/jenkins-workspace/workspace/operations-dns-lint/build
+ /usr/local/bin/authdns-lint /mnt/jenkins-workspace/workspace/operations-dns-lint /mnt/jenkins-workspace/workspace/operations-dns-lint/build
Using /mnt/jenkins-workspace/workspace/operations-dns-lint as the input working directory
Using /mnt/jenkins-workspace/workspace/operations-dns-lint/build as the output working directory (gdnsd chroot)
Generating zonefiles from zone templates
Generating gdnsd config
Copying GeoIP databases inside the chroot:
  /usr/share/GeoIP/GeoIPCity.dat
  /usr/share/GeoIP/GeoIP.dat
  /usr/share/GeoIP/GeoIPv6.dat
cp: cannot stat `/usr/share/GeoIP/GeoLiteCityv6.dat': No such file or directory
Finished: FAILURE

Details

SubjectRepoBranchLines +/-
contint: authdns::lint on light Jessie slaveoperations/puppetproduction+3 -0
contint: authdns::lint is only now only for Jessie labsoperations/puppetproduction+4 -3
Migrate operations-dns-lint to labs/Jessieintegration/configmaster+2 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.May 11 2015, 1:36 PM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added a project: Continuous-Integration-Config.
hashar subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 11 2015, 1:36 PM
hashar added a parent task: T86659: Migrate all jobs to labs slaves.May 19 2015, 9:34 AM
hashar triaged this task as Low priority.May 28 2015, 12:19 PM

The files are fetched from MaxMind on the puppetmaster under /volatile/ and are only fetched on production hosts.

hashar mentioned this in T98003: Upgrade prod DNS daemons to gdnsd 2.2.0.May 28 2015, 7:09 PM
faidon raised the priority of this task from Low to Medium.May 29 2015, 10:55 AM
faidon subscribed.
In T98737#1317341, @hashar wrote:

The files are fetched from MaxMind on the puppetmaster under /volatile/ and are only fetched on production hosts.

Tha'ts not exactly true. We do have a MaxMind subscription for production and indeed, we cannot share those files with Labs projects.

However, the freely-available GeoLite products can be installed in Labs and, in fact, we have puppet code to do exactly this. I'm not sure why GeoLiteCityv6.dat failed above. We can we try on a new Labs instance that runs jessie and has the geoip class included? I'll help troubleshoot that.

All that being said, it'd be ideal if Jenkins would actually check our configuration against the proper, production, non-free MaxMind databases (that cannot be installed in Labs in any way). Any ideas on how to accommodate this scenario?

hashar added a comment.May 29 2015, 12:53 PM

However, the freely-available GeoLite products can be installed in Labs and, in fact, we have puppet code to do exactly this. I'm not sure why GeoLiteCityv6.dat failed above. We can we try on a new Labs instance that runs jessie and has the geoip class included? I'll help troubleshoot that.

The CI slaves include chain is:

contint::packages
authdns::lint
geoip
geoip::data::puppet

Thus the geoip package is not installed. We have geoip-database on gallium, probably because the geoip puppet class used to install it.

Seems on labs we want to include geoip::data::package which provides geoip-database, but that is not sufficient, we need geoip-database and geoip-database-contrib as well:

geoip-database files
/usr/share/GeoIP/GeoIP.dat
/usr/share/GeoIP/GeoIPv6.dat

geoip-database-extra files
/usr/share/GeoIP/GeoIPASNum.dat
/usr/share/GeoIP/GeoIPCity.dat

geoip-database-contrib files
/usr/share/GeoIP/GeoIP.dat
/usr/share/GeoIP/GeoIPASNum.dat
/usr/share/GeoIP/GeoIPASNumv6.dat
/usr/share/GeoIP/GeoIPv6.dat
/usr/share/GeoIP/GeoLiteCity.dat
/usr/share/GeoIP/GeoLiteCityv6.dat

That later package also provides update scripts /usr/sbin/geoip-database-contrib_update
/usr/sbin/update-geoip-database. Not sure what they do nor whether they are enabled by default.

Seems to me geoip::data::package on Jessie should also install geoip-database-extra and geoip-database-contrib.

That would be a first step.

hashar added a comment.May 29 2015, 12:59 PM

@faidon wrote:

All that being said, it'd be ideal if Jenkins would actually check our configuration against the proper, production, non-free MaxMind databases (that cannot be installed in Labs in any way). Any ideas on how to accommodate this scenario?

If we had a Jessie server in prod realm (maybe on ganeti), we could attach it to the Jenkins master and then tie the job to run on that machine. The server would not run the rest of the CI job, so we would "just" need:

  • java7 / jenkins-slave user and its ssh public key (embedded in role::ci::slave)
  • authdns::lint puppet class

Then duplicate the current Jenkins job to run on that host which is trivial.

gerritbot subscribed.Jun 11 2015, 10:39 AM

Change 217467 had a related patch set uploaded (by Hashar):
contint: authdns::lint on light Jessie slave

https://gerrit.wikimedia.org/r/217467

gerritbot added a project: Patch-For-Review.Jun 11 2015, 10:39 AM
hashar added a comment.Jun 11 2015, 10:42 AM

I have pooled a Jessie slave that is very minimal. https://integration.wikimedia.org/ci/computer/integration-lightslave-jessie-1002/ with labels DebianJessie and contintLabsSlave. It is not suitable for other CI jobs but has authdns::lint included and would let us migrate the Jenkins job to it.

Seems the blocker is now having the GeoIP file GeoLiteCityv6.dat so either:

@hashar wrote in T98737#1320553

Seems to me geoip::data::package on Jessie should also install geoip-database-extra and geoip-database-contrib.

Or @hashar wrote in T98737#1320570

If we had a Jessie server in prod realm (maybe on ganeti), we could attach it to the Jenkins master and then tie the job to run on that machine.

hashar moved this task from Backlog to In progress on the Continuous-Integration-Config board.Jun 11 2015, 10:43 AM
faidon added a comment.Jun 11 2015, 11:21 AM

No, we don't rely on the packages for GeoIP databases. We use geoipupdate instead and GeoLiteCityv6.dat was a hack and isn't available there AFAIK (the database is essentially empty). The T98003 task is about gdnsd 2.2.0 which is the move to GeoIP2, so this is a very short-term problem.

I'll copy it manually on that slave (ugh, I know!) and remove it very soon once we migrate to 2.2.0.

gerritbot added a comment.Jun 11 2015, 11:41 AM

Change 217486 had a related patch set uploaded (by Hashar):
Migrate operations-dns-lint to labs/Jessie

https://gerrit.wikimedia.org/r/217486

hashar claimed this task.Jun 11 2015, 12:50 PM
gerritbot added a comment.Jun 11 2015, 12:57 PM

Change 217486 merged by jenkins-bot:
Migrate operations-dns-lint to labs/Jessie

https://gerrit.wikimedia.org/r/217486

hashar mentioned this in rCICF02f7be0c1d9a: Migrate operations-dns-lint to labs/Jessie.Jun 11 2015, 12:59 PM
gerritbot added a comment.Jun 11 2015, 1:06 PM

Change 217501 had a related patch set uploaded (by Hashar):
contint: authdns::lint is only now only for Jessie labs

https://gerrit.wikimedia.org/r/217501

gerritbot added a comment.Jun 11 2015, 1:09 PM

Change 217501 merged by Faidon Liambotis:
contint: authdns::lint is only now only for Jessie labs

https://gerrit.wikimedia.org/r/217501

hashar mentioned this in rOPUP264c4b6b2a0d: contint: authdns::lint is only now only for Jessie labs.Jun 11 2015, 1:09 PM
hashar closed this task as Resolved.Jun 11 2015, 1:13 PM

All resolved once @faidon found a workaround for the GeoIP file \O/

hashar moved this task from In progress to Archive on the Continuous-Integration-Config board.Jun 11 2015, 1:13 PM
hashar added a parent task: T98003: Upgrade prod DNS daemons to gdnsd 2.2.0.Jun 11 2015, 1:38 PM
gerritbot added a comment.Jul 6 2015, 1:33 PM

Change 217467 merged by Faidon Liambotis:
contint: authdns::lint on light Jessie slave

https://gerrit.wikimedia.org/r/217467

hashar mentioned this in rOPUP3e91574d6835: contint: authdns::lint on light Jessie slave.Jul 6 2015, 1:34 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits