post build failures for operations/puppet on operations-puppet-doc
Closed, ResolvedPublic

Description

Upstream bug: https://tickets.puppetlabs.com/browse/PUP-3261

I noticed on 2016-08-15, the post build process is successful. Example: https://gerrit.wikimedia.org/r/#/c/304220/

The on 2016-08-16, the post build for patches fails. Example: https://gerrit.wikimedia.org/r/#/c/302761/

It seems it is the same failure for all patchsets in operations/puppet. This doesn't affect our use, but it doesn't update our puppet documentation anymore, since it fails.

https://integration.wikimedia.org/ci/job/operations-puppet-doc/25626/console

22:26:11 + /usr/bin/puppet doc --mode rdoc --outputdir /mnt/jenkins-workspace/workspace/operations-puppet-doc/doc --modulepath modules --manifestdir manifests
22:26:15 Warning: Unrecognised escape sequence '\;' in file /mnt/jenkins-workspace/workspace/operations-puppet-doc/src/modules/mailman/manifests/cron.pp at line 6
22:26:15 Warning: Deprecation notice:  Resource references should now be capitalized on line 122 in file /mnt/jenkins-workspace/workspace/operations-puppet-doc/src/modules/statistics/manifests/wmde.pp
22:26:17 Warning: You cannot collect without storeconfigs being set on line 80 in file /mnt/jenkins-workspace/workspace/operations-puppet-doc/src/modules/ssh/manifests/server.pp
22:26:19 Warning: Unrecognised escape sequence '\?' in file /mnt/jenkins-workspace/workspace/operations-puppet-doc/src/modules/dataset/manifests/cron/wikitech_dumps.pp at line 27
22:27:32 Warning: You cannot collect without storeconfigs being set on line 84 in file /mnt/jenkins-workspace/workspace/operations-puppet-doc/src/modules/openstack/manifests/nova/compute.pp
22:27:32 Warning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored on line 87 in file /mnt/jenkins-workspace/workspace/operations-puppet-doc/src/modules/openstack/manifests/nova/compute.pp
22:27:33 Before reporting this, could you check that the file you're documenting
22:27:33 has proper syntax:
22:27:33 
22:27:33   /usr/bin/ruby1.9.1 -c /mnt/jenkins-workspace/workspace/operations-puppet-doc/src/modules/puppetdbquery/bin/find-nodes
22:27:33 
22:27:33 RDoc is not a full Ruby parser and will fail when fed invalid ruby programs.
22:27:33 
22:27:33 The internal error was:
22:27:33 
22:27:33 	(Errno::ENOENT) No such file or directory - dummy.rb
22:27:33 
22:27:33 Error: Could not generate documentation: No such file or directory - dummy.rb
22:27:33 Build step 'Execute shell' marked build as failure
22:27:33 Finished: FAILURE
RobH created this task.Aug 17 2016, 6:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 17 2016, 6:00 PM
bd808 added a subscriber: bd808.Aug 18 2016, 4:35 PM

It looks like the rdoc parser hates something about the modules/puppetdbquery/bin/find-nodes file introduced in rOPUPb1492be. That whole module is marked as being excluded from rubocop linting and appears to come from an upstream source.

bd808 added a comment.Aug 18 2016, 4:43 PM

FWIW, I don't get that failure when I run bundle exec rake doc locally:

$ bundle exec rake doc
Running puppet doc --mode rdoc --all --manifestdir manifests --modulepath modules
Warning: You cannot collect without storeconfigs being set on line 80 in file /Users/bd808/projects/wmf/operations/puppet/modules/bacula/manifests/client.pp
Warning: You cannot collect without storeconfigs being set on line 44 in file /Users/bd808/projects/wmf/operations/puppet/modules/bacula/manifests/client/job.pp
Warning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored on line 27 in file /Users/bd808/projects/wmf/operations/puppet/modules/bacula/manifests/console.pp
Warning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored on line 45 in file /Users/bd808/projects/wmf/operations/puppet/modules/bacula/manifests/director.pp
Warning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored on line 50 in file /Users/bd808/projects/wmf/operations/puppet/modules/bacula/manifests/director.pp
Warning: You cannot collect without storeconfigs being set on line 141 in file /Users/bd808/projects/wmf/operations/puppet/modules/bacula/manifests/director.pp
Warning: You cannot collect without storeconfigs being set on line 65 in file /Users/bd808/projects/wmf/operations/puppet/modules/bacula/manifests/storage/device.pp
Warning: Unrecognised escape sequence '\.' in file /Users/bd808/projects/wmf/operations/puppet/modules/confluent/manifests/kafka/mirror/alerts.pp at line 23
Warning: Unrecognised escape sequence '\?' in file /Users/bd808/projects/wmf/operations/puppet/modules/dataset/manifests/cron/wikitech_dumps.pp at line 27
Warning: You cannot collect without storeconfigs being set on line 25 in file /Users/bd808/projects/wmf/operations/puppet/modules/ganglia/manifests/monitor.pp
Warning: Unrecognised escape sequence '\;' in file /Users/bd808/projects/wmf/operations/puppet/modules/mailman/manifests/cron.pp at line 6
Warning: You cannot collect without storeconfigs being set on line 49 in file /Users/bd808/projects/wmf/operations/puppet/modules/monitoring/manifests/host.pp
Warning: You cannot collect without storeconfigs being set on line 67 in file /Users/bd808/projects/wmf/operations/puppet/modules/monitoring/manifests/host.pp
Warning: You cannot collect without storeconfigs being set on line 94 in file /Users/bd808/projects/wmf/operations/puppet/modules/monitoring/manifests/service.pp
Warning: You cannot collect without storeconfigs being set on line 84 in file /Users/bd808/projects/wmf/operations/puppet/modules/openstack/manifests/nova/compute.pp
Warning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored on line 87 in file /Users/bd808/projects/wmf/operations/puppet/modules/openstack/manifests/nova/compute.pp
Warning: Unrecognised escape sequence '\.' in file /Users/bd808/projects/wmf/operations/puppet/modules/role/manifests/kafka/main/mirror.pp at line 47
Warning: You cannot collect without storeconfigs being set on line 80 in file /Users/bd808/projects/wmf/operations/puppet/modules/ssh/manifests/server.pp
Warning: Deprecation notice:  Resource references should now be capitalized on line 122 in file /Users/bd808/projects/wmf/operations/puppet/modules/statistics/manifests/wmde.pp
Warning: tag is a metaparam; this value will inherit to all contained resources in the toollabs::kubebuilder definition
Error: Could not generate documentation: error generating /Users/bd808/projects/wmf/operations/puppet/doc/modules/cassandra/manifests/sysctl_pp.html: Error while evaluating /Users/bd808/.rbenv/versions/2.1.3/lib/ruby/2.1.0/rdoc/generator/template/darkfish/page.rhtml: undefined method `intern' for []:Array (RDoc::Error)
$ echo $?
0
$ bundle exec ruby --version
ruby 2.1.3p242 (2014-09-19 revision 47630) [x86_64-darwin12.0]
RobH added a subscriber: Joe.Aug 18 2016, 5:27 PM

It appears that was a change from @Joe, so looping him onto this task.

bd808 added a comment.Aug 18 2016, 5:44 PM

This upstream bug seems related -- puppet doc passes files to rdoc too agressively. Sadly open for 2 years with no activity at all.

Maybe @hashar or @zeljkofilipin can take a look and figure out how to run under a different ruby or exclude the file that puppet doc hates from being processed?

Dzahn added a subscriber: Dzahn.Aug 24 2016, 8:32 PM

Yep, i noticed this too in operations/puppet.

The issue seems to be the "Could not generate documentation: No such file or directory - dummy.rb" part.

Thanks for the task. It is due to what Bryan said: puppet doc passes files to rdoc too agressively.

The issue is due to modules/puppetdbquery/bin/find-nodes which is passed to rdoc but that puppet rdoc fails to load.

I have no idea how to fix it. A lame way would be to have the Jenkins job to just rm -fR modules/*/bin

Change 307654 had a related patch set uploaded (by Paladox):
Add rm -fR "$WORKSPACE/modules/*/bin" to jenkins job operations-puppet-doc

https://gerrit.wikimedia.org/r/307654

hashar triaged this task as "High" priority.Sep 3 2016, 12:23 PM

@hashar the fix you wrote in T143233#2580487 I uploaded it to https://gerrit.wikimedia.org/r/307654 waiting for a review?

It will hopefully fix the problem.

hashar added a comment.Sep 8 2016, 1:49 PM

Puppet has a blacklist of files pattern to not pass to rdoc:

lib/puppet/util/rdoc.rb
# specify our own format & where to output
options = [ "--fmt", "puppet",
            "--quiet",
            "--exclude", "/modules/[^/]*/spec/.*$",
            "--exclude", "/modules/[^/]*/files/.*$",
            "--exclude", "/modules/[^/]*/tests/.*$",
            "--exclude", "/modules/[^/]*/templates/.*$",
            "--op", outputdir ]

rdoc supports env variable RDOCOPT to pass extra options, so potentially we could use:

RDOCOPT="--exclude='/modules/[^/]*/bin/.*$'"

Not sure whether it will merge the settings from the env variables with the other settings though.

I cant build the doc on my local machine for now :(

hashar edited the task description. (Show Details)
hashar added a comment.Sep 8 2016, 2:06 PM

I have manually hacked the Jenkins job via the web UI to add RDOCOPT='--exclude=/modules/[^/]*/bin/.*$' and --trace.

The doc has build in roughly 4 minutes https://integration.wikimedia.org/ci/job/operations-puppet-doc/26257/

Result: https://doc.wikimedia.org/puppet/

Needs to check whether the env var ends up overridding the --excludes which are build-in puppet. I guess if we see doc for spec/files/tests/templates, that is not good.

Need to update the JJB definition.

Change 309327 had a related patch set uploaded (by Paladox):
Add env params['RDOCOPT'] = '--exclude=/modules/[^/]*/bin/.*$'

https://gerrit.wikimedia.org/r/309327

Change 309328 had a related patch set (by Paladox) published:
Also add --exclude=/modules/[^/]*/bin/.*$ to the rdoc

https://gerrit.wikimedia.org/r/309328

hashar added a comment.Sep 8 2016, 3:00 PM

Looks like previously it took several minutes to build anyway:*

#256038 min 7 secintegration-slave-trusty-1001
#256026 min 24 secintegration-slave-trusty-1016
#256017 min 42 secintegration-slave-trusty-1001
#256006 min 50 secintegration-slave-trusty-1017
#255997 min 40 secintegration-slave-trusty-1001
#255986 min 36 secintegration-slave-trusty-1018
#255976 min 35 secintegration-slave-trusty-1012

Change 309328 abandoned by Paladox:
Also add --exclude=/modules/[^/]*/bin/.*$ to the rdoc

https://gerrit.wikimedia.org/r/309328

Change 309332 had a related patch set uploaded (by Hashar):
Puppet doc now ignore /bin files

https://gerrit.wikimedia.org/r/309332

hashar closed this task as "Resolved".Sep 8 2016, 3:22 PM
hashar claimed this task.

I have updated the job in JJB and refreshed it https://gerrit.wikimedia.org/r/#/c/309332/

Seems the hack is working. I have also made it to pass to puppet --trace which might help in case miserably fail for some random reason.

Paladox reopened this task as "Open".Sep 8 2016, 3:25 PM

Re opening per @hashar it seems that when you click on for example

https://doc.wikimedia.org/mnt/jenkins-workspace/workspace/operations-puppet-doc/src/modules/admin/README.html

it shows

The requested URL /mnt/jenkins-workspace/workspace/operations-puppet-doc/src/modules/admin/README.html was not found on this server.

And it happends for the other files too.

Change 307654 abandoned by Hashar:
Add rm -fR "$WORKSPACE/modules/*/bin" to jenkins job operations-puppet-doc

Reason:
Fixed differently by making puppet doc to ignore bin dirs https://gerrit.wikimedia.org/r/#/c/309332/

https://gerrit.wikimedia.org/r/307654

Change 309327 abandoned by Hashar:
Add env params['RDOCOPT'] = '--exclude=/modules/[^/]*/bin/.*$'

Reason:
Fixed differently

https://gerrit.wikimedia.org/r/309327

Change 309332 merged by jenkins-bot:
Puppet doc now deletes bin directories

https://gerrit.wikimedia.org/r/309332

Change 319665 had a related patch set uploaded (by Hashar):
Stop generating puppet doc on each change

https://gerrit.wikimedia.org/r/319665

Change 319665 merged by jenkins-bot:
Stop generating puppet doc on each change

https://gerrit.wikimedia.org/r/319665

The operations-puppet-doc no more triggers after a change is merged, it was simply too long and consumed instances. I have made it to poll the git repository hourly and generate the doc.

Now looking at the last build https://integration.wikimedia.org/ci/job/operations-puppet-doc/lastBuild/console , it fails:

Error: Could not generate documentation: error generating /home/jenkins/workspace/operations-puppet-doc/doc/modules/cassandra/manifests/twcs_pp.html: Error while evaluating /usr/lib/ruby/2.1.0/rdoc/generator/template/darkfish/page.rhtml: undefined method `intern' for []:Array (RDoc::Error)

And I cannot reproduce locally.

There is also an oddity that has rdoc use the absolute filepath when generating the href links :(

Change 309561 had a related patch set uploaded (by Hashar):
Puppet doc with strings/yard

https://gerrit.wikimedia.org/r/309561

hashar added a subscriber: dduvall.Jan 3 2017, 10:58 PM

Early September I looked at puppet-strings which is supposedly a replacement for puppet doc command. I gave it a few try in a draft change, and finally this evening all the bits connected :]

https://gerrit.wikimedia.org/r/#/c/309561/4 lets one generate documentation for ALL pp files using yard for documentation generation. It is arguably much better.

Once happy with the change, we will have to adjust https://integration.wikimedia.org/ci/job/operations-puppet-doc/ configuration to use bundle exec rake doc.

Change 330390 had a related patch set uploaded (by Hashar):
build: bump bundler puppet version 3.4 -> 3.7

https://gerrit.wikimedia.org/r/330390

Change 330390 merged by Alexandros Kosiaris:
build: bump bundler puppet version 3.4 -> 3.7

https://gerrit.wikimedia.org/r/330390

Change 330415 had a related patch set uploaded (by Hashar):
Switch ops/puppet doc to rake doc

https://gerrit.wikimedia.org/r/330415

Change 330423 had a related patch set uploaded (by Hashar):
Drop link to puppet source

https://gerrit.wikimedia.org/r/330423

Change 309561 merged by Filippo Giunchedi:
Puppet doc with strings/yard

https://gerrit.wikimedia.org/r/309561

Change 330415 merged by jenkins-bot:
Switch ops/puppet doc to rake doc

https://gerrit.wikimedia.org/r/330415

Mentioned in SAL (#wikimedia-releng) [2017-01-05T17:20:52Z] <hashar> Dropping puppet source from https://doc.wikimedia.org/puppetsource/ . contint1001: sudo rm -fR /srv/org/wikimedia/doc/puppetsource (T143233)

Change 330423 merged by jenkins-bot:
Drop link to puppet source

https://gerrit.wikimedia.org/r/330423

hashar closed this task as "Resolved".Jan 5 2017, 5:23 PM

I got rid of puppet rdoc entirely. We are now using puppet-strings and yard. End result can be seen at https://doc.wikimedia.org/puppet/

The job poll the git repo on an hourly basis and spurt the doc automagically. \O/