deployment-imagescaler0[12] puppet broken due to missing nginx lua support
Closed, ResolvedPublic

Description

krenair@deployment-imagescaler02:~$ sudo /usr/sbin/nginx -g 'daemon on; master_process on;'
nginx: [emerg] unknown directive "lua_shared_dict" in /etc/nginx/sites-enabled/thumbor:1

https://gerrit.wikimedia.org/r/#/c/372254/

Krenair created this task.Aug 31 2017, 11:20 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 31 2017, 11:20 PM
Krenair updated the task description. (Show Details)Aug 31 2017, 11:22 PM
Krenair added a comment.EditedAug 31 2017, 11:31 PM
  • Installed libnginx-mod-http-lua
  • Manually installed /etc/nginx/prometheus.lua file to avoid puppet catch-22 (specifically, puppet will error out before creating that file due to nginx failing due to that file being missing), need to make the nginx service depend on that file somehow?
  • Killed existing deployment-imagescaler01 (was fine on -imagescaler02) nginx processes that were refusing to stop (preventing a restart due to port 8800 being in use)
  • Puppet now works

Need to puppetise the package installation and lua file dependency still.

Mentioned in SAL (#wikimedia-releng) [2017-08-31T23:32:12Z] <Krenair> fixed deployment-imagescaler0[12] puppet by installing a package and file manually, some puppetisation still needed - https://phabricator.wikimedia.org/T174746

Gilles added a comment.EditedSep 4 2017, 10:39 AM

I think the issue is simply that beta hasn't been configured to use the "extras" variant of nginx, like we did for prod: https://gerrit.wikimedia.org/r/#/c/372254/6/hieradata/role/eqiad/thumbor/mediawiki.yaml

But the thumbor nginx config uses the lua code regardless of which nginx version is present: https://gerrit.wikimedia.org/r/#/c/372254/6/modules/thumbor/templates/nginx.conf.erb

We should probably make the thumbor nginx config depend on the variant, to avoid future breakages like this, and simply configure beta to use the extras variant.

Gilles triaged this task as High priority.Sep 4 2017, 10:40 AM
Gilles claimed this task.
Gilles added projects: Performance-Team, Thumbor.

Change 375772 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/puppet@production] Only uses lua in Thumbor Nginx config if "extras" variant

https://gerrit.wikimedia.org/r/375772

Tried to set nginx::extras in https://horizon.wikimedia.org/project/prefixpuppet/?tab=prefix_puppet__puppet-deployment-imagescaler

But when running puppet:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: 'variant' must be 'full', 'extras', or 'light' (got: 'extras'). at /etc/puppet/modules/nginx/manifests/init.pp:32 on node deployment-imagescaler02.deployment-prep.eqiad.wmflabs

Hmmm...

Ah, my bad, I had written it as an array.

Gilles closed this task as Resolved.Sep 4 2017, 11:47 AM

Change 375772 merged by Filippo Giunchedi:
[operations/puppet@production] Thumbor: only use lua in nginx config if "extras" variant

https://gerrit.wikimedia.org/r/375772