Page MenuHomePhabricator
Create Task
Maniphest T110836

nginx puppet manifest requires nfs so error page cannot be updated over puppet
Closed, ResolvedPublic
Actions

Description

for https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin/emergency_guides/labs_down_notification

Notice: /Stage[main]/Role::Labs::Instance/Exec[block-for-project-export]/returns: executed successfully
Notice: /Stage[main]/Toollabs/File[/data/project/.system/gridengine]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Toollabs/File[/data/project/.system/gridengine]: Skipping because of failed dependencies
Notice: /Stage[main]/Toollabs/File[/data/project/.system/store]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Toollabs/File[/data/project/.system/store]: Skipping because of failed dependencies
Notice: /Stage[main]/Toollabs/Exec[make_known_hosts]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Toollabs/Exec[make_known_hosts]: Skipping because of failed dependencies
Notice: /Stage[main]/Toollabs/Mount[/var/lib/gridengine]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Toollabs/Mount[/var/lib/gridengine]: Skipping because of failed dependencies
Notice: /Stage[main]/Gridengine/Package[gridengine-common]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Gridengine/Package[gridengine-common]: Skipping because of failed dependencies
Notice: /Stage[main]/Toollabs/Labsdebrepo[/data/project/.system/deb-trusty]/Package[dpkg-dev]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Toollabs/Labsdebrepo[/data/project/.system/deb-trusty]/Package[dpkg-dev]: Skipping because of failed dependencies
Notice: /Stage[main]/Misc::Labsdebrepo/Labsdebrepo[/data/project/repo]/Exec[Turn /data/project/repo into deb repo]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Misc::Labsdebrepo/Labsdebrepo[/data/project/repo]/Exec[Turn /data/project/repo into deb repo]: Skipping because of failed dependencies
Notice: /Stage[main]/Misc::Labsdebrepo/Labsdebrepo[/data/project/repo]/File[/etc/apt/sources.list.d/labsdebrepo.list]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Misc::Labsdebrepo/Labsdebrepo[/data/project/repo]/File[/etc/apt/sources.list.d/labsdebrepo.list]: Skipping because of failed dependencies
Notice: /Stage[main]/Toollabs/File[/data/project/.system/gridengine/collectors]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Toollabs/File[/data/project/.system/gridengine/collectors]: Skipping because of failed dependencies
Notice: /Stage[main]/Toollabs/File[/etc/ssh/ssh_known_hosts]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Toollabs/File[/etc/ssh/ssh_known_hosts]: Skipping because of failed dependencies
Notice: /Stage[main]/Role::Labs::Instance/Exec[block-for-home-export]/returns: executed successfully
Notice: /Stage[main]/Toollabs/File[/data/project/.system/store/hostkey-tools-webproxy-02.tools.eqiad.wmflabs]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Toollabs/File[/data/project/.system/store/hostkey-tools-webproxy-02.tools.eqiad.wmflabs]: Skipping because of failed dependencies
Notice: /Stage[main]/Nginx/Package[nginx-extras]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Nginx/Package[nginx-extras]: Skipping because of failed dependencies
Notice: /Stage[main]/Nginx/File[/etc/nginx/sites-enabled]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Nginx/File[/etc/nginx/sites-enabled]: Skipping because of failed dependencies
Notice: /Stage[main]/Nginx/File[/etc/nginx/sites-available]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Nginx/File[/etc/nginx/sites-available]: Skipping because of failed dependencies
Notice: /Stage[main]/Dynamicproxy/Diamond::Collector::Nginx[diamond-monitor-proxy]/Nginx::Status_site[status]/Nginx::Site[status]/File[/etc/nginx/sites-available/status]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Dynamicproxy/Diamond::Collector::Nginx[diamond-monitor-proxy]/Nginx::Status_site[status]/Nginx::Site[status]/File[/etc/nginx/sites-available/status]: Skipping because of failed dependencies
Notice: /Stage[main]/Dynamicproxy/File[/etc/nginx/lua]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Dynamicproxy/File[/etc/nginx/lua]: Skipping because of failed dependencies
Notice: /Stage[main]/Dynamicproxy/File[/etc/nginx/lua/list-proxy-entries.lua]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Dynamicproxy/File[/etc/nginx/lua/list-proxy-entries.lua]: Skipping because of failed dependencies
Notice: /Stage[main]/Dynamicproxy/Nginx::Site[proxymanager]/File[/etc/nginx/sites-available/proxymanager]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Dynamicproxy/Nginx::Site[proxymanager]/File[/etc/nginx/sites-available/proxymanager]: Skipping because of failed dependencies
Notice: /Stage[main]/Dynamicproxy/Nginx::Site[proxymanager]/File[/etc/nginx/sites-enabled/proxymanager]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Dynamicproxy/Nginx::Site[proxymanager]/File[/etc/nginx/sites-enabled/proxymanager]: Skipping because of failed dependencies
Notice: /Stage[main]/Dynamicproxy/File[/etc/nginx/lua/resty]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Dynamicproxy/File[/etc/nginx/lua/resty]: Skipping because of failed dependencies
Notice: /Stage[main]/Dynamicproxy/File[/etc/nginx/lua/resty/redis.lua]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Dynamicproxy/File[/etc/nginx/lua/resty/redis.lua]: Skipping because of failed dependencies
Notice: /Stage[main]/Dynamicproxy/File[/etc/nginx/lua/urlproxy.lua]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Dynamicproxy/File[/etc/nginx/lua/urlproxy.lua]: Skipping because of failed dependencies
Notice: /Stage[main]/Dynamicproxy/Nginx::Site[proxy]/File[/etc/nginx/sites-available/proxy]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Dynamicproxy/Nginx::Site[proxy]/File[/etc/nginx/sites-available/proxy]: Skipping because of failed dependencies
Notice: /Stage[main]/Dynamicproxy/Nginx::Site[proxy]/File[/etc/nginx/sites-enabled/proxy]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Dynamicproxy/Nginx::Site[proxy]/File[/etc/nginx/sites-enabled/proxy]: Skipping because of failed dependencies
Notice: /Stage[main]/Toollabs/Labsdebrepo[/data/project/.system/deb-trusty]/File[/data/project/.system/deb-trusty]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Toollabs/Labsdebrepo[/data/project/.system/deb-trusty]/File[/data/project/.system/deb-trusty]: Skipping because of failed dependencies
Notice: /Stage[main]/Toollabs/Labsdebrepo[/data/project/.system/deb-trusty]/Exec[Turn /data/project/.system/deb-trusty into deb repo]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Toollabs/Labsdebrepo[/data/project/.system/deb-trusty]/Exec[Turn /data/project/.system/deb-trusty into deb repo]: Skipping because of failed dependencies
Notice: /Stage[main]/Toollabs/Labsdebrepo[/data/project/.system/deb-trusty]/Exec[Turn /data/project/.system/deb-trusty into deb repo]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Toollabs/Labsdebrepo[/data/project/.system/deb-trusty]/Exec[Turn /data/project/.system/deb-trusty into deb repo]: Skipping because of failed dependencies
Notice: /Stage[main]/Toollabs/Labsdebrepo[/data/project/.system/deb-trusty]/File[/etc/apt/sources.list.d/tools-project.list]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Toollabs/Labsdebrepo[/data/project/.system/deb-trusty]/File[/etc/apt/sources.list.d/tools-project.list]: Skipping because of failed dependencies
Notice: /Stage[main]/Nginx/Package[nginx-common]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Nginx/Package[nginx-common]: Skipping because of failed dependencies
Notice: /Stage[main]/Dynamicproxy/File[/etc/nginx/nginx.conf]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Dynamicproxy/File[/etc/nginx/nginx.conf]: Skipping because of failed dependencies
Notice: /Stage[main]/Dynamicproxy/File[/etc/security/limits.conf]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Dynamicproxy/File[/etc/security/limits.conf]: Skipping because of failed dependencies
Notice: /Stage[main]/Dynamicproxy/Diamond::Collector::Nginx[diamond-monitor-proxy]/Nginx::Status_site[status]/Nginx::Site[status]/File[/etc/nginx/sites-enabled/status]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Dynamicproxy/Diamond::Collector::Nginx[diamond-monitor-proxy]/Nginx::Status_site[status]/Nginx::Site[status]/File[/etc/nginx/sites-enabled/status]: Skipping because of failed dependencies
Notice: /Stage[main]/Toollabs/Labsdebrepo[/data/project/.system/deb-trusty]/File[/etc/apt/preferences.d/tools-project.pref]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Toollabs/Labsdebrepo[/data/project/.system/deb-trusty]/File[/etc/apt/preferences.d/tools-project.pref]: Skipping because of failed dependencies
Notice: /Stage[main]/Nginx/File[/etc/nginx/conf.d]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Nginx/File[/etc/nginx/conf.d]: Skipping because of failed dependencies
Notice: /Stage[main]/Nginx/Service[nginx]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Nginx/Service[nginx]: Skipping because of failed dependencies
Notice: Finished catalog run in 1178.05 seconds

Related Objects

Event Timeline

valhallasw created this task.Aug 30 2015, 10:44 AM
valhallasw raised the priority of this task from to Needs Triage.
valhallasw updated the task description. (Show Details)
valhallasw added a project: Toolforge.
valhallasw subscribed.
Restricted Application added a project: Cloud-Services. · View Herald TranscriptAug 30 2015, 10:44 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
scfc added subscribers: yuvipanda, scfc.Aug 30 2015, 4:20 PM

At the moment, we include the "local" NFS Debian repository everywhere.

  1. I believe for the proxy we nowadays only need the package lua-json_1.3.2-1_all.deb (cf. https://gerrit.wikimedia.org/r/#/c/203313/) which – some day – will come through Ubuntu Backports (cf. T91874). After that, we could limit the inclusion of the local repository to the hosts that need them, or, even better, to the packages that come from there (package { 'x': ensure => installed, require => Labsdebrepo[$toollabs::repo], }). The latter would also solve Puppet errors on new instances where packages are required before the configuration for the local repository is in place.
  2. @yuvipanda recently created the aptly Puppet module which – AFAIUI – creates an apt repository accessible by network. This requires testing how the aptly server reacts to when its "source" file system is inaccessible or a copying mechanism/manual checklist to refresh an instance-local file system with the packages from the source, and it would be nice if we could include the aptly server on multiple instances and configure the clients via Hiera.
valhallasw updated the task description. (Show Details)Aug 30 2015, 8:24 PM
valhallasw set Security to None.
valhallasw added a comment.Aug 30 2015, 8:35 PM

Yes, I think you're right the root cause is the labsdebrepo. Specifically, I think the dependency chain is

Notice: /Stage[main]/Dynamicproxy/Nginx::Site[proxy]/File[/etc/nginx/sites-available/proxy]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Dynamicproxy/Nginx::Site[proxy]/File[/etc/nginx/sites-available/proxy]: Skipping because of failed dependencies
Notice: /Stage[main]/Dynamicproxy/Nginx::Site[proxy]/File[/etc/nginx/sites-enabled/proxy]: Dependency File[/data/project/.system] has failures: true
Warning: /Stage[main]/Dynamicproxy/Nginx::Site[proxy]/File[/etc/nginx/sites-enabled/proxy]: Skipping because of failed dependencies

which depends on Package["nginx-extras"] via the 'nginx' tag (module nginx). That Package is installed in module dynamicproxy using

class { 'nginx':
    variant => 'extras',
    require => Class['misc::labsdebrepo'],
}

and of course Class['misc::labsdebrepo'] depends on NFS to be online. The require in this nginx class is because of the lua-json issue you pointed out.

scfc added a comment.Aug 30 2015, 9:50 PM

I think the require in:

class { 'nginx':
    variant => 'extras',
    require => Class['misc::labsdebrepo'],
}

is an artifact when the proxy ran Precise and the nginx version needed was only available from Trusty. To clean up that manifest up (with a focus on the Lua modules) I had filed T95442.

There are of course also references to /data/project/.system and subdirectories themselves in class toollabs.

valhallasw added a comment.Aug 31 2015, 7:56 AM

*nod*. In general, it's OK if parts of the puppet manifest fail, but in this case, it prevented /etc/nginx/sites-enabled/proxy from being updated.

To switch to the 503 handler, puppet needs to do three things:

  • update /var/www/error/errorpage.html (worked correctly)
  • update /etc/nginx/sites-enabled/proxy to show /var/www/error/* instead of proxying (didn't work)
  • reload nginx configuration (also didn't work)
valhallasw triaged this task as Low priority.Oct 4 2015, 11:43 AM
valhallasw moved this task from Backlog to Ready to be worked on on the Toolforge board.Oct 4 2015, 11:47 AM
yuvipanda added a comment.Dec 21 2015, 7:03 PM

This should be fine now that we're using aptly and it doesn't use NFS.

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:47 PM
• GTirloni edited projects, added cloud-services-team (Kanban); removed Cloud-Services.Mar 23 2019, 9:32 PM
bd808 closed this task as Resolved.Mar 26 2019, 1:06 AM
bd808 claimed this task.
bd808 added subscribers: yuvipanda, bd808.
In T110836#1895472, @yuvipanda wrote:

This should be fine now that we're using aptly and it doesn't use NFS.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL