Mentioned in SAL (#wikimedia-releng) [2017-11-20T09:29:36Z] <hashar> deployment-tin: apt-mark hold scap | the apt-repo on deployment-tin is out of date | T180935

deployment-cache-text04 and deployment-cache-upload04 are broken because hieradata for roles are not applied on labs T120165. They are applied via the Horizon Prefix Puppet page.

cache::app_def_be_ops is missing between_bytes_timeout. I have synced the settings from puppet.git. For the prefix deployment-cache-text:

before:

cache::app_def_be_opts:
 connect_timeout: 5s
 first_byte_timeout: 180s
 max_connections: 1000
 port: 80

after:

cache::app_def_be_opts:
  port: 80
  connect_timeout: '3s'
  first_byte_timeout: '63s'     
  between_bytes_timeout: '31s'  
  max_connections: 1000

For the prefix deployment-cache-upload:

before:

cache::app_def_be_opts:
  connect_timeout: 5s
  first_byte_timeout: 35s
  max_connections: 10000
  port: 80

after:

cache::app_def_be_opts:
  port: 80
  connect_timeout: '5s'
  first_byte_timeout: '35s'
  between_bytes_timeout: '60s'
  max_connections: 10000

That fixed puppet on both deployment-cache-text04 and deployment-cache-upload04.

Mentioned in SAL (#wikimedia-releng) [2017-11-20T09:39:06Z] <hashar> deployment-prep added missing key between_bytes_timeout to cache::app_def_be_opts for deployment-cache-text04 and deployment-cache-upload04 | T180935

Mentioned in SAL (#wikimedia-releng) [2017-11-20T10:05:40Z] <hashar> deployment-phab : set hiera 'phabricator_cluster_search: []' trying to unblock puppet and soft rebooted the instance | T180935

If deployment-mx is still in use/needed, it should be reimaged to jessie or stretch.

deployment-tin seems failing because scap is put on hold, since 3.7.3 is also on apt.wikimedia.org "apt-mark unhold scap" should fix it.

I have marked scap on hold to get the version from apt.wikimedia.org, else it tries to get an outdated version generated by CI (from deployment-prep which has priority 1500).

$ apt-cache policy scap
scap:
  Installed: 3.7.3-1
  Candidate: 3.6.0-1~20171117182426.238
  Version table:
 *** 3.7.3-1 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status
     3.6.0-1~20171117182426.238 0
       1500 http://deployment-tin.deployment-prep.eqiad.wmflabs/repo/ jessie-deployment-prep/main amd64 Packages

Would have to sort that out with @mmodell

for deployment-phab we could do https://phabricator.wikimedia.org/P6353 (syntax untested but should work)

In T180935#3774154, @hashar wrote:

I have marked scap on hold to get the version from apt.wikimedia.org, else it tries to get an outdated version generated by CI (from deployment-prep which has priority 1500).

$ apt-cache policy scap
scap:
  Installed: 3.7.3-1
  Candidate: 3.6.0-1~20171117182426.238
  Version table:
 *** 3.7.3-1 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status
     3.6.0-1~20171117182426.238 0
       1500 http://deployment-tin.deployment-prep.eqiad.wmflabs/repo/ jessie-deployment-prep/main amd64 Packages

Would have to sort that out with @mmodell

@mmodell is out this week from the looks of it. The package built by CI is actually a more recent version of scap. CI scap is built from master which is the branch where scap development is done rather than release where the master branch is merged for production deployments after cooking on beta a while.

The version numbers looked wrong because although scap/version.py was being updated as part of development, the debian/changelog in the master branch was missing a bunch of stuff. I pushed up rMSCAfa9df0ce262b982bc0c3a5f0bdfb34836774902e and ran apt-mark unhold scap so this should be fixed now.

Thanks @thcipriani and indeed deployment-tin works just fine now :]

deployment-mx

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: You can only use systemd resources on systems with systemd, got upstart at /etc/puppet/modules/systemd/manifests/init.pp:8 on node deployment-mx.deployment-prep.eqiad.wmflabs

I did a git bisect which points at 052e3a87c143a3e736f7a9b84c140a8cb0ad7f22 T179565. It adds to role::mail::mx the class mtail::program which only comes with systemd template.

Looking at past puppet log, seems LetsEncrypt/Nginx has been broken for a while.

Anyway as Moritz said, we should rebuild it to Jessie/Stretch.

Per Mukunda, deployment-phab will be recreated anyway. So we can ignore it for now.

In T180935#3872752, @hashar wrote:

As for puppet being broken on several instances, indeed we could use some new tasks. The reasons listed in this are no more accurate, so I am declining as outdated.

What a mess. It was disappointing to find deployment-prep in this state. I've opened some more tasks.