Page MenuHomePhabricator
Create Task
Maniphest T377803

Cloud VPS: 2024-10-22 cloud-wide puppet problem related to java update
Closed, ResolvedPublic
Actions

Description

Today there was a cloud-wide puppet problem related to puppet-enc:

aborrero@bastion-restricted-eqiad1-3:~$ sudo run-puppet-agent
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed when searching for node bastion-restricted-eqiad1-3.bastion.eqiad1.wikimedia.cloud: Exception while executing '/usr/local/bin/puppet-enc': Cannot run program "/usr/local/bin/puppet-enc" (in directory "."): error=0, Failed to exec spawn helper: pid: 872033, exit value: 1
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
wmcs: puppetserver: introduce apt pin for openjdkoperations/puppetproduction+10 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

• aborrero created this task.Oct 22 2024, 8:46 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 22 2024, 8:46 AM
• aborrero changed the task status from Open to In Progress.Oct 22 2024, 8:47 AM
• aborrero triaged this task as Unbreak Now! priority.
• aborrero moved this task from Backlog to Doing on the User-aborrero board.
• aborrero added a comment.Oct 22 2024, 9:03 AM

the puppet-enc API seems to be up and running:

aborrero@cloudinfra-cloudvps-puppetserver-1:~$ curl https://puppet-enc.cloudinfra.wmcloud.org/v1/cloudinfra/node/cloudinfra-cloudvps-puppetserver-1
hiera:
  acmechief_host: cloudinfra-acme-chief-02.cloudinfra.eqiad1.wikimedia.cloud
  profile::puppet::agent::dns_alt_names:
  - puppet
  - puppetmaster.cloudinfra.wmflabs.org
  profile::puppet::agent::force_puppet7: true
  profile::puppetserver::autosign: /usr/local/sbin/validatelabsfqdn.py
  profile::puppetserver::git::repos:
    labs/private:
      branch: master
      hooks:
        post-checkout: puppet:///modules/profile/puppetserver/git/operations/hooks/deploy-code.sh
        post-commit: puppet:///modules/profile/puppetserver/git/operations/hooks/deploy-code.sh
        post-merge: puppet:///modules/profile/puppetserver/git/operations/hooks/deploy-code.sh
        pre-commit: puppet:///modules/puppetmaster/git/private/pre-commit
        pre-merge: puppet:///modules/puppetmaster/git/private/pre-merge
        pre-rebase: puppet:///modules/puppetmaster/git/private/pre-rebase
      link: /etc/puppet/private
    operations/puppet:
      branch: production
      hooks:
        post-checkout: puppet:///modules/profile/puppetserver/git/operations/hooks/deploy-code.sh
        post-commit: puppet:///modules/profile/puppetserver/git/operations/hooks/deploy-code.sh
        post-merge: puppet:///modules/profile/puppetserver/git/operations/hooks/deploy-code.sh
        pre-commit: puppet:///modules/puppetmaster/git/pre-commit
        pre-merge: puppet:///modules/puppetmaster/git/pre-merge
        pre-rebase: puppet:///modules/puppetmaster/git/pre-rebase
  profile::puppetserver::java_max_mem: 26g
  profile::puppetserver::server_certname: puppetmaster.cloudinfra.wmflabs.org
  puppetmaster: cloudinfra-internal-puppetserver-1.cloudinfra.eqiad1.wikimedia.cloud
roles:
- role::puppetserver::cloud_vps_global
• aborrero added a comment.Oct 22 2024, 9:04 AM

The puppetserver service seems to have some errors:

aborrero@cloudinfra-cloudvps-puppetserver-1:~$ sudo journalctl -u puppetserver --since today
[...]
Oct 22 06:57:59 cloudinfra-cloudvps-puppetserver-1 java[866423]: jspawnhelper version 17.0.13+11-Debian-2deb12u1
Oct 22 06:57:59 cloudinfra-cloudvps-puppetserver-1 java[866423]: This command is not for general use and should only be run as the result of a call to
Oct 22 06:57:59 cloudinfra-cloudvps-puppetserver-1 java[866423]: ProcessBuilder.start() or Runtime.exec() in a java application
Oct 22 06:58:10 cloudinfra-cloudvps-puppetserver-1 java[866426]: Incorrect number of arguments: 2
[...]
Stashbot added a comment.Oct 22 2024, 9:05 AM

Mentioned in SAL (#wikimedia-cloud) [2024-10-22T09:04:58Z] <arturo> restart puppetserver service in T377803

Stashbot added a comment.Oct 22 2024, 9:05 AM

Mentioned in SAL (#wikimedia-cloud) [2024-10-22T09:05:53Z] <arturo> restart puppetserver service for T377803

• aborrero lowered the priority of this task from Unbreak Now! to High.Oct 22 2024, 9:07 AM

restarting the puppetserver.service unit in the corresponding puppetserver VM seems to fix the problem. Lowering priority.

Don-vip awarded a token.Oct 22 2024, 9:10 AM
Don-vip subscribed.
• aborrero added a comment.EditedOct 22 2024, 9:11 AM

There was an unattended java upgrade today:

aborrero@cloudinfra-cloudvps-puppetserver-1:~$ sudo tail /var/log/apt/history.log
[...]
Start-Date: 2024-10-22  06:56:59
Commandline: /usr/bin/unattended-upgrade
Upgrade: openjdk-17-jre-headless:amd64 (17.0.12+7-2~deb12u1, 17.0.13+11-2~deb12u1)
End-Date: 2024-10-22  06:57:02

aborrero@tools-puppetserver-01:~$ sudo tail /var/log/apt/history.log
[...]
Start-Date: 2024-10-22  06:54:32
Commandline: /usr/bin/unattended-upgrade
Upgrade: openjdk-17-jre-headless:amd64 (17.0.12+7-2~deb12u1, 17.0.13+11-2~deb12u1)
End-Date: 2024-10-22  06:54:36

This is likely the root cause of the problem.

LSobanski subscribed.Oct 22 2024, 9:15 AM
• aborrero renamed this task from Cloud VPS: cloud-wide puppet problem related to puppet-enc 2024-10-22 to Cloud VPS: 2024-10-22 cloud-wide puppet problem related to java update.Oct 22 2024, 9:22 AM
• aborrero added subscribers: MoritzMuehlenhoff, elukey.Oct 22 2024, 9:28 AM

chat on IRC #wikimedia-sre channel:

11:25 <elukey> arturo: o/ yes we are aware of this issue, when openjdk is installed we need to immediately restart puppetserver :(
11:26 <elukey> IIUC a restart fixed it right?
11:26 <arturo> elukey: apparently yes
11:26 <moritzm> yeah, you should exempt openjdk-17 from unattended-upgrades for the puppetserver role
11:26 <moritzm> puppetserver needs an immediate restart after the upgrade
11:27 <moritzm> I think it's related to jruby, jruby-compiled artefacts generated by two different JREs don't mix
gerritbot added a comment.Oct 22 2024, 12:43 PM

Change #1082201 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] wmcs: puppetserver: introduce apt pin for openjdk

https://gerrit.wikimedia.org/r/1082201

gerritbot added a project: Patch-For-Review.Oct 22 2024, 12:43 PM
• aborrero updated the task description. (Show Details)Oct 22 2024, 1:04 PM
bking subscribed.Oct 22 2024, 1:23 PM
taavi added a project: Cloud-VPS.Oct 22 2024, 5:32 PM
gerritbot added a comment.Oct 23 2024, 9:36 AM

Change #1082201 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] wmcs: puppetserver: introduce apt pin for openjdk

https://gerrit.wikimedia.org/r/1082201

• aborrero closed this task as Resolved.Oct 23 2024, 9:40 AM
• aborrero claimed this task.
Maintenance_bot removed a project: Patch-For-Review.Oct 23 2024, 10:31 AM
• aborrero added a subtask: T385553: Cloud VPS puppet breakage on 2025-02-04 related to puppet-enc.Feb 4 2025, 9:20 AM
• aborrero closed subtask T385553: Cloud VPS puppet breakage on 2025-02-04 related to puppet-enc as Resolved.Feb 4 2025, 11:37 AM
taavi mentioned this in T386755: Multiple *-pipeline-test jobs failing to load pipelinelib with git error.Feb 19 2025, 6:35 PM
MoritzMuehlenhoff mentioned this in T388629: puppet error at the end of the run on prometheus2008: Could not autoload puppet/reports/logstash: Cannot invoke "jnr.netdb.Service.getName()" because "service" is null.Mar 18 2025, 11:05 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits