Page MenuHomePhabricator
Create Task
Maniphest T94195

puppet fail on praseodymium / cassandra db not running
Closed, ResolvedPublic
Actions

Description

praseodymium

Cassandra database CRITICAL 2015-03-27 16:49:12 0d 17h 30m 13s 3/3 PROCS CRITICAL: 0 processes with UID = 109 (cassandra), command name 'java', args 'CassandraDaemon'

puppet last run
CRITICAL 2015-03-27 16:49:39 0d 17h 14m 14s 3/3 CRITICAL: Puppet has 1 failures

Error: Cannot create /var/lib/cassandra/data; parent directory /var/lib/cassandra does not exist
Error: /Stage[main]/Cassandra/File[/var/lib/cassandra/data]/ensure: change from absent to directory failed: Cannot create /var/lib/cassandra/data; parent directory /var/lib/cassandra does not exist

Event Timeline

Dzahn created this task.Mar 27 2015, 4:51 PM
Dzahn raised the priority of this task from to Needs Triage.
Dzahn updated the task description. (Show Details)
Dzahn added projects: acl*sre-team, RESTBase-Cassandra.
Dzahn subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 27 2015, 4:51 PM
Dzahn added a comment.Mar 27 2015, 4:57 PM

@praseodymium:/var/lib# file cassandra
cassandra: broken symbolic link to `/mnt/data/cassandra/'

Dzahn added a comment.Mar 27 2015, 5:02 PM

i did: mount /mnt/data

and ran puppet again, which fixed the immediate issue:

/Service[cassandra]/ensure: ensure changed 'stopped' to 'running'

< icinga-wm> RECOVERY - Cassandra database on praseodymium is OK: PROCS OK: 1 process with UID = 109 (cassandra), command name java, args CassandraDaemon

puppet running again as well

Dzahn added a comment.Mar 27 2015, 5:04 PM

this was because the device name changed after a recent reboot. /etc/fstab had been adjusted but this just needed to be mounted.

i'm resolving, though, the mounts should probably be puppetized

Dzahn closed this task as Resolved.Mar 27 2015, 5:05 PM
Dzahn claimed this task.
Dzahn set Security to None.
Dzahn added a subscriber: • GWicke.
Dzahn renamed this task from puppet fail on prasedymium / cassandra db not running to puppet fail on praseodymium / cassandra db not running.Mar 27 2015, 6:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL