Page MenuHomePhabricator

Cassandra broken in beta
Closed, InvalidPublic

Description

root@deployment-restbase02:~# puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-restbase02.deployment-prep.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1530904172'
Notice: /Stage[main]/Cassandra/Cassandra::Instance[default]/Systemd::Service[cassandra]/Service[cassandra]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Cassandra/Cassandra::Instance[default]/Systemd::Service[cassandra]/Service[cassandra]: Unscheduling refresh on Service[cassandra]
Notice: Applied catalog in 14.85 seconds
root@deployment-restbase02:~# service cassandra status
● cassandra.service - distributed storage system for structured data
   Loaded: loaded (/lib/systemd/system/cassandra.service; static)
   Active: failed (Result: exit-code) since Fri 2018-07-06 19:10:00 UTC; 23s ago
  Process: 28719 ExecStart=/usr/sbin/cassandra -p /var/run/cassandra/cassandra.pid (code=exited, status=1/FAILURE)
 Main PID: 28719 (code=exited, status=1/FAILURE)

Jul 06 19:09:59 deployment-restbase02 systemd[1]: Started distributed storage system for structured data.
Jul 06 19:10:00 deployment-restbase02 cassandra[28636]: Error opening zip file or JAR manifest missing : /usr/share/java/prometheus/jmx_prometheus_javaagent.jar
Jul 06 19:10:00 deployment-restbase02 cassandra[28636]: Error occurred during initialization of VM
Jul 06 19:10:00 deployment-restbase02 cassandra[28636]: agent library failed to init: instrument
Jul 06 19:10:00 deployment-restbase02 systemd[1]: cassandra.service: main process exited, code=exited, status=1/FAILURE
Jul 06 19:10:00 deployment-restbase02 systemd[1]: Unit cassandra.service entered failed state.

Event Timeline

Reedy created this task.Jul 6 2018, 8:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 6 2018, 8:14 PM
Reedy triaged this task as High priority.Jul 6 2018, 8:14 PM
Krenair added a subscriber: Krenair.

Something's wrong with that host per T195709
What about 01?
See also T186750 and T186994

Reedy added a comment.Jul 6 2018, 8:34 PM

01 has the same error trying to start cassandra

Pchelolo closed this task as Invalid.Jul 7 2018, 8:27 AM
Pchelolo added a subscriber: Pchelolo.

It's all fine, Cassandra lives on deployment-cassandra3-01/2 now. We still need to reset the environment properly there, but I think we're waiting to co-locate that with stretch upgrade.