Page MenuHomePhabricator

Create parent directories for JBOD data_directories and e.g. commitlog directories
Closed, ResolvedPublic

Description

Puppet should create parent directories on the JBOD devices for Cassandra's data_file_directories. Additionally, it should create the parent directories for the commitlog, saved caches, and hints.

Event Timeline

Eevans created this task.Sep 7 2017, 3:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 7 2017, 3:33 PM
mobrovac moved this task from Backlog to next on the Services board.Sep 12 2017, 8:39 AM
mobrovac edited projects, added Services (next); removed Services.

Indeed having this would be nice as it would simplify reinstalls/reimages we have to do for T184100: Reprovision legacy Cassandra nodes into new cluster

Eevans added a comment.EditedJan 11 2018, 5:02 PM

From a discussion on IRC:

11:35 < _joe_>  urandom: puppet 4 has loops
11:35 <+urandom> and then, there is also a common parent directory that needs to be created as well (though i think that is more straightforward)
11:36 < _joe_> $directories.each |$directory| { <do what you need with $directory> }
11:36 < _joe_> so you can just cycle through them
11:42 <+urandom> _joe_: it's probably a mistake to be asking this before getting my head back into the module, but i think the confusion came from trying to do this with a file/ensure, you can't use a loop there,  can you?
11:42 < _joe_> urandom: you can do things like this, lemme show you
11:44 < _joe_> https://gerrit.wikimedia.org/r/#/c/403388/7/modules/puppetmaster/manifests/puppetdb/database.pp,sidebyside line 61 of the new code
11:44 < _joe_> there $puppetdb_users is a hash
11:44 < _joe_> but say you have a list of directories
11:45 < _joe_> and of each of those you have to create the parent too
11:45 < _joe_> you could just do a similar loop and declare the parents via a file resource
Eevans added a comment.EditedJan 11 2018, 5:44 PM

Here is some more detail:

There seems to exist some code in instance.pp that is meant to handle this, though it fails with errors like:

...
Error: Cannot create /srv/sda4/cassandra-a/data; parent directory /srv/sda4/cassandra-a does not exist
...

Where /srv/sda4/cassandra-a/data is an element in $instance_data_file_directories that cannot be created because /srv/sda4/cassandra-a does not exist. This seems to be a commonly encountered problem, where puppet's file type does not create directories using the equivalent of mkdir -p.

This makes a solution more complex, because the data file directories are treated as fully-qualified arbitrary paths (i.e. you'd need to treat each recursively).

Eevans added a subscriber: Joe.Jan 11 2018, 5:48 PM
Eevans claimed this task.Jan 17 2018, 3:48 PM

Change 404705 had a related patch set uploaded (by Eevans; owner: Eevans):
[operations/puppet@production] [WIP] cassandra: create parent data directories with exec

https://gerrit.wikimedia.org/r/404705

Eevans moved this task from Backlog to In-Progress on the User-Eevans board.Feb 13 2018, 8:18 PM
Eevans moved this task from In-Progress to Blocked on the User-Eevans board.Feb 13 2018, 9:31 PM

Change 404705 merged by Filippo Giunchedi:
[operations/puppet@production] cassandra: create parent data directories with exec

https://gerrit.wikimedia.org/r/404705

Change 413121 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] cassandra: create data directories only when needed

https://gerrit.wikimedia.org/r/413121

Change 413121 merged by Filippo Giunchedi:
[operations/puppet@production] cassandra: create data directories only when needed

https://gerrit.wikimedia.org/r/413121

Eevans closed this task as Resolved.Jun 11 2018, 3:23 PM