Change 898803 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Fix an error with the ceph::server::firewall profile.

https://gerrit.wikimedia.org/r/898803

Change 898803 merged by Btullis:

[operations/puppet@production] Fix an error with the ceph::server::firewall profile.

https://gerrit.wikimedia.org/r/898803

Change 898819 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Fix the srange format for the ceph servers srange

https://gerrit.wikimedia.org/r/898819

Change 898819 merged by Btullis:

[operations/puppet@production] Fix the srange format for the ceph servers srange

https://gerrit.wikimedia.org/r/898819

I have created the initial monmap for bootstrapping the cluster with the following command.

btullis@cephosd1001:~$ monmaptool --create --fsid 6d4278e1-ea45-4d29-86fe-85b44c150813 --add cephosd1001 10.64.130.13 --add cephosd1002 10.64.131.21 --add cephosd1003 10.64.132.23 --add cephosd1004 10.64.134.12 --add cephosd1005 10.64.135.21 --enable-all-features --set-min-mon-release quincy monmap
monmaptool: monmap file monmap
setting min_mon_release = quincy
monmaptool: set fsid to 6d4278e1-ea45-4d29-86fe-85b44c150813
monmaptool: writing epoch 0 to monmap (5 monitors)

This is a binary file, but we can validate its contents with:

btullis@cephosd1001:~$ monmaptool --print monmap
monmaptool: monmap file monmap
epoch 0
fsid 6d4278e1-ea45-4d29-86fe-85b44c150813
last_changed 2023-03-16T09:56:13.788886+0000
created 2023-03-16T09:56:13.788886+0000
min_mon_release 17 (quincy)
election_strategy: 1
0: [v2:10.64.130.13:3300/0,v1:10.64.130.13:6789/0] mon.cephosd1001
1: [v2:10.64.131.21:3300/0,v1:10.64.131.21:6789/0] mon.cephosd1002
2: [v2:10.64.132.23:3300/0,v1:10.64.132.23:6789/0] mon.cephosd1003
3: [v2:10.64.134.12:3300/0,v1:10.64.134.12:6789/0] mon.cephosd1004
4: [v2:10.64.135.21:3300/0,v1:10.64.135.21:6789/0] mon.cephosd1005

The monitor processes are now running.

btullis@cephosd1001:~$ sudo ceph -s
  cluster:
    id:     6d4278e1-ea45-4d29-86fe-85b44c150813
    health: HEALTH_WARN
            mons are allowing insecure global_id reclaim
 
  services:
    mon: 5 daemons, quorum cephosd1001,cephosd1002,cephosd1003,cephosd1004,cephosd1005 (age 24s)
    mgr: no daemons active
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

The bootstrapping process was a little more manual than I would like and the documentation a little scattered.
I followed the steps outlined here:

https://docs.ceph.com/en/quincy/dev/mon-bootstrap/
https://docs.ceph.com/en/quincy/install/manual-deployment/
https://docs.ceph.com/en/latest/man/8/monmaptool/

What was confusing is that puppet attempts to create keyrings for each monitor:
https://github.com/wikimedia/operations-puppet/blob/production/hieradata/role/common/ceph/server.yaml#L7-L21
e.g.

mon.cephosd1001:
  caps:
    mon: allow *
mon.cephosd1002:
  caps:
    mon: allow *

However, when attempting to use these individual keys with the commands to bootstrap the cluster, the first mon process would work but all subsequent versions would not work.

I reverted to using the instructions from the manual installation, which instead refer to creating a single key for mon. like this:

sudo ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'

I checked the cloudceph cluster and verified that indeed the contents of each mon server's /var/lib/ceph/mon/ceph-$hostname/keyring are in fact a key named mon. and they are identical across all monitor modes.

Once I did that I was able to use the generated ceph.client.admin.keyring with:

sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring

I created an initial monmap with:

monmaptool --create --fsid 6d4278e1-ea45-4d29-86fe-85b44c150813 --add cephosd1001 10.64.130.13 --add cephosd1002 10.64.131.21 --add cephosd1003 10.64.132.23 --add cephosd1004 10.64.134.12 --add cephosd1005 10.64.135.21 --enable-all-features --set-min-mon-release quincy monmap

Then I used this monmap with the ceph-mon bootstrap command on each of the five servers.

sudo -u ceph ceph-mon --mkfs -i cephosd1001 --monmap monmap --keyring /tmp/ceph.mon.keyring

Then I was able to start the ceph-mon service on each server with:

sudo systemctl start ceph-mon@cephosd1001.service

Change 900694 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update the location of the mgr and mon keyrings for the new ceph cluster

https://gerrit.wikimedia.org/r/900694

Change 900694 merged by Btullis:

[operations/puppet@production] Update the location of the mgr and mon keyrings for the new ceph cluster

https://gerrit.wikimedia.org/r/900694

Change 900697 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Correct an error in the location of the ceph mon and mgr keyrings

https://gerrit.wikimedia.org/r/900697

Change 900697 merged by Btullis:

[operations/puppet@production] Correct an error in the location of the ceph mon and mgr keyrings

https://gerrit.wikimedia.org/r/900697

Change 900699 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Remove the specified location of the mon keys

https://gerrit.wikimedia.org/r/900699

Change 900699 merged by Btullis:

[operations/puppet@production] Remove the specified location of the mon keys

https://gerrit.wikimedia.org/r/900699

These mon and mgr daemons are now deployed to the cluster.

btullis@cephosd1001:~$ sudo ceph -s
  cluster:
    id:     6d4278e1-ea45-4d29-86fe-85b44c150813
    health: HEALTH_WARN
            28 daemons have recently crashed
            OSD count 0 < osd_pool_default_size 3
 
  services:
    mon: 5 daemons, quorum cephosd1001,cephosd1002,cephosd1003,cephosd1004,cephosd1005 (age 24h)
    mgr: cephosd1001(active, since 25h), standbys: cephosd1005, cephosd1002, cephosd1003, cephosd1004
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

There is still a potential issue with the bootstrapping process, which I have tried to outline in T332987: Review and fix any bugs found in the automated bootstrap process for a ceph mon/mgr server.
I believe that before we promote this cluster to production we will want to try to identify and fix any of these bugs and reimage at least one, or possibly all nodes in the cluster.

Nevertheless, I believe that we can call this ticket done.