Page MenuHomePhabricator

tools-k8s-master-01 etcd fail loop
Closed, DuplicatePublic

Description

This repeats over and over again:

Oct 07 22:42:12 tools-k8s-master-01 systemd[1]: etcd.service holdoff time over, scheduling restart.
Oct 07 22:42:12 tools-k8s-master-01 systemd[1]: Stopping etcd...
Oct 07 22:42:12 tools-k8s-master-01 systemd[1]: Starting etcd...
Oct 07 22:42:12 tools-k8s-master-01 systemd[1]: Started etcd.
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://tools-k8s-master-01.tools.eqiad.wmflabs:2379
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: recognized and used environment variable ETCD_CERT_FILE=/var/lib/etcd/ssl/certs/cert.pem
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/tools-k8s
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://tools-k8s-master-01.tools.eqiad.wmflabs:2380
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: recognized and used environment variable ETCD_INITIAL_CLUSTER=tools-k8s-master-01=http://tools-k8s-master-01.tools.eqiad.wmflabs:2380
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: recognized and used environment variable ETCD_KEY_FILE=/var/lib/etcd/ssl/private_keys/server.key
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://tools-k8s-master-01.tools.eqiad.wmflabs:2379
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://tools-k8s-master-01.tools.eqiad.wmflabs:2380
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: recognized and used environment variable ETCD_NAME=tools-k8s-master-01
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: etcd Version: 2.2.1
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: Git SHA: Not provided (use ./build instead of go build)
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: Go Version: go1.5.1
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: Go OS/Arch: linux/amd64
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: setting maximum number of CPUs to 2, total number of available CPUs is 2
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: the server is already initialized as member before, starting as etcd member...
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: listening for peers on http://tools-k8s-master-01.tools.eqiad.wmflabs:2380
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: clientTLS: cert = /var/lib/etcd/ssl/certs/cert.pem, key = /var/lib/etcd/ssl/private_keys/server.key, ca = , trusted-ca = , client-cert-auth = false
Oct 07 22:42:12 tools-k8s-master-01 etcd[15099]: stopping listening for peers on http://tools-k8s-master-01.tools.eqiad.wmflabs:2380
Oct 07 22:42:12 tools-k8s-master-01 [15099]: open /var/lib/etcd/ssl/certs/cert.pem: no such file or directory
Oct 07 22:42:12 tools-k8s-master-01 systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE
Oct 07 22:42:12 tools-k8s-master-01 systemd[1]: Unit etcd.service entered failed state.

Etcd is practically not running on this host since at least Sept 30. /var/lib/etcd/ssl/ is empty.