Page MenuHomePhabricator

CloudVPS: create a wrapper script to ease virsh console to VMs
Open, LowPublic

Description

If trying to access the serial console of a VM in CloudVPS one must perform the following steps:

  • investigate in which hypervisor is the VM running
  • investigate the libvirt ID of the VM (something like i-000045b2)
  • jump by ssh to the hypervisor and run the command sudo virsh console --devname serial1 $ID

This is documented here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Root_console_access

The procedure is very time-consuming and can easily be automated (all the information is in keystone)
When developing new services or projects (example: T228500: Toolforge: evaluate ingress mechanism) this is a very common operation, so having a wmcs-root-console script that does all the above would save us a lot of time.

First step would be to develop some helper library to help us translate between the main identifiers for a given VM:

  • short name (like tools-worker-1003)
  • full hostname (like tools-worker-1003.tools.eqiad.wmflabs)
  • nova ID (like d268926f-32a4-4201-ad70-b6956a66d929)
  • libvirt ID (like i-000045b2)

(BTW this small helper library should be really helpful in a lot of other small scripts we have.)

Ideally, the script would be able run from anywhere with enough credentials to do all the steps (keystone creds + ssh creds for the cloudvirt).

Example usage:
$ wmcs-root-console tools-worker-1003

Mind: https://xkcd.com/1205/ https://xkcd.com/1319/

Event Timeline

aborrero created this task.Oct 9 2019, 12:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 9 2019, 12:12 PM
aborrero triaged this task as Low priority.Oct 22 2019, 10:45 AM

I created this for personal use. Intended to be run in cloudcontrol servers for example. I didn't want to invest more time investigating proper SSH credentials. This script just prints the SSH command you can use in your laptop.

Example usage:

aborrero@cloudcontrol1004:~ $ python3 wmcs-vm-console.py toolsbeta-test-k8s-etcd-2 toolsbeta
VM nova ID:    59922260-bd39-4bdc-b233-feb78ddb2665
VM Hypervisor: cloudvirt1014.eqiad.wmnet
VM libvirt id: i-0000ddf4

Try this in your laptop:

	ssh -t cloudvirt1014.eqiad.wmnet "sudo virsh console --devname serial1 i-0000ddf4"

(I would really love to do that myself, but SSH creds, etc...)

I don't intend to persist this in puppet or the like. The source code.

#!/usr/bin/env python3

import os
import sys
import argparse
import yaml
import keystoneauth1
from keystoneauth1.identity import v3
from keystoneauth1 import session as keystone_session
from novaclient import client as novaclient


def exit_with_msg(msg):
    print("ERROR: {}".format(msg), file=sys.stderr)
    exit(1)


def main():
    parser = argparse.ArgumentParser(description="Jump by SSH to a VM console")
    parser.add_argument("name", help="VM name")
    parser.add_argument(
        "project",
        help="OpenStack project scope",
    )
    parser.add_argument(
        "--observer-pass",
        default="",
        help="Password for the OpenStack observer account",
    )
    parser.add_argument(
        "--auth-url",
        default="",
        help="Keystone URL -- can be obtained from novaobserver.yaml",
    )
    args = parser.parse_args()

    if not args.observer_pass:
        if os.path.isfile("/etc/novaobserver.yaml"):
            with open("/etc/novaobserver.yaml") as conf_fh:
                nova_observer_config = yaml.safe_load(conf_fh)

            args.observer_pass = nova_observer_config["OS_PASSWORD"]
            args.auth_url = nova_observer_config["OS_AUTH_URL"]
        else:
            exit_with_msg("The --observer-pass argument is required without /etc/novaobserver.yaml")

    if not args.auth_url:
        exit_with_msg("the --auth-url argument is required without /etc/novaobserver.yaml")

    auth = v3.Password(
        auth_url=args.auth_url,
        username="novaobserver",
        password=args.observer_pass,
        user_domain_name='Default',
        project_domain_name='Default',
        project_name=args.project
    )
    session = keystoneauth1.session.Session(auth=auth)
    mynovaclient = novaclient.Client("2.0", session=session)

    try:
        server = mynovaclient.servers.list(search_opts={'name': '^{}$'.format(args.name)})
    except keystoneauth1.exceptions.http.Unauthorized:
        exit_with_msg("couldn't query the openstack API. Operation not authorized.")
    except:
        raise

    if len(server) > 1:
        exit_with_msg("more than 1 VM found with that name.")

    if len(server) == 0:
        exit_with_msg("no VM found with that name.")

    vm_info = server[0]._info
    vm_id = server[0].id
    vm_hypervisor = vm_info['OS-EXT-SRV-ATTR:hypervisor_hostname']
    vm_libvirt_id = vm_info['OS-EXT-SRV-ATTR:instance_name']

    print("VM nova ID:    {}".format(vm_id))
    print("VM hypervisor: {}".format(vm_hypervisor))
    print("VM libvirt id: {}".format(vm_libvirt_id))
    print()

    print("Try this in your laptop:\n")
    print("\tssh -t {} \"sudo virsh console --devname serial1 {}\"".format(
          vm_hypervisor, vm_libvirt_id))

    print("\n(I would really love to do that myself, but SSH creds etc...)")

if __name__ == "__main__":
    main()

This already speeds up our workflow by removing the need to navigate horizon and/or openstack-browser.