Page MenuHomePhabricator

eqiad: (1) VM for static-bugzilla
Closed, ResolvedPublic

Description

Labs Project Tested: n/a (production)
Site/Location: eqiad
Number of systems: 1
Service: static html store of bugzilla things
Networking Requirements: Internal, misc-web is used
Processor Requirements: 1
Memory: 1GB
Disks: 20GB
Other Requirements: just misc-web being able to communicate

Event Timeline

JohnLewis raised the priority of this task from to Needs Triage.
JohnLewis updated the task description. (Show Details)
JohnLewis added subscribers: JohnLewis, Dzahn.
Dzahn added a subscriber: akosiaris.

@akosiaris next VM for next zirconium service please :)

do we really have to follow the "1001" kind of naming for stuff like this? static-bugzilla1001 is a bit odd but i don't have a strong opinion :)

statbugs1001 or just staticbugs or something

Dzahn triaged this task as Medium priority.Jun 26 2015, 11:11 PM

there will very likely just be a single host so staticbugs.eqiad.wmmet or something would be best following the misc naming style as this isn't really a 'service' perse.

that said, "misc naming style" would mean to use element names in eqiad and star names in codfw :)

..and then have service names that are not equal to the hostname.. which is ok

I am fine with either approach. Wanna propose a name ?

I propose staticbugs.eqiad.wmnet simply because the 100x part makes it look ugly and it won't have another host unless this somehow becomes more popular than actual wikis :)

The 100x part indicates site as well as member in a series.

staticbugs.eqiad.wmnet is against the policy. TL;DR It's either element/star (eqiad/codfw respectively) or serviceXNNN where X=[1,2,3,4] for eqiad, codfw, esams,ulsfo.

Next element in line then I guess is my view point though I'm not against the staticbugs1001 name if that's what seems best?

That sounds like a bit of a silly policy to me...

If this is a VM, and not bare metal, it shouldnt use up element names. The element names are allocated for bare metal misc servers.

This was an ongoing discussion on 'how do we name misc vms' that I didn't think had been resolved. I'm against them sharing the naming scheme of bare metal misc servers.

If its a VM, I think it should be named by its role and all VMs should have some designation in their hostnames to denote they aren't on bare metal.

I'd call this vm something like: staticbugzilla1001, rather than staticbugs, since bugs can be ANY bug tracker.

Infrastructure naming convention discussions are the worst.

There is no really nice way to name these in regards to misc bare-metal versus misc vm, considering their use cases will overlap. As such, Alex has convinced me to change stance and revoke my statement not to use element names/star names for misc vm servers.

Change 222149 had a related patch set uploaded (by Alexandros Kosiaris):
Introducing bromine.eqiad.wmnet

https://gerrit.wikimedia.org/r/222149

Change 222149 merged by Alexandros Kosiaris:
Introducing bromine.eqiad.wmnet

https://gerrit.wikimedia.org/r/222149

bromine.eqiad.wmnet is ready on ganeti01.svc.eqiad.wmnet with MAC: aa:00:00:f1:36:3a. DHCP configuration and installation are not done yet.

Change 222172 had a related patch set uploaded (by Dzahn):
add bromine to dhcp

https://gerrit.wikimedia.org/r/222172

merged DHCP config (thanks John)

tried to get console:

[ganeti1003:~] $ sudo gnt-instance console bromine.eqiad.wmnet
2015/07/01 22:03:13 socat[47275] E connect(5, AF=1 "/var/run/ganeti/kvm-hypervisor/ctrl/bromine.eqiad.wmnet.serial", 64): No such file or directory

                                                                   Connection to ganeti1003.eqiad.wmnet closed.

Failure: command execution error:
Connection to console of instance bromine.eqiad.wmnet failed, please check cluster configuration

after waiting a bit and restarting it i got console and saw the installer.
then, as with planet1001 at first:

    ┌────────┤ [!!] Install the GRUB boot loader on a hard disk ├─────────┐
    │                                                                     │
    │                      GRUB installation failed                       │
    │ The 'grub-pc' package failed to install into /target/. Without the  │
<Tab> moves; <Space> selects; <Enter> activates buttons───────────────────┘
 ┌─────────────────┤ [!!] Finish the installation ├─────────────────┐
┌───│                                                                  │ ─┐
│   │                 Failed to run preseeded command                  │  │
│   │ Execution of preseeded command "wget -O /tmp/late_command        │  │
│   │ http://apt.wikimedia.org/autoinstall/scripts/late_command && sh  │  │
│ Ru│ /tmp/late_command" failed with exit code 100.                    │  │
│   │                                                                  │  │
└───│                            <Continue>                            │ ─┘
    │                                                                  │
    └──────────────────────────────────────────────────────────────────┘

after that the installation finished and the system powered down.

when i powered it up again the installer started from scratch

next attempt the installer finished without these error messages above. after finishing it shuts down the machine.

after you power it up again, installing starts again.

sure i am not supposed to manually switch the boot order (as i tried with planet1001)

merged DHCP config (thanks John)

tried to get console:

[ganeti1003:~] $ sudo gnt-instance console bromine.eqiad.wmnet
2015/07/01 22:03:13 socat[47275] E connect(5, AF=1 "/var/run/ganeti/kvm-hypervisor/ctrl/bromine.eqiad.wmnet.serial", 64): No such file or directory

                                                                   Connection to ganeti1003.eqiad.wmnet closed.

Failure: command execution error:
Connection to console of instance bromine.eqiad.wmnet failed, please check cluster configuration

That means the VM was powered off

after waiting a bit and restarting it i got console and saw the installer.
then, as with planet1001 at first:

    ┌────────┤ [!!] Install the GRUB boot loader on a hard disk ├─────────┐
    │                                                                     │
    │                      GRUB installation failed                       │
    │ The 'grub-pc' package failed to install into /target/. Without the  │
<Tab> moves; <Space> selects; <Enter> activates buttons───────────────────┘

This is the second time this issue is encountered. I 'd like to debug it but alas, it resolved itself on the next reboot

next attempt the installer finished without these error messages above. after finishing it shuts down the machine.

after you power it up again, installing starts again.

sure i am not supposed to manually switch the boot order (as i tried with planet1001)

You are supposed to do exactly that. https://wikitech.wikimedia.org/wiki/Ganeti#Set_boot_order_to_disk

This is a limitation of the current version of the Ganeti software and is expected to be solved (upstream is aware)

I did that and set the boot order to disk during install. After shutdown i started it up again.

Then i tried to connect to console.. no output

[ganeti1003:~] $ sudo gnt-instance modify --hypervisor-parameters=boot_order=disk bromine.eqiad.wmnet
Modified instance bromine.eqiad.wmnet
 - hv/boot_order -> disk
  ┌─────────────────────┤ Finishing the installation ├──────────────────────┐
  │                                                                         │
  │                                   95%                                   │
  │                                                                         │
The system is going down NOW!ystem...                                       │
Sent SIGTERM to all processes                                               │
Sent SIGKILL to all processes───────────────────────────────────────────────┘
Requesting system poweroff
                          [ 1119.188806] reboot: Power down
Connection to ganeti1003.eqiad.wmnet closed.
[ganeti1003:~] $ sudo gnt-instance startup bromine.eqiad.wmnet
Waiting for job 10657 for bromine.eqiad.wmnet ...
[ganeti1003:~] $ sudo gnt-instance console bromine.eqiad.wmnet

De

all that said, it still worked and i could ssh to it with the "new_install" key and it's up and running :)

Dzahn set Security to None.

a couple minutes later the screen with the "frozen" console came to life and works now:

bromine login:
Debian GNU/Linux 8 bromine ttyS0

there was a typo in DNS that lead to issues when trying to sign the puppet cert for initial run (and possibly some of the other issues)

https://gerrit.wikimedia.org/r/#/c/222367/1

reinstalled after the fix (also no fail when installing grub, console works immediately after starting it up)

hostname is still ending in .wmmet though

after it was installed and the initial puppet run i could not login with either SSH key nor root password from console

that was my fault applying a role on initial run that caused a puppet fail before users were setup and not applying "standard' first.

works now. resolving.