Labs Project Tested: n/a (production)
Site/Location: eqiad
Number of systems: 1
Service: static html store of bugzilla things
Networking Requirements: Internal, misc-web is used
Processor Requirements: 1
Memory: 1GB
Disks: 20GB
Other Requirements: just misc-web being able to communicate
Description
Details
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| add bromine to dhcp | John F. Lewis | operations/puppet | production | +8 -1 | |
| Introducing bromine.eqiad.wmnet | Alexandros Kosiaris | operations/dns | master | +2 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Cmjohnson | T105510 reclaim zirconium (wipe disk) | |||
| Resolved | Dzahn | T104946 move all misc services from zirconium to a VM | |||
| Resolved | Dzahn | T101734 Move static-bugzilla from zirconium to ganeti | |||
| Resolved | akosiaris | T103604 eqiad: (1) VM for static-bugzilla |
Event Timeline
@akosiaris next VM for next zirconium service please :)
do we really have to follow the "1001" kind of naming for stuff like this? static-bugzilla1001 is a bit odd but i don't have a strong opinion :)
there will very likely just be a single host so staticbugs.eqiad.wmmet or something would be best following the misc naming style as this isn't really a 'service' perse.
that said, "misc naming style" would mean to use element names in eqiad and star names in codfw :)
..and then have service names that are not equal to the hostname.. which is ok
I propose staticbugs.eqiad.wmnet simply because the 100x part makes it look ugly and it won't have another host unless this somehow becomes more popular than actual wikis :)
staticbugs.eqiad.wmnet is against the policy. TL;DR It's either element/star (eqiad/codfw respectively) or serviceXNNN where X=[1,2,3,4] for eqiad, codfw, esams,ulsfo.
Next element in line then I guess is my view point though I'm not against the staticbugs1001 name if that's what seems best?
For the interested, this is documented in:
https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions
@JohnLewis: Next element in line it is, then
If this is a VM, and not bare metal, it shouldnt use up element names. The element names are allocated for bare metal misc servers.
This was an ongoing discussion on 'how do we name misc vms' that I didn't think had been resolved. I'm against them sharing the naming scheme of bare metal misc servers.
If its a VM, I think it should be named by its role and all VMs should have some designation in their hostnames to denote they aren't on bare metal.
I'd call this vm something like: staticbugzilla1001, rather than staticbugs, since bugs can be ANY bug tracker.
Infrastructure naming convention discussions are the worst.
There is no really nice way to name these in regards to misc bare-metal versus misc vm, considering their use cases will overlap. As such, Alex has convinced me to change stance and revoke my statement not to use element names/star names for misc vm servers.
Change 222149 had a related patch set uploaded (by Alexandros Kosiaris):
Introducing bromine.eqiad.wmnet
bromine.eqiad.wmnet is ready on ganeti01.svc.eqiad.wmnet with MAC: aa:00:00:f1:36:3a. DHCP configuration and installation are not done yet.
merged DHCP config (thanks John)
tried to get console:
[ganeti1003:~] $ sudo gnt-instance console bromine.eqiad.wmnet
2015/07/01 22:03:13 socat[47275] E connect(5, AF=1 "/var/run/ganeti/kvm-hypervisor/ctrl/bromine.eqiad.wmnet.serial", 64): No such file or directory
Connection to ganeti1003.eqiad.wmnet closed.
Failure: command execution error:
Connection to console of instance bromine.eqiad.wmnet failed, please check cluster configuration
after waiting a bit and restarting it i got console and saw the installer.
then, as with planet1001 at first:
┌────────┤ [!!] Install the GRUB boot loader on a hard disk ├─────────┐
│ │
│ GRUB installation failed │
│ The 'grub-pc' package failed to install into /target/. Without the │
<Tab> moves; <Space> selects; <Enter> activates buttons───────────────────┘ ┌─────────────────┤ [!!] Finish the installation ├─────────────────┐
┌───│ │ ─┐
│ │ Failed to run preseeded command │ │
│ │ Execution of preseeded command "wget -O /tmp/late_command │ │
│ │ http://apt.wikimedia.org/autoinstall/scripts/late_command && sh │ │
│ Ru│ /tmp/late_command" failed with exit code 100. │ │
│ │ │ │
└───│ <Continue> │ ─┘
│ │
└──────────────────────────────────────────────────────────────────┘after that the installation finished and the system powered down.
when i powered it up again the installer started from scratch
next attempt the installer finished without these error messages above. after finishing it shuts down the machine.
after you power it up again, installing starts again.
sure i am not supposed to manually switch the boot order (as i tried with planet1001)
This is the second time this issue is encountered. I 'd like to debug it but alas, it resolved itself on the next reboot
You are supposed to do exactly that. https://wikitech.wikimedia.org/wiki/Ganeti#Set_boot_order_to_disk
This is a limitation of the current version of the Ganeti software and is expected to be solved (upstream is aware)
I did that and set the boot order to disk during install. After shutdown i started it up again.
Then i tried to connect to console.. no output
[ganeti1003:~] $ sudo gnt-instance modify --hypervisor-parameters=boot_order=disk bromine.eqiad.wmnet Modified instance bromine.eqiad.wmnet - hv/boot_order -> disk
┌─────────────────────┤ Finishing the installation ├──────────────────────┐
│ │
│ 95% │
│ │
The system is going down NOW!ystem... │
Sent SIGTERM to all processes │
Sent SIGKILL to all processes───────────────────────────────────────────────┘
Requesting system poweroff
[ 1119.188806] reboot: Power down
Connection to ganeti1003.eqiad.wmnet closed.
[ganeti1003:~] $ sudo gnt-instance startup bromine.eqiad.wmnet
Waiting for job 10657 for bromine.eqiad.wmnet ...
[ganeti1003:~] $ sudo gnt-instance console bromine.eqiad.wmnet
Deall that said, it still worked and i could ssh to it with the "new_install" key and it's up and running :)
a couple minutes later the screen with the "frozen" console came to life and works now:
bromine login:
Debian GNU/Linux 8 bromine ttyS0
there was a typo in DNS that lead to issues when trying to sign the puppet cert for initial run (and possibly some of the other issues)
https://gerrit.wikimedia.org/r/#/c/222367/1
reinstalled after the fix (also no fail when installing grub, console works immediately after starting it up)
hostname is still ending in .wmmet though
after it was installed and the initial puppet run i could not login with either SSH key nor root password from console
that was my fault applying a role on initial run that caused a puppet fail before users were setup and not applying "standard' first.
works now. resolving.