Page MenuHomePhabricator

eqiad: 1 VM request for mailman
Closed, ResolvedPublic

Description

Labs Project Tested: mailman
Site/Location: eqiad
Number of systems: 1
Service: lists.wikimedia.org (staging, upgrade test)
Networking Requirements: 1 private IP
Processor Requirements: 1
Memory: 8G
Disks: 200G
Other Requirements:

Related Objects

Event Timeline

Dzahn created this task.Aug 5 2015, 5:30 PM
Dzahn raised the priority of this task from to Needs Triage.
Dzahn updated the task description. (Show Details)
Dzahn added projects: acl*sre-team, vm-requests.
Dzahn added a subscriber: Dzahn.
Restricted Application added subscribers: Matanya, Aklapper. · View Herald TranscriptAug 5 2015, 5:30 PM
Dzahn renamed this task from eqiad: 1 VM %request for mailman to eqiad: 1 VM request for mailman.Aug 5 2015, 5:30 PM
Dzahn set Security to None.
Dzahn updated the task description. (Show Details)
Dzahn assigned this task to RobH.Aug 5 2015, 6:34 PM

per IRC talk, thank you for trying to create this so akosiaris doesn't have to respond to all VM requests

RobH added a subscriber: mark.EditedAug 5 2015, 6:36 PM

This is typically approved by Alex, as it is his process. However, this was already approved by @mark. As such, we are still filing all the tickets for full audit. (Much like folks sometimes allocate pre-approved systems from spares and file the tasks as they do so.) Also we just had the discussion on naming standards for the VMs, so I'm aware of them (match bare metal, these are production level misc vms).

@Dzahn and I had discussed this previously, and as I haven't done a ganeti install yet, I requested that I be allowed to handle this one.

RobH added a comment.Aug 5 2015, 6:45 PM

Actually, there are no existing ganeti VMs using public IP space. I'm not sure of the support of that.

Dzahn updated the task description. (Show Details)Aug 5 2015, 6:46 PM

i adjusted the request to "private IP". we probably don't need a public one just yet, at least not for testing the config and archive import

RobH added a subscriber: akosiaris.Aug 5 2015, 6:50 PM

The next available name in elements is fermium.

I'd like to create these, but I don't want to overstep into @akosiaris' process. Perhaps we can get him to sign off on this and kick it back to me for implementation? (I want to do the work, I just do not want to upset anyone =)

Private is fine for now as we're just testing the process. Perhaps add to misc do we can view and physically interact but that's not a strict requirement per se.

RobH reassigned this task from RobH to akosiaris.Aug 5 2015, 8:36 PM
RobH added a subscriber: RobH.

I'll note that upon further reflection, I'm not entirely comfortable circumventing Alex in this process. When I setup the process with him awhile ago, all of these were very specifically supposed to go through him. (Much like all spare requests filter through me.) The documentation exists so when he is on vacation, or an emergency allocation is needed, proper processes can be followed.

I'm not really comfortable circumventing the process simply because we want to work on this today. (I really do want to work on this! I even have all the commands ready to go to create things.) So as long as someone else is willing to say this is approved, and overrriding Alex, I'm ok to continue. (I'm just not comfortable making that call.)

RobH added a comment.Aug 5 2015, 8:49 PM

Note: While the testing instance can use a private IP, the eventual live instance will need a public IP.

Since Alex will have to review this task anyone, perhaps he can advise. When looking at the eqiad ganeti serivce nodes (1001-1004), all only have an internal network connection for their system network connections. Do we support VM instances having public IP addresses?

Hello, well, I am fine running the process, but I did create the documentation so anyone can do it. For whatever reasons, I might be available for longer than wanted periods of time, so it would be awesome if someone else can do it as well. @RobH, thanks for wanting to get involved into this.

Technically, public IPs are possible in ganeti, although restricted to public1-c-eqiad for eqiad at this point. That is a rack row restriction we plan to work on resolving in the future. For what is worth, private IPs are also restricted to private1-c-eqiad as well for the same reason. Also, ganeti nodes do have an internal IP only, but it is possible to have public IPs for the VMs. In fact I 've updated the documentation on how to do that in https://wikitech.wikimedia.org/wiki/Ganeti#Create_the_VM_.28public_IP.29. That being said, we haven't had a public IP VM yet so I would like to be present for that first one please.

That being said, I like @Dzahn's argument about using private IP for now so let's go for that. @RobH, I 'll be around to assist but let's see if anyone else apart from me can run this process.

Needless to say, this is approved

JohnLewis reassigned this task from akosiaris to RobH.Aug 6 2015, 11:58 AM

Thanks @akosiaris.

Presumably since we'll be testing an import here we'd want to view mailman physically so when you do this and install it Rob, perhaps add it to misc? An import can look fine server side but can be a different question on the front end which is why I recommend this. Anyone else's decision can be different though.

Not sure I understand what "view mailman physically" exactly means but I am gonna assume it means have the web interface publicly available in which case, a temporary misc-web rule is probably worth it.

On an off topic note. I assume full puppetization of the service is in order right ? That will be especially helpful since it will make it possible to just reinstall the VM with minimal fuss when we want to finally go with a public IP. Not that other ways are not possible to do that, but it is a specially easy way and also guarantees the service will be correctly pupptized (since it will not be working otherwise)

@akosiaris yes, misc-web to view the interface is what I meant, perhaps not the best wording in my part :)

The service is fully puppetised and has been for years to my knowledge. It recently had changes which makes it less dependent on running on sodium with its IPs and config plus is more condensed (thanks to Faidon). The idea of this first stage then go into production is to test a migration and also to change puppet if needed as well.

JohnLewis triaged this task as Medium priority.Aug 6 2015, 12:15 PM
Dzahn added a comment.Aug 6 2015, 3:55 PM

Or we can use ssh -D port forwarding when connecting to a bastion host and put localhost in SOCKS proxy settings of the browser. Then we will also be able to see the web interface while running on a private IP.

Dzahn added a comment.Aug 6 2015, 6:11 PM

Let's go ahead then and create the instance please.

RobH added a comment.Aug 6 2015, 7:26 PM

So slow...

robh@ganeti1003:~$ sudo gnt-instance add \

-t drbd \
-I hail \
--net 0:link=br0 \
--hypervisor-parameters=kvm:boot_order=network \
-o debootstrap+default \
--no-install \
-B vcpus=1,memory=8g --disk 0:size=200g \
fermium.eqiad.wmnet

Thu Aug 6 19:21:18 2015 - INFO: No-installation mode selected, disabling startup
Thu Aug 6 19:21:22 2015 - INFO: Selected nodes for instance fermium.eqiad.wmnet via iallocator hail: ganeti1004.eqiad.wmnet, ganeti1003.eqiad.wmnet
Thu Aug 6 19:21:24 2015 * creating instance disks...
Thu Aug 6 19:21:34 2015 adding instance fermium.eqiad.wmnet to cluster config
Thu Aug 6 19:21:36 2015 adding disks to cluster config
Thu Aug 6 19:21:37 2015 - INFO: Waiting for instance fermium.eqiad.wmnet to sync disks
Thu Aug 6 19:21:39 2015 - INFO: - device disk/0: 0.10% done, 1h 23m 9s remaining (estimated)
Thu Aug 6 19:22:39 2015 - INFO: - device disk/0: 1.20% done, 1h 29m 39s remaining (estimated)
Thu Aug 6 19:23:41 2015 - INFO: - device disk/0: 2.40% done, 1h 25m 47s remaining (estimated)
Thu Aug 6 19:24:41 2015 - INFO: - device disk/0: 3.50% done, 1h 21m 29s remaining (estimated)
Thu Aug 6 19:25:42 2015 - INFO: - device disk/0: 4.60% done, 1h 21m 23s remaining (estimated)

RobH added a comment.Aug 6 2015, 7:28 PM

I'll note the wikitech directions have: -B vcpus=<x>,memory=<y>g --disk 0:size=<z>g \ which is missing the , after the memory line. I don't think it matters though, as -- seems to be the required delimiter.

Dzahn closed this task as Resolved.Aug 6 2015, 10:55 PM

resolved by @RobH

akosiaris added a comment.EditedAug 7 2015, 8:34 AM

I'll note the wikitech directions have: -B vcpus=<x>,memory=<y>g --disk 0:size=<z>g \ which is missing the , after the memory line. I don't think it matters though, as -- seems to be the required delimiter.

Yeah, those are too parameters in one single line. the \ is a line continuation operator. You can just issue the entire command in one single line. In any case I did split it up that in 2 lines