Expand RAM on arclamp hosts and move them to baremetal
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krinkle
	Aug 25 2022, 1:43 PM

Description

Location:
- Eqiad webperf1004 VM -> arclamp1001,
- Codfw webperf2004 VM -> arclamp2001
Memory: 16GB -> 32GB

The number of entry point and flamegraph variants (excimer-wall, excimer-cpu, k8s, etc) has increased, and the amount of daily samples has increased as well now that MW REST API and Parsoid PHP exist and essentially start to take over traffic from what was or would-have-been RESTBase.

It seems we were already hugging the limits before, but we're now over the limit more often than not during each run, which causes some flamegraphs to not get generated until a lucky attempt later on.

Details at T315056: arclamp_generate_svgs OOMs.

We'll try to find upstream solutions that make it more memory efficient overall, but for now, given this isn't run-away or uncontrolled growth, a doubling of memory shouhld easily gain us a few years.

Prior history:

In T260192#6397791, @Dzahn wrote:

[..]
18:28 < mutante> i will claim it's resolved and if it turns out you need more than 16 then let's reopen it and that discussion from the start
18:29 < mutante> because then they would become special cases among the other VMs

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	arclamp: Remove rsync::quickdatacopy	operations/puppet	production	+0 -9

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Krinkle	T200108 Increase retention of ArcLamp SVGs to 2 years
Resolved	fgiunchedi	T316223 Expand RAM on arclamp hosts and move them to baremetal
		Unknown Object (Task)
Resolved	Papaul	T319428 Q2:rack/setup/install arclamp2001.codfw.wmnet
Resolved	fgiunchedi	T319429 service implementation tracking: arclamp2001.codfw.wmnet
		Unknown Object (Task)
Resolved	• Cmjohnson	T319433 Q2:rack/setup/install arclamp1001.eqiad.wmnet
Resolved	fgiunchedi	T319434 service implementation tracking: arclamp1001.eqiad.wmnet

Event Timeline

Krinkle created this task.Aug 25 2022, 1:43 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 25 2022, 1:43 PM

Krinkle updated the task description. (Show Details)Aug 25 2022, 1:43 PM

Instead let's move these to a baremetal host instead? We're hitting some limits of what makes sense with Ganeti for these, one other issue is high rate of memory changes, which currently allows to migrate these off to a different host with hacks like temporarily stopping arclamp processes. We always buy a few spare hosts for unforeseen machine needs, so I guess the hardware is available.

That would also be a fine opportunity to move away from the confusing naming scheme, given that webperf1003 and 1004 are totally different services, something like xenon* or arclamp* instead.

RhinosF1 subscribed.Aug 25 2022, 2:00 PM

+1 to not using the same names for the different webperf roles, thought the same before, should match more the puppet role

And yea, like the history says the discussion was to start from scratch once we get over the 16GB RAM limit. Hardware sounds the right way indeed.

Looking at CPU and disk usage (currently 150ish since some data is now on Swift) and the desired RAM, servers with "config A" would do just fine.

Krinkle mentioned this in T315056: arclamp_generate_svgs OOMs.Aug 29 2022, 3:28 PM

Dzahn mentioned this in Unknown Object (Task).Sep 1 2022, 8:26 PM

Dzahn mentioned this in Unknown Object (Task).Sep 1 2022, 8:30 PM

Dzahn added subtasks: Unknown Object (Task), Unknown Object (Task).

@MoritzMuehlenhoff @Krinkle I made procurement subtasks. There are 2 because the template says it needs to be limited to a specific DC. Please take a look if you have any comments for example on the sections about disk space and RAID. I am just saying "as long as it has 32GB RAM and matches the VM (or more obviously) to keep it as flexible as possible for dcops.

@Dzahn LGTM. Thanks for filing those. I would suggest for the hostnames to go with arclamp#001.

jbond triaged this task as Medium priority.Sep 6 2022, 2:41 PM

Retitling the task and dropping vm-requests

MoritzMuehlenhoff renamed this task from Resize webperf1004/2004 VM for arc-lamp to Expand RAM on arclamp hosts and move them to baremetal.Sep 6 2022, 2:43 PM

MoritzMuehlenhoff removed projects: vm-requests, Infrastructure-Foundations.

Krinkle updated the task description. (Show Details)Sep 6 2022, 3:37 PM

Krinkle added a project: Arc-Lamp.Oct 15 2022, 10:22 PM

Restricted Application added a project: Performance-Team. · View Herald TranscriptOct 15 2022, 10:22 PM

Krinkle edited projects, added Performance-Team (Radar); removed Performance-Team.Oct 15 2022, 10:22 PM

Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.

Is there a more specific tag we can use for this instead of SRE? perhaps serviceops?

Jclark-ctr closed subtask Unknown Object (Task) as Resolved.Nov 4 2022, 2:23 PM

akosiaris edited projects, added serviceops-deprecated; removed SRE.Nov 8 2022, 3:01 PM

akosiaris subscribed.

In T316223#8362915, @jbond wrote:

Is there a more specific tag we can use for this instead of SRE? perhaps serviceops?

Yeah, this one is a bit in a grey area regarding which SRE team is the best one to deal with it, but ServiceOps new is the most adjacent one given MediaWiki. We 'll take over this one.

akosiaris moved this task from Incoming 🐫 to 🛠 Upgrades and Hardware on the serviceops-deprecated board.Nov 9 2022, 7:45 AM

Dzahn awarded a token.Nov 9 2022, 5:55 PM

Dzahn mentioned this in T319434: service implementation tracking: arclamp1001.eqiad.wmnet.

Dzahn mentioned this in T319429: service implementation tracking: arclamp2001.codfw.wmnet.

@akosiaris Thank you! So what this is is:

hardware has been procured (I reviewed/approved in T316906 (eqiad) and T316907 (codfw). done, i think. Though the dcops-codfw task looks open ->

dcops racks hardware T319433 (eqiad) and T319428 (codfw). pending ->

T319434 (eqiad) and T319429 (codfw) are supposed to be assigned to someone by dcops and that would be basically be this ticket

Dzahn mentioned this in T319428: Q2:rack/setup/install arclamp2001.codfw.wmnet.Nov 14 2022, 6:33 PM

Dzahn mentioned this in T319433: Q2:rack/setup/install arclamp1001.eqiad.wmnet.

Papaul closed subtask Unknown Object (Task) as Resolved.Nov 14 2022, 11:49 PM

• lmata added a project: SRE Observability (FY2022/2023-Q3).Nov 25 2022, 1:53 PM

• lmata moved this task from Inbox to Up next on the SRE Observability (FY2022/2023-Q3) board.Nov 30 2022, 6:40 PM

In T316223#8381863, @akosiaris wrote:

In T316223#8362915, @jbond wrote:

Is there a more specific tag we can use for this instead of SRE? perhaps serviceops?

Yeah, this one is a bit in a grey area regarding which SRE team is the best one to deal with it, but ServiceOps new is the most adjacent one given MediaWiki. We 'll take over this one.

Per 28f86674054b7 observability has taken over arclamp. It aligns more closely with their area of expertise and focus than ServiceOps new. I thank them for this.