⚓ T257118 Beta cluster has reached its quota

	Subject	Repo	Branch	Lines +/-
	Drop deployment-sca04 from betacluster deploys	mediawiki/services/recommendation-api/deploy	master	+0 -1

		Status	Subtype	Assigned	Task
		Open		None	T215217 deployment-prep (beta cluster): Code stewardship request
		Resolved		None	T257118 Beta cluster has reached its quota

Ladsgroup created this task.Jul 4 2020, 3:17 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 4 2020, 3:17 PM

DannyS712 subscribed.Jul 4 2020, 3:17 PM

taavi subscribed.Jul 4 2020, 3:17 PM

List of instances:

Instance Name	VCPUs	RAM (MB)	Disk (GB)	Usage (Hours)	Age (Seconds)	State
deployment-acme-chief03	1	2048	20	39.46	40926995	Active
deployment-acme-chief04	1	2048	20	39.46	40924947	Active
deployment-aqs01	2	4096	40	39.46	43385428	Active
deployment-aqs02	2	4096	40	39.46	43372228	Active
deployment-aqs03	2	4096	40	39.46	43372210	Active
deployment-cache-text06	2	4096	40	39.46	rEEVL7260971c7df2	Active
deployment-cache-upload06	2	4096	40	39.46	7261651	Active
deployment-changeprop	1	2048	20	39.46	51182405	Stopped
deployment-chromium01	1	2048	20	39.46	51215972	Active
deployment-chromium02	1	2048	20	39.46	51224204	Active
deployment-cpjobqueue	2	4096	40	39.46	51210191	Stopped
deployment-cumin02	1	2048	20	39.46	38848120	Active
deployment-cumin	1	2048	20	39.46	1055004	Active
deployment-db05	8	16384	160	39.46	43792600	Active
deployment-db06	8	16384	160	39.46	40429759	Active
deployment-deploy01	8	8192	60	39.46	51223631	Active
deployment-deploy02	8	8192	60	39.46	51223229	Active
deployment-docker-changeprop01	1	2048	20	39.46	4413318	Active
deployment-docker-citoid01	1	2048	20	39.46	36043832	Active
deployment-docker-cpjobqueue01	1	2048	20	39.46	2071846	Active
deployment-docker-cxserver01	1	2048	20	39.46	35988148	Active
deployment-docker-mathoid01	1	2048	20	39.46	36472431	Active
deployment-echostore01	1	2048	20	39.46	17170994	Active
deployment-elastic05	4	8192	80	39.46	51154703	Active
deployment-elastic06	4	8192	80	39.46	51155828	Active
deployment-elastic07	4	8192	80	39.46	51152564	Active
deployment-etcd-01	1	2048	20	39.46	51201269	Active
deployment-eventgate-3	1	2048	20	39.46	21411224	Active
deployment-eventlog05	4	8192	80	39.46	51208642	Active
deployment-eventstreams-1	1	2048	20	39.46	15548061	Active
deployment-fluorine02	1	2048	80	39.46	51193096	Active
deployment-imagescaler01	2	4096	40	39.46	51190311	Active
deployment-imagescaler02	2	4096	40	39.46	51202702	Active
deployment-imagescaler03	2	4096	40	39.46	49250051	Active
deployment-ircd	1	2048	20	39.46	51194397	Active
deployment-jobrunner03	4	8192	80	39.46	51218925	Active
deployment-kafka-jumbo-1	1	2048	80	39.46	51203682	Active
deployment-kafka-jumbo-2	1	2048	80	39.46	51201263	Active
deployment-kafka-main-1	1	2048	20	39.46	51213003	Active
deployment-kafka-main-2	1	2048	20	39.46	51218210	Active
deployment-logstash03	8	16384	160	39.46	37060051	Active
deployment-logstash2	8	16384	160	39.46	51182252	Active
deployment-mailman01	1	2048	20	39.46	2419715	Active
deployment-maps05	4	8192	80	39.46	39724237	Active
deployment-mcs01	1	2048	20	39.46	51195371	Active
deployment-mdb01	1	2048	80	39.46	2122503	Active
deployment-mediawiki-07	4	8192	80	39.46	51211839	Active
deployment-mediawiki-09	4	8192	80	39.46	51216632	Active
deployment-memc04	2	4096	40	39.46	51191933	Active
deployment-memc05	2	4096	40	39.46	51183858	Active
deployment-memc06	2	4096	40	39.46	51184243	Active
deployment-memc07	2	4096	40	39.46	51204969	Active
deployment-memc08	2	4096	40	39.46	22661725	Active
deployment-ms-be05	8	16384	160	39.46	38761194	Active
deployment-ms-be06	8	16384	160	39.46	38761194	Active
deployment-ms-fe03	1	2048	20	39.46	38232895	Active
deployment-mwmaint01	1	2048	80	39.46	51223997	Active
deployment-mx02	1	2048	20	39.46	51211782	Active
deployment-ores01	4	8192	80	39.46	51219547	Active
deployment-parsoid11	2	4096	40	39.46	10590936	Active
deployment-poolcounter06	1	2048	20	39.46	18754786	Active
deployment-prometheus02	4	8192	80	39.46	46161569	Active
deployment-puppetdb03	1	2048	20	39.46	14229837	Active
deployment-puppetmaster04	2	4096	40	39.46	12932186	Active
deployment-push-notifications01	1	2048	20	39.46	2832947	Active
deployment-restbase01	4	8192	80	39.46	51180967	Active
deployment-restbase02	4	8192	80	39.46	51189611	Active
deployment-restbase03	4	8192	80	39.46	7229633	Active
deployment-sca01	1	2048	20	39.46	51184163	Active
deployment-sca02	1	2048	20	39.46	51165143	Active
deployment-sca04	2	4096	40	39.46	51198853	Active
deployment-schema-2	1	2048	20	39.46	34200080	Active
deployment-sentry01	2	4096	40	39.46	51183953	Active
deployment-sessionstore03	1	2048	20	39.46	17171015	Active
deployment-snapshot01	2	4096	40	39.46	51207377	Active
deployment-urldownloader02	1	2048	20	39.46	51224813	Active
deployment-wdqs01	4	8192	80	39.46	21527681	Active
deployment-webperf11	1	2048	20	39.46	51219955	Active
deployment-webperf12	1	2048	20	39.46	51221874	Active
deployment-wikifeeds01	1	2048	20	39.46	33600161	Active
deployment-xhgui01	1	2048	20	39.46	19310473	Active
deployment-xhgui02	1	2048	20	39.46	2042306	Active
deployment-zookeeper02	1	2048	20	39.46	17299874	Active

• Mholloway subscribed.Jul 4 2020, 4:11 PM

We have two appservers but five memcached nodes, that seems off.

RhinosF1 subscribed.Jul 4 2020, 6:45 PM

Deleted deployment-sentry01 according to T106915#6279270

In T257118#6279152, @Ladsgroup wrote:

We have two appservers but five memcached nodes, that seems off.

Not really, given the amount of traffic beta receives, and the fact we don't have load-balancing so it's hard to distribute requests more.

Most of the VMs you listed above are still in use, even if no one logs into them or touches them since some time.

I would ask why do we have mailman VMs in deployment-prep, OTOH. It seems quite off-topic there.

The two stopped changeprop VMs can be removed, I think, but I'd ask @hnowlan to confirm that's the case

In T257118#6279378, @Joe wrote:

I would ask why do we have mailman VMs in deployment-prep, OTOH. It seems quite off-topic there.

It's the Mailman v3 testing instance as per T52864.

In T257118#6279378, @Joe wrote:

Most of the VMs you listed above are still in use, even if no one logs into them or touches them since some time.

Indeed but this project is huge and this is list of all VMs, if we just audit and clean 10-20%, it frees up 20-40 VCPUs. For comparison, "meet" project has 8 VCPUs as the quota.

I would ask why do we have mailman VMs in deployment-prep, OTOH. It seems quite off-topic there.

Yup, that's what I'm working on: https://lists-beta.wmflabs.org I can also request a dedicated project if you think that's better.

• dpifke subscribed.Jul 6 2020, 11:32 PM

Ladsgroup updated the task description. (Show Details)Jul 6 2020, 11:44 PM

• dpifke updated the task description. (Show Details)Jul 6 2020, 11:47 PM

hnowlan updated the task description. (Show Details)Jul 7 2020, 9:18 AM

• Mholloway updated the task description. (Show Details)Jul 7 2020, 1:59 PM

• Mholloway updated the task description. (Show Details)Jul 7 2020, 2:19 PM

• Mholloway updated the task description. (Show Details)Jul 7 2020, 3:27 PM

Urbanecm updated the task description. (Show Details)Oct 5 2020, 10:21 PM

Urbanecm updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-releng) [2020-10-05T22:29:21Z] <Amir1> deleted deployment-imagescaler01 and deployment-imagescaler02 (T257118)

Mentioned in SAL (#wikimedia-releng) [2020-10-05T22:31:06Z] <Amir1> deleted deployment-mailman01 (T257118)

deployment-poolcounter06

I've marked this as "Needed" on behalf of Platform Engineering team. This is the primary and only poolcounter node, and is indeed being used per LabsServices.

Ladsgroup updated the task description. (Show Details)Oct 5 2020, 11:45 PM

Krinkle updated the task description. (Show Details)Oct 5 2020, 11:51 PM

EBernhardson updated the task description. (Show Details)Oct 7 2020, 12:47 AM

ssastry updated the task description. (Show Details)Oct 7 2020, 12:54 AM

cache-text, cache-upload, etcd, ircd, and memc are most certainly also still in use and we need at least one of each.

For memc we can most likely size down, I don't think we added this many intentionally but rather added new ones during rebuilds/upgrades and maybe left old ones?

cc @RLazarus Do you know if any of these were used for tests, and whether we could e.g. get a way with just one or two of the deployment-memc* instances? If so, where do we need to check for configs and update things etc. is the latest one the "right" one to keep?

Peachey88 updated the task description. (Show Details)Oct 7 2020, 2:02 AM

Peachey88 mentioned this in T244642: Do something about deployment-imagescaler01.

• dpifke updated the task description. (Show Details)Oct 7 2020, 2:50 AM

• dpifke updated the task description. (Show Details)Oct 7 2020, 2:53 AM

In T257118#6523783, @Krinkle wrote:

cache-text, cache-upload, etcd, ircd, and memc are most certainly also still in use and we need at least one of each.

For memc we can most likely size down, I don't think we added this many intentionally but rather added new ones during rebuilds/upgrades and maybe left old ones?

cc @RLazarus Do you know if any of these were used for tests, and whether we could e.g. get a way with just one or two of the deployment-memc* instances? If so, where do we need to check for configs and update things etc. is the latest one the "right" one to keep?

I don't know much about this, but @Joe addressed the same question in T257118#6279378:

We have two appservers but five memcached nodes, that seems off.

Not really, given the amount of traffic beta receives, and the fact we don't have load-balancing so it's hard to distribute requests more.

I will be deleting snapshot01 in a week and a half, per https://phabricator.wikimedia.org/T245402#6514449

hnowlan updated the task description. (Show Details)Oct 9 2020, 11:43 AM

I suspect some or all of the Kafka hosts (it's required for changeprop to behave normally) need to be kept but I don't know which.

The situation of the memcached servers is amazingly telling of how deployment-prep is unmanaged, and we should really dedicate some resources to it, but also work better when we do stuff there.

The memcached servers are used primarily for memcached, and then for redis (mainstash) and redis (locking).

See below the usage matrix:

server	memcached	mainstash	locks
memc04	x	x	x
memc05	x	x	x
memc06		x
memc07		x
memc08	x

My proposal would be to move everything to memc06-08, and remove memc04/05 soon, but this will need to be scheduled and done by someone. In the meantime, all those servers are currently serving traffic.

Joe updated the task description. (Show Details)Oct 12 2020, 11:18 AM

Joe updated the task description. (Show Details)

In T257118#6536304, @Joe wrote:

The situation of the memcached servers is amazingly telling of how deployment-prep is unmanaged, and we should really dedicate some resources to it, but also work better when we do stuff there.

T215217: deployment-prep (beta cluster): Code stewardship request

• ema updated the task description. (Show Details)Oct 12 2020, 11:22 AM

Joe updated the task description. (Show Details)Oct 13 2020, 6:23 AM

Joe mentioned this in T215217: deployment-prep (beta cluster): Code stewardship request.Oct 13 2020, 6:29 AM

AntiCompositeNumber mentioned this in T265580: 404 error for SVG image thumbnail in UploadWizard on Beta Cluster.Oct 15 2020, 3:55 PM

Ottomata updated the task description. (Show Details)Oct 21 2020, 8:04 PM

bd808 added a parent task: T215217: deployment-prep (beta cluster): Code stewardship request.Nov 16 2020, 7:17 PM

thcipriani updated the task description. (Show Details)Nov 16 2020, 7:19 PM

Mentioned in SAL (#wikimedia-releng) [2020-11-19T00:25:33Z] <Amir1> shutting off deployment-aqs* instances (T257118)

Change 641855 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/services/recommendation-api/deploy@master] Drop deployment-sca04 from betacluster deploys

https://gerrit.wikimedia.org/r/641855

gerritbot added a project: Patch-For-Review.Nov 19 2020, 12:28 AM

Change 641855 merged by jenkins-bot:
[mediawiki/services/recommendation-api/deploy@master] Drop deployment-sca04 from betacluster deploys

https://gerrit.wikimedia.org/r/641855

Maintenance_bot removed a project: Patch-For-Review.Nov 19 2020, 3:10 PM

Mentioned in SAL (#wikimedia-releng) [2020-11-19T20:30:29Z] <Amir1> delete deployment-aqs* instances (T257118)

Mentioned in SAL (#wikimedia-releng) [2020-11-19T20:30:39Z] <Amir1> shut down deployment-sca04 (T257118)

Ladsgroup mentioned this in rMSRD9ca186bddb44: Drop deployment-sca04 from betacluster deploys.Nov 19 2020, 8:40 PM

taavi mentioned this in T276650: Re-consider setting up a Kubernetes cluster on the Beta cluster.Mar 6 2021, 7:34 AM

The project is now down to 67 / 100 instances, Should the task now be closed off.

Number of VCPUs is still at 84% but that's not that bad

Zabe updated the task description. (Show Details)Feb 28 2022, 6:16 PM

In T257118#6947852, @Peachey88 wrote:

The project is now down to 67 / 100 instances, Should the task now be closed off.

Yes.

Beta cluster has reached its quota
Closed, ResolvedPublic
Actions

Description

List of instances

Details

Related Objects
Search...

Event Timeline

	F34191490: image.png
	Mar 26 2021, 7:48 AM

	F31917101: image.png
	Jul 4 2020, 3:17 PM

	F31917103: image.png
	Jul 4 2020, 3:17 PM

Beta cluster has reached its quotaClosed, ResolvedPublicActions