Page MenuHomePhabricator

eqiad: 1 VM request for testreduce
Closed, ResolvedPublic

Description

Cloud VPS Project Tested: deployment-prep ?
Site/Location: eqiad
Number of systems: 1
Service: testreduce (nodejs, formerly parsoid testing)
Networking Requirements: internal
Processor Requirements: 2
Memory: 4G
Disks: 40G
Other Requirements: similar to scandium (hardware)

Event Timeline

Dzahn created this task.Jul 14 2020, 4:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 14 2020, 4:21 PM

This is supposed to be a VM with buster, to host testreduce (nodejs, formerly on scandium, the parsoid test host on real hardware).

More details on T257906

CPU and disk requirements still to be determined. CCing @SubrahamanyamVarma

ssastry edited subscribers, added: ssastry; removed: SubrahamanyamVarma.Jul 14 2020, 5:05 PM

@Dzahn please note that my handle is @ssastry - I know it is confusing with multiple subbus and almost identical first names. Removing the other subbu from the task.

ssastry added a comment.EditedJul 14 2020, 5:16 PM

So, here is how rt-testing works. There is a testreduce server which is nodejs code that needs a connection to a mysql db. There are testreduce clients that run nodejs code to fetch page titles from the server and then run custom code. In the rt-testing use-case, it runs bin/roundtrip-test.js code form the parsoid repo which contacts Parsoid API to do a bunch of work on its behalf, runs some diffs, analyzes diffs and computes a test score and sends the test score with the title back to the testreduce server.

The testreduce server, testreduce clients and Parsoid API can all run on different servers. So, multiple VMs can run testreduce clients for example .. they just ping the server periodically for work, and so, we control new test runs by updating the testreduce server test run control file.

In the current hardware setup, everything runs on scandium. And, the most resource intensive part of the work is Parsoid API itself. The testreduce clients are less resource intensive (cpu-wise and maybe memory wise too). So, as long as scandium runs Parsoid API to respond to testreduce client requests, we can set up this other VM (or VMs) to run the server and clients. They can all be on the same single VM. The # of testreduce clients we can run is bounded by scandium's response rate and resource usage. Right now, we run 28 on scandium. I imagine, we can probably run 32 if we move testreduce scripts off scandium.

Dzahn added a comment.Jul 14 2020, 5:17 PM

@Dzahn please note that my handle is @ssastry - I know it is confusing with multiple subbus and almost identical first names. Removing the other subbu from the task.

I knew that but for some reason thought you actually changed your full name in phab :p

Change 612971 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] add IP for testreduce1001

https://gerrit.wikimedia.org/r/612971

Change 612971 merged by Dzahn:
[operations/dns@master] add IP for testreduce1001

https://gerrit.wikimedia.org/r/612971

Dzahn updated the task description. (Show Details)Jul 15 2020, 10:58 PM

Change 612983 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] DHCP/partman: add MAC for testreduce1001

https://gerrit.wikimedia.org/r/612983

Change 612983 merged by Dzahn:
[operations/puppet@production] DHCP/partman: add MAC for testreduce1001

https://gerrit.wikimedia.org/r/612983

Change 613201 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: add testreduce1001 with insetup role

https://gerrit.wikimedia.org/r/613201

Change 613201 merged by Dzahn:
[operations/puppet@production] site: add testreduce1001 with insetup role

https://gerrit.wikimedia.org/r/613201

Change 613278 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] parsoid: create new role to install just testreduce vd-server

https://gerrit.wikimedia.org/r/613278

Dzahn closed this task as Resolved.Jul 16 2020, 8:23 PM

VM has been created and runs with insetup role. Next will be applying a new role.

Change 613278 merged by Dzahn:
[operations/puppet@production] parsoid: create new role to install just testreduce

https://gerrit.wikimedia.org/r/613278