Page MenuHomePhabricator

Replace/refresh carbon
Closed, ResolvedPublic

Description

According to racktables, carbon was purchased back on 2011-01-27, so it's up for a much-needed refresh. It's also precise right now, so this is good timing :) Per @mark, this needs to happen quick to fit it into our current FY.

Per T132757, we're splitting off the install server and apt repository parts of carbon e.g. to install1001 (and install2001). So that should leave carbon (and hence its replacement) purely as the mirror server (also see T84817) , which is good for security, among other things.

We currently use ~1.6TB of space on carbon, out of 2TB available, most of which for mirrors. These are split right now like this:

  • 1.1T for Ubuntu
  • 435G for Debian
  • 6.3G for tails

Since we're refreshing, I'd like us to expand our disk space considerably, to account for:

  • organic growth in the next 5 years or so
  • expand our mirror to cover the entire Debian archive (1.5T right now, so +1T, not counting debian-cd which is a nice-to-have). This would allow us to become an official mirror, which is a) a nice thing to contribute back to Debian, but also b) makes us a push mirror, getting updates faster.
  • add RAID-10, since this service is quite a SPOF for our infrastructure
  • possibly mirroring other important software as well

So, I'd say:

  • around 8-12TB of usable capacity, in the cheapest config (say, 6x3TB)
  • 96GB of RAM or so (for pagecache)
  • RAID controller capable of RAID10 and with a 512MB BBU
  • Ideally, a 10G NIC, as this is a fileserver

Event Timeline

RobH mentioned this in Unknown Object (Task).Jun 6 2016, 6:24 PM

We do not have any spare systems with 3TB * 6, only 4TB * 4 and it would max out the disk bays of those particular systems:

Example system WMF4723. Dual Intel® Xeon® Processor E5- 2623 V3 with 32GB RAM and 4 * 4TB disks.

Per the request, this isn't anywhere near the amount of RAM or disk space, so this will need to be a specific quotation.

The dbstore servers have the right amount of disk space (1.6TB disks * 16 = 25.6 RAW / 2 = 12.8 usable space AFTER raid10. They have far too much memory (512GB), and need to have a 10GbE added. Additonally, it is a dual CPU config, where carbon only has single.

I've created task T137132 to track the quotation of a replacement server.

4*4TB makes for 8TB usable, so it could work, although it's kind of suboptimal (just 4 spindles); 32GB would probably work fine for now (carbon has just 8GB!), and 10GbE… can probably wait until we actually reach that amount of traffic, when we can just pay for an extra card.

(my specs were quite inflated because I assumed it'd be a new purchase and made sense to spend e.g. $200 extra for 10GbE now, rather than upgrade later when we potentially need it when we e.g. become an official mirror)

The dbstore config is quite the overkill — the amount of RAM in that spec is insane, as is the use of 1.6TB $1000+ SSDs which are not needed _at all_ for this use case.

Since you started the quoting process, let's continue that regardless and hear from Dell. We can thus make an informed decision about this when we have dollar amounts.

I had Dell just quote to the specification, and I advised HP to reconfigure the quote and chassis as needed since we're reducing the CPU sockets by half and the memory by 75%.

I should have all quotes back today, I'll create a matrix of the different options, including using an on site spare machine.

This system has arrived on site and is being setup via T139171.

Or should this stay open for replacing the other aspects of carbon? (The install server side.)

RobH closed subtask Unknown Object (Task) as Resolved.Oct 12 2016, 5:46 PM