Change Details

%%%Right now we have the server "carbon" in eqiad to serve as what we call an install server: a DHCP server, a TFTP server, a Squid (for webproxy), our own apt repository (reprepro) and an Ubuntu mirror.%%% %%%Right now, tftpboot consumes 144M, reprepro consumes 6G, squid's spool 1.9G and the Ubuntu mirror 761G. Bandwidth-wise, we we pushing 200-250Mbps, but we're down to less than 10Mbps since the switch from lighttpd to nginx, presumably due to If-Modified-Since support. I/O-wise, the workload is very read intensive and has a lot of hot files, so pagecache helps to keep IOPS really low.%%% %%%If we add Debian into the mix (we don't strictly need to), this will be another ~400G in size for i386/amd64 (https://www.debian.org/mirror/size). carbon has the capacity for this -- it currently has 1.1T free.%%% %%%However, I think it's a bit of a pity for us to spend 1-1.5T in disk space and have them sitting idle. Therefore, I propose that we a) split the mirror server from the install server(s) b) publically advertise the install server as an official Debian/Ubuntu mirror. (a) is not strictly a prerequisite for (b), but it's my view that it would be better to split those two roles, as i) we frequently tinker with carbon (disable puppet and do manual hacks while troubleshooting, ii) a mirror would require more software to run, like FTP and rsync and possibly SSH (for push mirroring), and we shouldn't expose our apt repository/install server to a larger attack vector.%%% %%%Since carbon has the necessary space, this is would actually entail procuring a new *install* server. The requirements for this should be really tiny, a small misc server would probably do it. Note that we already have install2001 in codfw and it seems overprovisioned for an install server and underprovisioned for a mirror (it doesn't even fit Ubuntu).%%% %%%Moreover, this would also entail "paying" for the additional resources an official mirror would need compared to an unofficial one like we have now. For Debian, I asked around and even big country mirrors have < 100Mbps in traffic, which I think wouldn't be a problem for us. Ubuntu may be more popular but primary mirrors are Canonical's, so we'd only be a secondary mirror and one out of many, so I don't expect much usage there either.%%% %%%I'm soliciting opinions and approval to go forward; let me know and I'll file a separate procurement request.%%%

Right now we have the server "carbon" in eqiad to serve as what we call an install server: a DHCP server, a TFTP server, a Squid (for webproxy), our own apt repository (reprepro) and an Ubuntu mirror. tftpboot consumes 144M, reprepro consumes 6G, squid's spool 1.9G and the Ubuntu mirror 761G. Bandwidth-wise, we we pushing 200-250Mbps, but we're down to less than 10Mbps since the switch from lighttpd to nginx, presumably due to If-Modified-Since support. I/O-wise, the workload is very read intensive and has a lot of hot files, so pagecache helps to keep IOPS really low. If we add Debian into the mix (we don't strictly need to), this will be another ~400G in size for i386/amd64 (https://www.debian.org/mirror/size). carbon has the capacity for this -- it currently has 1.1T free. However, I think it's a bit of a pity for us to spend 1-1.5T in disk space and have them sitting idle. Therefore, I propose that we a) split the mirror server from the install server(s) b) publically advertise the install server as an official Debian/Ubuntu mirror. (a) is not strictly a prerequisite for (b), but it's my view that it would be better to split those two roles, as i) we frequently tinker with carbon (disable puppet and do manual hacks while troubleshooting, ii) a mirror would require more software to run, like FTP and rsync and possibly SSH (for push mirroring), and we shouldn't expose our apt repository/install server to a larger attack vector. Since carbon has the necessary space, this is would actually entail procuring a new *install* server. The requirements for this should be really tiny, a small misc server would probably do it. Note that we already have install2001 in codfw and it seems overprovisioned for an install server and underprovisioned for a mirror (it doesn't even fit Ubuntu). Moreover, this would also entail "paying" for the additional resources an official mirror would need compared to an unofficial one like we have now. For Debian, I asked around and even big country mirrors have < 100Mbps in traffic, which I think wouldn't be a problem for us. Ubuntu may be more popular but primary mirrors are Canonical's, so we'd only be a secondary mirror and one out of many, so I don't expect much usage there either. I'm soliciting opinions and approval to go forward; let me know and I'll file a separate procurement request.

%%%Right now we have the server "carbon" in eqiad to serve as what we call an install server: a DHCP server, a TFTP server, a Squid (for webproxy), our own apt repository (reprepro) and an Ubuntu mirror.%%% %%%Right now, tftpboot consumes 144M, reprepro consumes 6G, squid's spool 1.9G and the Ubuntu mirror 761G. Bandwidth-wise, we we pushing 200-250Mbps, but we're down to less than 10Mbps since the switch from lighttpd to nginx, presumably due to If-Modified-Since support. I/O-wise, the workload is very read intensive and has a lot of hot files, so pagecache helps to keep IOPS really low.%%% %%%If we add Debian into the mix (we don't strictly need to), this will be another ~400G in size for i386/amd64 (https://www.debian.org/mirror/size). carbon has the capacity for this -- it currently has 1.1T free.%%% %%% However, I think it's a bit of a pity for us to spend 1-1.5T in disk space and have them sitting idle. Therefore, I propose that we a) split the mirror server from the install server(s) b) publically advertise the install server as an official Debian/Ubuntu mirror. (a) is not strictly a prerequisite for (b), but it's my view that it would be better to split those two roles, as i) we frequently tinker with carbon (disable puppet and do manual hacks while troubleshooting, ii) a mirror would require more software to run, like FTP and rsync and possibly SSH (for push mirroring), and we shouldn't expose our apt repository/install server to a larger attack vector.%%% %%% Since carbon has the necessary space, this is would actually entail procuring a new *install* server. The requirements for this should be really tiny, a small misc server would probably do it. Note that we already have install2001 in codfw and it seems overprovisioned for an install server and underprovisioned for a mirror (it doesn't even fit Ubuntu).%%% %%% Moreover, this would also entail "paying" for the additional resources an official mirror would need compared to an unofficial one like we have now. For Debian, I asked around and even big country mirrors have < 100Mbps in traffic, which I think wouldn't be a problem for us. Ubuntu may be more popular but primary mirrors are Canonical's, so we'd only be a secondary mirror and one out of many, so I don't expect much usage there either.%%% %%%I'm I'm soliciting opinions and approval to go forward; let me know and I'll file a separate procurement request.%%%