Explicitly define each 'service' toollabs provides, and what the documented interfaces to them are. Once this is done, we'll codify them into scripts that emit boolean values (up / down), and use those to monitor uptime.

1. Bastion Hosts
   1. SSH works
   2. CPU / Memory usage within sane levels, so interactive use is possible
2. Grid engine
   1. Job starts executing within X seconds
   2. Continuous jobs execute continuously
   3. Queues / Jobs in Errored state don't exceed threshold
   4. check that the number of pending jobs is under some threshold
3. Webservices
   1. Webservice starts in X secods
   2. Webservice stays up as long as there is no developer error, executing continuously
   3. Webservice available to the external world continuously
4. Cron services
   1. Crons run when they should
5. Redis
   1. Write an entry (appropriate timeout)
   2. Read an entry just written (appropriate timeout)
6. Tools DB
   1. Write some data to it (appropriate timeout)
   2. Read some data from it (appropriate timeout)
7. LabsDB
   1. Replag is 0
   2. Read queries execute reasonably quickly (set up a stable set of 'reference' queries, and measure their output)
   3. Write access on user databases (appropriate timeout)
8. NFS (Homes)
   1. Read random file (appropriate timeout)
   2. Write random file (appropriate timeout)
9. NFS (Dumps)
   1. Read random file (appropriate timeout)
   2. New dumps available within X minutes / hours of them appearing on dumps.wikimedia.org
10. Misc
    1. Tools home page is available and showing sensible things