We rely on the stock mysql SHOW STATUS functions to detect slave lag, but these only give log -> DB lag not the "lag behind master DB". We want to properly detect lag even when:
a) The slave can't talk to the master
b) Binlogs relay at a trickle due to network issues
c) Replication uses chaining (eg. master -> slave1 -> slave2)
A few possibilities:
a) We have pt-heartbeat set up, and it would be nice to use something like that. We can change how DatabaseMysql::getLag() works using a config flag. I'd prefer this.
b) Use some schema (post-send lazy loading and local memcached) to check slave vs master position/timestamp instead of only using the Mysql slave lag functions which only check the slaves application lag (not the full relay lag).
Also make sure the read only mode triggers when remote DC slaves are lagged too.