Make the default robot policy a configurable option.
Also replace some odd hardcoded robot policies (for Special:Specialpages and for nonexistent images) with 'noindex,nofollow', which seems more
appropriate at least for the latter. I'm not quite sure _why_ Special:Specialpages was marked as 'index,nofollow' before; it's been that way since
r1284. It might be safe to just leave it at the default setting, but I don't really think it makes much difference either way.