A missing null-check in an EntityLinkFormatter recently caused issues: T208142. LexemeIdHtmlFormatter and LexemeLinkFormatter (fixing patch https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseLexeme/+/470601) both are missing the null-check as well.
Other EntityIdFormatters implement different ways to fall back in case the entity does not exist. Maybe the right thing to do is to handle this within the dispatching formatter service, either by ensuring the entity that is to be formatted exists before it gets to the concrete formatter service, or by defining an exception that should be thrown and handled by the upper layer. The DispatchingEntityIdHtmlLinkFormatter already holds a NonExistingEntityIdHtmlFormatter that it could use.