I think we only use the site link diff. maybe we could deserialize the diff in more of a lazy way that only instantiates the site link part of the diff.
NOTE: We may not need diffs in changes at all, if we can remove the need of programmatic filtering. See in particular T111161: Subscribe client wikis immediately after adding sitelinks