Token transformers return one of:
- { async: false, token: .., }
- { async: false, tokens: [..] }
- { async: true }
- { async: true, token: ... }
- { async: true, tokens: [..] }
- { async: true, tokens: [] }
And, then, we have verifyTokenIntegrity converting the return value to a normalized { tokens: .. } format. This is done in the token accumulators used in the Async TTM.
We should instead normalize all return values to canonical forms that minimizes sanity checks and inter-conversions between return type formats. This is good from both the code readability and performance standpoints.