Simplify SyncTTM and handlers
- The updated SyncTTM code eliminates the use of ranks, add, remove, get transforms and instead uses a prioritized list of handlers for every transformer.
Every transformer now provides upto 4 handlers: onTag, onNewline, onEnd, onAny
onEnd, onNewline are the most specific and invoked on EOFTk and NLTk respetively. If not, onTag is invoked on all other tokens.
Lastly, onAny is invoked on all tokens unless disabled by the more specific handlers that were run previously. It is the fallback handler if the other more specific handlers leave the token unmodified.
The abstract TokenHandler class provides default implementations for the above handlers which are identity transformations.
- Updated all token transformers to implement the above interface. In most cases, this required renaming or forwarding requests to existing handlers.
To mimic the previous rank-based skip functionality, the non-onAny handlers can set a skip flag in the return object which causes the onAny handler to be skipped.
To mimic the add/remove handler functionlity, the non-onAny handlers can mark the transformer inactive which causes the onAny handler to be skipped until it is renabled again.
- Further simplification of this interface is likely possible in later rounds of refactoring in the future.
- Updated genTest code and transformTests.js runner as well. The transformTest.js runner is much simplifed with the above changes.
- All parser tests pass. Initial rt testing results show no regressions.
- Performance is not affected either. In fact, timing mode runs of transformTests.js show ~2x speedup in the core testing code. But, this doesn't seem to translate into any measurable parse time improvements for the entire parse itself.
- TODO for later.
- Potentially simplify code in the individual handlers in the transformers.