The tokenizer I've written is "one token per line", so it's not quite a real tokenizer. It would be useful, if we want to emit nice errors, to actually record locations of things. That would help integrating with the unified ecosystem, like vfile, for reporters.
The tokenizer I've written is "one token per line", so it's not quite a real tokenizer. It would be useful, if we want to emit nice errors, to actually record locations of things. That would help integrating with the unified ecosystem, like
vfile, for reporters.