Our lexer builds a list of tokens before parsing begins. The usual implementation is to stream tokens one by one. We've done some work to optimize our unusual architecture so we're not too far behind. At this point switching to a streaming lexer is likely to give us modest performance improvements by allowing us to iterate over ordered objects only once.