introduction of mocking on "Functions should only be one level of abstraction" should be removed

https://github.com/jupeter/clean-code-php#functions-should-only-be-one-level-of-abstraction

I think your refactoring results in two helper-classes "Lexer" and "Tokenizer". These types of classes should not be mocked nor handled by the di-framework.

Uncle Bob suggests to

Mock across architecturally significant boundaries, but not within those boundaries.

http://blog.cleancoder.com/uncle-bob/2014/05/10/WhenToMock.html