Basically LGTM. However, there are many things I would want to rename/remodel for clarity purposes. Right now, I find this piece of our codebase quite hard to grasp in comparison to its complexity.
A first step IMO is to add several performance regression tests. That way we can refactor with more confidence.