Consider caching GriddedPerms and Perms (singleton model)

[jaypantone]

This would cost some memory, but on the other hand it would reduce a lot of load on the GC.

We could also cache, e.g., to_bytes on GriddedPerms and maybe some symmetries.