feat: new quantizer for vllm

[llcnt]

Description

This PR is the little sister of this PR in pruna_pro.
This PR in pruna add some flexibility around hqq quantization and saving:

• a new parameter patch_for_inference is added to allow the user to quantize in 4bits without patching the layers (because can make them un-loadable);
• a new parameter default_to_hf allows the user to select a specific implementation of hqq quantization (previously only a try:...except was done, but no freedom was given to the user). this is important in particular because vllm expects a models that has been saved with the 'transformers' structure, not the AutoHQQModel one.

Related Issue

Fixes #(issue number)

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Unit test pass locally.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

Additional Notes

[github-actions]

This PR has been inactive for 10 days and is now marked as stale.

[sharpenb]

This is functional codeso it is approved. Some comments that would be important to consider before merging:

• Simplify/Clairfy the interface in this PR
• Test in this PR the compatibility fo the new introduced configs with the compatible algorithms in HQQ.
• Add step in the roadmap which corresponds to refactorization of the quantization code aiming at homogenizing the quant UX between quantizers, and the quant features (e.g. save/load), making easier the implementation of custom quants.

[sharpenb]

When is that important for the user to bypass AutoHQQHFModel?

[sharpenb]

Could we maybe find a way to avoir having this additional hyperparameter?

[llcnt]

In the context of vllm, we need to bypass AutoHQQ and use AutoModel (because the former produces a qmodel.pt while vllm needs .safetensors)

[llcnt]

I did't find a clean way to avoid the use of additional hyperparameter. If you have any idea I will be happy to add it :)
In general, do you remember why we still have this try...except with AutoHQQ and AutoModel in hqq.py? Why didn't we have since the beginning a hyperparameter to let the user select which type of implementation s/he want to use?

[llcnt]

This is functional codeso it is approved. Some comments that would be important to consider before merging:

* Simplify/Clairfy the interface in this PR

* Test in this PR the compatibility fo the new introduced configs with the compatible algorithms in HQQ.

* Add step in the roadmap which corresponds to refactorization of the quantization code aiming at homogenizing the quant UX between quantizers, and the quant features (e.g. save/load), making easier the implementation of custom quants.

I re-ran locally all unit test related to hqq, and there are valid (the 2 hyperparameters introduced have some default value that yields to the exact previous code) :)

[github-actions]

This PR has been inactive for 10 days and is now marked as stale.

[johnrachwan123]

bugbot run

[johnrachwan123]

LGTM

[sharpenb]

Reapproving

[cursor]

Bug: Quantized Model Retrieval Fails Post-Context Exit

After the ModelContext exits, the code attempts to retrieve the quantized model via pipeline.working_model. However, the context manager's __exit__ method reassigns the working_model to a specific pipeline attribute (e.g., transformer, unet) and then deletes pipeline.working_model. This causes hasattr(pipeline, "working_model") to return False, resulting in the unquantized pipeline object being returned instead of the quantized model.

Furthermore, moving the move_to_device call outside the context manager, after safe_memory_cleanup() (which includes torch.cuda.empty_cache()) in __exit__, can lead to device inconsistencies. The original approach of returning the model object was correct as the context manager automatically updates it and handles device placement.