E2E test for the experimental compress algorithm based on https://arxiv.org/abs/2411.19146

[danielkorzekwa]

What does this PR do?

Type of change: ?
new feature

Overview: ?
E2E test for the experimental compress algorithm based on https://arxiv.org/abs/2411.19146

Usage

See tests/gpu/torch/_compress/test_compress.py

# Add a code snippet demonstrating how to use this

See tests/gpu/torch/_compress/test_compress.py

Testing

See tests/gpu/torch/_compress/test_compress.py

Before your PR is "Ready for review"

• Make sure you read and follow Contributor guidelines and your commits are signed.
• Is this change backward compatible?: Yes
• Did you write any new necessary tests?: Yes
• Did you add or update any necessary documentation?: No
• Did you update Changelog?: No

Additional Information

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.40%. Comparing base (230cee1) to head (d942e0a).
⚠️ Report is 2 commits behind head on feature/compress.

Additional details and impacted files

@@                Coverage Diff                @@
##           feature/compress     #464   +/-   ##
=================================================
  Coverage             73.40%   73.40%           
=================================================
  Files                   180      180           
  Lines                 18077    18077           
=================================================
  Hits                  13270    13270           
  Misses                 4807     4807           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kevalmorabia97]

Is the resources/tokenizer used as a toy tokenizer for testing instead of using original llama tokenizer?

We can instead re-use test toy models and tokenizers used in other tests. See comment below in gpu test file

[danielkorzekwa]

created an internal issue to address this in the next MR: issues/12

[kevalmorabia97]

Unrelated to this PR, but do we also plan to simplify the yaml files as part of roadmap? Currently there are too many things to be configured and in too many yaml files, which we can move to one common base yaml hidden from users and only require user to provide 4-5 most important inputs to keep things simpler

[danielkorzekwa]

this is captured in the Nvidia internal roadmap

[kevalmorabia97]

Looks good to merge. Thanks for addressing my comments