Skip to content

feat: add arniqa#183

Merged
begumcig merged 3 commits intomainfrom
feat/arniqa-metric
Jul 16, 2025
Merged

feat: add arniqa#183
begumcig merged 3 commits intomainfrom
feat/arniqa-metric

Conversation

@begumcig
Copy link
Copy Markdown
Member

@begumcig begumcig commented Jun 6, 2025

Description

This PR adds support for the arniqa image quality metric. Since ARNIQA uses an internal dataset, it now runs with the new call_type="y" (output-only).

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Comment thread src/pruna/evaluation/metrics/metric_torch.py Outdated
Copy link
Copy Markdown
Member

@davidberenstein1957 davidberenstein1957 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, looks good. Should we update the documentation somewhere?

Comment thread tests/evaluation/test_torch_metrics.py Outdated
Comment thread docs/user_manual/customize_metric.rst Outdated
Copy link
Copy Markdown
Member

@davidberenstein1957 davidberenstein1957 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LFTM

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jul 9, 2025

This PR has been inactive for 10 days and is now marked as stale.

@github-actions github-actions bot added the stale label Jul 9, 2025
@begumcig begumcig force-pushed the feat/arniqa-metric branch from 63d675a to bb440ac Compare July 14, 2025 09:30
@begumcig begumcig requested a review from johannaSommer July 14, 2025 09:34
@github-actions github-actions bot removed the stale label Jul 15, 2025
@begumcig begumcig requested review from sharpenb and removed request for nifleisch July 16, 2025 11:23
Copy link
Copy Markdown
Member

@sharpenb sharpenb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That looks very nice! If you can just fix the two comments before merging that would be perfect :)

Comment thread docs/user_manual/evaluate.rst Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we say "Stateful metrics can generally be evaluated in single-model and pariwise modes. Single-mode allows to compare a mode against ground-truth data, while pairwise mode allows to compare the fidelity of model against another model. "

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My suggestion would be to return an error here instead of silently fixing the problem :)

Copy link
Copy Markdown
Member Author

@begumcig begumcig Jul 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes so much sense! I also moved throwing this error to the general utils file where i check call types for all metrics, so we would throw this error not just for tochmetrics but for all of our metrics!

@begumcig begumcig force-pushed the feat/arniqa-metric branch 2 times, most recently from de73860 to a29d9c4 Compare July 16, 2025 17:22
@begumcig begumcig force-pushed the feat/arniqa-metric branch from a29d9c4 to d8aa4f4 Compare July 16, 2025 17:23
@begumcig begumcig removed the request for review from johannaSommer July 16, 2025 17:24
@begumcig begumcig merged commit 8c17a58 into main Jul 16, 2025
6 checks passed
@begumcig begumcig deleted the feat/arniqa-metric branch July 16, 2025 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants