Update benchmarking configs #650
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #650 +/- ##
=======================================
Coverage 97.52% 97.52%
=======================================
Files 86 86
Lines 8373 8373
=======================================
Hits 8166 8166
Misses 207 207 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
Collaborator
|
Good catch with the ("common", "env_name") -> ("environment", "gym_id") transition! I missed that during refactoring. Also seams sensible to download demonstration instead of computing from scratch. LGTM from my side. This is mostly configuration file changes. |
hacobe
added a commit
that referenced
this pull request
Jan 6, 2023
This commit patches in the changes from PR #650. It also fixes an error in the dagger configs by replacing "py/object" with "py/type". "py/object" failed the assert in sacred.config.utils.assert_is_valid_key. Finally, it adds a minimal test suite for the benchmarking configs. This commit makes progress on testing, but still does not test that all the benchmarking configs work. Running all the configs with the fast configs applied is too slow. Calling sacred's print_config command for all the configs is also too slow.
hacobe
added a commit
that referenced
this pull request
Jan 6, 2023
This commit patches in the changes from PR #650. It also fixes an error in the dagger configs by replacing "py/object" with "py/type". "py/object" failed the assert in sacred.config.utils.assert_is_valid_key. Finally, it adds a minimal test suite for the benchmarking configs. This commit makes progress on testing, but still does not test that all the benchmarking configs work. Running all the configs with the fast configs applied is too slow. Calling sacred's print_config command for all the configs is also too slow.
AdamGleave
approved these changes
Jan 6, 2023
AdamGleave
added a commit
that referenced
this pull request
Jan 8, 2023
#653) * Fix benchmarking configs + Add a minimal test suite for those configs. This commit patches in the changes from PR #650. It also fixes an error in the dagger configs by replacing "py/object" with "py/type". "py/object" failed the assert in sacred.config.utils.assert_is_valid_key. Finally, it adds a minimal test suite for the benchmarking configs. This commit makes progress on testing, but still does not test that all the benchmarking configs work. Running all the configs with the fast configs applied is too slow. Calling sacred's print_config command for all the configs is also too slow. * Update the tests for benchmarking configs so that they do not require MuJoCo. * Remove unused global in test_benchmarking.py * Test all the benchmarking configs using print_config. Avoid the overhead of subprocess by running the print_config using the experiment's run method. Also, use a path relative to the script for BENCHMARKING_DIR instead of creating a path and then checking if it exists. * Minor fixes Co-authored-by: Adam Gleave <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Added
demonstrations.rollout_typein all the configs to download rollouts from the huggingface hub. Also, updated config key names changed in #604.Testing
Tested the configs locally using:
python -m imitation.scripts.<train_script> <algo> with benchmarking/<config_name>.json