Reinvent Keynote3 Elastic Training Feature Support

[mollyheamazon]

What's changing and why?

We are adding six new command line arguments related to elastic training for HyperPodTrainingOperator. Elastic training is a method that dynamically scales distributed machine learning operations, which helps solve the resource inefficiency or poor fault tolerance issues being seen often in traditional distributed training jobs.

Before/After UX

Before:

hyp create hyp-pytorch-job
--job-name <job_name>
--image <image_name>
--node-count 1

After:

(Case 1)
hyp create hyp-pytorch-job
--job-name <job_name>
--image <image_name>
--node-count 1
--elastic-replica-increment-step 1
--max-node-count 2
--elastic-graceful-shutdown-timeout-in-seconds 180
--elastic-scaling-timeout-in-seconds 60
--elastic-scale-up-snooze-time-in-seconds 60
(Case 2)
hyp create hyp-pytorch-job
--job-name <job_name>
--image <image_name>
--node-count 1
--max-node-count 8
--elastic-graceful-shutdown-timeout-in-seconds 180
--elastic-scaling-timeout-in-seconds 60
--elastic-scale-up-snooze-time-in-seconds 60
--elastic-replica-discrete-values '[2, 4, 8]'

How was this change tested?

• Manual testing: Testing local changes by updating with pip install . in both template and project root folders. Then run the hyp create hyp-pytorch-job command with newly added command line arguments.
  • Could get the message Successfully submitted HyperPodPytorchJob '<job_name>'!
  • Run the command kubectl get hyperpodpytorchjob <job_name> -o yaml and check the output yaml file
• Run the unit test and integ test

Are unit tests added?

Yes

Are integration tests added?

Yes

Reviewer Guidelines

‼️ Merge Requirements: PRs with failing integration tests cannot be merged without justification.

One of the following must be true:

• All automated PR checks pass
• Failed tests include local run results/screenshots proving they work
• Changes are documentation-only