feature: Add mpi support for mxnet estimator api & remove local_mode test for horovod [tf+mx] by ChaiBapchya · Pull Request #1581 · aws/sagemaker-python-sdk

ChaiBapchya · 2020-06-12T00:22:37Z

Issue #, if available: None

Description of changes:
Mxnet supports horovod for distributed training.
However, sagemaker-mxnet python sdk doesn't expose the horovod support.
This PR includes changes to allow end-users to invoke horovod [via mpi] as a key:value parameter in "distributions" dictionary

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
I have passed the region in to any/all clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2020-06-12T00:27:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 5650ab5
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-12T00:51:00Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 5650ab5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-12T01:05:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 5650ab5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-12T02:15:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 1247c01
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-12T02:37:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 1247c01
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-12T02:48:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 84102c9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-12T02:52:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 1247c01
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-12T03:10:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 84102c9
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-12T03:14:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 84102c9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-12T03:29:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 84102c9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…rparam

…pu, remove script_mode since mx/pt doesnt hv it

sagemaker-bot · 2020-06-12T18:21:15Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 3c05a2b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-12T18:47:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 3c05a2b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-12T18:59:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 3c05a2b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-12T19:19:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 3c05a2b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-12T19:37:37Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: aa9d5cd
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-12T19:38:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: aa9d5cd
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-12T20:02:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: aa9d5cd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-06-12T20:53:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: aa9d5cd
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-21T21:58:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 9fc4128
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-21T21:59:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 9fc4128
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-21T22:20:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 9fc4128
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-22T18:12:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 3d50c46
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

src/sagemaker/fw_utils.py

src/sagemaker/mxnet/estimator.py

tests/integ/test_horovod_mx.py

laurenyu · 2020-07-22T18:21:25Z

tests/integ/test_horovod_mx.py

+
+@pytest.mark.local_mode
+@pytest.mark.parametrize("instances, processes", [[1, 2], (2, 1), (2, 2)])
+def test_horovod_local_mode(sagemaker_local_session, instances, processes, tmpdir):


other than reading the output file, it doesn't seem like this test is providing much value for testing Python SDK functionality. would it be possible to instead put the assert at the end of the CPU test above?

I'm following the same approach as horovod test for tensorflow in tests/integ/test_horovod.py
So for consistency purposes I'm sticking to a similar approach.

I could see both the TF and this one being removed. testing for the processes being set up correctly, etc. should live in the framework toolkit repositories, and here we just need to make sure that the hyperparameters get passed along correctly.

sagemaker-bot · 2020-07-22T18:24:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 13b09a3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-22T18:28:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 6470c09
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-22T18:45:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 13b09a3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-22T18:46:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 6470c09
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…ral and remove unnecessary condition

sagemaker-bot · 2020-07-22T19:18:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 6470c09
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-22T19:25:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: d008108
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-22T19:42:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: d008108
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-22T20:14:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: d008108
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-22T20:16:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: d008108
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-23T02:25:13Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 8845d48
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-23T02:42:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 8845d48
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-23T03:11:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 8845d48
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-23T03:51:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 8845d48
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-23T15:12:48Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 484a12f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-23T15:30:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 484a12f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-23T15:37:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 484a12f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2020-07-23T16:00:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 484a12f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

add mpi support for mxnet estimator api

5650ab5

update function definition

1247c01

add hvd mnist mxnet test scripts

84102c9

ChaiBapchya added 2 commits June 12, 2020 18:12

replace self.dist with dist and additional_hyperparam with self._hype…

751327e

…rparam

no ps warning if mpi, rename job name, train_instance_type to local_g…

3c05a2b

…pu, remove script_mode since mx/pt doesnt hv it

add non-local test for sm-hvd-mx

aa9d5cd

This was referenced Jun 13, 2020

[mxnet] [build, test] [sagemaker] update dockerfile with MPI/OpenSSH & add horovod test aws/deep-learning-containers#273

Merged

Add mpi support for distributed training in MXNet aws/sagemaker-mxnet-training-toolkit#178

Merged

remove imagename var

9fc4128

laurenyu changed the title ~~[feature] Add mpi support for mxnet estimator api~~ feature: Add mpi support for mxnet estimator api Jul 22, 2020

chuyang-deng previously approved these changes Jul 22, 2020

View reviewed changes

Merge branch 'master' into mx_estimator_horovod_mpi

6470c09

laurenyu reviewed Jul 22, 2020

View reviewed changes

update import grouping acc to Pep8, fix restructured text inline lite…

d008108

…ral and remove unnecessary condition

ChaiBapchya dismissed chuyang-deng’s stale review via d008108 July 22, 2020 19:15

remove local mode test from tf & mx horovod

8845d48

laurenyu approved these changes Jul 23, 2020

View reviewed changes

Merge branch 'master' into mx_estimator_horovod_mpi

484a12f

laurenyu merged commit 78f0b8b into aws:master Jul 23, 2020

ChaiBapchya deleted the mx_estimator_horovod_mpi branch July 23, 2020 17:10

ChaiBapchya changed the title ~~feature: Add mpi support for mxnet estimator api~~ feature: Add mpi support for mxnet estimator api & remove local_mode test for horovod [tf+mx] Jul 23, 2020

Conversation

ChaiBapchya commented Jun 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jun 12, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 21, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 21, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 21, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Jul 22, 2020

AWS CodeBuild CI Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

laurenyu Jul 22, 2020

Choose a reason for hiding this comment

ChaiBapchya commented Jun 12, 2020 •

edited

Loading