TEZ-4605: Enable hadoop CallerContext in TezChild and DAGAppMaster#394
Merged
ayushtkn merged 6 commits intoapache:masterfrom Feb 21, 2025
Merged
TEZ-4605: Enable hadoop CallerContext in TezChild and DAGAppMaster#394ayushtkn merged 6 commits intoapache:masterfrom
ayushtkn merged 6 commits intoapache:masterfrom
Conversation
Contributor
Author
|
@tez-yetus @abstractdog |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
ayushtkn
reviewed
Feb 20, 2025
Comment on lines
+2393
to
+2395
| org.apache.hadoop.ipc.CallerContext.setCurrent(new org.apache.hadoop.ipc.CallerContext | ||
| .Builder("tez_appmaster_" + containerId.getApplicationAttemptId() | ||
| ).build()); |
Member
There was a problem hiding this comment.
This will overwrite any existing CallerContext, You need to append the values not overwrite
Contributor
Author
There was a problem hiding this comment.
Thanks for you review,@ayushtkn
I just fixed this line, is it OK now?
Contributor
Author
There was a problem hiding this comment.
This will overwrite any existing
CallerContext, You need to append the values not overwrite
But in my opinion, this org.apache.hadoop.ipc.CallerContextis different from org.apache.tez.client.CallerContext, in this main function, it is the first time I set a new value to org.apache.hadoop.ipc.CallerContext, so there will be no existing CallerContext.
is it correct?
Member
There was a problem hiding this comment.
if any code or application which invokes the DagAppMaster sets it then?
Contributor
Author
There was a problem hiding this comment.
org.apache.hadoop.ipc.CallerContext is a threadLocal variable, if other application invokes the DagAppMaster, it should start a new thread, and create a new callercontext variable.
Member
There was a problem hiding this comment.
Ahh, then we can live without it.
is there a possibility to extend some tests?
Contributor
Author
There was a problem hiding this comment.
I tried to implement some unit tests, but the ThreadLocal variable could not be accessed. In Hadoop, this ThreadLocal variable is tested by checking the audit log, but I am unable to start a Hadoop Mini Cluster and correctly invoke DAGAppMaster, so I cannot implement this unit test.
However, the code implementation is simple, and there is a similar implementation in this Hadoop pr which is implemented by my coworker, so it should definitely work.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
abstractdog
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As described in TEZ-4605
What did I change
I modified
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.javaandtez-runtime-internals/src/main/java/org/apache/tez/runtime/task/TezChild.javaWhy did I change
I want to enable org.apache.hadoop.ipc.CallerContext when running TEZ on yarn, so that we can trace how the application interacted with hdfs files.
How did I test
I modified the code and build the whole project, then I test on my own hadoop cluster, we can see the callcontext information in hdfs-audit.log.
here is a screenshot of it:
