Skip to content

Failing streaming endpoints do return trace IDs#45

Merged
bulldozer-bot[bot] merged 3 commits intopalantir:developfrom
buffer51:pmustiere/streaming-trace
Nov 29, 2018
Merged

Failing streaming endpoints do return trace IDs#45
bulldozer-bot[bot] merged 3 commits intopalantir:developfrom
buffer51:pmustiere/streaming-trace

Conversation

@buffer51
Copy link
Copy Markdown
Contributor

Before this PR

A failing streaming endpoint would not return the trace ID as a response header.

After this PR

Currently this PR doesn't fix the issue, it just adds unit tests that show the problem.

@buffer51 buffer51 requested a review from a team as a code owner November 27, 2018 00:54
@buffer51
Copy link
Copy Markdown
Contributor Author

The issue seems to be that the filter is called twice:

  • first for the initial response, where the trace is set, but the response is eventually discarded because of the exception
  • then a second response is created based on the exception alone, which goes to the filter but the trace has been cleared and nothing is set.

@Test
public void testTraceState_withoutRequestHeadersGeneratesValidTraceResponseHeadersWhenFailing() {
Response response = target.path("/failing-trace").request().get();
assertThat(response.getHeaderString(TraceHttpHeaders.TRACE_ID), not(nullValue()));
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

taking a step back, it's not clear to me that the response needs to contain a trace id header at all (even in the success case).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the general case of streaming one? I'm confused, this is how it's been working so far everywhere. If not provided on the request headers, a trace will be created and returned so that you can find out what happens. This makes even more sense IMHO when things go wrong.

For what it's worth, here even forcing the trace with request headers is ignored in the failing streaming case. Doesn't really matter if you force it or let it pick, the exception handling breaks it.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool, that sounds reasonable.

@buffer51
Copy link
Copy Markdown
Contributor Author

I don't know if that best effort else condition I put introduces some bad behaviour in other cases. I suspect it might, otherwise why wouldn't the logic be as simple as that? Perhaps because people don't only use automatic tracing, but the more advanced capabilities.

if (maybeSpan.isPresent()) {
Span span = maybeSpan.get();
headers.putSingle(TraceHttpHeaders.TRACE_ID, span.getTraceId());
} else {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's pretty funky that this filter is called twice in the case of a failing streaming response... indicates to me that we don't fully understand the jersey behavior here.

ignoring our ignorance for a moment, the fix seems ok to me, but i'd like to see a comment indicating why this else block exists and what it does and how.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, will do

}

@Test
public void testTraceState_withoutRequestHeadersGeneratesValidTraceResponseHeadersWhenFailing() {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note that this test succeeded before this fix

}

@Test
public void testTraceState_withoutRequestHeadersGeneratesValidTraceResponseHeadersWhenStreaming() {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note that this test succeeded before this fix

}

@Test
public void testTraceState_withoutRequestHeadersGeneratesValidTraceResponseHeadersWhenFailingToStream() {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test fails before this fix, and succeeds with this fix

Copy link
Copy Markdown

@uschi2000 uschi2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. @iamdanfox ?

Copy link
Copy Markdown
Contributor

@iamdanfox iamdanfox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @buffer51!

@bulldozer-bot bulldozer-bot bot merged commit df887c3 into palantir:develop Nov 29, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants