Allow for request bodies larger than 32768 bytes

[evgenii-petrov-arrival]

requestBody.Read truncates the body to the length of the byte slice that is passed to it, since it returns io.EOF immediately. This can be observed by running ioutil.ReadAll(req.Body), which will return a byte slice of size bytes.MinRead. Apparently default http transport uses a byte slice of size 32768 (or 32767, I'm not sure and couldn't find the code), which results in request body being truncated to 32768 bytes.

As far as I understand, changes proposed in this pull request make requests with large bodies not safe to use in multiple goroutines concurrently. One solution can be to clone the request before sending it to the http client, consume the body into a buffer, and then copy the body from the buffer into original request and new request. This sounds a bit convoluted, so I'll be happy to hear suggestions on how this can be done better.

[manugarg]

@evgenii-petrov-arrival thanks for identifying this issue and the pull request.

Since running requests in parallel is a pretty core feature of HTTP probes, I think we cannot have a shared request body with state.

I think the only option we have is to clone the HTTP request object right before executing it:

cloudprober/probes/http/http.go

Line 231 in 5715f21

We can do something like:

Let's say we store the user provided body bytes in p.reqBody

r := req.Clone()
r.Body = ioutil.NopCloser(bytes.NewReader(p.reqBody))
req = r
....
p.client.Do(req)

(For efficiency we can do this only if len(p.reqBody) > largeBodyThreshold.)

Multiple bytes.Reader can operate on the same underlying bytes storage.

Does that make sense?

[evgenii-petrov-arrival]

Hi @manugarg , thank you for the suggestion!

I've git reverted the original commit, and added a new one that follows your suggestion, please take a look.

[manugarg]

Sorry for the late review. Please see my comments above. I don't think we need to build the request -> body-bytes map. We can just store bytes once for the probe and create a new reader on top of those bytes.

[evgenii-petrov-arrival]

Sorry for the late review

No worries at all!

I've made the changes as per the suggestion.

[manugarg]

Thanks a lot for your patience! There is one comment on the context usage, but looks good otherwise. I'll start the process of importing this change, following the process described at:
https://github.com/google/cloudprober/blob/master/CONTRIBUTING.md#source-of-truth-sot-and-commit-process

[manugarg]

This change has been merged with #448.

Commit: 07644a2.

I'll close this PR now. Thank you once again @evgenii-petrov-arrival for the contribution.