storage: support resumable uploads

[stephenplusplus]

• Allow a choice in upload and createWriteStream between simple/resumable upload technique
• upload: Stat the incoming file for size, default to simple for < 5mb, resumable for > 5mb
• createWriteStream: default to resumable uploads
• Integrate { resumableThreshold: n } option on storage instantiation (defaults to 5mb)
• test integrity upload stream
• finalize error messaging & objects

Fixes #298

createWriteStream uses the Resumable Upload API: http://goo.gl/jb0e9D.

The process involves these steps:

1. POST the file's metadata. We get a resumable upload URI back, then cache it with ConfigStore.
2. PUT data to that URI with a Content-Range header noting what position the data is beginning from. We also cache, at most, the first 16 bytes of the data being uploaded.
3. Delete the ConfigStore cache after the upload completes.

If the initial upload operation is interrupted, the next time the user uploads the file, these steps occur:

1. Detect the presence of a cached URI in ConfigStore.
2. Make an empty PUT request to that URI to get the last byte written to the remote file.
3. PUT data to the URI starting from the first byte after the last byte returned from the call above.

If the user tries to upload entirely different data to the remote file:

1. -- same as above --
2. -- same as above --
3. -- same as above --
4. Compare the first chunk of the new data with the chunk in cache. If it's different, start a new resumable upload (Step 1 of the first example).

[ryanseys]

Overall this looks good. No big issues. RETRY_LIMIT might want to be increased if we put in exponential backoff as suggested. 5 seems like a more sane default (as suggested here).

[stephenplusplus]

Recent best practices have emerged, so I figured we should put these in place before merging. I've added a task list in the initial post with the intended revisions so far; more will likely be coming.

One of the revisions is allowing a user to specify a preference of a simple or resumable upload. I'm seeking opinions on converting the upload method to a different signature than what we have currently.

Current:

myBucket.upload("./photo.jpg", myFile, { my: "metadata" }, function (err, file) {})

Suggested:

myBucket.upload("./photo.jpg", {
  destination: myFile,
  metadata: {
    my: "metadata"
  },
  resumable: (true || false)
}, function (err, file) {})

file.createWriteStream() will also need this functionality added.

Current:

myFile.createWriteStream({ my: "metadata" })

Suggested:

myFile.createWriteStream({
  resumable: false, // default: true
  metadata: {
    my: "metadata"
  }
})

Any better ideas?

[silvolu]

Looks good, but I'd like the user to be able to change the resumable_threshold (that defaults to 5MB). Could we expose a configuration for storage, or add setters for similar values? In the future we might need it for the chunk size as well, and we could use it to allow the user to change the default for createWriteStream at a global level.

[stephenplusplus]

Config on the storage level makes sense to me.

var gcloud = require("gcloud")({ /* conn info */ })
gcloud.storage({ resumableThreshold: n })

&

var gcloud = require("gcloud")
var storage = gcloud.storage({ /* conn info */, resumableThreshold: n  })

1. What format do we accept for n? (bytes, kb, mb?)
2. Is it ok to rely on our docs to explain resumableThreshold won't have an effect on createWriteStream uploads? I can see that being a point of confusion

[ryanseys]

What format do we accept for n? (bytes, kb, mb?)

Bytes. The header is in bytes, so this seems like a simple choice.

Is it ok to rely on our docs to explain resumableThreshold won't have an effect on createWriteStream uploads?

Wait, why not?

[stephenplusplus]

In a stream, we can't stat a file for its size. It comes to us in small kb chunks, meaning we don't know if it's over a threshold until after we've already formed the request.

I suppose if we wanted to, we could buffer n threshold into memory before beginning the request (which is the time we have to say resumable vs simple), but that seems like a dangerous approach.

& +1 on bytes.

[ryanseys]

Fair enough. Plus you don't really know that the readable stream is a file at all. That being said, should resumable even work with streams unless they explicitly give us the filename to use?

[stephenplusplus]

That's a great question, but I think it's impossible to answer. Still, I anticipate resumable will be a desirable default, and speaking technically, we have a solution for if we resume an upload, but are sent different data than we were originally (we bail and start a new upload).

And in any case, the user knows best what they are doing, so we [will] allow them to be explicit about what type of upload to use at the time of the upload.

[ryanseys]

Can we get access to the readable stream that is piping their data to our writable stream? In theory, if we can, we could try to yank the fd (file descriptor) from it and then we could sneakily stat the file to find out its name and size? This is a total shot in the dark though.

[stephenplusplus]

With a stream, we should only be aware of the data coming in, and not about how/where/etc. It would also be a bit magical if we tried to implement something like that. And usually, whenever there's magic, the solution is to add an option or variation of the method that gives the user explicit control of the outcome. We will have both of those things ({ resumable: false } and bucket.upload)

[ryanseys]

Yeah, that would be too much magic, agreed. Getting back to the original question, I think that it's safe to say that if the developer is uploading using a stream, they know that resumableThreshold won't work. I would only expect it to work if we explicitly give the filename i.e. like in .upload(), so if you can do better than that, that's exceeding expectations in my mind.