Skip to content

datastore: re-use query object for continuation queries#1635

Merged
callmehiphop merged 1 commit intogoogleapis:masterfrom
stephenplusplus:spp--1634
Sep 28, 2016
Merged

datastore: re-use query object for continuation queries#1635
callmehiphop merged 1 commit intogoogleapis:masterfrom
stephenplusplus:spp--1634

Conversation

@stephenplusplus
Copy link
Contributor

@stephenplusplus stephenplusplus commented Sep 27, 2016

Fixes #1634

We've been calculating continuation query properties, like limit, from the original query the user provided. We should be calculating those properties from the last query that ran.

Example scenario: Datastore seems to return results in chunks of 300. So, with a dataset with 1,000 entities, imagine a query for just 600 of them:

var query = datastore.createQuery('table').limit(600);

How our code works behind the scenes:

  1. Run query
  2. Get Datastore response with 300 results and a NOT_FINISHED message
  3. Because of the NOT_FINISHED, we build a new query that sets the new limit to 300:
    original_limit (600) - results_returned (300)
  4. Get Datastore response with 300 results and a MORE_RESULTS_AFTER_LIMIT message
  5. We stop here because we let the user decide if they want to run further queries for the rest of the results after it reached their limit

Now imagine a query for 601:

var query = datastore.createQuery('table').limit(601);

How our code works behind the scenes:

  1. Run query
  2. Get Datastore response with 300 results and a NOT_FINISHED message
  3. Because of the NOT_FINISHED, we build a new query that sets the new limit to 301:
    original_limit (601) - results_returned (300)
  4. Get Datastore response with 300 results and a NOT_FINISHED message
  5. Because of the NOT_FINISHED, we build a new query that sets the new limit to 301:
    original_limit (601) - results_returned (300)
  6. Get Datastore response with 300 results and a NOT_FINISHED message
  7. Because of the NOT_FINISHED, we build a new query that sets the new limit to 301:
    original_limit (601) - results_returned (300)
  8. Get Datastore response with 100 results and a NO_MORE_RESULTS message
  9. We stop here and give all of the results we collected back to the user

The problem above is the formula, original_limit - results_returned. It should have been last_query_limit - results_returned. So now, we clone the query up front, and modify it as we continue to make queries, so that these calculations are accurate.

@stephenplusplus stephenplusplus added the api: datastore Issues related to the Datastore API. label Sep 27, 2016
@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Sep 27, 2016
@coveralls
Copy link

Coverage Status

Changes Unknown when pulling e3e053c on stephenplusplus:spp--1634 into * on GoogleCloudPlatform:master*.

@coveralls
Copy link

Coverage Status

Changes Unknown when pulling e3e053c on stephenplusplus:spp--1634 into * on GoogleCloudPlatform:master*.

@coveralls
Copy link

Coverage Status

Changes Unknown when pulling e3e053c on stephenplusplus:spp--1634 into * on GoogleCloudPlatform:master*.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: datastore Issues related to the Datastore API. cla: yes This human has signed the Contributor License Agreement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants