paged listServiceOffering response returns duplicated records in different pages

[borisstoyanov]

ISSUE TYPE

• Bug Report

COMPONENT NAME

API

CLOUDSTACK VERSION

Main and older

SUMMARY

Create a large number of offerings, in my case about 250. list the offerings and divide the response into 3 pages of 100
While comparing the records from page1 and 2 I was able to find an id on the both pages. Checkout the attached file.

STEPS TO REPRODUCE

See attachment for details

EXPECTED RESULTS

Pages should contain unique ids

ACTUAL RESULTS

pages contained a duplicated id

compare paged api response.txt

[shwstppr]

I am able to reproduce this with a Simulator env for both compute and disk offerings. Not sure if the problem happens with other resources as well.
I created 1000 compute and disk offerings. Tried listing pages for disk offering with pagesize=500. It resulted in many duplicates. Logs for disk offerings:
diskoffering.log

While setting global config default.page.size to 1200 and listing 1000 items at once there were no duplicates,

shwstppr@shwstppr-ThinkPad:~/Desktop/temp|
⇒  cmk list diskofferings filter=id page=1 pagesize=1000 > ./dpage3
shwstppr@shwstppr-ThinkPad:~/Desktop/temp|
⇒  sort ./dpage3 | uniq -d
+--------------------------------------+

@rhtyd @weizhouapache @nvazquez could this be a caching issue with ApiServer?

[nvazquez]

Thanks for testing @shwstppr, I checked the code for listDiskOfferings and the QueryManager is simply obtaining data from the DAO layer using a search criteria and a filter based on the start index and the page size value. Perhaps the filter can be replaced by this method: StringUtils.applyPagination?

[weizhouapache]

@shwstppr @borisstoyanov
is it similar as #3934 which has been fixed by #4494 ?

try ?

UPDATE `cloud`.`disk_offering` JOIN (SELECT COUNT(1) AS count FROM `cloud`.`disk_offering` WHERE `sort_key` != 0) AS tbl_tmp SET `sort_key` = `id` WHERE count = 0;

what should the default value of sort_key be, 0 or id ?

[shwstppr]

@weizhouapache @nvazquez
By default value for sort_key field remains 0. Maybe @borisstoyanov can confirm but this issue is seen with older versions as well 4.9 probably.
I've created a PR to use id field as secondary sort criteria which fixes the issue in my tests.

[weizhouapache]

@weizhouapache @nvazquez
By default value for sort_key field remains 0. Maybe @borisstoyanov can confirm but this issue is seen with older versions as well 4.9 probably.
I've created a PR to use id field as secondary sort criteria which fixes the issue in my tests.

@shwstppr sounds like a better idea.