Issue #3931
closedPagination of content units returns less than requested limit
Description
For a pulp repository with the following amount of content units:
26 "docker_blob"
16 "docker_image"
39 "docker_manifest"
10 "docker_manifest_list"
52 "docker_tag"
When paginating via all content units in repository, pulp groups content units by their type, which is probably ok.
However, if the requested page size is less than the size of one of the groups, pulp returns partial results.
With the example repository above, notice the skip and limit value and the amount of items that are returned:
- POST /pulp/api/v2/repositories/redhat-e2e-container-e2e-container-test-product/search/units/ {"criteria": {"sort": {"unit": "unit_id", "ascending"}, "skip": 0, "type_ids": ["docker_image", "docker_manifest", "docker_blob", "docker_tag", "docker_manifest_list", "iso"], "limit": 25, "filters": {"unit": {"_last_updated": {"$lte": 1535038100.200567}}}}} - Returns 25 items
- POST /pulp/api/v2/repositories/redhat-e2e-container-e2e-container-test-product/search/units/ {"criteria": {"sort": {"unit": "unit_id", "ascending"}, "skip": 25, "type_ids": ["docker_image", "docker_manifest", "docker_blob", "docker_tag", "docker_manifest_list", "iso"], "limit": 25, "filters": {"unit": {"_last_updated": {"$lte": 1535038100.200567}}}}} - Returns 1 item
- POST /pulp/api/v2/repositories/redhat-e2e-container-e2e-container-test-product/search/units/ {"criteria": {"sort": {"unit": "unit_id", "ascending"}, "skip": 26, "type_ids": ["docker_image", "docker_manifest", "docker_blob", "docker_tag", "docker_manifest_list", "iso"], "limit": 25, "filters": {"unit": {"_last_updated": {"$lte": 1535038100.200567}}}}} - Returns 25 items
If limit is set to, at least, the amount of content units in the largest group (52 in example above), then this behavior doesn't happen.
Why is this important?
I'd like to implement a "safe" pagination process in dockpulp[0]. This would be done by paginating through content units with one item overlap. Then I can check for the expected overlap and have a bit more confidence that no content units were skipped. When pulp returns less items than expected, it's not clear whether there are more items to be fetched or pulp is just done processing a particular content unit type.
[0] https://github.com/release-engineering/dockpulp/
$ rpm -qa '*pulp*'
python-pulp-client-lib-2.15.1-1.el7.noarch
pulp-docker-plugins-3.1.1-2.el7eng.noarch
python-pulp-common-2.15.1-1.el7.noarch
python-pulp-agent-lib-2.15.1-1.el7.noarch
pulp-selinux-2.15.1-1.el7.noarch
python-pulp-repoauth-2.15.1-1.el7.noarch
pulp-admin-client-2.15.1-1.el7.noarch
python-pulp-docker-common-3.1.1-2.el7eng.noarch
pulp-server-2.15.1-1.el7.noarch
python-pulp-bindings-2.15.1-1.el7.noarch
pulp-docker-admin-extensions-3.1.1-2.el7eng.noarch