Project

Profile

Help

Issue #3931

closed

Pagination of content units returns less than requested limit

Added by lucarval over 5 years ago. Updated almost 4 years ago.

Status:
CLOSED - WONTFIX
Priority:
High
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Master
Platform Release:
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

For a pulp repository with the following amount of content units:

26 "docker_blob"
16 "docker_image"
39 "docker_manifest"
10 "docker_manifest_list"
52 "docker_tag"

When paginating via all content units in repository, pulp groups content units by their type, which is probably ok.
However, if the requested page size is less than the size of one of the groups, pulp returns partial results.

With the example repository above, notice the skip and limit value and the amount of items that are returned:

  1. POST /pulp/api/v2/repositories/redhat-e2e-container-e2e-container-test-product/search/units/ {"criteria": {"sort": {"unit": "unit_id", "ascending"}, "skip": 0, "type_ids": ["docker_image", "docker_manifest", "docker_blob", "docker_tag", "docker_manifest_list", "iso"], "limit": 25, "filters": {"unit": {"_last_updated": {"$lte": 1535038100.200567}}}}} - Returns 25 items
  2. POST /pulp/api/v2/repositories/redhat-e2e-container-e2e-container-test-product/search/units/ {"criteria": {"sort": {"unit": "unit_id", "ascending"}, "skip": 25, "type_ids": ["docker_image", "docker_manifest", "docker_blob", "docker_tag", "docker_manifest_list", "iso"], "limit": 25, "filters": {"unit": {"_last_updated": {"$lte": 1535038100.200567}}}}} - Returns 1 item
  3. POST /pulp/api/v2/repositories/redhat-e2e-container-e2e-container-test-product/search/units/ {"criteria": {"sort": {"unit": "unit_id", "ascending"}, "skip": 26, "type_ids": ["docker_image", "docker_manifest", "docker_blob", "docker_tag", "docker_manifest_list", "iso"], "limit": 25, "filters": {"unit": {"_last_updated": {"$lte": 1535038100.200567}}}}} - Returns 25 items

If limit is set to, at least, the amount of content units in the largest group (52 in example above), then this behavior doesn't happen.

Why is this important?
I'd like to implement a "safe" pagination process in dockpulp[0]. This would be done by paginating through content units with one item overlap. Then I can check for the expected overlap and have a bit more confidence that no content units were skipped. When pulp returns less items than expected, it's not clear whether there are more items to be fetched or pulp is just done processing a particular content unit type.

[0] https://github.com/release-engineering/dockpulp/

$ rpm -qa '*pulp*'
python-pulp-client-lib-2.15.1-1.el7.noarch
pulp-docker-plugins-3.1.1-2.el7eng.noarch
python-pulp-common-2.15.1-1.el7.noarch
python-pulp-agent-lib-2.15.1-1.el7.noarch
pulp-selinux-2.15.1-1.el7.noarch
python-pulp-repoauth-2.15.1-1.el7.noarch
pulp-admin-client-2.15.1-1.el7.noarch
python-pulp-docker-common-3.1.1-2.el7eng.noarch
pulp-server-2.15.1-1.el7.noarch
python-pulp-bindings-2.15.1-1.el7.noarch
pulp-docker-admin-extensions-3.1.1-2.el7eng.noarch

Also available in: Atom PDF