Pulp

Interesting detail a REST query to a task[0] is fast when the task is running. As soon as it finishes, it becomes slow.

[0]: http --auth admin:admin http://dev.pulp3:8000/pulp/api/v3/tasks/1/

One possible solution is to de-normalize repository versions. They're immutable, so when creating one, you can just take a snapshot of the current counts of content, and possibly the URLs, and store that instead of having fields that calculate it anew every time it's serialized.

There's 2 separate issues here - repository versions are extremely slow to render for large repositories due to the calculations required for each one, and that problem should be resolved by just doing the calculation at repository version creation time and saving them in the database.

The second issue is that this slowness bleeds into rendering tasks due to CreatedResource. Fixing the former will "fix" the latter but it's better that we also remove this dependency.

dalley wrote:

The second issue is that this slowness bleeds into rendering tasks due to CreatedResource.

Can you elaborate on this? I'm assuming that the created-resource for a sync is a RepositoryVersion and when serialized (in the task), is just the URL and not the entire serialized RepositoryVersion, right?

I'm not able to reproduce this. I installed the latest pulpcore, pulpcore-plugin, and pulp_rpm. I then lazy sync epel6 with commands like:

http POST http://localhost:8000/pulp/api/v3/repositories/ name=foo
export REPO_HREF=$(http :8000/pulp/api/v3/repositories/ | jq -r '.results[] | select(.name == "foo") | ._href')
http POST http://localhost:8000/pulp/api/v3/remotes/rpm/rpm/ name='bar' url='https://dl.fedoraproject.org/pub/epel/6Server/x86_64/' policy='on_demand'
export REMOTE_HREF=$(http :8000/pulp/api/v3/remotes/rpm/rpm/ | jq -r '.results[] | select(.name == "bar") | ._href')
http POST :8000${REMOTE_HREF}sync/ repository=$REPO_HREF

It completes successfully and downloads 16673, which should be enough to trigger the bug. However when I list the task it always completes in ~ 0.5 seconds. I can't reproduce the ~ 5 second slowdown.

(pulp) [vagrant@pulp3-source-fedora28 pulp_rpm]$ time http :/pulp/api/v3/tasks/1/
HTTP/1.1 200 OK
Allow: GET, DELETE, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 967
Content-Type: application/json
Date: Fri, 22 Feb 2019 17:12:10 GMT
Server: nginx/1.12.1
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "_created": "2019-02-21T21:42:59.168700Z",
    "_href": "/pulp/api/v3/tasks/1/",
    "created_resources": [
        "/pulp/api/v3/repositories/1/versions/1/"
    ],
    "error": null,
    "finished_at": "2019-02-22T00:48:22.166977Z",
    "job_id": "7103bb64-a426-4710-bd5d-4a65b341c47b",
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
    "non_fatal_errors": [],
    "parent": null,
    "progress_reports": [
        {
            "done": 5,
            "message": "Downloading and Parsing Metadata",
            "state": "completed",
            "suffix": "",
            "task": "/pulp/api/v3/tasks/1/",
            "total": 5
        },
        {
            "done": 0,
            "message": "Downloading Artifacts",
            "state": "completed",
            "suffix": "",
            "task": "/pulp/api/v3/tasks/1/",
            "total": null
        },
        {
            "done": 16673,
            "message": "Associating Content",
            "state": "completed",
            "suffix": "",
            "task": "/pulp/api/v3/tasks/1/",
            "total": 16673
        },
        {
            "done": 0,
            "message": "Un-Associating Content",
            "state": "completed",
            "suffix": "",
            "task": "/pulp/api/v3/tasks/1/",
            "total": null
        }
    ],
    "spawned_tasks": [],
    "started_at": "2019-02-21T21:42:59.266729Z",
    "state": "completed",
    "worker": "/pulp/api/v3/workers/2/"
}

real    0m0.589s
user    0m0.351s
sys     0m0.047s

This is definitely still an issue and I can reproduce it easily. You simply need more tasks/repository versions, or much bigger tasks/repository versions. I expect that moderate production usage will encounter this but doing a simple sync of EPEL in an otherwise empty installation will not.

After 10 syncs totalling about 500,000 content units, viewing a single (1) task takes about 2 seconds, because the associated repository via CreatedResource is so large (and serializing it performs read-intensive ops to do the counts and so forth).

Serializing all 10 tasks takes 14.7 seconds.

Serializing 8 repository versions takes 10.7 seconds.

Serializing 1 repository version takes between 1.1 and 2.5 seconds, depending on which repository version it is.

So this is definitely a problem and fixing it later will require a data migration. Also, we might take this opportunity to also change the representation when serialized to be more in-sync with the new data layout underneath we adopt, which would be a backwards incompatible change.

I see the slowdown now so I can fix this I believe. Thanks dalley for the alternate reproducer!

pulp smash PR is here: https://github.com/PulpQE/pulp-smash/pull/1174