Project

Profile

Help

Issue #8568

Overall task status remains in 'running' state while all the reports are in the final state

Added by ipanova@redhat.com 27 days ago. Updated 12 days ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 96
Quarter:
Q2-2021

Description

This is happens on master but not 100% reproducible.

After pclean and prestart create a repo and a remote( with corrupted upstream repo) , trigger sync. In my case I had a container repo which was missing files. on 404 pulp gave up on backoff, reports states were flipped to canceled meanwhile overall task state remained running

(pulp) [vagrant@pulp3-source-fedora32 _scripts]$ http  :24817/pulp/api/v3/tasks/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 1102
Content-Type: application/json
Correlation-ID: 83f0319d8b464bf8b150b6ef0c67e5dd
Date: Thu, 15 Apr 2021 15:58:35 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "child_tasks": [],
            "created_resources": [],
            "error": null,
            "finished_at": null,
            "logging_cid": "a47f9ae4316447dab5d287acb64ac652",
            "name": "pulp_container.app.tasks.synchronize.synchronize",
            "parent_task": null,
            "progress_reports": [
                {
                    "code": "sync.downloading.tag_list",
                    "done": 1,
                    "message": "Downloading tag list",
                    "state": "completed",
                    "suffix": null,
                    "total": 1
                },
                {
                    "code": "sync.downloading.artifacts",
                    "done": 0,
                    "message": "Downloading Artifacts",
                    "state": "canceled",
                    "suffix": null,
                    "total": null
                },
                {
                    "code": "associating.content",
                    "done": 0,
                    "message": "Associating Content",
                    "state": "canceled",
                    "suffix": null,
                    "total": null
                }
            ],
            "pulp_created": "2021-04-15T15:50:25.872155Z",
            "pulp_href": "/pulp/api/v3/tasks/3e8611c8-03d7-4bc6-ab4d-704ca903fe55/",
            "reserved_resources_record": [
                "/pulp/api/v3/repositories/container/container/27afa91e-5e99-444f-8ef1-4beac1626eb8/",
                "/pulp/api/v3/remotes/container/container/70bb5002-ad17-4f35-adbf-8518d90fa4ef/"
            ],
            "started_at": "2021-04-15T15:50:25.984691Z",
            "state": "running",
            "task_group": null,
            "worker": "/pulp/api/v3/workers/fe8928f3-589d-43bc-973b-47974661ebbc/"
        }
    ]
}

Apr 15 16:03:50 pulp3-source-fedora32.lan rq[43524]: pulp [46ce1d38b8244cfda5abc50347138e14]: backoff:ERROR: Giving up _run(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('http://pulp>
Apr 15 16:03:50 pulp3-source-fedora32.lan rq[43524]: Giving up _run(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('http://pulp2-nightly-pulp3-source-centos7.lan/pulp/container/ocp3-c>
Apr 15 16:03:50 pulp3-source-fedora32.lan rq[43524]: pulp [46ce1d38b8244cfda5abc50347138e14]: pulp_container.app.tasks.synchronize:INFO: Synchronizing: repository=cnbhm remote=my-hello-repo

Related issues

Related to Pulp - Issue #8637: Possible race condition where task's reservations get deletedNEW<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

History

#1 Updated by ipanova@redhat.com 27 days ago

  • Subject changed from Overall task status remain in 'running' state while all the reports' are in the final state to Overall task status remains in 'running' state while all the reports are in the final state

#2 Updated by ipanova@redhat.com 27 days ago

also syncing this repo https://fixtures.pulpproject.org/file-invalid/ brings a task in a similar state.

$ http GET :24817/pulp/api/v3/tasks/dd0859e5-ae5c-49de-b9a3-5b17d2c6ec23/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 1149
Content-Type: application/json
Correlation-ID: 7a7f4113004e4987b5a770eb2fdb9d4e
Date: Thu, 15 Apr 2021 17:54:26 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "child_tasks": [],
    "created_resources": [],
    "error": null,
    "finished_at": null,
    "logging_cid": "22eed21a7ebe41e6829e49d12837d21b",
    "name": "pulp_file.app.tasks.synchronizing.synchronize",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "sync.downloading.metadata",
            "done": 1,
            "message": "Downloading Metadata",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "sync.downloading.artifacts",
            "done": 2,
            "message": "Downloading Artifacts",
            "state": "failed",
            "suffix": null,
            "total": null
        },
        {
            "code": "associating.content",
            "done": 0,
            "message": "Associating Content",
            "state": "canceled",
            "suffix": null,
            "total": null
        },
        {
            "code": "sync.parsing.metadata",
            "done": 4,
            "message": "Parsing Metadata Lines",
            "state": "completed",
            "suffix": null,
            "total": 4
        }
    ],
    "pulp_created": "2021-04-15T17:54:14.707879Z",
    "pulp_href": "/pulp/api/v3/tasks/dd0859e5-ae5c-49de-b9a3-5b17d2c6ec23/",
    "reserved_resources_record": [
        "/pulp/api/v3/repositories/file/file/27e5b637-ebed-4312-8b57-7a749f110331/",
        "/pulp/api/v3/remotes/file/file/c001ada1-22ba-45ec-a628-bf5f128411a7/"
    ],
    "started_at": "2021-04-15T17:54:14.892632Z",
    "state": "running",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/d88b3b05-bd2a-4d45-92e4-7dbef012d73a/"
}

#3 Updated by fao89 26 days ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 95
  • Quarter set to Q2-2021

#4 Updated by dalley 15 days ago

  • Related to Issue #8637: Possible race condition where task's reservations get deleted added

#5 Updated by rchan 12 days ago

  • Sprint changed from Sprint 95 to Sprint 96

Please register to edit this issue

Also available in: Atom PDF