Project

Profile

Help

Issue #8568

Issue #8912: [EPIC] Issues with the traditional tasking system

Overall task status remains in 'running' state while all the reports are in the final state

Added by ipanova@redhat.com 5 months ago. Updated 2 months ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 100
Quarter:
Q2-2021

Description

This is happens on master but not 100% reproducible.

After pclean and prestart create a repo and a remote( with corrupted upstream repo) , trigger sync. In my case I had a container repo which was missing files. on 404 pulp gave up on backoff, reports states were flipped to canceled meanwhile overall task state remained running

(pulp) [vagrant@pulp3-source-fedora32 _scripts]$ http  :24817/pulp/api/v3/tasks/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 1102
Content-Type: application/json
Correlation-ID: 83f0319d8b464bf8b150b6ef0c67e5dd
Date: Thu, 15 Apr 2021 15:58:35 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "child_tasks": [],
            "created_resources": [],
            "error": null,
            "finished_at": null,
            "logging_cid": "a47f9ae4316447dab5d287acb64ac652",
            "name": "pulp_container.app.tasks.synchronize.synchronize",
            "parent_task": null,
            "progress_reports": [
                {
                    "code": "sync.downloading.tag_list",
                    "done": 1,
                    "message": "Downloading tag list",
                    "state": "completed",
                    "suffix": null,
                    "total": 1
                },
                {
                    "code": "sync.downloading.artifacts",
                    "done": 0,
                    "message": "Downloading Artifacts",
                    "state": "canceled",
                    "suffix": null,
                    "total": null
                },
                {
                    "code": "associating.content",
                    "done": 0,
                    "message": "Associating Content",
                    "state": "canceled",
                    "suffix": null,
                    "total": null
                }
            ],
            "pulp_created": "2021-04-15T15:50:25.872155Z",
            "pulp_href": "/pulp/api/v3/tasks/3e8611c8-03d7-4bc6-ab4d-704ca903fe55/",
            "reserved_resources_record": [
                "/pulp/api/v3/repositories/container/container/27afa91e-5e99-444f-8ef1-4beac1626eb8/",
                "/pulp/api/v3/remotes/container/container/70bb5002-ad17-4f35-adbf-8518d90fa4ef/"
            ],
            "started_at": "2021-04-15T15:50:25.984691Z",
            "state": "running",
            "task_group": null,
            "worker": "/pulp/api/v3/workers/fe8928f3-589d-43bc-973b-47974661ebbc/"
        }
    ]
}

Apr 15 16:03:50 pulp3-source-fedora32.lan rq[43524]: pulp [46ce1d38b8244cfda5abc50347138e14]: backoff:ERROR: Giving up _run(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('http://pulp>
Apr 15 16:03:50 pulp3-source-fedora32.lan rq[43524]: Giving up _run(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('http://pulp2-nightly-pulp3-source-centos7.lan/pulp/container/ocp3-c>
Apr 15 16:03:50 pulp3-source-fedora32.lan rq[43524]: pulp [46ce1d38b8244cfda5abc50347138e14]: pulp_container.app.tasks.synchronize:INFO: Synchronizing: repository=cnbhm remote=my-hello-repo

Related issues

Related to Pulp - Issue #8637: Possible race condition where task's reservations get deletedCLOSED - CURRENTRELEASE<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

History

#1 Updated by ipanova@redhat.com 5 months ago

  • Subject changed from Overall task status remain in 'running' state while all the reports' are in the final state to Overall task status remains in 'running' state while all the reports are in the final state

#2 Updated by ipanova@redhat.com 5 months ago

also syncing this repo https://fixtures.pulpproject.org/file-invalid/ brings a task in a similar state.

$ http GET :24817/pulp/api/v3/tasks/dd0859e5-ae5c-49de-b9a3-5b17d2c6ec23/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 1149
Content-Type: application/json
Correlation-ID: 7a7f4113004e4987b5a770eb2fdb9d4e
Date: Thu, 15 Apr 2021 17:54:26 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "child_tasks": [],
    "created_resources": [],
    "error": null,
    "finished_at": null,
    "logging_cid": "22eed21a7ebe41e6829e49d12837d21b",
    "name": "pulp_file.app.tasks.synchronizing.synchronize",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "sync.downloading.metadata",
            "done": 1,
            "message": "Downloading Metadata",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "sync.downloading.artifacts",
            "done": 2,
            "message": "Downloading Artifacts",
            "state": "failed",
            "suffix": null,
            "total": null
        },
        {
            "code": "associating.content",
            "done": 0,
            "message": "Associating Content",
            "state": "canceled",
            "suffix": null,
            "total": null
        },
        {
            "code": "sync.parsing.metadata",
            "done": 4,
            "message": "Parsing Metadata Lines",
            "state": "completed",
            "suffix": null,
            "total": 4
        }
    ],
    "pulp_created": "2021-04-15T17:54:14.707879Z",
    "pulp_href": "/pulp/api/v3/tasks/dd0859e5-ae5c-49de-b9a3-5b17d2c6ec23/",
    "reserved_resources_record": [
        "/pulp/api/v3/repositories/file/file/27e5b637-ebed-4312-8b57-7a749f110331/",
        "/pulp/api/v3/remotes/file/file/c001ada1-22ba-45ec-a628-bf5f128411a7/"
    ],
    "started_at": "2021-04-15T17:54:14.892632Z",
    "state": "running",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/d88b3b05-bd2a-4d45-92e4-7dbef012d73a/"
}

#3 Updated by fao89 5 months ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 95
  • Quarter set to Q2-2021

#4 Updated by dalley 5 months ago

  • Related to Issue #8637: Possible race condition where task's reservations get deleted added

#5 Updated by rchan 5 months ago

  • Sprint changed from Sprint 95 to Sprint 96

#6 Updated by rchan 4 months ago

  • Sprint changed from Sprint 96 to Sprint 97

#7 Updated by rchan 4 months ago

  • Sprint changed from Sprint 97 to Sprint 98

#8 Updated by mdellweg 3 months ago

  • Parent task set to #8912

#9 Updated by rchan 3 months ago

  • Sprint changed from Sprint 98 to Sprint 99

#10 Updated by rchan 3 months ago

  • Sprint changed from Sprint 99 to Sprint 100

#11 Updated by bmbouter 2 months ago

  • Status changed from NEW to CLOSED - WONTFIX

Users are recommended to upgrade to the new tasking system as the resolution to this issue. If someone is not able to do that, please comment here with information on why we should reopen the issue.

#12 Updated by mdepaulo@redhat.com about 2 months ago

  • Has duplicate Issue #9180: pulp-installer CI doesn't properly test upgrading RPM packages added

#13 Updated by mdepaulo@redhat.com about 2 months ago

  • Has duplicate deleted (Issue #9180: pulp-installer CI doesn't properly test upgrading RPM packages)

Please register to edit this issue

Also available in: Atom PDF