Project

Profile

Help

Issue #8568

closed

Issue #8912: [EPIC] Issues with the traditional tasking system

Overall task status remains in 'running' state while all the reports are in the final state

Added by ipanova@redhat.com over 1 year ago. Updated over 1 year ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 100
Quarter:
Q2-2021

Description

This is happens on master but not 100% reproducible.

After pclean and prestart create a repo and a remote( with corrupted upstream repo) , trigger sync. In my case I had a container repo which was missing files. on 404 pulp gave up on backoff, reports states were flipped to canceled meanwhile overall task state remained running

(pulp) [vagrant@pulp3-source-fedora32 _scripts]$ http  :24817/pulp/api/v3/tasks/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 1102
Content-Type: application/json
Correlation-ID: 83f0319d8b464bf8b150b6ef0c67e5dd
Date: Thu, 15 Apr 2021 15:58:35 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "child_tasks": [],
            "created_resources": [],
            "error": null,
            "finished_at": null,
            "logging_cid": "a47f9ae4316447dab5d287acb64ac652",
            "name": "pulp_container.app.tasks.synchronize.synchronize",
            "parent_task": null,
            "progress_reports": [
                {
                    "code": "sync.downloading.tag_list",
                    "done": 1,
                    "message": "Downloading tag list",
                    "state": "completed",
                    "suffix": null,
                    "total": 1
                },
                {
                    "code": "sync.downloading.artifacts",
                    "done": 0,
                    "message": "Downloading Artifacts",
                    "state": "canceled",
                    "suffix": null,
                    "total": null
                },
                {
                    "code": "associating.content",
                    "done": 0,
                    "message": "Associating Content",
                    "state": "canceled",
                    "suffix": null,
                    "total": null
                }
            ],
            "pulp_created": "2021-04-15T15:50:25.872155Z",
            "pulp_href": "/pulp/api/v3/tasks/3e8611c8-03d7-4bc6-ab4d-704ca903fe55/",
            "reserved_resources_record": [
                "/pulp/api/v3/repositories/container/container/27afa91e-5e99-444f-8ef1-4beac1626eb8/",
                "/pulp/api/v3/remotes/container/container/70bb5002-ad17-4f35-adbf-8518d90fa4ef/"
            ],
            "started_at": "2021-04-15T15:50:25.984691Z",
            "state": "running",
            "task_group": null,
            "worker": "/pulp/api/v3/workers/fe8928f3-589d-43bc-973b-47974661ebbc/"
        }
    ]
}

Apr 15 16:03:50 pulp3-source-fedora32.lan rq[43524]: pulp [46ce1d38b8244cfda5abc50347138e14]: backoff:ERROR: Giving up _run(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('http://pulp>
Apr 15 16:03:50 pulp3-source-fedora32.lan rq[43524]: Giving up _run(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('http://pulp2-nightly-pulp3-source-centos7.lan/pulp/container/ocp3-c>
Apr 15 16:03:50 pulp3-source-fedora32.lan rq[43524]: pulp [46ce1d38b8244cfda5abc50347138e14]: pulp_container.app.tasks.synchronize:INFO: Synchronizing: repository=cnbhm remote=my-hello-repo

Related issues

Related to Pulp - Issue #8637: Possible race condition where task's reservations get deletedCLOSED - CURRENTRELEASEdaviddavisActions
Actions #1

Updated by ipanova@redhat.com over 1 year ago

  • Subject changed from Overall task status remain in 'running' state while all the reports' are in the final state to Overall task status remains in 'running' state while all the reports are in the final state
Actions #2

Updated by ipanova@redhat.com over 1 year ago

also syncing this repo https://fixtures.pulpproject.org/file-invalid/ brings a task in a similar state.

$ http GET :24817/pulp/api/v3/tasks/dd0859e5-ae5c-49de-b9a3-5b17d2c6ec23/
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 1149
Content-Type: application/json
Correlation-ID: 7a7f4113004e4987b5a770eb2fdb9d4e
Date: Thu, 15 Apr 2021 17:54:26 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "child_tasks": [],
    "created_resources": [],
    "error": null,
    "finished_at": null,
    "logging_cid": "22eed21a7ebe41e6829e49d12837d21b",
    "name": "pulp_file.app.tasks.synchronizing.synchronize",
    "parent_task": null,
    "progress_reports": [
        {
            "code": "sync.downloading.metadata",
            "done": 1,
            "message": "Downloading Metadata",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "sync.downloading.artifacts",
            "done": 2,
            "message": "Downloading Artifacts",
            "state": "failed",
            "suffix": null,
            "total": null
        },
        {
            "code": "associating.content",
            "done": 0,
            "message": "Associating Content",
            "state": "canceled",
            "suffix": null,
            "total": null
        },
        {
            "code": "sync.parsing.metadata",
            "done": 4,
            "message": "Parsing Metadata Lines",
            "state": "completed",
            "suffix": null,
            "total": 4
        }
    ],
    "pulp_created": "2021-04-15T17:54:14.707879Z",
    "pulp_href": "/pulp/api/v3/tasks/dd0859e5-ae5c-49de-b9a3-5b17d2c6ec23/",
    "reserved_resources_record": [
        "/pulp/api/v3/repositories/file/file/27e5b637-ebed-4312-8b57-7a749f110331/",
        "/pulp/api/v3/remotes/file/file/c001ada1-22ba-45ec-a628-bf5f128411a7/"
    ],
    "started_at": "2021-04-15T17:54:14.892632Z",
    "state": "running",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/d88b3b05-bd2a-4d45-92e4-7dbef012d73a/"
}

Actions #3

Updated by fao89 over 1 year ago

  • Triaged changed from No to Yes
  • Sprint set to Sprint 95
  • Quarter set to Q2-2021
Actions #4

Updated by dalley over 1 year ago

  • Related to Issue #8637: Possible race condition where task's reservations get deleted added
Actions #5

Updated by rchan over 1 year ago

  • Sprint changed from Sprint 95 to Sprint 96
Actions #6

Updated by rchan over 1 year ago

  • Sprint changed from Sprint 96 to Sprint 97
Actions #7

Updated by rchan over 1 year ago

  • Sprint changed from Sprint 97 to Sprint 98
Actions #8

Updated by mdellweg over 1 year ago

  • Parent task set to #8912
Actions #9

Updated by rchan over 1 year ago

  • Sprint changed from Sprint 98 to Sprint 99
Actions #10

Updated by rchan over 1 year ago

  • Sprint changed from Sprint 99 to Sprint 100
Actions #11

Updated by bmbouter over 1 year ago

  • Status changed from NEW to CLOSED - WONTFIX

Users are recommended to upgrade to the new tasking system as the resolution to this issue. If someone is not able to do that, please comment here with information on why we should reopen the issue.

Actions #12

Updated by mdepaulo@redhat.com over 1 year ago

  • Has duplicate Issue #9180: pulp-installer CI doesn't properly test upgrading RPM packages added
Actions #13

Updated by mdepaulo@redhat.com over 1 year ago

  • Has duplicate deleted (Issue #9180: pulp-installer CI doesn't properly test upgrading RPM packages)

Also available in: Atom PDF