Task #5712: In case of exception in finalize_new_version, no repo version is saved for the subsequent reruns of the same sync - Pulp

Actions

Send by e-mail Copy link

Task #5712

closed

In case of exception in finalize_new_version, no repo version is saved for the subsequent reruns of the same sync

Added by ttereshc over 4 years ago. Updated over 4 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

Normal

Assignee:

Category:

Sprint/Milestone:

3.0.0

Start date:

Due date:

% Done:

100%

Estimated time:

Platform Release:

Groomed:

Sprint Candidate:

Tags:

Sprint:

Quarter:

Description

If exception is raised in finalize_new_version , a repo version is not marked as complete but it's also not removed from the db. Subsequent syncs compare to this incomplete version and as a consequence in the subsequent attempts to add the same content no version is created.

Steps to reproduce:
- raise any error in finalize_new_version
- create a repo
- create a remote with URL1 and sync from it -> task will fail due to your exception
- remove exception, restart services and try to resync from the same remote -> no repo verison
- create a remote with URL2 and sync from it -> you'll get a repo version number 2

# create a repository and remotes
http POST http://localhost:24817/pulp/api/v3/repositories/rpm/rpm/ name=foo2
export REPO_HREF=$(http :24817/pulp/api/v3/repositories/rpm/rpm/ | jq -r '.results[] | select(.name == "foo2") | pulp_href')

http POST http://localhost:24817/pulp/api/v3/remotes/rpm/rpm/ name='fix'  url='https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-unsigned/' policy='on_demand'
export REMOTE_HREF=$(http :24817/pulp/api/v3/remotes/rpm/rpm/ | jq -r '.results[] | select(.name == "fix") | .pulp_href')

http POST http://localhost:24817/pulp/api/v3/remotes/rpm/rpm/ name='fix2' url='https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-updated-updateinfo/' policy='on_demand'
export REMOTE_HREF2=$(http :24817/pulp/api/v3/remotes/rpm/rpm/ | jq -r '.results[] | select(.name == "fix2") | .pulp_href')

Here is the result from the steps steps described earlier.

(pulp) [vagrant@pulp3-source-fedora30 pulp_rpm]$ http POST :24817${REPO_HREF}sync/ remote=$REMOTE_HREF
HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Thu, 14 Nov 2019 12:46:24 GMT
Server: gunicorn/19.9.0
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "task": "/pulp/api/v3/tasks/395408b7-ff8c-4bae-9591-e0f992477ef2/"
}

(pulp) [vagrant@pulp3-source-fedora30 pulp_rpm]$ http :/pulp/api/v3/tasks/395408b7-ff8c-4bae-9591-e0f992477ef2/
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 2528
Content-Type: application/json
Date: Thu, 14 Nov 2019 12:46:45 GMT
Server: nginx/1.16.1
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "created_resources": [
        null
    ],
    "error": {
        "description": "blah-blah",
        "traceback": "  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/worker.py\", line 822, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/job.py\", line 605, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib64/python3.7/site-packages/rq/job.py\", line 611, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py\", line 133, in synchronize\n    dv.create()\n  File \"/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py\", line 149, in create\n    loop.run_until_complete(pipeline)\n  File \"/home/vagrant/devel/pulpcore/pulpcore/app/models/repository.py\", line 631, in __exit__\n    self.repository.finalize_new_version(self)\n  File \"/home/vagrant/devel/pulp_rpm/pulp_rpm/app/models.py\", line 1164, in finalize_new_version\n    raise Exception(\"blah-blah\")\n"
    },
    "finished_at": "2019-11-14T12:46:28.590618Z",
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
    "parent": null,
    "progress_reports": [
        {
            "code": "parsing.packages",
            "done": 35,
            "message": "Parsed Packages",
            "state": "completed",
            "suffix": null,
            "total": 35
        },
        {
            "code": "parsing.errata",
            "done": 4,
            "message": "Parsed Erratum",
            "state": "completed",
            "suffix": null,
            "total": 4
        },
        {
            "code": "parsing.comps",
            "done": 3,
            "message": "Parsed Comps",
            "state": "completed",
            "suffix": null,
            "total": 3
        },
        {
            "code": "downloading.metadata",
            "done": 5,
            "message": "Downloading Metadata Files",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "downloading.artifacts",
            "done": 0,
            "message": "Downloading Artifacts",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "associating.content",
            "done": 43,
            "message": "Associating Content",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "parsing.modulemds",
            "done": 0,
            "message": "Parse Modulemd",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "parsing.modulemddefaults",
            "done": 0,
            "message": "Parse Modulemd-defaults",
            "state": "completed",
            "suffix": null,
            "total": null
        }
    ],
    "pulp_created": "2019-11-14T12:46:24.348275Z",
    "pulp_href": "/pulp/api/v3/tasks/395408b7-ff8c-4bae-9591-e0f992477ef2/",
    "reserved_resources_record": [
        "/pulp/api/v3/remotes/rpm/rpm/f42ce3df-46e6-40c2-8c52-b5fce1b4c8df/",
        "/pulp/api/v3/repositories/rpm/rpm/9f5d2411-9142-44ce-b23e-cc743ff18139/"
    ],
    "spawned_tasks": [],
    "started_at": "2019-11-14T12:46:24.471893Z",
    "state": "failed",
    "worker": "/pulp/api/v3/workers/b170cd20-9196-4631-845e-f90f002e269c/"
}

# removed the exception
(pulp) [vagrant@pulp3-source-fedora30 pulp_rpm]$ prestart
systemctl restart pulpcore-content pulpcore-worker@1 pulpcore-worker@2 pulpcore-resource-manager pulpcore-api
(pulp) [vagrant@pulp3-source-fedora30 pulp_rpm]$ http POST :24817${REPO_HREF}sync/ remote=$REMOTE_HREF
HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Thu, 14 Nov 2019 12:47:22 GMT
Server: gunicorn/19.9.0
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "task": "/pulp/api/v3/tasks/aa34f6fc-c129-43ca-99a6-256b70e5307a/"
}

# completed but no created resources!
(pulp) [vagrant@pulp3-source-fedora30 pulp_rpm]$ http :/pulp/api/v3/tasks/aa34f6fc-c129-43ca-99a6-256b70e5307a/
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 1540
Content-Type: application/json
Date: Thu, 14 Nov 2019 12:47:31 GMT
Server: nginx/1.16.1
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "created_resources": [],
    "error": null,
    "finished_at": "2019-11-14T12:47:26.441165Z",
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
    "parent": null,
    "progress_reports": [
        {
            "code": "downloading.metadata",
            "done": 5,
            "message": "Downloading Metadata Files",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "parsing.packages",
            "done": 35,
            "message": "Parsed Packages",
            "state": "completed",
            "suffix": null,
            "total": 35
        },
        {
            "code": "parsing.errata",
            "done": 4,
            "message": "Parsed Erratum",
            "state": "completed",
            "suffix": null,
            "total": 4
        },
        {
            "code": "parsing.modulemds",
            "done": 0,
            "message": "Parse Modulemd",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "parsing.modulemddefaults",
            "done": 0,
            "message": "Parse Modulemd-defaults",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "parsing.comps",
            "done": 3,
            "message": "Parsed Comps",
            "state": "completed",
            "suffix": null,
            "total": 3
        },
        {
            "code": "downloading.artifacts",
            "done": 0,
            "message": "Downloading Artifacts",
            "state": "completed",
            "suffix": null,
            "total": null
        },
        {
            "code": "associating.content",
            "done": 0,
            "message": "Associating Content",
            "state": "completed",
            "suffix": null,
            "total": null
        }
    ],
    "pulp_created": "2019-11-14T12:47:22.026779Z",
    "pulp_href": "/pulp/api/v3/tasks/aa34f6fc-c129-43ca-99a6-256b70e5307a/",
    "reserved_resources_record": [
        "/pulp/api/v3/remotes/rpm/rpm/f42ce3df-46e6-40c2-8c52-b5fce1b4c8df/",
        "/pulp/api/v3/repositories/rpm/rpm/9f5d2411-9142-44ce-b23e-cc743ff18139/"
    ],
    "spawned_tasks": [],
    "started_at": "2019-11-14T12:47:22.134781Z",
    "state": "completed",
    "worker": "/pulp/api/v3/workers/28bc601e-d871-4477-81c4-1a4e4d135bee/"
}

(pulp) [vagrant@pulp3-source-fedora30 pulp_rpm]$ http :/pulp/api/v3/repositories/rpm/rpm/9f5d2411-9142-44ce-b23e-cc743ff18139/versions/
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 52
Content-Type: application/json
Date: Thu, 14 Nov 2019 12:49:25 GMT
Server: nginx/1.16.1
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "count": 0,
    "next": null,
    "previous": null,
    "results": []
}

# sync from the other remote
(pulp) [vagrant@pulp3-source-fedora30 pulp_rpm]$ http POST :24817${REPO_HREF}sync/ remote=$REMOTE_HREF2
HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Thu, 14 Nov 2019 12:50:05 GMT
Server: gunicorn/19.9.0
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "task": "/pulp/api/v3/tasks/06f0ef04-f4b7-4de0-b7d6-c5632d9a3b8c/"
}

# now there is a version #2
(pulp) [vagrant@pulp3-source-fedora30 pulp_rpm]$ http :/pulp/api/v3/repositories/rpm/rpm/9f5d2411-9142-44ce-b23e-cc743ff18139/versions/
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 1370
Content-Type: application/json
Date: Thu, 14 Nov 2019 12:50:33 GMT
Server: nginx/1.16.1
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "base_version": null,
            "content_summary": {
                "added": {
                    "rpm.advisory": {
                        "count": 4,
                        "href": "/pulp/api/v3/content/rpm/advisories/?repository_version_added=/pulp/api/v3/repositories/rpm/rpm/9f5d2411-9142-44ce-b23e-cc743ff18139/versions/2/"
                    }
                },
                "present": {
                    "rpm.advisory": {
                        "count": 8,
                        "href": "/pulp/api/v3/content/rpm/advisories/?repository_version=/pulp/api/v3/repositories/rpm/rpm/9f5d2411-9142-44ce-b23e-cc743ff18139/versions/2/"
                    },
                    "rpm.package": {
                        "count": 35,
                        "href": "/pulp/api/v3/content/rpm/packages/?repository_version=/pulp/api/v3/repositories/rpm/rpm/9f5d2411-9142-44ce-b23e-cc743ff18139/versions/2/"
                    },
                    "rpm.packagecategory": {
                        "count": 1,
                        "href": "/pulp/api/v3/content/rpm/packagecategories/?repository_version=/pulp/api/v3/repositories/rpm/rpm/9f5d2411-9142-44ce-b23e-cc743ff18139/versions/2/"
                    },
                    "rpm.packagegroup": {
                        "count": 2,
                        "href": "/pulp/api/v3/content/rpm/packagegroups/?repository_version=/pulp/api/v3/repositories/rpm/rpm/9f5d2411-9142-44ce-b23e-cc743ff18139/versions/2/"
                    },
                    "rpm.packagelangpacks": {
                        "count": 1,
                        "href": "/pulp/api/v3/content/rpm/packagelangpacks/?repository_version=/pulp/api/v3/repositories/rpm/rpm/9f5d2411-9142-44ce-b23e-cc743ff18139/versions/2/"
                    }
                },
                "removed": {}
            },
            "number": 2,
            "pulp_created": "2019-11-14T12:50:07.432468Z",
            "pulp_href": "/pulp/api/v3/repositories/rpm/rpm/9f5d2411-9142-44ce-b23e-cc743ff18139/versions/2/"
        }
    ]
}

If you look in the DB there is one incomplete verison.

pulp=> select * from core_repositoryversion;
               pulp_id                |         pulp_created          |       pulp_last_updated       | number | complete | base_version_id |            repository_id             
--------------------------------------+-------------------------------+-------------------------------+--------+----------+-----------------+--------------------------------------
 9d03a798-f9ad-49f4-9cd4-67f1bf421fe8 | 2019-11-14 12:46:26.004132+00 | 2019-11-14 12:46:26.004199+00 |      1 | f        |                 | 9f5d2411-9142-44ce-b23e-cc743ff18139
 fe391907-ad69-4445-aa2e-ae6e6ac3d931 | 2019-11-14 12:50:07.432468+00 | 2019-11-14 12:50:09.746473+00 |      2 | t        |                 | 9f5d2411-9142-44ce-b23e-cc743ff18139
(2 rows)