Issue #6058: re-migrating after deleting a pulp2 repo errors with: AttributeError: 'NoneType' object has no attribute 'publication_set' - Migration Plugin - Pulp

Actions

Send by e-mail Copy link

Issue #6058

closed

re-migrating after deleting a pulp2 repo errors with: AttributeError: 'NoneType' object has no attribute 'publication_set'

Added by jsherril@redhat.com almost 5 years ago. Updated over 4 years ago.

Status:

CLOSED - CURRENTRELEASE

Priority:

Normal

Assignee:

ipanova@redhat.com

Sprint/Milestone:

0.1.0

Start date:

Due date:

Estimated time:

Severity:

2. Medium

Platform Release:

OS:

Triaged:

Yes

Groomed:

Sprint Candidate:

Tags:

Katello

Sprint:

Sprint 65

Quarter:

Description

Here's what i did:

Created a migration plan to migrate 2 repositories:

{"plugins":\[{"type":"iso","repositories":\[{"name":"5b08c8a6-d3bb-41af-9cd4-cadf8fec2a55","repository_versions":\[{"pulp2_repository_id":"5b08c8a6-d3bb-41af-9cd4-cadf8fec2a55","pulp2_distributor_repository_ids":\["5b08c8a6-d3bb-41af-9cd4-cadf8fec2a55"\]}\],"pulp2_importer_repository_id":"5b08c8a6-d3bb-41af-9cd4-cadf8fec2a55"},{"name":"file_test-1-2","repository_versions":\[{"pulp2_repository_id":"1-file_test-v1_0-5b08c8a6-d3bb-41af-9cd4-cadf8fec2a55","pulp2_distributor_repository_ids":\["1-file_test-Library-5b08c8a6-d3bb-41af-9cd4-cadf8fec2a55"\]}\]}\]}\]}

Then deleted one of the repos in pulp2, and created a new plan to migrate the one that is remaining:

{"plugins":[{"type":"iso","repositories":[{"name":"5b08c8a6-d3bb-41af-9cd4-cadf8fec2a55","repository_versions":[{"pulp2_repository_id":"5b08c8a6-d3bb-41af-9cd4-cadf8fec2a55","pulp2_distributor_repository_ids":["5b08c8a6-d3bb-41af-9cd4-cadf8fec2a55"]}],"pulp2_importer_repository_id":"5b08c8a6-d3bb-41af-9cd4-cadf8fec2a55"}]}]}

I then tried to list from the pulp2repositories api, and i get an error:

pulp: django.request:ERROR: Internal Server Error: /pulp/api/v3/pulp2repositories/
Traceback (most recent call last):
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/django/views/decorators/csrf.py", line 71, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/viewsets.py", line 114, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 505, in dispatch
response = self.handle_exception(exc)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 465, in handle_exception
self.raise_uncaught_exception(exc)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
raise exc
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/views.py", line 502, in dispatch
response = handler(request, *args, **kwargs)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/mixins.py", line 43, in list
return self.get_paginated_response(serializer.data)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/serializers.py", line 757, in data
ret = super().data
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/serializers.py", line 261, in data
self._data = self.to_representation(self.instance)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/serializers.py", line 675, in to_representation
self.child.to_representation(item) for item in iterable
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/serializers.py", line 675, in <listcomp>
self.child.to_representation(item) for item in iterable
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/serializers.py", line 526, in to_representation
ret[field.field_name] = field.to_representation(attribute)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/fields.py", line 1873, in to_representation
return method(value)
File "/usr/local/lib/pulp/src/pulp/pulp_2to3_migration/app/serializers.py", line 184, in get_pulp3_publication_href
pub_list = obj.pulp3_repository_version.publication_set.all()
AttributeError: 'NoneType' object has no attribute 'publication_set'

Actions

Copy link

Updated by dalley almost 5 years ago

I can't seem to reproduce

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulpcore]$ http GET :24817/pulp/api/v3/pulp2repositories/                                                                                                                                  
....
....

{
    "count": 2,
    "next": null,
    "previous": null,
    "results": [
        {
            "is_migrated": true,
            "not_in_plan": false,
            "pulp2_object_id": "5e2f60dc6e50c9325cda8f53",
            "pulp2_repo_id": "file",
            "pulp3_distribution_hrefs": [
                "/pulp/api/v3/distributions/file/file/a1fbff03-4095-4b03-944d-495042d76591/"
            ],
            "pulp3_publication_href": [
                "/pulp/api/v3/publications/file/file/b3029362-6ee9-4748-9f23-aa460cb50c4e/"
            ],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/595cc126-5042-47b6-af6b-7de4d8e4b596/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/0ee9f786-1c4c-4529-bdeb-3739a4b262ff/versions/1/",
            "pulp_created": "2020-01-27T23:05:52.466059Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/3d5ea4c5-cb7b-47f1-b7b7-ef4579b8ec56/"
        },
        {
            "is_migrated": true,
            "not_in_plan": false,
            "pulp2_object_id": "5e2f60dc6e50c9325cda8f59",
            "pulp2_repo_id": "file-many",
            "pulp3_distribution_hrefs": [
                "/pulp/api/v3/distributions/file/file/143bc56d-3c10-4b08-a093-b5e126887f1e/"
            ],
            "pulp3_publication_href": [
                "/pulp/api/v3/publications/file/file/112b9ed5-93b1-4d77-9a51-d1987b46c579/"
            ],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/eed00d69-0bce-4ffc-a0fe-9dbfa1f21a4d/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/c5450ac6-9382-4d34-9792-ef8c075c54b4/versions/1/",
            "pulp_created": "2020-01-27T23:05:52.415607Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/ced895c7-43a4-4cb4-bfd2-c0f6b3435bc4/"
        }
    ]
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulpcore]$ pulp-admin iso repo delete --repo-id file-many
....

Repository [file-many] successfully deleted

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulpcore]$ http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{"type": "iso", "repositories": [{"name": "pulp3file", "pulp2_importer_repository_id": "file", "repository_vers
ions": [{"pulp2_repository_id": "file", "pulp2_distributor_repository_ids": ["file"]}]}]}]}'
....
....

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulpcore]$ http POST :24817/pulp/api/v3/migration-plans/a1c5c759-6596-4dfa-a2af-fc19681f1015/run/
....
....

{
    "task": "/pulp/api/v3/tasks/8301c9b8-a0f6-4ee1-9076-95fbde3c62e1/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulpcore]$ http GET :24817/pulp/api/v3/pulp2repositories/
....
....
{
    "count": 2,
    "next": null,
    "previous": null,
    "results": [
        {
            "is_migrated": true,
            "not_in_plan": true,
            "pulp2_object_id": "5e2f60dc6e50c9325cda8f59",
            "pulp2_repo_id": "file-many",
            "pulp3_distribution_hrefs": [],
            "pulp3_publication_href": [],
            "pulp3_remote_href": null,
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/c5450ac6-9382-4d34-9792-ef8c075c54b4/versions/1/",
            "pulp_created": "2020-01-27T23:05:52.415607Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/ced895c7-43a4-4cb4-bfd2-c0f6b3435bc4/"
        },
        {
            "is_migrated": true,
            "not_in_plan": false,
            "pulp2_object_id": "5e2f60dc6e50c9325cda8f53",
            "pulp2_repo_id": "file",
            "pulp3_distribution_hrefs": [
                "/pulp/api/v3/distributions/file/file/1096a093-e3ea-4b94-a74b-6bf67e741992/"
            ],
            "pulp3_publication_href": [
                "/pulp/api/v3/publications/file/file/1d0fc161-6e06-438b-bee1-d365623838ab/"
            ],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/5e51aea5-48bd-46d5-9deb-1f5581f218f5/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/0ee9f786-1c4c-4529-bdeb-3739a4b262ff/versions/1/",
            "pulp_created": "2020-01-27T23:05:52.466059Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/3d5ea4c5-cb7b-47f1-b7b7-ef4579b8ec56/"
        }
    ]
}

Actions

Copy link

Updated by dalley almost 5 years ago

Granted, I'm not sure that repository should still be there?

Actions

Copy link

Updated by jsherril@redhat.com almost 5 years ago

I will try to reproduce!

Actions

Copy link

Updated by jsherril@redhat.com almost 5 years ago

I suspect it might have had something to do with: https://pulp.plan.io/issues/6056

where by the 2nd repo was never fully migrated.

Actions

Copy link

Updated by ipanova@redhat.com almost 5 years ago

The only way i managed to reproduce this is to query pulp2repositories while task is still running. When we serialize the results the repo_version has not been created at that time that is why we get that traceback. After task successfully completes i get 200 on pulp2repositories api endpoint

$ http POST :24817/pulp/api/v3/migration-plans/43f46baf-9168-4f85-bfb7-a9ad1ea5f0ad/run/
HTTP/1.1 202 Accepted
Allow: POST, OPTIONS
Connection: close
Content-Length: 67
Content-Type: application/json
Date: Tue, 28 Jan 2020 22:09:34 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "task": "/pulp/api/v3/tasks/7ded9d41-c338-4143-aad2-71f3f2127e41/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 devel]$ http GET :24817/pulp/api/v3/pulp2repositories/
HTTP/1.1 500 Internal Server Error
Connection: close
Content-Length: 27
Content-Type: text/html
Date: Tue, 28 Jan 2020 22:09:36 GMT
Server: gunicorn/20.0.4
Vary: Cookie
X-Frame-Options: SAMEORIGIN

<h1>Server Error (500)</h1>

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 devel]$ http GET :24817/pulp/api/v3/pulp2repositories/
HTTP/1.1 500 Internal Server Error
Connection: close
Content-Length: 27
Content-Type: text/html
Date: Tue, 28 Jan 2020 22:09:37 GMT
Server: gunicorn/20.0.4
Vary: Cookie
X-Frame-Options: SAMEORIGIN

<h1>Server Error (500)</h1>

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 devel]$ pjournal 
-- Logs begin at Tue 2020-01-28 15:21:34 UTC. --
Jan 28 22:09:37 pulp2-nightly-pulp3-source-centos7.fluffy.example.com gunicorn[11859]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/serializers.py", line 675, in <listcomp>
Jan 28 22:09:37 pulp2-nightly-pulp3-source-centos7.fluffy.example.com gunicorn[11859]: self.child.to_representation(item) for item in iterable
Jan 28 22:09:37 pulp2-nightly-pulp3-source-centos7.fluffy.example.com gunicorn[11859]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/serializers.py", line 526, in to_representation
Jan 28 22:09:37 pulp2-nightly-pulp3-source-centos7.fluffy.example.com gunicorn[11859]: ret[field.field_name] = field.to_representation(attribute)
Jan 28 22:09:37 pulp2-nightly-pulp3-source-centos7.fluffy.example.com gunicorn[11859]: File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rest_framework/fields.py", line 1873, in to_representation
Jan 28 22:09:37 pulp2-nightly-pulp3-source-centos7.fluffy.example.com gunicorn[11859]: return method(value)
Jan 28 22:09:37 pulp2-nightly-pulp3-source-centos7.fluffy.example.com gunicorn[11859]: File "/home/vagrant/devel/pulp-2to3-migration/pulp_2to3_migration/app/serializers.py", line 184, in get_pulp3_publication_href
Jan 28 22:09:37 pulp2-nightly-pulp3-source-centos7.fluffy.example.com gunicorn[11859]: pub_list = obj.pulp3_repository_version.publication_set.all()
Jan 28 22:09:37 pulp2-nightly-pulp3-source-centos7.fluffy.example.com gunicorn[11859]: AttributeError: 'NoneType' object has no attribute 'publication_set'
Jan 28 22:09:37 pulp2-nightly-pulp3-source-centos7.fluffy.example.com gunicorn[11859]: 127.0.0.1 - admin [28/Jan/2020:22:09:37 +0000] "GET /pulp/api/v3/pulp2repositories/ HTTP/1.1" 500 27 "-" "HTTPie/1.0.3"
Jan 28 22:09:40 pulp2-nightly-pulp3-source-centos7.fluffy.example.com rq[11856]: pulp: rq.worker:INFO: 11856@pulp2-nightly-pulp3-source-centos7.fluffy.example.com: Job OK (7ded9d41-c338-4143-aad2-71f3f2127e41)
Jan 28 22:09:40 pulp2-nightly-pulp3-source-centos7.fluffy.example.com rq[11856]: pulp: rq.worker:INFO: 11856@pulp2-nightly-pulp3-source-centos7.fluffy.example.com: b053f51d-d2cc-4b43-9b31-5458e7a93362
Jan 28 22:09:40 pulp2-nightly-pulp3-source-centos7.fluffy.example.com rq[11856]: pulp: rq.worker:INFO: 11856@pulp2-nightly-pulp3-source-centos7.fluffy.example.com: Job OK (b053f51d-d2cc-4b43-9b31-5458e7a93362)
^[[A^[[B^C
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 devel]$ http GET :24817/pulp/api/v3/pulp2repositories/
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 2667
Content-Type: application/json
Date: Tue, 28 Jan 2020 22:09:48 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "count": 4,
    "next": null,
    "previous": null,
    "results": [
        {
            "is_migrated": true,
            "not_in_plan": false,
            "pulp2_object_id": "5e1c932ec998ac367cd60d85",
            "pulp2_repo_id": "file2",
            "pulp3_distribution_hrefs": [
                "/pulp/api/v3/distributions/file/file/2eef2d4f-f4ca-4c61-ac08-1f42089d2293/"
            ],
            "pulp3_publication_href": [
                "/pulp/api/v3/publications/file/file/0607d4ea-ce79-4ec5-b94d-8389b0f5066d/"
            ],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/7549b885-f930-4475-9c0c-a37956960f3c/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/bfc30d9b-1dc7-4d3a-9801-602b2d00fe4a/versions/1/",
            "pulp_created": "2020-01-28T22:09:34.824798Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/d6fd4544-17a4-4aab-a633-25ad7bd4c88c/"
        },
        {
       <snip>

@jsherril can we close this out? in case your issue was caused by https://pulp.plan.io/issues/6058#note-4 that would mean that the task failed and with failed task we cannot guarantee serialization of the data since it has not been migrated fully/correctly.

Actions

Copy link

Updated by jsherril@redhat.com almost 5 years ago

Yes, i do think that this only happens when a migration failure occurs. However i think we need to either:

1) ensure that 'dirty' data that causes a 500 on error can never get created (or gets cleaned up)
2) relax the rendering of the pulp2repositories api to handle this situation
3) don't return those pulp2repositories as part of the api

i've seen this multiple times, essentially whenever a migration fails for whatever reason, it leaves the pulp2Repository api returning a 500 ISE, which i think it should never do.

Thoughts?

Actions

Copy link

Updated by ipanova@redhat.com almost 5 years ago

Description updated (diff)
Status changed from NEW to POST
Assignee set to ipanova@redhat.com
Triaged changed from No to Yes
Sprint set to Sprint 65

https://github.com/pulp/pulp-2to3-migration/pull/86/files

Actions

Copy link

Updated by ipanova@redhat.com almost 5 years ago

@justin i eliminated the 500 ISE error, but we still are displaying repos that did not fully migrate( for example publication has not been created for it, you will just need null on the publication href)

Actions

Copy link