Actions
Task #6718
closedTask #6672: [Epic] Upload Process Improvements
Stop using artifacts when importing collections
Start date:
Due date:
% Done:
100%
Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:
Description
UploadGalaxyCollectionMixin - https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/galaxy/mixins.py Update UploadGalaxyCollectionMixin to have a method for creating an S3-compatible temporary file, replace artifact_pk with temp_file_path on _dispatch_import_collection_task method
Related issues
Updated by fao89 about 4 years ago
worker_1 | Couldn't parse task at /tmp/tmpah_bmbw9/ansible_collections/newswangerd/collection_demo/roles/deltoid/tasks/main.yml:1 (couldn't resolve module/action 'newswangerd.collection_demo.real_facts'. This often indicates a misspelling, missing collection, or incorrect module path.
worker_1 |
worker_1 | The error appears to be in '<unicode string>': line 1, column 3, but may
worker_1 | be elsewhere in the file depending on the exact syntax problem.
worker_1 |
worker_1 | (could not open file to display line))
worker_1 | { 'name': 'Gather some real facts FROM A DELTOID ROLE!',
worker_1 | 'newswangerd.collection_demo.real_facts': { '__file__': '/tmp/tmpah_bmbw9/ansible_collections/newswangerd/collection_demo/roles/deltoid/tasks/main.yml',
worker_1 | '__line__': 2,
worker_1 | 'name': 'John Doe'},
worker_1 | 'register': 'testout',
worker_1 | 'skipped_rules': []}
worker_1 | Couldn't parse task at /tmp/tmpah_bmbw9/ansible_collections/newswangerd/collection_demo/roles/factoid/tasks/main.yml:1 (couldn't resolve module/action 'newswangerd.collection_demo.real_facts'. This often indicates a misspelling, missing collection, or incorrect module path.
worker_1 |
worker_1 | The error appears to be in '<unicode string>': line 1, column 3, but may
worker_1 | be elsewhere in the file depending on the exact syntax problem.
worker_1 |
worker_1 | (could not open file to display line))
worker_1 | { 'name': 'Gather some real facts FROM A ROLE!',
worker_1 | 'newswangerd.collection_demo.real_facts': { '__file__': '/tmp/tmpah_bmbw9/ansible_collections/newswangerd/collection_demo/roles/factoid/tasks/main.yml',
worker_1 | '__line__': 2,
worker_1 | 'name': 'John Doe'},
worker_1 | 'register': 'testout',
worker_1 | 'skipped_rules': []}
worker_1 | pulp: rq.worker:ERROR: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'Artifact' because they are referenced through a protected foreign key: 'ContentArtifact.artifact'", <QuerySet [<ContentArtifact: pk=27aeabe9-7c71-4da3-bd6b-a7682f3ff2f0>]>)
worker_1 | pulp: rq.worker:ERROR: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'Artifact' because they are referenced through a protected foreign key: 'ContentArtifact.artifact'", <QuerySet [<ContentArtifact: pk=4974e2ad-ec69-4138-96fb-7b4c0d192c3c>]>)
worker_1 | Traceback (most recent call last):
worker_1 | File "/venv/lib64/python3.6/site-packages/pulp_ansible/app/tasks/collections.py", line 231, in _artifact_guard
worker_1 | yield artifact
worker_1 | File "/venv/lib64/python3.6/site-packages/pulp_ansible/app/tasks/collections.py", line 148, in import_collection
worker_1 | CreatedResource.objects.create(content_object=collection_version)
worker_1 | File "/venv/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
worker_1 | return getattr(self.get_queryset(), name)(*args, **kwargs)
worker_1 | File "/venv/lib64/python3.6/site-packages/django/db/models/query.py", line 420, in create
worker_1 | obj = self.model(**kwargs)
worker_1 | File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 21, in __init__
worker_1 | self._initial_state = self._snapshot_state()
worker_1 | File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 26, in _snapshot_state
worker_1 | for watched_related_field in self._watched_fk_model_fields:
worker_1 | File "/venv/lib64/python3.6/site-packages/django/utils/functional.py", line 80, in __get__
worker_1 | res = instance.__dict__[self.name] = self.func(instance)
worker_1 | File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 169, in _watched_fk_model_fields
worker_1 | for method in self._potentially_hooked_methods:
worker_1 | File "/venv/lib64/python3.6/site-packages/django/utils/functional.py", line 80, in __get__
worker_1 | res = instance.__dict__[self.name] = self.func(instance)
worker_1 | File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 145, in _potentially_hooked_methods
worker_1 | skip = set(get_unhookable_attribute_names(self))
worker_1 | File "/venv/lib64/python3.6/site-packages/django_lifecycle/utils.py", line 68, in get_unhookable_attribute_names
worker_1 | + ["_run_hooked_methods"]
worker_1 | File "/venv/lib64/python3.6/site-packages/django_lifecycle/utils.py", line 57, in _get_field_names
worker_1 | if instance._meta.get_field(f.name).get_internal_type() == "ForeignKey":
worker_1 | AttributeError: 'GenericForeignKey' object has no attribute 'get_internal_type'
worker_1 |
worker_1 | During handling of the above exception, another exception occurred:
worker_1 |
worker_1 | Traceback (most recent call last):
worker_1 | File "/venv/lib64/python3.6/site-packages/rq/worker.py", line 883, in perform_job
worker_1 | rv = job.perform()
worker_1 | File "/venv/lib64/python3.6/site-packages/rq/job.py", line 657, in perform
worker_1 | self._result = self._execute()
worker_1 | File "/venv/lib64/python3.6/site-packages/rq/job.py", line 663, in _execute
worker_1 | return self.func(*self.args, **self.kwargs)
worker_1 | File "/src/galaxy_ng/galaxy_ng/app/tasks/publishing.py", line 21, in import_and_auto_approve
worker_1 | import_collection(artifact_pk=artifact_pk, repository_pk=kwargs.get('repository_pk', None))
worker_1 | File "/venv/lib64/python3.6/site-packages/pulp_ansible/app/tasks/collections.py", line 155, in import_collection
worker_1 | CreatedResource.objects.create(content_object=repository)
worker_1 | File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
worker_1 | self.gen.throw(type, value, traceback)
worker_1 | File "/venv/lib64/python3.6/site-packages/pulp_ansible/app/tasks/collections.py", line 233, in _artifact_guard
worker_1 | artifact.delete()
worker_1 | File "/src/pulpcore/pulpcore/app/models/content.py", line 185, in delete
worker_1 | super().delete(*args, **kwargs)
worker_1 | File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 140, in delete
worker_1 | super().delete(*args, **kwargs)
worker_1 | File "/venv/lib64/python3.6/site-packages/django/db/models/base.py", line 918, in delete
worker_1 | collector.collect([self], keep_parents=keep_parents)
worker_1 | File "/venv/lib64/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
worker_1 | field.remote_field.on_delete(self, field, sub_objs, self.using)
worker_1 | File "/venv/lib64/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
worker_1 | sub_objs
worker_1 | django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'Artifact' because they are referenced through a protected foreign key: 'ContentArtifact.artifact'", <QuerySet [<ContentArtifact: pk=4974e2ad-ec69-4138-96fb-7b4c0d192c3c>]>)
worker_1 | Traceback (most recent call last):
worker_1 | File "/venv/lib64/python3.6/site-packages/pulp_ansible/app/tasks/collections.py", line 231, in _artifact_guard
worker_1 | yield artifact
worker_1 | File "/venv/lib64/python3.6/site-packages/pulp_ansible/app/tasks/collections.py", line 148, in import_collection
worker_1 | CreatedResource.objects.create(content_object=collection_version)
worker_1 | File "/venv/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
worker_1 | return getattr(self.get_queryset(), name)(*args, **kwargs)
worker_1 | File "/venv/lib64/python3.6/site-packages/django/db/models/query.py", line 420, in create
worker_1 | obj = self.model(**kwargs)
worker_1 | File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 21, in __init__
worker_1 | self._initial_state = self._snapshot_state()
worker_1 | File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 26, in _snapshot_state
worker_1 | for watched_related_field in self._watched_fk_model_fields:
worker_1 | File "/venv/lib64/python3.6/site-packages/django/utils/functional.py", line 80, in __get__
worker_1 | res = instance.__dict__[self.name] = self.func(instance)
worker_1 | File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 169, in _watched_fk_model_fields
worker_1 | for method in self._potentially_hooked_methods:
worker_1 | File "/venv/lib64/python3.6/site-packages/django/utils/functional.py", line 80, in __get__
worker_1 | res = instance.__dict__[self.name] = self.func(instance)
worker_1 | File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 145, in _potentially_hooked_methods
worker_1 | skip = set(get_unhookable_attribute_names(self))
worker_1 | File "/venv/lib64/python3.6/site-packages/django_lifecycle/utils.py", line 68, in get_unhookable_attribute_names
worker_1 | + ["_run_hooked_methods"]
worker_1 | File "/venv/lib64/python3.6/site-packages/django_lifecycle/utils.py", line 57, in _get_field_names
worker_1 | if instance._meta.get_field(f.name).get_internal_type() == "ForeignKey":
worker_1 | AttributeError: 'GenericForeignKey' object has no attribute 'get_internal_type'
worker_1 |
worker_1 | During handling of the above exception, another exception occurred:
worker_1 |
worker_1 | Traceback (most recent call last):
worker_1 | File "/venv/lib64/python3.6/site-packages/rq/worker.py", line 883, in perform_job
worker_1 | rv = job.perform()
worker_1 | File "/venv/lib64/python3.6/site-packages/rq/job.py", line 657, in perform
worker_1 | self._result = self._execute()
worker_1 | File "/venv/lib64/python3.6/site-packages/rq/job.py", line 663, in _execute
worker_1 | return self.func(*self.args, **self.kwargs)
worker_1 | File "/src/galaxy_ng/galaxy_ng/app/tasks/publishing.py", line 21, in import_and_auto_approve
worker_1 | import_collection(artifact_pk=artifact_pk, repository_pk=kwargs.get('repository_pk', None))
worker_1 | File "/venv/lib64/python3.6/site-packages/pulp_ansible/app/tasks/collections.py", line 155, in import_collection
worker_1 | CreatedResource.objects.create(content_object=repository)
worker_1 | File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
worker_1 | self.gen.throw(type, value, traceback)
worker_1 | File "/venv/lib64/python3.6/site-packages/pulp_ansible/app/tasks/collections.py", line 233, in _artifact_guard
worker_1 | artifact.delete()
worker_1 | File "/src/pulpcore/pulpcore/app/models/content.py", line 185, in delete
worker_1 | super().delete(*args, **kwargs)
worker_1 | File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 140, in delete
worker_1 | super().delete(*args, **kwargs)
worker_1 | File "/venv/lib64/python3.6/site-packages/django/db/models/base.py", line 918, in delete
worker_1 | collector.collect([self], keep_parents=keep_parents)
worker_1 | File "/venv/lib64/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
worker_1 | field.remote_field.on_delete(self, field, sub_objs, self.using)
worker_1 | File "/venv/lib64/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
worker_1 | sub_objs
worker_1 | django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'Artifact' because they are referenced through a protected foreign key: 'ContentArtifact.artifact'", <QuerySet [<ContentArtifact: pk=4974e2ad-ec69-4138-96fb-7b4c0d192c3c>]>)
Updated by fao89 about 4 years ago
- Related to Story #6749: As a core or plugin developer, i want to be able to share temporary files between the viewset and triggered tasks added
Updated by fao89 about 4 years ago
- Related to Issue #5666: Failed imports leave orphaned artifacts blocking subsequent attempts added
Updated by fao89 about 4 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to fao89
Updated by pulpbot about 4 years ago
- Status changed from ASSIGNED to POST
Added by Fabricio Aguiar about 4 years ago
Updated by Anonymous about 4 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulp_ansible|f6dbec732a5f7c17b34911a9a71ad84b47e7570d.
Updated by pulpbot almost 4 years ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions
Stop using artifacts when importing collections
Required PR: https://github.com/pulp/pulpcore/pull/793
https://pulp.plan.io/issues/6718 closes #6718