Project

Profile

Help

Actions
Send by e-mail Copy link

Task #6718

closed

Task #6672: [Epic] Upload Process Improvements

Stop using artifacts when importing collections

Added by fao89 over 4 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
fao89
Sprint/Milestone:
0.2.0
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

UploadGalaxyCollectionMixin - https://github.com/pulp/pulp_ansible/blob/master/pulp_ansible/app/galaxy/mixins.py Update UploadGalaxyCollectionMixin to have a method for creating an S3-compatible temporary file, replace artifact_pk with temp_file_path on _dispatch_import_collection_task method

Related issues

Related to Pulp - Story #6749: As a core or plugin developer, i want to be able to share temporary files between the viewset and triggered tasksCLOSED - CURRENTRELEASEfao89

Actions
Related to Ansible Plugin - Issue #5666: Failed imports leave orphaned artifacts blocking subsequent attemptsCLOSED - DUPLICATEActions
Actions
Copy link
 #1

Updated by fao89 over 4 years ago 

worker_1            | Couldn't parse task at /tmp/tmpah_bmbw9/ansible_collections/newswangerd/collection_demo/roles/deltoid/tasks/main.yml:1 (couldn't resolve module/action 'newswangerd.collection_demo.real_facts'. This often indicates a misspelling, missing collection, or incorrect module path.
worker_1            |
worker_1            | The error appears to be in '<unicode string>': line 1, column 3, but may
worker_1            | be elsewhere in the file depending on the exact syntax problem.
worker_1            |
worker_1            | (could not open file to display line))
worker_1            | { 'name': 'Gather some real facts FROM A DELTOID ROLE!',
worker_1            |   'newswangerd.collection_demo.real_facts': { '__file__': '/tmp/tmpah_bmbw9/ansible_collections/newswangerd/collection_demo/roles/deltoid/tasks/main.yml',
worker_1            |                                               '__line__': 2,
worker_1            |                                               'name': 'John Doe'},
worker_1            |   'register': 'testout',
worker_1            |   'skipped_rules': []}
worker_1            | Couldn't parse task at /tmp/tmpah_bmbw9/ansible_collections/newswangerd/collection_demo/roles/factoid/tasks/main.yml:1 (couldn't resolve module/action 'newswangerd.collection_demo.real_facts'. This often indicates a misspelling, missing collection, or incorrect module path.
worker_1            |
worker_1            | The error appears to be in '<unicode string>': line 1, column 3, but may
worker_1            | be elsewhere in the file depending on the exact syntax problem.
worker_1            |
worker_1            | (could not open file to display line))
worker_1            | { 'name': 'Gather some real facts FROM A ROLE!',
worker_1            |   'newswangerd.collection_demo.real_facts': { '__file__': '/tmp/tmpah_bmbw9/ansible_collections/newswangerd/collection_demo/roles/factoid/tasks/main.yml',
worker_1            |                                               '__line__': 2,
worker_1            |                                               'name': 'John Doe'},
worker_1            |   'register': 'testout',
worker_1            |   'skipped_rules': []}
worker_1            | pulp: rq.worker:ERROR: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'Artifact' because they are referenced through a protected foreign key: 'ContentArtifact.artifact'", <QuerySet [<ContentArtifact: pk=27aeabe9-7c71-4da3-bd6b-a7682f3ff2f0>]>)

worker_1            | pulp: rq.worker:ERROR: django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'Artifact' because they are referenced through a protected foreign key: 'ContentArtifact.artifact'", <QuerySet [<ContentArtifact: pk=4974e2ad-ec69-4138-96fb-7b4c0d192c3c>]>)
worker_1            | Traceback (most recent call last):
worker_1            |   File "/venv/lib64/python3.6/site-packages/pulp_ansible/app/tasks/collections.py", line 231, in _artifact_guard
worker_1            |     yield artifact
worker_1            |   File "/venv/lib64/python3.6/site-packages/pulp_ansible/app/tasks/collections.py", line 148, in import_collection
worker_1            |     CreatedResource.objects.create(content_object=collection_version)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
worker_1            |     return getattr(self.get_queryset(), name)(*args, **kwargs)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django/db/models/query.py", line 420, in create
worker_1            |     obj = self.model(**kwargs)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 21, in __init__
worker_1            |     self._initial_state = self._snapshot_state()
worker_1            |   File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 26, in _snapshot_state
worker_1            |     for watched_related_field in self._watched_fk_model_fields:
worker_1            |   File "/venv/lib64/python3.6/site-packages/django/utils/functional.py", line 80, in __get__
worker_1            |     res = instance.__dict__[self.name] = self.func(instance)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 169, in _watched_fk_model_fields
worker_1            |     for method in self._potentially_hooked_methods:
worker_1            |   File "/venv/lib64/python3.6/site-packages/django/utils/functional.py", line 80, in __get__
worker_1            |     res = instance.__dict__[self.name] = self.func(instance)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 145, in _potentially_hooked_methods
worker_1            |     skip = set(get_unhookable_attribute_names(self))
worker_1            |   File "/venv/lib64/python3.6/site-packages/django_lifecycle/utils.py", line 68, in get_unhookable_attribute_names
worker_1            |     + ["_run_hooked_methods"]
worker_1            |   File "/venv/lib64/python3.6/site-packages/django_lifecycle/utils.py", line 57, in _get_field_names
worker_1            |     if instance._meta.get_field(f.name).get_internal_type() == "ForeignKey":
worker_1            | AttributeError: 'GenericForeignKey' object has no attribute 'get_internal_type'
worker_1            |
worker_1            | During handling of the above exception, another exception occurred:
worker_1            |
worker_1            | Traceback (most recent call last):
worker_1            |   File "/venv/lib64/python3.6/site-packages/rq/worker.py", line 883, in perform_job
worker_1            |     rv = job.perform()
worker_1            |   File "/venv/lib64/python3.6/site-packages/rq/job.py", line 657, in perform
worker_1            |     self._result = self._execute()
worker_1            |   File "/venv/lib64/python3.6/site-packages/rq/job.py", line 663, in _execute
worker_1            |     return self.func(*self.args, **self.kwargs)
worker_1            |   File "/src/galaxy_ng/galaxy_ng/app/tasks/publishing.py", line 21, in import_and_auto_approve
worker_1            |     import_collection(artifact_pk=artifact_pk, repository_pk=kwargs.get('repository_pk', None))
worker_1            |   File "/venv/lib64/python3.6/site-packages/pulp_ansible/app/tasks/collections.py", line 155, in import_collection
worker_1            |     CreatedResource.objects.create(content_object=repository)
worker_1            |   File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
worker_1            |     self.gen.throw(type, value, traceback)
worker_1            |   File "/venv/lib64/python3.6/site-packages/pulp_ansible/app/tasks/collections.py", line 233, in _artifact_guard
worker_1            |     artifact.delete()
worker_1            |   File "/src/pulpcore/pulpcore/app/models/content.py", line 185, in delete
worker_1            |     super().delete(*args, **kwargs)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 140, in delete
worker_1            |     super().delete(*args, **kwargs)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django/db/models/base.py", line 918, in delete
worker_1            |     collector.collect([self], keep_parents=keep_parents)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
worker_1            |     field.remote_field.on_delete(self, field, sub_objs, self.using)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
worker_1            |     sub_objs
worker_1            | django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'Artifact' because they are referenced through a protected foreign key: 'ContentArtifact.artifact'", <QuerySet [<ContentArtifact: pk=4974e2ad-ec69-4138-96fb-7b4c0d192c3c>]>)
worker_1            | Traceback (most recent call last):
worker_1            |   File "/venv/lib64/python3.6/site-packages/pulp_ansible/app/tasks/collections.py", line 231, in _artifact_guard
worker_1            |     yield artifact
worker_1            |   File "/venv/lib64/python3.6/site-packages/pulp_ansible/app/tasks/collections.py", line 148, in import_collection
worker_1            |     CreatedResource.objects.create(content_object=collection_version)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
worker_1            |     return getattr(self.get_queryset(), name)(*args, **kwargs)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django/db/models/query.py", line 420, in create
worker_1            |     obj = self.model(**kwargs)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 21, in __init__
worker_1            |     self._initial_state = self._snapshot_state()
worker_1            |   File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 26, in _snapshot_state
worker_1            |     for watched_related_field in self._watched_fk_model_fields:
worker_1            |   File "/venv/lib64/python3.6/site-packages/django/utils/functional.py", line 80, in __get__
worker_1            |     res = instance.__dict__[self.name] = self.func(instance)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 169, in _watched_fk_model_fields
worker_1            |     for method in self._potentially_hooked_methods:
worker_1            |   File "/venv/lib64/python3.6/site-packages/django/utils/functional.py", line 80, in __get__
worker_1            |     res = instance.__dict__[self.name] = self.func(instance)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 145, in _potentially_hooked_methods
worker_1            |     skip = set(get_unhookable_attribute_names(self))
worker_1            |   File "/venv/lib64/python3.6/site-packages/django_lifecycle/utils.py", line 68, in get_unhookable_attribute_names
worker_1            |     + ["_run_hooked_methods"]
worker_1            |   File "/venv/lib64/python3.6/site-packages/django_lifecycle/utils.py", line 57, in _get_field_names
worker_1            |     if instance._meta.get_field(f.name).get_internal_type() == "ForeignKey":
worker_1            | AttributeError: 'GenericForeignKey' object has no attribute 'get_internal_type'
worker_1            |
worker_1            | During handling of the above exception, another exception occurred:
worker_1            |
worker_1            | Traceback (most recent call last):
worker_1            |   File "/venv/lib64/python3.6/site-packages/rq/worker.py", line 883, in perform_job
worker_1            |     rv = job.perform()
worker_1            |   File "/venv/lib64/python3.6/site-packages/rq/job.py", line 657, in perform
worker_1            |     self._result = self._execute()
worker_1            |   File "/venv/lib64/python3.6/site-packages/rq/job.py", line 663, in _execute
worker_1            |     return self.func(*self.args, **self.kwargs)
worker_1            |   File "/src/galaxy_ng/galaxy_ng/app/tasks/publishing.py", line 21, in import_and_auto_approve
worker_1            |     import_collection(artifact_pk=artifact_pk, repository_pk=kwargs.get('repository_pk', None))
worker_1            |   File "/venv/lib64/python3.6/site-packages/pulp_ansible/app/tasks/collections.py", line 155, in import_collection
worker_1            |     CreatedResource.objects.create(content_object=repository)
worker_1            |   File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
worker_1            |     self.gen.throw(type, value, traceback)
worker_1            |   File "/venv/lib64/python3.6/site-packages/pulp_ansible/app/tasks/collections.py", line 233, in _artifact_guard
worker_1            |     artifact.delete()
worker_1            |   File "/src/pulpcore/pulpcore/app/models/content.py", line 185, in delete
worker_1            |     super().delete(*args, **kwargs)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django_lifecycle/mixins.py", line 140, in delete
worker_1            |     super().delete(*args, **kwargs)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django/db/models/base.py", line 918, in delete
worker_1            |     collector.collect([self], keep_parents=keep_parents)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django/db/models/deletion.py", line 224, in collect
worker_1            |     field.remote_field.on_delete(self, field, sub_objs, self.using)
worker_1            |   File "/venv/lib64/python3.6/site-packages/django/db/models/deletion.py", line 27, in PROTECT
worker_1            |     sub_objs
worker_1            | django.db.models.deletion.ProtectedError: ("Cannot delete some instances of model 'Artifact' because they are referenced through a protected foreign key: 'ContentArtifact.artifact'", <QuerySet [<ContentArtifact: pk=4974e2ad-ec69-4138-96fb-7b4c0d192c3c>]>)
Actions
Copy link
 #2

Updated by fao89 over 4 years ago

  • Related to Story #6749: As a core or plugin developer, i want to be able to share temporary files between the viewset and triggered tasks added
Actions
Copy link
 #3

Updated by fao89 over 4 years ago

  • Related to Issue #5666: Failed imports leave orphaned artifacts blocking subsequent attempts added
Actions
Copy link
 #4

Updated by fao89 over 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to fao89
Actions
Copy link
 #5

Updated by pulpbot over 4 years ago

  • Status changed from ASSIGNED to POST

Added by Fabricio Aguiar over 4 years ago

Revision f6dbec73 | View on GitHub

Stop using artifacts when importing collections

Required PR: https://github.com/pulp/pulpcore/pull/793

https://pulp.plan.io/issues/6718 closes #6718

Actions
Copy link
 #6

Updated by Anonymous over 4 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions
Copy link
 #7

Updated by fao89 over 4 years ago

  • Sprint/Milestone set to 0.2.0
Actions
Copy link
 #8

Updated by pulpbot over 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions
Send by e-mail Copy link

Also available in: Atom PDF