https://pulp.plan.io/https://pulp.plan.io/favicon.ico2018-10-09T13:15:15ZPulpAnsible Plugin - Story #4066: Add a role support to existing one-shot uploader for pulp_ansiblehttps://pulp.plan.io/issues/4066?journal_id=313022018-10-09T13:15:15Zdaviddavis
<ul><li><strong>Tracker</strong> changed from <i>Issue</i> to <i>Story</i></li><li><strong>% Done</strong> set to <i>0</i></li></ul> Ansible Plugin - Story #4066: Add a role support to existing one-shot uploader for pulp_ansiblehttps://pulp.plan.io/issues/4066?journal_id=313322018-10-10T21:00:21Zbmbouterbmbouter@redhat.com
<ul><li><strong>Status</strong> changed from <i>ASSIGNED</i> to <i>POST</i></li></ul><p>PR available at: <a href="https://github.com/pulp/pulp_ansible/pull/65" class="external">https://github.com/pulp/pulp_ansible/pull/65</a></p> Ansible Plugin - Story #4066: Add a role support to existing one-shot uploader for pulp_ansiblehttps://pulp.plan.io/issues/4066?journal_id=319492018-11-16T14:18:42Zbmbouterbmbouter@redhat.com
<ul><li><strong>Sprint</strong> set to <i>Sprint 46</i></li></ul><p>This story was done a months ago, but it's currently blocked because of an on-going discussion with Ansible about adding a 'version' field to Roles.</p> Ansible Plugin - Story #4066: Add a role support to existing one-shot uploader for pulp_ansiblehttps://pulp.plan.io/issues/4066?journal_id=319992018-11-19T10:29:49Zttereshcttereshc@redhat.com
<ul><li><strong>Sprint</strong> changed from <i>Sprint 46</i> to <i>Sprint 45</i></li></ul> Ansible Plugin - Story #4066: Add a role support to existing one-shot uploader for pulp_ansiblehttps://pulp.plan.io/issues/4066?journal_id=323392018-11-29T14:31:49Zrchan
<ul><li><strong>Sprint</strong> changed from <i>Sprint 45</i> to <i>Sprint 46</i></li></ul> Ansible Plugin - Story #4066: Add a role support to existing one-shot uploader for pulp_ansiblehttps://pulp.plan.io/issues/4066?journal_id=330892019-01-04T21:05:19Zrchan
<ul><li><strong>Sprint</strong> deleted (<del><i>Sprint 46</i></del>)</li></ul> Ansible Plugin - Story #4066: Add a role support to existing one-shot uploader for pulp_ansiblehttps://pulp.plan.io/issues/4066?journal_id=457192019-07-15T12:58:36Zbmbouterbmbouter@redhat.com
<ul><li><strong>Subject</strong> changed from <i>Add a one-shot upload to pulp_ansible</i> to <i>Add a role support to existing one-shot uploader for pulp_ansible</i></li></ul> Ansible Plugin - Story #4066: Add a role support to existing one-shot uploader for pulp_ansiblehttps://pulp.plan.io/issues/4066?journal_id=457202019-07-15T13:01:42Zbmbouterbmbouter@redhat.com
<ul><li><strong>Status</strong> changed from <i>POST</i> to <i>NEW</i></li><li><strong>Assignee</strong> deleted (<del><i>bmbouter</i></del>)</li></ul><p>This PR had issues importing the same content twice because Pulp without a 'version' in the metadata can't recognize the tarball contains the same Role data. Adding a 'version' to the metadata field for role data would resolve this, but we need to work with the broader Galaxy community for that.</p>
<p>Here is a copy of the patch:</p>
<pre><code class="diff syntaxhl" data-language="diff"><span class="p">commit b0038261704c50f5b44423b65de8621ee4e6b536
Author: Brian Bouterse <bbouters@redhat.com>
Date: Wed Oct 10 16:58:14 2018 -0400
</span>
Adds a one-shot upload
This one-shot upload will auto-discover any roles in the tarball
uploaded, create necessary Roles, and RoleVersion objects, and then
associate the RoleVersion objects with a new RepositoryVersion.
https://pulp.plan.io/issues/4066
closes #4066
diff --git a/README.rst b/README.rst
<span class="gh">index 03fa20f..a7e7390 100644
</span><span class="gd">--- a/README.rst
</span><span class="gi">+++ b/README.rst
</span><span class="p">@@ -151,11 +151,35 @@</span> Look at the new Repository Version created
"number": 1
}
+Upload one or more Roles to Pulp (the easy way)
<span class="gi">+-----------------------------------------------
</span>
-Upload a Role to Pulp
<span class="gd">----------------------
</span><span class="gi">+The upload API accepts a tarball which is opened up and any roles present will be imported and
+associated with the repository to create a new repository version.
</span>
-Download a role version.
<span class="gi">+The created roles are assigned the following data:
+
+- The namespace is your username.
+- The role name is the role name of the directory in the uploaded tarball.
+- The version is an invented UUID due to version not being part of the Role metadata format. You can
+ assign versions later through the API.
+
+Here is a tarball with 6 roles in it.
+
+``curl -L https://github.com/pulp/ansible-pulp3/archive/master.tar.gz -o pulp.tar.gz``
+
+Upload it to Pulp and associate it with the repository:
+
+``http --form POST :8000/pulp_ansible/upload/ repository=$REPO_HREF file@pulp.tar.gz sha256=$(sha256sum pulp.tar.gz | awk '{ print $1 }')``
+
+
+Upload a Role to Pulp (the hard way)
+------------------------------------
+
+Uploading content this way lets you specify a namespace, name, and version which are automatically
+determined with the upload API above.
+
+To start "the hard way", download a role version.
</span>
``curl -L https://github.com/pulp/ansible-pulp3/archive/master.tar.gz -o pulp.tar.gz``
@@ -165,7 +189,7 @@ Create an Artifact by uploading the role version tarball to Pulp.
Create a Role content unit
<span class="gd">---------------------------
</span><span class="gi">+^^^^^^^^^^^^^^^^^^^^^^^^^^
</span>
Create an Ansible role in Pulp.
@@ -173,7 +197,7 @@ Create an Ansible role in Pulp.
Create a ``role version`` from the Role and Artifact
<span class="gd">------------------------------------------------------
</span><span class="gi">+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
</span>
Create a content unit and point it to your Artifact and Role
@@ -181,13 +205,13 @@ Create a content unit and point it to your Artifact and Role
Add content to repository ``foo``
<span class="gd">----------------------------------
</span><span class="gi">+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
</span>
``$ http POST ':8000'$REPO_HREF'versions/' add_content_units:="[\"$CONTENT_HREF\"]"``
Create a Publication
<span class="gd">--------------------------------------------------
</span><span class="gi">+--------------------
</span>
``$ http POST :8000/pulp/api/v3/ansible/publications/ repository=$REPO_HREF``
diff --git a/pulp_ansible/app/serializers.py b/pulp_ansible/app/serializers.py
<span class="gh">index 7552559..addcae5 100644
</span><span class="gd">--- a/pulp_ansible/app/serializers.py
</span><span class="gi">+++ b/pulp_ansible/app/serializers.py
</span><span class="p">@@ -1,8 +1,10 @@</span>
<span class="gi">+from gettext import gettext as _
+
</span> from rest_framework import serializers
from pulpcore.plugin.serializers import ContentSerializer, IdentityField, NestedIdentityField, \
RelatedField, RemoteSerializer
<span class="gd">-from pulpcore.plugin.models import Artifact
</span><span class="gi">+from pulpcore.plugin.models import Artifact, Repository
</span>
from .models import AnsibleRemote, AnsibleRole, AnsibleRoleVersion
@@ -61,3 +63,26 @@ class AnsibleRemoteSerializer(RemoteSerializer):
class Meta:
fields = RemoteSerializer.Meta.fields
model = AnsibleRemote
<span class="gi">+
+
+class OneShotUploadSerializer(serializers.Serializer):
+ """
+ A serializer for the One Shot Upload API.
+ """
+
+ repository = serializers.HyperlinkedRelatedField(
+ help_text=_('A URI of the repository.'),
+ required=True,
+ queryset=Repository.objects.all(),
+ view_name='repositories-detail',
+ )
+
+ file = serializers.FileField(
+ help_text=_("The collection file."),
+ required=True,
+ )
+
+ sha256 = serializers.CharField(
+ required=False,
+ default=None,
+ )
</span><span class="gh">diff --git a/pulp_ansible/app/tasks/__init__.py b/pulp_ansible/app/tasks/__init__.py
index 67e6aac..925e12a 100644
</span><span class="gd">--- a/pulp_ansible/app/tasks/__init__.py
</span><span class="gi">+++ b/pulp_ansible/app/tasks/__init__.py
</span><span class="p">@@ -1,2 +1,3 @@</span>
<span class="gi">+from .upload import import_content_from_tarball # noqa
</span> from .synchronizing import synchronize # noqa
from .publishing import publish # noqa
<span class="gh">diff --git a/pulp_ansible/app/tasks/upload.py b/pulp_ansible/app/tasks/upload.py
</span><span class="p">new file mode 100644
</span><span class="gh">index 0000000..6349d69
</span><span class="gd">--- /dev/null
</span><span class="gi">+++ b/pulp_ansible/app/tasks/upload.py
</span><span class="p">@@ -0,0 +1,69 @@</span>
<span class="gi">+import os
+import re
+import tarfile
+import uuid
+
+from pulpcore.plugin.models import Artifact, ProgressBar, Repository, RepositoryVersion
+
+from pulp_ansible.app.models import AnsibleRole, AnsibleRoleVersion
+
+
+def import_content_from_tarball(namespace, artifact_pk=None, repository_pk=None):
+ """
+ Import Ansible content from a tarball saved as an Artifact.
+
+ The artifact is only a temporary storage area, and is deleted after being analyzed for more
+ content. Currently this task correctly handles: AnsibleRole and AnsibleRoleVersion content.
+
+ Args:
+ namespace (str): The namespace for any Ansible content to create
+ artifact_pk (int): The pk of the tarball Artifact to analyze and then delete
+ repository_pk (int): The repository that all created content should be associated with.
+ """
+ repository = Repository.objects.get(pk=repository_pk)
+ artifact = Artifact.objects.get(pk=artifact_pk)
+ role_paths = set()
+ with tarfile.open(str(artifact.file), "r") as tar:
+ artifact.delete() # this artifact is only stored between the frontend and backend
+ for tarinfo in tar:
+ match = re.search('(.*)/(tasks|handlers|defaults|vars|files|templates|meta)/main.yml',
+ tarinfo.path)
+ if match:
+ # This is a role asset
+ role_path = match.group(1)
+ role_paths.add(role_path)
+
+ tar.extractall()
+
+ role_version_pks = []
+ with ProgressBar(message='Importing Roles', total=len(role_paths)) as pb:
+ for role_path in role_paths:
+ match = re.search('(.*/)(.*)$', role_path)
+ role_name = match.group(2)
+ for tarinfo in tar:
+ if tarinfo.path == role_path:
+ # This is the role itself
+ assert tarinfo.isdir()
+ tarball_name = "{name}.tar.gz".format(name=role_name)
+ with tarfile.open(tarball_name, "w:gz") as newtar:
+ current_dir = os.getcwd()
+ os.chdir(match.group(1))
+ newtar.add(role_name)
+ os.chdir(current_dir)
+ full_path = os.path.abspath(tarball_name)
+ new_artifact = Artifact.init_and_validate(full_path)
+ new_artifact.save()
+ role, created = AnsibleRole.objects.get_or_create(namespace=namespace,
+ name=role_name)
+ version = uuid.uuid4()
+ role_version = AnsibleRoleVersion(
+ role=role,
+ version=version
+ )
+ role_version.artifact = new_artifact
+ role_version.save()
+ role_version_pks.append(role_version.pk)
+ pb.increment()
+ with RepositoryVersion.create(repository) as new_version:
+ qs = AnsibleRoleVersion.objects.filter(pk__in=role_version_pks)
+ new_version.add_content(qs)
</span><span class="gh">diff --git a/pulp_ansible/app/urls.py b/pulp_ansible/app/urls.py
index 4b7f872..efabbb3 100644
</span><span class="gd">--- a/pulp_ansible/app/urls.py
</span><span class="gi">+++ b/pulp_ansible/app/urls.py
</span><span class="p">@@ -3,10 +3,13 @@</span> from django.conf.urls import url
from pulp_ansible.app.galaxy.views import (
AnsibleGalaxyVersionView,
AnsibleRoleList,
<span class="gd">- AnsibleRoleVersionList
</span><span class="gi">+ AnsibleRoleVersionList,
</span> )
<span class="gi">+from .viewsets import OneShotUploadView
+
</span>
urlpatterns = [
<span class="gi">+ url(r'pulp_ansible/upload/$', OneShotUploadView.as_view()),
</span> url(r'pulp_ansible/galaxy/(?P<path>.+)/api/$', AnsibleGalaxyVersionView.as_view()),
url(r'pulp_ansible/galaxy/(?P<path>.+)/api/v1/roles/$', AnsibleRoleList.as_view()),
url(r'pulp_ansible/galaxy/(?P<path>.+)/api/v1/roles/(?P<role_pk>[0-9a-f-]+)/versions/$',
<span class="gh">diff --git a/pulp_ansible/app/viewsets.py b/pulp_ansible/app/viewsets.py
index 8106467..7b03209 100644
</span><span class="gd">--- a/pulp_ansible/app/viewsets.py
</span><span class="gi">+++ b/pulp_ansible/app/viewsets.py
</span><span class="p">@@ -1,7 +1,7 @@</span>
from django.db import transaction
from drf_yasg.utils import swagger_auto_schema
from rest_framework.decorators import detail_route
<span class="gd">-from rest_framework import mixins, status
</span><span class="gi">+from rest_framework import mixins, status, views
</span> from rest_framework.response import Response
from pulpcore.plugin.models import Artifact, RepositoryVersion, Publication
<span class="p">@@ -22,7 +22,7 @@</span> from pulpcore.plugin.viewsets import (
from . import tasks
from .models import AnsibleRemote, AnsibleRole, AnsibleRoleVersion
from .serializers import (AnsibleRemoteSerializer, AnsibleRoleSerializer,
<span class="gd">- AnsibleRoleVersionSerializer)
</span><span class="gi">+ AnsibleRoleVersionSerializer, OneShotUploadSerializer)
</span>
class AnsibleRoleFilter(BaseFilterSet):
<span class="p">@@ -188,3 +188,32 @@</span> class AnsiblePublicationsViewSet(NamedModelViewSet,
}
)
return OperationPostponedResponse(result, request)
<span class="gi">+
+
+class OneShotUploadView(views.APIView):
+ """
+ ViewSet for One Shot Upload API.
+ """
+
+ @transaction.atomic
+ def post(self, request):
+ """Upload an Ansible Role."""
+ serializer = OneShotUploadSerializer(
+ data=request.data, context={'request': request})
+ serializer.is_valid(raise_exception=True)
+ data = serializer.validated_data
+ expected_digests = {'sha256': data['sha256']}
+
+ artifact = Artifact.init_and_validate(request.data['file'],
+ expected_digests=expected_digests)
+ artifact.save()
+
+ repository = data['repository']
+ async_result = enqueue_with_reservation(
+ tasks.import_content_from_tarball, [repository],
+ kwargs={
+ 'namespace': request.user.username,
+ 'artifact_pk': artifact.pk,
+ 'repository_pk': repository.pk
+ })
+ return OperationPostponedResponse(async_result, request)
</span></code></pre> Ansible Plugin - Story #4066: Add a role support to existing one-shot uploader for pulp_ansiblehttps://pulp.plan.io/issues/4066?journal_id=459002019-07-22T11:27:54Zrchan
<ul></ul><p>Since this issue is blocked by a new feature/change in Galaxy, can we add a tracker.discussion in the Ansible Galaxy project that can be added to this issue indicating that relationship?</p> Ansible Plugin - Story #4066: Add a role support to existing one-shot uploader for pulp_ansiblehttps://pulp.plan.io/issues/4066?journal_id=459012019-07-22T12:09:21Zbmbouterbmbouter@redhat.com
<ul></ul><p>My perspective on what is blocking this issue isn't a Galaxy change, it's that a user or stakeholder hasn't prioritized it because it's "role" content (where the focus currently is "collection" content). If it did have a champion, we could move forward without any code changes in external projects by advising pulp users to add a 'version' to their role metadata.</p>
<p>I want to share some background on the placeholders idea. I agree we don't really have a good way to indicate what this ticket is waiting on and a tracker would do that. We used to have the 'External' Redmine project to have placeholders like that. We ran into a challenge with those placeholders where they got unblocked but never updated in Redmine so we mistakenly thought unblocked work was blocked. At some point on pulp-dev to delete the External Redmine project and instead link to external to Redmine where the work is happening (fedora bugs, upstream bugs, etc). Thoughts or suggestions on how to make this better are welcome.</p> Ansible Plugin - Story #4066: Add a role support to existing one-shot uploader for pulp_ansiblehttps://pulp.plan.io/issues/4066?journal_id=772002021-11-17T19:42:56Zpulpbot
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/77200/diff?detail_id=77588">diff</a>)</li><li><strong>Status</strong> changed from <i>NEW</i> to <i>CLOSED - DUPLICATE</i></li></ul>