Project

Profile

Help

Story #3285

closed

Story #3209: As a user, I have Repository Versions

As a plugin writer, I have a tool that helps me write tasks that create RepositoryVersions

Added by amacdona@redhat.com almost 7 years ago. Updated over 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

0%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Sprint 31
Quarter:

Description

Tasks that create new RepositoryVersions have to write to pulpcore tables in transactions, set up a working directory, finalize the new version and clean up the database if the task fails. All of this should be handled by a context manager that is a part of the plugin API.

Proposal: Wrapper for the RepositoryVersion object in the plugin API

The object can be used as a context manager that creates RepositoryVersions. The pulpcore model for RepositoryVersion will be removed from the plugin API, so the wrapper will be the only supported way for plugin writers to create new RepositoryVersions, and must be performed only in celery tasks. It will contain all business logic for creation, as well as adding and removing content. The wrapper will be passed to the changeset.

example usage

Create a new RepositoryVersion. Plugin writers can use this object to add/remove content. Or they can pass it to the ChangeSet.


with RepositoryVersion.create(repository) as created_version:
    created_version.add_content(some_content)
    changeset = ChangeSet(created_version, ...)

The wrapper object can also be used to access the RepositoryVersion model layer, which replaces the use of plugin.models.RepositoryVersion.objects.get() with:


latest_version = RepositoryVersion.latest(repository)
for content in latest_version.content():

The wrapper will provide any functions that plugin writers need to act safely upon the RepositoryVersion and related tables. It will enforce correct usage (and therefore RepositoryVersion immutability) by:

  1. ensuring it is only run in tasks
  2. exclusive control of the RepositoryVersion.complete flag
  3. only allowing repository versions to add and remove content if RepositoryVersion.complete = True
  4. using transactions to create a new version and update related tables
  5. handle failure scenarios by cleaning up incomplete work

The business logic that will live in this class is psuedo-implemented in this comment: https://pulp.plan.io/issues/3285#note-6

Working code that will be moved to this class currently lives in:

  1. the File plugin's sync https://github.com/asmacdo/pulp_file/blob/a8997383e2a64f915769db629435ae4abecf576c/pulp_file/app/tasks.py#L52-L79
  2. RepositoryVersion django model in models.repository.RepositoryVersion

Review of plugin API changes

The RepositoryVersion django model (models.repository.RepositoryVersion) will be removed from the plugin API. Plugin writers should use the RepositoryVersion wrapper/facade instead of the model directly. New/incomplete repository versions will be used exclusively within the context_manager. The ChangeSet will no longer accept a models.repository.RepositoryVersion object, and will instead accept an instance of the class created by this story.

The changes above allow us to make the following promise to users: "Once a RepositoryVersion is marked complete, it cannot be altered by any plugin."

They also limit the access to the pulpcore data model, which reduces the likelihood of plugin writer errors. We can promise the plugin writer:

  1. When dealing with new RepositoryVersions, your code will either run successfully or the pulpcore data layer will be cleaned up.
  2. All pulpcore actions that must take place together in transactions is handled for you under the hood.
  3. The object you have access to is guarded against unsafe use.

Related issues

Copied to Pulp - Story #3295: As a plugin writer, I have a tool that helps me write tasks that create PublicationsCLOSED - CURRENTRELEASEjortel@redhat.com

Actions
Actions #1

Updated by amacdona@redhat.com almost 7 years ago

  • Tracker changed from Issue to Story
  • % Done set to 0
Actions #2

Updated by amacdona@redhat.com almost 7 years ago

  • Description updated (diff)
Actions #3

Updated by amacdona@redhat.com almost 7 years ago

  • Parent issue set to #3209
Actions #4

Updated by amacdona@redhat.com almost 7 years ago

  • Copied to Story #3295: As a plugin writer, I have a tool that helps me write tasks that create Publications added
Actions #5

Updated by daviddavis almost 7 years ago

+1 to this task. We should also use these tools internally for #3234.

Actions #6

Updated by jortel@redhat.com almost 7 years ago

This might be a good fit for the Facade[1] design pattern. The goal is to provide the plugin writer a domain object that can be safely used like a RepositoryVersion but encapsulates the complexity of managing them. By doing so we avoid adding methods that manage other models on the RepositoryVersion model goes against django best practices. And, provides an intuitive and appropriate place for the context manger.

The plugin API could include something like this in a module named repository.py (see similar domain object in tasking.py).

from contextlib import suppress
from gettext import gettext

from pulpcore.app import models
from pulpcore.plugin.tasking import Task

class RepositoryVersion:

    @classmethod
    def latest(cls, repository):
        model = repository.versions.exclude(complete=False).latest()
        return RepositoryVersionFacade(model)

    @classmethod
    def create(cls, repository):
        with transaction.atomic():
            version = models.RepositoryVersion(
                repository=repository,
                number=repository.last_version + 1)
            repository.last_version = version.number
            repository.save()
            version.save()
            resource = models.CreatedResource(content_object=version)
            resource.save()
            return cls(version)

    def __init__(self, model):
        self._model = model

    @property
    def content(self):
        # moved from RepositoryVersion model.
        associations = models.RepositoryContent.objects.filter(
            repository=self._model.repository,
            version_added__number__lte=self.number).exclude(
            version_removed__number__lte=self.number)
        content_model = self._repository.content.model
        return content_model.objects.filter(version_memberships__in=associations)

    def add_content(self, content):
        # moved from RepositoryVersion model.
        assert not self._model.complete, _('Complete repository version is immutable.')
        with suppress(IntegrityError):
            association = RepositoryContent(
                repository=self._model.repository,
                content=content,
                version_added=self._model)
            association.save()

    def remove_content(self, content):
        # moved from RepositoryVersion model.
        assert not self._model.complete, _('Complete repository version is immutable.')
        q_set = models.RepositoryContent.objects.filter(
            repository=self._model.repository,
            content=content,
            version_removed=None)
        q_set.update(version_removed=self._model)

    def complete(self):
        self._model.complete = True
        self._model.save()

    def delete(self):
        with transaction.atomic():
            models.CreatedResource.objcets.delete(content_object=self._model)
            self._model.delete()

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        if exc_value:
            self.delete()

# List content on latest version.
latest_version = RepositoryVersion.latest(repository)
for content in latest_version.content():
    ...

# Create new version.
with RepositoryVersion.create(repository) as created_version:
    changeset = ChangeSet(created_version, ...)

[1] https://en.wikipedia.org/wiki/Facade_pattern

Actions #7

Updated by amacdona@redhat.com almost 7 years ago

I'd like this to also include CreatedResource.

Since RepositoryVersions should always be created in a Task, it will always need a CreatedResource. Also, if things go badly, the CreatedResource needs to be cleaned up at the same time as the RepositoryVersion.

Actions #8

Updated by jortel@redhat.com almost 7 years ago

  • Groomed changed from No to Yes
  • Tags Pulp 3 added
Actions #9

Updated by amacdona@redhat.com almost 7 years ago

  • Description updated (diff)
  • Sprint/Milestone set to 53
  • Sprint Candidate changed from No to Yes
Actions #10

Updated by bizhang almost 7 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bizhang

Added by werwty almost 7 years ago

Revision 7e9205d4 | View on GitHub

Working sync with repoversion context manager

re #3285 https://pulp.plan.io/issues/3285

Added by werwty almost 7 years ago

Revision 13a28151 | View on GitHub

Refactor RepositoryVersion context manager, so it lives in pulpcore. Bring back the RepositoryVersion create endpoint.

re #3285 https://pulp.plan.io/issues/3285

Added by werwty almost 7 years ago

Revision 13a28151 | View on GitHub

Refactor RepositoryVersion context manager, so it lives in pulpcore. Bring back the RepositoryVersion create endpoint.

re #3285 https://pulp.plan.io/issues/3285

Actions #11

Updated by bizhang almost 7 years ago

  • Status changed from ASSIGNED to MODIFIED
Actions #12

Updated by bmbouter over 6 years ago

  • Sprint set to Sprint 31
Actions #13

Updated by bmbouter over 6 years ago

  • Sprint/Milestone deleted (53)
Actions #14

Updated by daviddavis over 5 years ago

  • Sprint/Milestone set to 3.0.0
Actions #15

Updated by bmbouter over 5 years ago

  • Tags deleted (Pulp 3)
Actions #16

Updated by bmbouter almost 5 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF