Story #3209: As a user, I have Repository Versions
As a plugin writer, I have a tool that helps me write tasks that create RepositoryVersions
Tasks that create new RepositoryVersions have to write to pulpcore tables in transactions, set up a working directory, finalize the new version and clean up the database if the task fails. All of this should be handled by a context manager that is a part of the plugin API.
Proposal: Wrapper for the RepositoryVersion object in the plugin API¶
The object can be used as a context manager that creates RepositoryVersions. The pulpcore model for RepositoryVersion will be removed from the plugin API, so the wrapper will be the only supported way for plugin writers to create new RepositoryVersions, and must be performed only in celery tasks. It will contain all business logic for creation, as well as adding and removing content. The wrapper will be passed to the changeset.
Create a new RepositoryVersion. Plugin writers can use this object to add/remove content. Or they can pass it to the ChangeSet.
with RepositoryVersion.create(repository) as created_version: created_version.add_content(some_content) changeset = ChangeSet(created_version, ...)
The wrapper object can also be used to access the RepositoryVersion model layer, which replaces the use of plugin.models.RepositoryVersion.objects.get() with:
latest_version = RepositoryVersion.latest(repository) for content in latest_version.content():
The wrapper will provide any functions that plugin writers need to act safely upon the RepositoryVersion and related tables. It will enforce correct usage (and therefore RepositoryVersion immutability) by:
- ensuring it is only run in tasks
- exclusive control of the RepositoryVersion.complete flag
- only allowing repository versions to add and remove content if RepositoryVersion.complete = True
- using transactions to create a new version and update related tables
- handle failure scenarios by cleaning up incomplete work
The business logic that will live in this class is psuedo-implemented in this comment: https://pulp.plan.io/issues/3285#note-6
Working code that will be moved to this class currently lives in:
- the File plugin's sync https://github.com/asmacdo/pulp_file/blob/a8997383e2a64f915769db629435ae4abecf576c/pulp_file/app/tasks.py#L52-L79
- RepositoryVersion django model in models.repository.RepositoryVersion
Review of plugin API changes¶
The RepositoryVersion django model (models.repository.RepositoryVersion) will be removed from the plugin API. Plugin writers should use the RepositoryVersion wrapper/facade instead of the model directly. New/incomplete repository versions will be used exclusively within the context_manager. The ChangeSet will no longer accept a models.repository.RepositoryVersion object, and will instead accept an instance of the class created by this story.
The changes above allow us to make the following promise to users: "Once a RepositoryVersion is marked complete, it cannot be altered by any plugin."
They also limit the access to the pulpcore data model, which reduces the likelihood of plugin writer errors. We can promise the plugin writer:
- When dealing with new RepositoryVersions, your code will either run successfully or the pulpcore data layer will be cleaned up.
- All pulpcore actions that must take place together in transactions is handled for you under the hood.
- The object you have access to is guarded against unsafe use.
#6 Updated by firstname.lastname@example.org about 4 years ago
This might be a good fit for the Facade design pattern. The goal is to provide the plugin writer a domain object that can be safely used like a RepositoryVersion but encapsulates the complexity of managing them. By doing so we avoid adding methods that manage other models on the RepositoryVersion model goes against django best practices. And, provides an intuitive and appropriate place for the context manger.
The plugin API could include something like this in a module named repository.py (see similar domain object in tasking.py).
from contextlib import suppress from gettext import gettext from pulpcore.app import models from pulpcore.plugin.tasking import Task class RepositoryVersion: @classmethod def latest(cls, repository): model = repository.versions.exclude(complete=False).latest() return RepositoryVersionFacade(model) @classmethod def create(cls, repository): with transaction.atomic(): version = models.RepositoryVersion( repository=repository, number=repository.last_version + 1) repository.last_version = version.number repository.save() version.save() resource = models.CreatedResource(content_object=version) resource.save() return cls(version) def __init__(self, model): self._model = model @property def content(self): # moved from RepositoryVersion model. associations = models.RepositoryContent.objects.filter( repository=self._model.repository, version_added__number__lte=self.number).exclude( version_removed__number__lte=self.number) content_model = self._repository.content.model return content_model.objects.filter(version_memberships__in=associations) def add_content(self, content): # moved from RepositoryVersion model. assert not self._model.complete, _('Complete repository version is immutable.') with suppress(IntegrityError): association = RepositoryContent( repository=self._model.repository, content=content, version_added=self._model) association.save() def remove_content(self, content): # moved from RepositoryVersion model. assert not self._model.complete, _('Complete repository version is immutable.') q_set = models.RepositoryContent.objects.filter( repository=self._model.repository, content=content, version_removed=None) q_set.update(version_removed=self._model) def complete(self): self._model.complete = True self._model.save() def delete(self): with transaction.atomic(): models.CreatedResource.objcets.delete(content_object=self._model) self._model.delete() def __enter__(self): return self def __exit__(self, exc_type, exc_value, traceback): if exc_value: self.delete() # List content on latest version. latest_version = RepositoryVersion.latest(repository) for content in latest_version.content(): ... # Create new version. with RepositoryVersion.create(repository) as created_version: changeset = ChangeSet(created_version, ...)
#7 Updated by email@example.com about 4 years ago
I'd like this to also include CreatedResource.
Since RepositoryVersions should always be created in a Task, it will always need a CreatedResource. Also, if things go badly, the CreatedResource needs to be cleaned up at the same time as the RepositoryVersion.