Project

Profile

Help

Story #5096

[epic] As a user, I can export the content of a RepositoryVersion from one Pulp3 system and import on an air gapped Pulp3 system

Added by bmbouter 3 months ago. Updated 29 days ago.

Status:
NEW
Priority:
Normal
Assignee:
-
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

0%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2 to 3 Migrate
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:

Description

Problem

The Pulp3-to-Pulp3 sync mechanisms are typically one Pulp syncing from another Pulp over the network. When the two Pulp3 systems are air gapped this sync mechanism won't work.

Plugins could make individual import/export functions, but can pulpcore do it once for all Content types?

Use Case: Full Export+Import

As a Pulp3 user I can export all of the content from one RepositoryVersion to a disk. Then walk that disk into the air gapped environment and mount it. Add the RepositoryVersion to an air gapped Pulp as part of a specified, existing Repository.

Use Case: Incremental Export+Import

As a Pulp3 user who has exported a RepositoryVersion earlier, I can specify that RepositoryVersion and it will export the repository containing all content differences between that RepositoryVersion and the latest version of that Repository.

Challenges

Artifact and ContentArtifact objects

The Content units created need to also create corresponding Artifact and ContentArtifact objects. Artifacts already existing on the import system should not be disturbed but used.

Foreign Keys to/from Content Model

Content units sometimes have other non-Content models that are related to them, e.g. in pulp_rpm the UpdateRecord has related UpdateCollection and UpdateCollectionPackage records. These may or may not need to be duplicated and likely need plugin writers to be involved.

Solution

Export

Make an export API that takes a 'repository_version' and it will generate a task that will create the export archive for the user. All Content types in the repository version, their Artifacts, and ContentArtifact objects are exported with django-import-export. This will make the file available for downloading via the pulp-content app somehow as a single archive.

Additionally the Export takes a 'prev_export_repository_version' option which will be used as the diff for all content between 'prev_export_repository_version' and 'repository_version'.

Import

Using the upload API the tarball can be uploaded in parallel efficiently. An import API that takes a 'repository' and it will generate a task that will import the archive for the user.

Technical Plan

To use django-import-export, plugin writers would provide a customized resource for their Content model and this would be put into a resources.py and imported by the Django plugin loader causing it to import for each plugin.

If the plugin writer doesn't provide one and the generic one will work for them, they do nothing, and pulpcore will create a generic one at Runtime

pulpcore would also provide Resources for Artifact and ContentArtifact which the import task would also handle.

Open Questions

How will users at import time specify if they want dry-run or not?


Related issues

Duplicated by Ansible Plugin - Story #5088: support ansible collection content export CLOSED - DUPLICATE Actions

History

#1 Updated by bmbouter 3 months ago

  • Description updated (diff)

#2 Updated by bmbouter 3 months ago

  • Description updated (diff)

#3 Updated by daviddavis 3 months ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#4 Updated by rchan 3 months ago

  • Sprint set to Sprint 56

#5 Updated by bmbouter 3 months ago

  • Duplicated by Story #5088: support ansible collection content export added

#6 Updated by bmbouter 3 months ago

  • Tags Katello-P2 added

This duplicates a Katello P2 ( https://pulp.plan.io/issues/5088. ), so this should receive the Katello P2 label.

#7 Updated by rchan 3 months ago

  • Sprint changed from Sprint 56 to Sprint 57

#8 Updated by rchan about 2 months ago

  • Sprint changed from Sprint 57 to Sprint 58

#9 Updated by rchan 30 days ago

  • Sprint changed from Sprint 58 to Sprint 59

#10 Updated by daviddavis 29 days ago

Removing from the sprint since this is not a 3.0 blocker.

#11 Updated by daviddavis 29 days ago

  • Sprint deleted (Sprint 59)

#12 Updated by daviddavis 29 days ago

  • Tags Pulp 2 to 3 Migrate added
  • Tags deleted (Katello-P2)

Please register to edit this issue

Also available in: Atom PDF