Story #3202
As a user, I can sync RPM/SRPM/Erratum from a remote Yum/DNF repository
67%
Description
As a user, I can sync RPM/SRPM/Erratum from a remote Yum/DNF repository.
Other unknown types of a content should be ignored during sync.
This story is complete when:
- I can initiate a sync from remote Yum/DNF repo
- The sync completes without error
- I can see that the expected content was added to the repo
Pseudo Code First Stage Example¶
The sync will create a Stages API first stage and then pass all of them to DeclarativeVersion. Here is some pseudocode that can run inside of the __call__()
of the first stage.
with ProgressBar(message='Downloading Metadata') as pb:
repomd_url = url = self.remote.url + '/repodata/repomd.xml'
downloader = self.remote.get_downloader(repomd_url) # get the downloader
result = await downloader.run() # downloading happens here
pb.increment() # tell the user we downloading something
other_metadata_files = get_other_metadata_files_as_list(result) # gives urls of comps, updateinfo, etc.
downloaders = []
for url in other_metadata_files:
downloader = self.remote.get_downloader(repomd_url) # get the downloader
downloaders.append(downloader.run()) # This only creates the coroutine, it doesn't run it
while downloaders:
done, downloaders = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED)
for finished_downloader in done:
pb.increment() # tell the user we downloading something
in_memory_parsed_metadata = createrepo_c.load_xml(finished_downloader.path)
for package in in_memory_parsed_metadata:
if is_updaterecord(package):
content = UpdateRecord(content_data_from_package)
artifact = Artifact(digest_data_from_package)
da = DeclarativeArtifact(artifact, url, entry.relative_path, self.remote)
dc = DeclarativeContent(content=file, d_artifacts=[da])
await out_q.put(dc)
if is_packagerecord(package):
# Similar to what we did only using RpmContent / SrpmContent
The call function will become long, so consider breaking it up into additional coroutine methods on the object. For example if we made a build_da_dc_and_emit() coroutine it would be defined like:
async def build_da_dc_and_emit(self, content, artifact):
da = DeclarativeArtifact(artifact, url, entry.relative_path, self.remote)
dc = DeclarativeContent(content=file, d_artifacts=[da])
await out_q.put(dc)
# Then in your code call it with:
await self.build_da_dc_and_emit(content, artifact)
Subtasks
Related issues
Associated revisions
Revision fcd156b3
View on GitHub
Parsing erratum relationships and storing them on the model
History
#1
Updated by ttereshc about 3 years ago
- Sprint/Milestone set to Pulp 3 RPM MVP
#2
Updated by ttereshc about 3 years ago
- Blocked by Task #3199: Create model(s) for a Package content type added
#4
Updated by ttereshc almost 3 years ago
- Subject changed from As a user, I can sync RPM/SRPM from a remote Yum/DNF repository to As a user, I can sync RPM/SRPM/Erratum from a remote Yum/DNF repository
#5
Updated by ttereshc almost 3 years ago
- Description updated (diff)
#6
Updated by ttereshc over 2 years ago
- Sprint set to Sprint 37
#7
Updated by rchan over 2 years ago
- Sprint changed from Sprint 37 to Sprint 38
#8
Updated by rchan over 2 years ago
- Sprint changed from Sprint 38 to Sprint 39
#9
Updated by dkliban@redhat.com over 2 years ago
- Sprint deleted (
Sprint 39)
#10
Updated by ttereshc over 2 years ago
- Sprint set to Sprint 41
#12
Updated by bmbouter over 2 years ago
- Description updated (diff)
Adding an example about breaking the code up into smaller coroutines.
#13
Updated by milan over 2 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to milan
#14
Updated by milan over 2 years ago
- Status changed from ASSIGNED to NEW
- Assignee deleted (
milan)
#15
Updated by ttereshc over 2 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to ttereshc
#16
Updated by ttereshc over 2 years ago
#17
Updated by rchan over 2 years ago
- Sprint changed from Sprint 41 to Sprint 42
#18
Updated by daviddavis over 2 years ago
- Status changed from ASSIGNED to MODIFIED
#19
Updated by kersom about 2 years ago
- Related to Test #4108: Test syncing RPM/DRPM/SRPM/Erratum added
#20
Updated by bmbouter over 1 year ago
- Tags deleted (
Pulp 3)
#21
Updated by ttereshc about 1 year ago
- Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Please register to edit this issue
Initial pass at syncing rpm and erratum content
This handles creating packages and update records from an RPM remote. UpdateCollections and their packages are not included.
ref #3202 https://pulp.plan.io/issues/3202