Project

Profile

Help

Story #3202

As a user, I can sync RPM/SRPM/Erratum from a remote Yum/DNF repository

Added by ttereshc about 2 years ago. Updated about 1 month ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

67%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
No
Sprint Candidate:
No
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 42

Description

As a user, I can sync RPM/SRPM/Erratum from a remote Yum/DNF repository.

Other unknown types of a content should be ignored during sync.

This story is complete when:

  • I can initiate a sync from remote Yum/DNF repo
  • The sync completes without error
  • I can see that the expected content was added to the repo

Pseudo Code First Stage Example

The sync will create a Stages API first stage and then pass all of them to DeclarativeVersion. Here is some pseudocode that can run inside of the __call__() of the first stage.

with ProgressBar(message='Downloading Metadata') as pb:
            repomd_url = url = self.remote.url + '/repodata/repomd.xml'
            downloader = self.remote.get_downloader(repomd_url)  # get the downloader
            result = await downloader.run()  # downloading happens here
            pb.increment()  # tell the user we downloading something

            other_metadata_files = get_other_metadata_files_as_list(result)  # gives urls of comps, updateinfo, etc.
            downloaders = []
            for url in other_metadata_files:
                       downloader = self.remote.get_downloader(repomd_url)  # get the downloader
                       downloaders.append(downloader.run())  # This only creates the coroutine, it doesn't run it

            while downloaders:
                       done, downloaders = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED)
                       for finished_downloader in done:
                                     pb.increment()  # tell the user we downloading something
                                     in_memory_parsed_metadata = createrepo_c.load_xml(finished_downloader.path)
                                     for package in in_memory_parsed_metadata:
                                                 if is_updaterecord(package):
                                                              content = UpdateRecord(content_data_from_package)
                                                              artifact = Artifact(digest_data_from_package)
                                                              da = DeclarativeArtifact(artifact, url, entry.relative_path, self.remote)
                                                              dc = DeclarativeContent(content=file, d_artifacts=[da])
                                                              await out_q.put(dc)
                                                 if is_packagerecord(package):
                                                              #  Similar to what we did  only using RpmContent / SrpmContent

The call function will become long, so consider breaking it up into additional coroutine methods on the object. For example if we made a build_da_dc_and_emit() coroutine it would be defined like:

async def build_da_dc_and_emit(self, content, artifact):
               da = DeclarativeArtifact(artifact, url, entry.relative_path, self.remote)
               dc = DeclarativeContent(content=file, d_artifacts=[da])
               await out_q.put(dc)

# Then in your code call it with:
await self.build_da_dc_and_emit(content, artifact)

Subtasks

Task #3933: Save UpdateCollection and UpdateCollectionPackage in bulkCLOSED - CURRENTRELEASEbmbouter

Actions
Pulp - Story #3934: As a plugin writer, I can have a stage that removes duplicatesCLOSED - CURRENTRELEASEamacdona@redhat.com

Actions
Task #3942: Store a digest of the fields for an UpdateRecordNEW

Actions

Related issues

Related to RPM Support - Test #4108: Test syncing RPM/DRPM/SRPM/Erratum ASSIGNED Actions
Blocked by RPM Support - Task #3199: Create model(s) for a Package content type CLOSED - CURRENTRELEASE Actions

Associated revisions

Revision d9f95bd8 View on GitHub
Added by daviddavis over 1 year ago

Initial pass at syncing rpm and erratum content

This handles creating packages and update records from an RPM remote.
UpdateCollections and their packages are not included.

ref #3202
https://pulp.plan.io/issues/3202

Revision fcd156b3 View on GitHub
Added by daviddavis over 1 year ago

Parsing erratum relationships and storing them on the model

ref #3202
https://pulp.plan.io/issues/3202

History

#1 Updated by ttereshc about 2 years ago

  • Sprint/Milestone set to Pulp 3 RPM MVP

#2 Updated by ttereshc about 2 years ago

  • Blocked by Task #3199: Create model(s) for a Package content type added

#4 Updated by ttereshc almost 2 years ago

  • Subject changed from As a user, I can sync RPM/SRPM from a remote Yum/DNF repository to As a user, I can sync RPM/SRPM/Erratum from a remote Yum/DNF repository

#5 Updated by ttereshc almost 2 years ago

  • Description updated (diff)

#6 Updated by ttereshc over 1 year ago

  • Sprint set to Sprint 37

#7 Updated by rchan over 1 year ago

  • Sprint changed from Sprint 37 to Sprint 38

#8 Updated by rchan over 1 year ago

  • Sprint changed from Sprint 38 to Sprint 39

#9 Updated by dkliban@redhat.com over 1 year ago

  • Sprint deleted (Sprint 39)

#10 Updated by ttereshc over 1 year ago

  • Sprint set to Sprint 41

#11 Updated by bmbouter over 1 year ago

  • Description updated (diff)

Posting some sync pseudocode

#12 Updated by bmbouter over 1 year ago

  • Description updated (diff)

Adding an example about breaking the code up into smaller coroutines.

#13 Updated by milan over 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to milan

#14 Updated by milan over 1 year ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (milan)

#15 Updated by ttereshc over 1 year ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc

#17 Updated by rchan over 1 year ago

  • Sprint changed from Sprint 41 to Sprint 42

#18 Updated by daviddavis over 1 year ago

  • Status changed from ASSIGNED to MODIFIED

#19 Updated by kersom about 1 year ago

  • Related to Test #4108: Test syncing RPM/DRPM/SRPM/Erratum added

#20 Updated by bmbouter 9 months ago

  • Tags deleted (Pulp 3)

#21 Updated by ttereshc about 1 month ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF