Project

Profile

Help

Story #3202

closed

As a user, I can sync RPM/SRPM/Erratum from a remote Yum/DNF repository

Added by ttereshc almost 6 years ago. Updated almost 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
(Total: 0:00 h)
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 42
Quarter:

Description

As a user, I can sync RPM/SRPM/Erratum from a remote Yum/DNF repository.

Other unknown types of a content should be ignored during sync.

This story is complete when:

  • I can initiate a sync from remote Yum/DNF repo
  • The sync completes without error
  • I can see that the expected content was added to the repo

Pseudo Code First Stage Example

The sync will create a Stages API first stage and then pass all of them to DeclarativeVersion. Here is some pseudocode that can run inside of the __call__() of the first stage.

with ProgressBar(message='Downloading Metadata') as pb:
            repomd_url = url = self.remote.url + '/repodata/repomd.xml'
            downloader = self.remote.get_downloader(repomd_url)  # get the downloader
            result = await downloader.run()  # downloading happens here
            pb.increment()  # tell the user we downloading something

            other_metadata_files = get_other_metadata_files_as_list(result)  # gives urls of comps, updateinfo, etc.
            downloaders = []
            for url in other_metadata_files:
                       downloader = self.remote.get_downloader(repomd_url)  # get the downloader
                       downloaders.append(downloader.run())  # This only creates the coroutine, it doesn't run it

            while downloaders:
                       done, downloaders = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED)
                       for finished_downloader in done:
                                     pb.increment()  # tell the user we downloading something
                                     in_memory_parsed_metadata = createrepo_c.load_xml(finished_downloader.path)
                                     for package in in_memory_parsed_metadata:
                                                 if is_updaterecord(package):
                                                              content = UpdateRecord(content_data_from_package)
                                                              artifact = Artifact(digest_data_from_package)
                                                              da = DeclarativeArtifact(artifact, url, entry.relative_path, self.remote)
                                                              dc = DeclarativeContent(content=file, d_artifacts=[da])
                                                              await out_q.put(dc)
                                                 if is_packagerecord(package):
                                                              #  Similar to what we did  only using RpmContent / SrpmContent

The call function will become long, so consider breaking it up into additional coroutine methods on the object. For example if we made a build_da_dc_and_emit() coroutine it would be defined like:

async def build_da_dc_and_emit(self, content, artifact):
               da = DeclarativeArtifact(artifact, url, entry.relative_path, self.remote)
               dc = DeclarativeContent(content=file, d_artifacts=[da])
               await out_q.put(dc)

# Then in your code call it with:
await self.build_da_dc_and_emit(content, artifact)

Subtasks 3 (0 open3 closed)

Task #3933: Save UpdateCollection and UpdateCollectionPackage in bulkCLOSED - CURRENTRELEASEbmbouter

Actions
Pulp - Story #3934: As a plugin writer, I can have a stage that removes duplicatesCLOSED - CURRENTRELEASEamacdona@redhat.com

Actions
Task #3942: Store a digest of the fields for an UpdateRecordCLOSED - DUPLICATE

Actions

Related issues

Related to RPM Support - Test #4108: Test syncing RPM/DRPM/SRPM/ErratumCLOSED - COMPLETEActions
Blocked by RPM Support - Task #3199: Create model(s) for a Package content typeCLOSED - CURRENTRELEASEdalley

Actions
Actions #1

Updated by ttereshc almost 6 years ago

  • Sprint/Milestone set to Pulp 3 RPM MVP
Actions #2

Updated by ttereshc almost 6 years ago

  • Blocked by Task #3199: Create model(s) for a Package content type added
Actions #4

Updated by ttereshc over 5 years ago

  • Subject changed from As a user, I can sync RPM/SRPM from a remote Yum/DNF repository to As a user, I can sync RPM/SRPM/Erratum from a remote Yum/DNF repository
Actions #5

Updated by ttereshc over 5 years ago

  • Description updated (diff)
Actions #6

Updated by ttereshc over 5 years ago

  • Sprint set to Sprint 37
Actions #7

Updated by rchan over 5 years ago

  • Sprint changed from Sprint 37 to Sprint 38
Actions #8

Updated by rchan over 5 years ago

  • Sprint changed from Sprint 38 to Sprint 39
Actions #9

Updated by dkliban@redhat.com about 5 years ago

  • Sprint deleted (Sprint 39)
Actions #10

Updated by ttereshc about 5 years ago

  • Sprint set to Sprint 41
Actions #11

Updated by bmbouter about 5 years ago

  • Description updated (diff)

Posting some sync pseudocode

Actions #12

Updated by bmbouter about 5 years ago

  • Description updated (diff)

Adding an example about breaking the code up into smaller coroutines.

Actions #13

Updated by milan about 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to milan
Actions #14

Updated by milan about 5 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (milan)
Actions #15

Updated by ttereshc about 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc

Added by daviddavis about 5 years ago

Revision d9f95bd8

Initial pass at syncing rpm and erratum content

This handles creating packages and update records from an RPM remote. UpdateCollections and their packages are not included.

ref #3202 https://pulp.plan.io/issues/3202

Added by daviddavis about 5 years ago

Revision fcd156b3

Parsing erratum relationships and storing them on the model

ref #3202 https://pulp.plan.io/issues/3202

Actions #17

Updated by rchan about 5 years ago

  • Sprint changed from Sprint 41 to Sprint 42
Actions #18

Updated by daviddavis about 5 years ago

  • Status changed from ASSIGNED to MODIFIED
Actions #19

Updated by kersom almost 5 years ago

  • Related to Test #4108: Test syncing RPM/DRPM/SRPM/Erratum added
Actions #20

Updated by bmbouter over 4 years ago

  • Tags deleted (Pulp 3)
Actions #21

Updated by ttereshc almost 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF