Project

Profile

Help

Story #3202

closed

As a user, I can sync RPM/SRPM/Erratum from a remote Yum/DNF repository

Added by ttereshc almost 7 years ago. Updated almost 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
(Total: 0:00 h)
Platform Release:
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Sprint 42
Quarter:

Description

As a user, I can sync RPM/SRPM/Erratum from a remote Yum/DNF repository.

Other unknown types of a content should be ignored during sync.

This story is complete when:

  • I can initiate a sync from remote Yum/DNF repo
  • The sync completes without error
  • I can see that the expected content was added to the repo

Pseudo Code First Stage Example

The sync will create a Stages API first stage and then pass all of them to DeclarativeVersion. Here is some pseudocode that can run inside of the __call__() of the first stage.

with ProgressBar(message='Downloading Metadata') as pb:
            repomd_url = url = self.remote.url + '/repodata/repomd.xml'
            downloader = self.remote.get_downloader(repomd_url)  # get the downloader
            result = await downloader.run()  # downloading happens here
            pb.increment()  # tell the user we downloading something

            other_metadata_files = get_other_metadata_files_as_list(result)  # gives urls of comps, updateinfo, etc.
            downloaders = []
            for url in other_metadata_files:
                       downloader = self.remote.get_downloader(repomd_url)  # get the downloader
                       downloaders.append(downloader.run())  # This only creates the coroutine, it doesn't run it

            while downloaders:
                       done, downloaders = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED)
                       for finished_downloader in done:
                                     pb.increment()  # tell the user we downloading something
                                     in_memory_parsed_metadata = createrepo_c.load_xml(finished_downloader.path)
                                     for package in in_memory_parsed_metadata:
                                                 if is_updaterecord(package):
                                                              content = UpdateRecord(content_data_from_package)
                                                              artifact = Artifact(digest_data_from_package)
                                                              da = DeclarativeArtifact(artifact, url, entry.relative_path, self.remote)
                                                              dc = DeclarativeContent(content=file, d_artifacts=[da])
                                                              await out_q.put(dc)
                                                 if is_packagerecord(package):
                                                              #  Similar to what we did  only using RpmContent / SrpmContent

The call function will become long, so consider breaking it up into additional coroutine methods on the object. For example if we made a build_da_dc_and_emit() coroutine it would be defined like:

async def build_da_dc_and_emit(self, content, artifact):
               da = DeclarativeArtifact(artifact, url, entry.relative_path, self.remote)
               dc = DeclarativeContent(content=file, d_artifacts=[da])
               await out_q.put(dc)

# Then in your code call it with:
await self.build_da_dc_and_emit(content, artifact)

Sub-issues 3 (0 open3 closed)

Task #3933: Save UpdateCollection and UpdateCollectionPackage in bulkCLOSED - CURRENTRELEASEbmbouter

Actions
Pulp - Story #3934: As a plugin writer, I can have a stage that removes duplicatesCLOSED - CURRENTRELEASEamacdona@redhat.com

Actions
Task #3942: Store a digest of the fields for an UpdateRecordCLOSED - DUPLICATE

Actions

Related issues

Related to RPM Support - Test #4108: Test syncing RPM/DRPM/SRPM/ErratumCLOSED - COMPLETEActions
Blocked by RPM Support - Task #3199: Create model(s) for a Package content typeCLOSED - CURRENTRELEASEdalley

Actions
Actions #1

Updated by ttereshc almost 7 years ago

  • Sprint/Milestone set to Pulp 3 RPM MVP
Actions #2

Updated by ttereshc almost 7 years ago

  • Blocked by Task #3199: Create model(s) for a Package content type added
Actions #4

Updated by ttereshc almost 7 years ago

  • Subject changed from As a user, I can sync RPM/SRPM from a remote Yum/DNF repository to As a user, I can sync RPM/SRPM/Erratum from a remote Yum/DNF repository
Actions #5

Updated by ttereshc almost 7 years ago

  • Description updated (diff)
Actions #6

Updated by ttereshc over 6 years ago

  • Sprint set to Sprint 37
Actions #7

Updated by rchan over 6 years ago

  • Sprint changed from Sprint 37 to Sprint 38
Actions #8

Updated by rchan over 6 years ago

  • Sprint changed from Sprint 38 to Sprint 39
Actions #9

Updated by dkliban@redhat.com over 6 years ago

  • Sprint deleted (Sprint 39)
Actions #10

Updated by ttereshc over 6 years ago

  • Sprint set to Sprint 41
Actions #11

Updated by bmbouter over 6 years ago

  • Description updated (diff)

Posting some sync pseudocode

Actions #12

Updated by bmbouter over 6 years ago

  • Description updated (diff)

Adding an example about breaking the code up into smaller coroutines.

Actions #13

Updated by milan over 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to milan
Actions #14

Updated by milan over 6 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (milan)
Actions #15

Updated by ttereshc over 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc

Added by daviddavis over 6 years ago

Revision d9f95bd8 | View on GitHub

Initial pass at syncing rpm and erratum content

This handles creating packages and update records from an RPM remote. UpdateCollections and their packages are not included.

ref #3202 https://pulp.plan.io/issues/3202

Added by daviddavis over 6 years ago

Revision fcd156b3 | View on GitHub

Parsing erratum relationships and storing them on the model

ref #3202 https://pulp.plan.io/issues/3202

Actions #17

Updated by rchan over 6 years ago

  • Sprint changed from Sprint 41 to Sprint 42
Actions #18

Updated by daviddavis about 6 years ago

  • Status changed from ASSIGNED to MODIFIED
Actions #19

Updated by kersom about 6 years ago

  • Related to Test #4108: Test syncing RPM/DRPM/SRPM/Erratum added
Actions #20

Updated by bmbouter over 5 years ago

  • Tags deleted (Pulp 3)
Actions #21

Updated by ttereshc almost 5 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF