Project

Profile

Help

Story #2

As a user, my rpm sync finishes quickly when upstream metadata hasn't changed

Added by Anonymous over 5 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.6.2
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
April 2015

Description

The repomd.xml file has its own timestamp in the XML, and then each file it references has its own timestamp. The treeinfo file contains its own timestamp, which already gets checked for freshness. We should check the repomd.xml timestamp and skip the rest of the sync if it hasn't changed. This will reduce the sync time for a big repo from possibly 1-2 minutes to a small number of seconds when the upstream repo hasn't changed. This will also help us be smarter about when the distribution files get re-downloaded.

Deliverables:

  • rpm sync checks the timestamp in repomd.xml, and skips all unit types but "distribution" if it hasn't changed since the last sync.
  • tell Katello team (specifically jsherrill) about this change once it's in Pulp. They might be able to remove some code on their end once this is in place.
  • release notes

Associated revisions

Revision 183cb1eb View on GitHub
Added by mhrivnak over 5 years ago

Sync now skips most steps if the feed metadata has not changed.

closes #2

History

#2 Updated by mhrivnak over 5 years ago

  • Description updated (diff)

#3 Updated by mhrivnak over 5 years ago

  • Tags Sprint Candidate added

#4 Updated by mhrivnak over 5 years ago

  • Priority changed from Normal to High

#5 Updated by cduryee over 5 years ago

  • Description updated (diff)

#6 Updated by mhrivnak over 5 years ago

  • Description updated (diff)
  • Tags Groomed added

#7 Updated by mhrivnak over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to mhrivnak

#8 Updated by mhrivnak over 5 years ago

  • Description updated (diff)

I'm reducing the scope of this story because doing per-unit-type timestamp checking proved to be far more complex than it's worth. We can revisit once this workflow gets stepified, which should make that much more achievable.

#9 Updated by mhrivnak over 5 years ago

  • % Done changed from 0 to 50

#10 Updated by mhrivnak over 5 years ago

  • Status changed from ASSIGNED to POST
  • % Done changed from 50 to 90

#11 Updated by mhrivnak over 5 years ago

  • Platform Release set to 2.6.2

closed above PR, opened this one instead: https://github.com/pulp/pulp_rpm/pull/676

#12 Updated by mhrivnak over 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 90 to 100

#13 Updated by mhrivnak over 5 years ago

  • Sprint/Milestone set to 15

#14 Updated by bmbouter over 5 years ago

  • Groomed set to Yes
  • Tags deleted (Groomed)

#15 Updated by bmbouter over 5 years ago

  • Sprint Candidate set to Yes
  • Tags deleted (Sprint Candidate)

#16 Updated by dkliban@redhat.com about 5 years ago

  • Status changed from MODIFIED to 5

#18 Updated by pthomas@redhat.com about 5 years ago

  • Status changed from 5 to 6

Verified

[root@mgmt12 ~]# rpm -qa pulp-server
pulp-server-2.6.2-0.2.beta.el6.noarch
[root@mgmt12 ~]# 
[root@mgmt12 ~]# time pulp-admin rpm repo sync run --repo-id rad-5Client
+----------------------------------------------------------------------+
                 Synchronizing Repository [rad-5Client]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Task Succeeded

Copying files
[-]
... completed

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[                                                  ] 0%
0 of 76 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
1487 of 1487 items
... completed

Publishing Comps file
[==================================================] 100%
27 of 27 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
[\]
... completed

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

real    0m32.350s
user    0m1.026s
sys    0m0.151s
[root@mgmt12 ~]# time pulp-admin rpm repo sync run --repo-id rhel6
+----------------------------------------------------------------------+
                    Synchronizing Repository [rhel6]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Task Succeeded

Copying files
[|]
... completed

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[/]
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
2913 of 2913 items
... completed

Publishing Comps file
[==================================================] 100%
212 of 212 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[/]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

real    1m17.722s
user    0m1.785s
sys    0m0.203s

#20 Updated by rbarlow over 4 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#22 Updated by bmbouter over 2 years ago

  • Sprint set to April 2015

#23 Updated by bmbouter over 2 years ago

  • Sprint/Milestone deleted (15)

#24 Updated by bmbouter over 1 year ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF