Project

Profile

Help

Story #2

As a user, my rpm sync finishes quickly when upstream metadata hasn't changed

Added by Anonymous about 5 years ago. Updated 9 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Platform Release:
2.6.2
Blocks Release:
Backwards Incompatible:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
April 2015

Description

The repomd.xml file has its own timestamp in the XML, and then each file it references has its own timestamp. The treeinfo file contains its own timestamp, which already gets checked for freshness. We should check the repomd.xml timestamp and skip the rest of the sync if it hasn't changed. This will reduce the sync time for a big repo from possibly 1-2 minutes to a small number of seconds when the upstream repo hasn't changed. This will also help us be smarter about when the distribution files get re-downloaded.

Deliverables:
  • rpm sync checks the timestamp in repomd.xml, and skips all unit types but "distribution" if it hasn't changed since the last sync.
  • tell Katello team (specifically jsherrill) about this change once it's in Pulp. They might be able to remove some code on their end once this is in place.
  • release notes

Associated revisions

Revision 183cb1eb View on GitHub
Added by mhrivnak almost 5 years ago

Sync now skips most steps if the feed metadata has not changed.

closes #2

History

#1 Updated by rbarlow about 5 years ago

  • Project changed from Import to RPM Support

#2 Updated by mhrivnak almost 5 years ago

  • Description updated (diff)

#3 Updated by mhrivnak almost 5 years ago

  • Tags Sprint Candidate added

#4 Updated by mhrivnak almost 5 years ago

  • Priority changed from Normal to High

#5 Updated by cduryee almost 5 years ago

  • Description updated (diff)

#6 Updated by mhrivnak almost 5 years ago

  • Description updated (diff)
  • Tags Groomed added

#7 Updated by mhrivnak almost 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to mhrivnak

#8 Updated by mhrivnak almost 5 years ago

  • Description updated (diff)

I'm reducing the scope of this story because doing per-unit-type timestamp checking proved to be far more complex than it's worth. We can revisit once this workflow gets stepified, which should make that much more achievable.

#9 Updated by mhrivnak almost 5 years ago

  • % Done changed from 0 to 50

#10 Updated by mhrivnak almost 5 years ago

  • Status changed from ASSIGNED to POST
  • % Done changed from 50 to 90

#11 Updated by mhrivnak almost 5 years ago

  • Platform Release set to 2.6.2

closed above PR, opened this one instead: https://github.com/pulp/pulp_rpm/pull/676

#12 Updated by mhrivnak over 4 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 90 to 100

#13 Updated by mhrivnak over 4 years ago

  • Sprint/Milestone set to 15

#14 Updated by bmbouter over 4 years ago

  • Groomed set to Yes
  • Tags deleted (Groomed)

#15 Updated by bmbouter over 4 years ago

  • Sprint Candidate set to Yes
  • Tags deleted (Sprint Candidate)

#16 Updated by dkliban@redhat.com over 4 years ago

  • Status changed from MODIFIED to ON_QA

#17 Updated by pthomas@redhat.com over 4 years ago

  • QA Contact set to pthomas@redhat.com

#18 Updated by pthomas@redhat.com over 4 years ago

  • Status changed from ON_QA to VERIFIED

Verified

[root@mgmt12 ~]# rpm -qa pulp-server
pulp-server-2.6.2-0.2.beta.el6.noarch
[root@mgmt12 ~]# 

[root@mgmt12 ~]# time pulp-admin rpm repo sync run --repo-id rad-5Client
+----------------------------------------------------------------------+
                 Synchronizing Repository [rad-5Client]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Task Succeeded

Copying files
[-]
... completed

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[                                                  ] 0%
0 of 76 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
1487 of 1487 items
... completed

Publishing Comps file
[==================================================] 100%
27 of 27 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
[\]
... completed

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

real    0m32.350s
user    0m1.026s
sys    0m0.151s

[root@mgmt12 ~]# time pulp-admin rpm repo sync run --repo-id rhel6
+----------------------------------------------------------------------+
                    Synchronizing Repository [rhel6]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Downloading metadata...
[\]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed

Task Succeeded

Copying files
[|]
... completed

Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[/]
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
2913 of 2913 items
... completed

Publishing Comps file
[==================================================] 100%
212 of 212 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[/]
... completed

Writing Listings File
[-]
... completed

Task Succeeded

real    1m17.722s
user    0m1.785s
sys    0m0.203s

#20 Updated by rbarlow almost 4 years ago

  • Status changed from VERIFIED to CLOSED - CURRENTRELEASE

#22 Updated by bmbouter almost 2 years ago

  • Sprint set to April 2015

#23 Updated by bmbouter almost 2 years ago

  • Sprint/Milestone deleted (15)

#24 Updated by bmbouter 9 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF