Story #2
closed
As a user, my rpm sync finishes quickly when upstream metadata hasn't changed
Status:
CLOSED - CURRENTRELEASE
Description
The repomd.xml file has its own timestamp in the XML, and then each file it references has its own timestamp. The treeinfo file contains its own timestamp, which already gets checked for freshness. We should check the repomd.xml timestamp and skip the rest of the sync if it hasn't changed. This will reduce the sync time for a big repo from possibly 1-2 minutes to a small number of seconds when the upstream repo hasn't changed. This will also help us be smarter about when the distribution files get re-downloaded.
Deliverables:
- rpm sync checks the timestamp in repomd.xml, and skips all unit types but "distribution" if it hasn't changed since the last sync.
- tell Katello team (specifically jsherrill) about this change once it's in Pulp. They might be able to remove some code on their end once this is in place.
- release notes
- Project changed from 22 to RPM Support
- Description updated (diff)
- Tags Sprint Candidate added
- Priority changed from Normal to High
- Description updated (diff)
- Description updated (diff)
- Tags Groomed added
- Status changed from NEW to ASSIGNED
- Assignee set to mhrivnak
- Description updated (diff)
I'm reducing the scope of this story because doing per-unit-type timestamp checking proved to be far more complex than it's worth. We can revisit once this workflow gets stepified, which should make that much more achievable.
- % Done changed from 0 to 50
- Status changed from ASSIGNED to POST
- % Done changed from 50 to 90
- Platform Release set to 2.6.2
- Status changed from POST to MODIFIED
- % Done changed from 90 to 100
- Sprint/Milestone set to 15
- Groomed set to Yes
- Tags deleted (
Groomed)
- Sprint Candidate set to Yes
- Tags deleted (
Sprint Candidate)
- Status changed from MODIFIED to 5
- Status changed from 5 to 6
Verified
[root@mgmt12 ~]# rpm -qa pulp-server
pulp-server-2.6.2-0.2.beta.el6.noarch
[root@mgmt12 ~]#
[root@mgmt12 ~]# time pulp-admin rpm repo sync run --repo-id rad-5Client
+----------------------------------------------------------------------+
Synchronizing Repository [rad-5Client]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[\]
... completed
Downloading repository content...
[==================================================] 100%
RPMs: 0/0 items
Delta RPMs: 0/0 items
... completed
Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed
Importing errata...
[-]
... completed
Importing package groups/categories...
[-]
... completed
Task Succeeded
Copying files
[-]
... completed
Initializing repo metadata
[-]
... completed
Publishing Distribution files
[-]
... completed
Publishing RPMs
[ ] 0%
0 of 76 items
... completed
Publishing Delta RPMs
... skipped
Publishing Errata
[==================================================] 100%
1487 of 1487 items
... completed
Publishing Comps file
[==================================================] 100%
27 of 27 items
... completed
Publishing Metadata.
[-]
... completed
Closing repo metadata
[-]
... completed
Generating sqlite files
[\]
... completed
Publishing files to web
[-]
... completed
Writing Listings File
[-]
... completed
Writing Listings File
[-]
... completed
Task Succeeded
real 0m32.350s
user 0m1.026s
sys 0m0.151s
[root@mgmt12 ~]# time pulp-admin rpm repo sync run --repo-id rhel6
+----------------------------------------------------------------------+
Synchronizing Repository [rhel6]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[\]
... completed
Downloading repository content...
[==================================================] 100%
RPMs: 0/0 items
Delta RPMs: 0/0 items
... completed
Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed
Importing errata...
[-]
... completed
Importing package groups/categories...
[-]
... completed
Task Succeeded
Copying files
[|]
... completed
Initializing repo metadata
[-]
... completed
Publishing Distribution files
[-]
... completed
Publishing RPMs
[/]
... completed
Publishing Delta RPMs
... skipped
Publishing Errata
[==================================================] 100%
2913 of 2913 items
... completed
Publishing Comps file
[==================================================] 100%
212 of 212 items
... completed
Publishing Metadata.
[-]
... completed
Closing repo metadata
[-]
... completed
Generating sqlite files
... skipped
Publishing files to web
[/]
... completed
Writing Listings File
[-]
... completed
Task Succeeded
real 1m17.722s
user 0m1.785s
sys 0m0.203s
- Status changed from 6 to CLOSED - CURRENTRELEASE
- Sprint/Milestone deleted (
15)
Also available in: Atom
PDF
.
Sync now skips most steps if the feed metadata has not changed.
closes #2