Project

Profile

Help

Issue #2783

CopyDirectoryStep should copy mtime

Added by rmcgover over 2 years ago. Updated 10 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Severity:
2. Medium
Version:
Platform Release:
2.14.0
Blocks Release:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
Yes
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
Yes
Verification Required:
No
Sprint:
Sprint 23

Description

In Pulp platform there is a CopyDirectoryStep used by various publishers. It's notably used by pulp_rpm to copy an existing yum repo and incrementally update it.

Currently, it will copy file contents and discard attributes, including mtime.

It would be better if this copy retained the mtime on files for the following reasons:

Improved rsync performance

When yum and rsync distributors are used together, an rsync publish may reference many files which have already been published with the same content (e.g. sqlite files which are retained over publishes).

Because these files were copied discarding timestamps, the rsync "quick check" algorithm (which checks if size and last modified time are equal between local and remote) will fail and rsync must read the entire file on both sides to calculate its checksum, which can be much slower if the repo has many files.

I see this in publishes where the rsync itemized output contains many lines like this:

<f..t...... 0258070a02e25a56bb6c05130c1ae275164a58fd4f07585bb43a8476d3cf1cc9-filelists.sqlite.bz2
<f..t...... 0259762a377462d92d93277b10b70b05b2356fdc474c0ac427db4f282df50dc1-primary.sqlite.bz2
<f..t...... 0264321f868112ee7bc735f7f25e47c87485da7847389047a5b0e3f076d3c11d-other.sqlite.bz2
<f..t...... 02751717b265bd5f4644da81e8f1129fcb972867caed8998daff4709cd4cb9ec-primary.sqlite.bz2

...indicating that file content didn't change, but timestamp changed.

(Side note: about yum repos retaining files between publish, see related issues https://pulp.plan.io/issues/1684 , https://pulp.plan.io/issues/2788 )

More user-friendly

mtime is useful for humans, e.g. when looking at a directory listing on a Pulp-hosted yum repo, I would prefer to see the real mtime of each file (so I can tell old and new repodata apart) rather than the time when the CopyDirectoryStep ran.

Associated revisions

Revision f36a5b96 View on GitHub
Added by werwty over 2 years ago

Copy mtime for copytree operation

closes #2783 https://pulp.plan.io/issues/2783

Revision c8b56e44 View on GitHub
Added by werwty over 2 years ago

Copy mtime for copytree operation

closes #2783 https://pulp.plan.io/issues/2783

(cherry picked from commit f36a5b96ad60d594cc875faf0ce395d80d63ac75)

History

#2 Updated by rmcgover over 2 years ago

  • Description updated (diff)

#4 Updated by ttereshc over 2 years ago

  • Triaged changed from No to Yes
  • Sprint Candidate changed from No to Yes

#5 Updated by mhrivnak over 2 years ago

  • Sprint/Milestone set to 40

#6 Updated by mhrivnak over 2 years ago

  • Sprint/Milestone changed from 40 to 41

#7 Updated by bizhang over 2 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bizhang

#8 Updated by mhrivnak over 2 years ago

  • Sprint/Milestone changed from 41 to 42

#9 Updated by bizhang over 2 years ago

  • Smash Test set to 720

#10 Updated by bizhang over 2 years ago

  • Status changed from ASSIGNED to POST

#11 Updated by werwty over 2 years ago

  • Status changed from POST to MODIFIED

#12 Updated by pcreech over 2 years ago

  • Platform Release set to 2.14.0

#13 Updated by pcreech over 2 years ago

  • Status changed from MODIFIED to ON_QA

#14 Updated by sjlouder over 2 years ago

OS: Fedora 25 Server
Platform Release: 2.14rc3

Verified as part of Test Day

Followed steps listed here: https://github.com/PulpQE/pulp-smash/issues/720

pulp-admin rpm repo create --repo-id=test --feed=https://bizhang.fedorapeople.org/testrepo/ &&
pulp-admin rpm repo update --generate-sqlite=true --repo-id=test &&
pulp-admin rpm repo sync run --repo-id=test &&
sudo ls -l /var/lib/pulp/published/yum/https/repos/testrepo/repodata | grep .sqlite
-rw-r--r--. 1 apache apache  864 Aug  8 15:35 0ff18d12b601c1a80294a4dd3dc55347bcc4d1967298d53346852fc1ddf26a86-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2295 Aug  8 15:35 2f153315a50e3313a5e4b2cd796713ccd279297c13f2560f5ff0e53d30ca6eb1-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 1067 Aug  8 15:35 d19d500a8bc34537db43b8bf87b719a4f14a30cfc8501d955dc084f463319b0d-filelists.sqlite.bz2

pulp-admin rpm repo uploads rpm --file=kangaroo-0.2-1.noarch.rpm --repo-id=test &&
pulp-admin rpm repo sync run --repo-id=test &&
sudo ls -l /var/lib/pulp/published/yum/https/repos/testrepo/repodata | grep .sqlite
-rw-r--r--. 1 apache apache  864 Aug  8 15:35 0ff18d12b601c1a80294a4dd3dc55347bcc4d1967298d53346852fc1ddf26a86-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2295 Aug  8 15:35 2f153315a50e3313a5e4b2cd796713ccd279297c13f2560f5ff0e53d30ca6eb1-primary.sqlite.bz2
-rw-r--r--. 1 apache apache  961 Aug  8 15:36 a04328c06fb054a10fec63bedc9e9a33a18622e117ae4f584ffa25875b58db17-other.sqlite.bz2
-rw-r--r--. 1 apache apache 1207 Aug  8 15:36 cd1c7d91516dd580f3163137f7e07d82efb59f974835f17b7c9298f80268ba8e-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1067 Aug  8 15:35 d19d500a8bc34537db43b8bf87b719a4f14a30cfc8501d955dc084f463319b0d-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 2498 Aug  8 15:36 f532dd782fa1a680d9a13210699f666d153a9f9f788079c79699b160af29882f-primary.sqlite.bz2

pulp-admin rpm repo uploads rpm --file=walrus-0.71-1.noarch.rpm --repo-id=test &&
pulp-admin rpm repo sync run --repo-id=test &&
sudo ls -l /var/lib/pulp/published/yum/https/repos/testrepo/repodata | grep .sqlite
-rw-r--r--. 1 apache apache  864 Aug  8 15:35 0ff18d12b601c1a80294a4dd3dc55347bcc4d1967298d53346852fc1ddf26a86-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2295 Aug  8 15:35 2f153315a50e3313a5e4b2cd796713ccd279297c13f2560f5ff0e53d30ca6eb1-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 2818 Aug  8 15:37 53173c4875c80edca361a5040ab1325e27ed587b42e33178981ac4b0ef65f5ba-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 1332 Aug  8 15:37 867ce81dec2dbe4d72bfe6e42060400860d1762b9274a32067cb66d4b853b337-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache  961 Aug  8 15:36 a04328c06fb054a10fec63bedc9e9a33a18622e117ae4f584ffa25875b58db17-other.sqlite.bz2
-rw-r--r--. 1 apache apache 1207 Aug  8 15:36 cd1c7d91516dd580f3163137f7e07d82efb59f974835f17b7c9298f80268ba8e-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1067 Aug  8 15:35 d19d500a8bc34537db43b8bf87b719a4f14a30cfc8501d955dc084f463319b0d-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1036 Aug  8 15:37 e9970685d7ec7bc5a906c5d28d426613588e15f0d63fca155817fae6086469d9-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2498 Aug  8 15:36 f532dd782fa1a680d9a13210699f666d153a9f9f788079c79699b160af29882f-primary.sqlite.bz2

pulp-admin rpm repo uploads rpm --file=walrus-5.21-1.noarch.rpm --repo-id=test &&
pulp-admin rpm repo sync run --repo-id=test &&
sudo ls -l /var/lib/pulp/published/yum/https/repos/testrepo/repodata | grep .sqlite
-rw-r--r--. 1 apache apache  864 Aug  8 15:35 0ff18d12b601c1a80294a4dd3dc55347bcc4d1967298d53346852fc1ddf26a86-other.sqlite.bz2
-rw-r--r--. 1 apache apache 1120 Aug  8 15:38 23bbc47640e44413eadbe118d0b38e2042c7b92407124adbaeec708dd0604ef9-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2295 Aug  8 15:35 2f153315a50e3313a5e4b2cd796713ccd279297c13f2560f5ff0e53d30ca6eb1-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 2818 Aug  8 15:37 53173c4875c80edca361a5040ab1325e27ed587b42e33178981ac4b0ef65f5ba-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 1459 Aug  8 15:38 585a1af156c4c2f9ee4e3e3be972adf1cc044d6295497064d04e8df0b9a0a2de-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1332 Aug  8 15:37 867ce81dec2dbe4d72bfe6e42060400860d1762b9274a32067cb66d4b853b337-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache  961 Aug  8 15:36 a04328c06fb054a10fec63bedc9e9a33a18622e117ae4f584ffa25875b58db17-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2953 Aug  8 15:38 bc63d5b6d30328901600308d2ef2851ba99a78c967081a12c9df16e9c9e50d83-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 1207 Aug  8 15:36 cd1c7d91516dd580f3163137f7e07d82efb59f974835f17b7c9298f80268ba8e-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1067 Aug  8 15:35 d19d500a8bc34537db43b8bf87b719a4f14a30cfc8501d955dc084f463319b0d-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1036 Aug  8 15:37 e9970685d7ec7bc5a906c5d28d426613588e15f0d63fca155817fae6086469d9-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2498 Aug  8 15:36 f532dd782fa1a680d9a13210699f666d153a9f9f788079c79699b160af29882f-primary.sqlite.bz2

#15 Updated by kdelee@redhat.com over 2 years ago

  • Verified changed from No to Yes

#16 Updated by pcreech over 2 years ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#17 Updated by bmbouter almost 2 years ago

  • Sprint set to Sprint 23

#18 Updated by bmbouter almost 2 years ago

  • Sprint/Milestone deleted (42)

#19 Updated by bmbouter 11 months ago

  • Tags Pulp 2 added

#20 Updated by bmbouter 10 months ago

  • Tags deleted (RCM)

Please register to edit this issue

Also available in: Atom PDF