Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #2783

closed

CopyDirectoryStep should copy mtime

Added by rmcgover almost 7 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
bizhang
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.14.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 23
Quarter:

Description

In Pulp platform there is a CopyDirectoryStep used by various publishers. It's notably used by pulp_rpm to copy an existing yum repo and incrementally update it.

Currently, it will copy file contents and discard attributes, including mtime.

It would be better if this copy retained the mtime on files for the following reasons:

Improved rsync performance

When yum and rsync distributors are used together, an rsync publish may reference many files which have already been published with the same content (e.g. sqlite files which are retained over publishes).

Because these files were copied discarding timestamps, the rsync "quick check" algorithm (which checks if size and last modified time are equal between local and remote) will fail and rsync must read the entire file on both sides to calculate its checksum, which can be much slower if the repo has many files.

I see this in publishes where the rsync itemized output contains many lines like this:

<f..t...... 0258070a02e25a56bb6c05130c1ae275164a58fd4f07585bb43a8476d3cf1cc9-filelists.sqlite.bz2
<f..t...... 0259762a377462d92d93277b10b70b05b2356fdc474c0ac427db4f282df50dc1-primary.sqlite.bz2
<f..t...... 0264321f868112ee7bc735f7f25e47c87485da7847389047a5b0e3f076d3c11d-other.sqlite.bz2
<f..t...... 02751717b265bd5f4644da81e8f1129fcb972867caed8998daff4709cd4cb9ec-primary.sqlite.bz2

...indicating that file content didn't change, but timestamp changed.

(Side note: about yum repos retaining files between publish, see related issues https://pulp.plan.io/issues/1684 , https://pulp.plan.io/issues/2788 )

More user-friendly

mtime is useful for humans, e.g. when looking at a directory listing on a Pulp-hosted yum repo, I would prefer to see the real mtime of each file (so I can tell old and new repodata apart) rather than the time when the CopyDirectoryStep ran.

Actions
Copy link
 #2

Updated by rmcgover almost 7 years ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by ttereshc almost 7 years ago

  • Triaged changed from No to Yes
  • Sprint Candidate changed from No to Yes
Actions
Copy link
 #5

Updated by mhrivnak almost 7 years ago

  • Sprint/Milestone set to 40
Actions
Copy link
 #6

Updated by mhrivnak almost 7 years ago

  • Sprint/Milestone changed from 40 to 41
Actions
Copy link
 #7

Updated by bizhang over 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bizhang
Actions
Copy link
 #8

Updated by mhrivnak over 6 years ago

  • Sprint/Milestone changed from 41 to 42
Actions
Copy link
 #10

Updated by bizhang over 6 years ago

  • Status changed from ASSIGNED to POST

Added by werwty over 6 years ago

Revision f36a5b96 | View on GitHub

Copy mtime for copytree operation

closes #2783 https://pulp.plan.io/issues/2783

Actions
Copy link
 #11

Updated by werwty over 6 years ago

  • Status changed from POST to MODIFIED
Actions
Copy link
 #12

Updated by pcreech over 6 years ago

  • Platform Release set to 2.14.0
Actions
Copy link
 #13

Updated by pcreech over 6 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #14

Updated by sjlouder over 6 years ago

OS: Fedora 25 Server
Platform Release: 2.14rc3

Verified as part of Test Day

Followed steps listed here: https://github.com/PulpQE/pulp-smash/issues/720

pulp-admin rpm repo create --repo-id=test --feed=https://bizhang.fedorapeople.org/testrepo/ &&
pulp-admin rpm repo update --generate-sqlite=true --repo-id=test &&
pulp-admin rpm repo sync run --repo-id=test &&
sudo ls -l /var/lib/pulp/published/yum/https/repos/testrepo/repodata | grep .sqlite
-rw-r--r--. 1 apache apache  864 Aug  8 15:35 0ff18d12b601c1a80294a4dd3dc55347bcc4d1967298d53346852fc1ddf26a86-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2295 Aug  8 15:35 2f153315a50e3313a5e4b2cd796713ccd279297c13f2560f5ff0e53d30ca6eb1-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 1067 Aug  8 15:35 d19d500a8bc34537db43b8bf87b719a4f14a30cfc8501d955dc084f463319b0d-filelists.sqlite.bz2

pulp-admin rpm repo uploads rpm --file=kangaroo-0.2-1.noarch.rpm --repo-id=test &&
pulp-admin rpm repo sync run --repo-id=test &&
sudo ls -l /var/lib/pulp/published/yum/https/repos/testrepo/repodata | grep .sqlite
-rw-r--r--. 1 apache apache  864 Aug  8 15:35 0ff18d12b601c1a80294a4dd3dc55347bcc4d1967298d53346852fc1ddf26a86-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2295 Aug  8 15:35 2f153315a50e3313a5e4b2cd796713ccd279297c13f2560f5ff0e53d30ca6eb1-primary.sqlite.bz2
-rw-r--r--. 1 apache apache  961 Aug  8 15:36 a04328c06fb054a10fec63bedc9e9a33a18622e117ae4f584ffa25875b58db17-other.sqlite.bz2
-rw-r--r--. 1 apache apache 1207 Aug  8 15:36 cd1c7d91516dd580f3163137f7e07d82efb59f974835f17b7c9298f80268ba8e-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1067 Aug  8 15:35 d19d500a8bc34537db43b8bf87b719a4f14a30cfc8501d955dc084f463319b0d-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 2498 Aug  8 15:36 f532dd782fa1a680d9a13210699f666d153a9f9f788079c79699b160af29882f-primary.sqlite.bz2

pulp-admin rpm repo uploads rpm --file=walrus-0.71-1.noarch.rpm --repo-id=test &&
pulp-admin rpm repo sync run --repo-id=test &&
sudo ls -l /var/lib/pulp/published/yum/https/repos/testrepo/repodata | grep .sqlite
-rw-r--r--. 1 apache apache  864 Aug  8 15:35 0ff18d12b601c1a80294a4dd3dc55347bcc4d1967298d53346852fc1ddf26a86-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2295 Aug  8 15:35 2f153315a50e3313a5e4b2cd796713ccd279297c13f2560f5ff0e53d30ca6eb1-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 2818 Aug  8 15:37 53173c4875c80edca361a5040ab1325e27ed587b42e33178981ac4b0ef65f5ba-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 1332 Aug  8 15:37 867ce81dec2dbe4d72bfe6e42060400860d1762b9274a32067cb66d4b853b337-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache  961 Aug  8 15:36 a04328c06fb054a10fec63bedc9e9a33a18622e117ae4f584ffa25875b58db17-other.sqlite.bz2
-rw-r--r--. 1 apache apache 1207 Aug  8 15:36 cd1c7d91516dd580f3163137f7e07d82efb59f974835f17b7c9298f80268ba8e-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1067 Aug  8 15:35 d19d500a8bc34537db43b8bf87b719a4f14a30cfc8501d955dc084f463319b0d-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1036 Aug  8 15:37 e9970685d7ec7bc5a906c5d28d426613588e15f0d63fca155817fae6086469d9-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2498 Aug  8 15:36 f532dd782fa1a680d9a13210699f666d153a9f9f788079c79699b160af29882f-primary.sqlite.bz2

pulp-admin rpm repo uploads rpm --file=walrus-5.21-1.noarch.rpm --repo-id=test &&
pulp-admin rpm repo sync run --repo-id=test &&
sudo ls -l /var/lib/pulp/published/yum/https/repos/testrepo/repodata | grep .sqlite
-rw-r--r--. 1 apache apache  864 Aug  8 15:35 0ff18d12b601c1a80294a4dd3dc55347bcc4d1967298d53346852fc1ddf26a86-other.sqlite.bz2
-rw-r--r--. 1 apache apache 1120 Aug  8 15:38 23bbc47640e44413eadbe118d0b38e2042c7b92407124adbaeec708dd0604ef9-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2295 Aug  8 15:35 2f153315a50e3313a5e4b2cd796713ccd279297c13f2560f5ff0e53d30ca6eb1-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 2818 Aug  8 15:37 53173c4875c80edca361a5040ab1325e27ed587b42e33178981ac4b0ef65f5ba-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 1459 Aug  8 15:38 585a1af156c4c2f9ee4e3e3be972adf1cc044d6295497064d04e8df0b9a0a2de-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1332 Aug  8 15:37 867ce81dec2dbe4d72bfe6e42060400860d1762b9274a32067cb66d4b853b337-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache  961 Aug  8 15:36 a04328c06fb054a10fec63bedc9e9a33a18622e117ae4f584ffa25875b58db17-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2953 Aug  8 15:38 bc63d5b6d30328901600308d2ef2851ba99a78c967081a12c9df16e9c9e50d83-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 1207 Aug  8 15:36 cd1c7d91516dd580f3163137f7e07d82efb59f974835f17b7c9298f80268ba8e-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1067 Aug  8 15:35 d19d500a8bc34537db43b8bf87b719a4f14a30cfc8501d955dc084f463319b0d-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1036 Aug  8 15:37 e9970685d7ec7bc5a906c5d28d426613588e15f0d63fca155817fae6086469d9-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2498 Aug  8 15:36 f532dd782fa1a680d9a13210699f666d153a9f9f788079c79699b160af29882f-primary.sqlite.bz2
Actions
Copy link
 #16

Updated by pcreech over 6 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

Added by werwty over 6 years ago

Revision c8b56e44 | View on GitHub

Copy mtime for copytree operation

closes #2783 https://pulp.plan.io/issues/2783

(cherry picked from commit f36a5b96ad60d594cc875faf0ce395d80d63ac75)

Actions
Copy link
 #17

Updated by bmbouter about 6 years ago

  • Sprint set to Sprint 23
Actions
Copy link
 #18

Updated by bmbouter about 6 years ago

  • Sprint/Milestone deleted (42)
Actions
Copy link
 #19

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added
Actions
Copy link
 #20

Updated by bmbouter almost 5 years ago

  • Tags deleted (RCM)
Actions
Send by e-mail Copy link

Also available in: Atom PDF