Project

Profile

Help

Issue #2783

closed

CopyDirectoryStep should copy mtime

Added by rmcgover almost 5 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.14.0
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 23
Quarter:

Description

In Pulp platform there is a CopyDirectoryStep used by various publishers. It's notably used by pulp_rpm to copy an existing yum repo and incrementally update it.

Currently, it will copy file contents and discard attributes, including mtime.

It would be better if this copy retained the mtime on files for the following reasons:

Improved rsync performance

When yum and rsync distributors are used together, an rsync publish may reference many files which have already been published with the same content (e.g. sqlite files which are retained over publishes).

Because these files were copied discarding timestamps, the rsync "quick check" algorithm (which checks if size and last modified time are equal between local and remote) will fail and rsync must read the entire file on both sides to calculate its checksum, which can be much slower if the repo has many files.

I see this in publishes where the rsync itemized output contains many lines like this:

<f..t...... 0258070a02e25a56bb6c05130c1ae275164a58fd4f07585bb43a8476d3cf1cc9-filelists.sqlite.bz2
<f..t...... 0259762a377462d92d93277b10b70b05b2356fdc474c0ac427db4f282df50dc1-primary.sqlite.bz2
<f..t...... 0264321f868112ee7bc735f7f25e47c87485da7847389047a5b0e3f076d3c11d-other.sqlite.bz2
<f..t...... 02751717b265bd5f4644da81e8f1129fcb972867caed8998daff4709cd4cb9ec-primary.sqlite.bz2

...indicating that file content didn't change, but timestamp changed.

(Side note: about yum repos retaining files between publish, see related issues https://pulp.plan.io/issues/1684 , https://pulp.plan.io/issues/2788 )

More user-friendly

mtime is useful for humans, e.g. when looking at a directory listing on a Pulp-hosted yum repo, I would prefer to see the real mtime of each file (so I can tell old and new repodata apart) rather than the time when the CopyDirectoryStep ran.

Actions #2

Updated by rmcgover almost 5 years ago

  • Description updated (diff)
Actions #4

Updated by ttereshc almost 5 years ago

  • Triaged changed from No to Yes
  • Sprint Candidate changed from No to Yes
Actions #5

Updated by mhrivnak almost 5 years ago

  • Sprint/Milestone set to 40
Actions #6

Updated by mhrivnak almost 5 years ago

  • Sprint/Milestone changed from 40 to 41
Actions #7

Updated by bizhang almost 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to bizhang
Actions #8

Updated by mhrivnak almost 5 years ago

  • Sprint/Milestone changed from 41 to 42
Actions #10

Updated by bizhang almost 5 years ago

  • Status changed from ASSIGNED to POST

Added by werwty almost 5 years ago

Revision f36a5b96

Copy mtime for copytree operation

closes #2783 https://pulp.plan.io/issues/2783

Actions #11

Updated by werwty almost 5 years ago

  • Status changed from POST to MODIFIED
Actions #12

Updated by pcreech almost 5 years ago

  • Platform Release set to 2.14.0
Actions #13

Updated by pcreech almost 5 years ago

  • Status changed from MODIFIED to 5
Actions #14

Updated by sjlouder almost 5 years ago

OS: Fedora 25 Server
Platform Release: 2.14rc3

Verified as part of Test Day

Followed steps listed here: https://github.com/PulpQE/pulp-smash/issues/720

pulp-admin rpm repo create --repo-id=test --feed=https://bizhang.fedorapeople.org/testrepo/ &&
pulp-admin rpm repo update --generate-sqlite=true --repo-id=test &&
pulp-admin rpm repo sync run --repo-id=test &&
sudo ls -l /var/lib/pulp/published/yum/https/repos/testrepo/repodata | grep .sqlite
-rw-r--r--. 1 apache apache  864 Aug  8 15:35 0ff18d12b601c1a80294a4dd3dc55347bcc4d1967298d53346852fc1ddf26a86-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2295 Aug  8 15:35 2f153315a50e3313a5e4b2cd796713ccd279297c13f2560f5ff0e53d30ca6eb1-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 1067 Aug  8 15:35 d19d500a8bc34537db43b8bf87b719a4f14a30cfc8501d955dc084f463319b0d-filelists.sqlite.bz2

pulp-admin rpm repo uploads rpm --file=kangaroo-0.2-1.noarch.rpm --repo-id=test &&
pulp-admin rpm repo sync run --repo-id=test &&
sudo ls -l /var/lib/pulp/published/yum/https/repos/testrepo/repodata | grep .sqlite
-rw-r--r--. 1 apache apache  864 Aug  8 15:35 0ff18d12b601c1a80294a4dd3dc55347bcc4d1967298d53346852fc1ddf26a86-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2295 Aug  8 15:35 2f153315a50e3313a5e4b2cd796713ccd279297c13f2560f5ff0e53d30ca6eb1-primary.sqlite.bz2
-rw-r--r--. 1 apache apache  961 Aug  8 15:36 a04328c06fb054a10fec63bedc9e9a33a18622e117ae4f584ffa25875b58db17-other.sqlite.bz2
-rw-r--r--. 1 apache apache 1207 Aug  8 15:36 cd1c7d91516dd580f3163137f7e07d82efb59f974835f17b7c9298f80268ba8e-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1067 Aug  8 15:35 d19d500a8bc34537db43b8bf87b719a4f14a30cfc8501d955dc084f463319b0d-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 2498 Aug  8 15:36 f532dd782fa1a680d9a13210699f666d153a9f9f788079c79699b160af29882f-primary.sqlite.bz2

pulp-admin rpm repo uploads rpm --file=walrus-0.71-1.noarch.rpm --repo-id=test &&
pulp-admin rpm repo sync run --repo-id=test &&
sudo ls -l /var/lib/pulp/published/yum/https/repos/testrepo/repodata | grep .sqlite
-rw-r--r--. 1 apache apache  864 Aug  8 15:35 0ff18d12b601c1a80294a4dd3dc55347bcc4d1967298d53346852fc1ddf26a86-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2295 Aug  8 15:35 2f153315a50e3313a5e4b2cd796713ccd279297c13f2560f5ff0e53d30ca6eb1-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 2818 Aug  8 15:37 53173c4875c80edca361a5040ab1325e27ed587b42e33178981ac4b0ef65f5ba-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 1332 Aug  8 15:37 867ce81dec2dbe4d72bfe6e42060400860d1762b9274a32067cb66d4b853b337-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache  961 Aug  8 15:36 a04328c06fb054a10fec63bedc9e9a33a18622e117ae4f584ffa25875b58db17-other.sqlite.bz2
-rw-r--r--. 1 apache apache 1207 Aug  8 15:36 cd1c7d91516dd580f3163137f7e07d82efb59f974835f17b7c9298f80268ba8e-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1067 Aug  8 15:35 d19d500a8bc34537db43b8bf87b719a4f14a30cfc8501d955dc084f463319b0d-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1036 Aug  8 15:37 e9970685d7ec7bc5a906c5d28d426613588e15f0d63fca155817fae6086469d9-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2498 Aug  8 15:36 f532dd782fa1a680d9a13210699f666d153a9f9f788079c79699b160af29882f-primary.sqlite.bz2

pulp-admin rpm repo uploads rpm --file=walrus-5.21-1.noarch.rpm --repo-id=test &&
pulp-admin rpm repo sync run --repo-id=test &&
sudo ls -l /var/lib/pulp/published/yum/https/repos/testrepo/repodata | grep .sqlite
-rw-r--r--. 1 apache apache  864 Aug  8 15:35 0ff18d12b601c1a80294a4dd3dc55347bcc4d1967298d53346852fc1ddf26a86-other.sqlite.bz2
-rw-r--r--. 1 apache apache 1120 Aug  8 15:38 23bbc47640e44413eadbe118d0b38e2042c7b92407124adbaeec708dd0604ef9-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2295 Aug  8 15:35 2f153315a50e3313a5e4b2cd796713ccd279297c13f2560f5ff0e53d30ca6eb1-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 2818 Aug  8 15:37 53173c4875c80edca361a5040ab1325e27ed587b42e33178981ac4b0ef65f5ba-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 1459 Aug  8 15:38 585a1af156c4c2f9ee4e3e3be972adf1cc044d6295497064d04e8df0b9a0a2de-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1332 Aug  8 15:37 867ce81dec2dbe4d72bfe6e42060400860d1762b9274a32067cb66d4b853b337-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache  961 Aug  8 15:36 a04328c06fb054a10fec63bedc9e9a33a18622e117ae4f584ffa25875b58db17-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2953 Aug  8 15:38 bc63d5b6d30328901600308d2ef2851ba99a78c967081a12c9df16e9c9e50d83-primary.sqlite.bz2
-rw-r--r--. 1 apache apache 1207 Aug  8 15:36 cd1c7d91516dd580f3163137f7e07d82efb59f974835f17b7c9298f80268ba8e-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1067 Aug  8 15:35 d19d500a8bc34537db43b8bf87b719a4f14a30cfc8501d955dc084f463319b0d-filelists.sqlite.bz2
-rw-r--r--. 1 apache apache 1036 Aug  8 15:37 e9970685d7ec7bc5a906c5d28d426613588e15f0d63fca155817fae6086469d9-other.sqlite.bz2
-rw-r--r--. 1 apache apache 2498 Aug  8 15:36 f532dd782fa1a680d9a13210699f666d153a9f9f788079c79699b160af29882f-primary.sqlite.bz2
Actions #16

Updated by pcreech almost 5 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

Added by werwty over 4 years ago

Revision c8b56e44

Copy mtime for copytree operation

closes #2783 https://pulp.plan.io/issues/2783

(cherry picked from commit f36a5b96ad60d594cc875faf0ce395d80d63ac75)

Actions #17

Updated by bmbouter about 4 years ago

  • Sprint set to Sprint 23
Actions #18

Updated by bmbouter about 4 years ago

  • Sprint/Milestone deleted (42)
Actions #19

Updated by bmbouter about 3 years ago

  • Tags Pulp 2 added
Actions #20

Updated by bmbouter about 3 years ago

  • Tags deleted (RCM)

Also available in: Atom PDF