Project

Profile

Help

Story #2594

closed

As a User I want Pulp OSTree to create a new unit for each commit version on a branch

Added by paji@redhat.com about 7 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Sprint/Milestone:
-
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
2.14.0
Target Release - OSTree:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 19
Quarter:

Description

With the "depth" option pulp today can sync an ostree repo "completely" as opposed to "only the latest". However pulp only a creates a unit per branch head. For example try the following

1) edit /etc/pulp/server/plugins.conf.d/ostree_importer.json
{
"depth": -1
}
restart workers ..

2) pulp-admin ostree repo create --repo-id=synthesize --feed=https://partha.fedorapeople.org/test-repos/ostree-zoo/
3) pulp-admin ostree repo sync run --repo-id=synthesize

4) pulp-admin ostree repo list --repo-id=synthesize --detail

notice that the content units show quantity of 2

However in reality each branch in that repo has many versions. You can notice this by try something like this

cd /var/lib/pulp/content/shared/ostree
ls -lt  # find the topmost  recent dir which is likely to be of the repo you just synced
$ ostree log --repo=<top-dir>/content fedora-atomic/f21/x86_64/updates-testing/docker-host
commit b954defc0c21b23705bf5776a78ca434b2569260e64e19fb61bdbd5526b3bf0f
Date:  2016-03-03 15:42:30 +0000
Version: 23.3

    testing

commit b5478067c6ed79acde91d09cd3effcd12885a3cc749c390735de18a91ee75c9c
Date:  2016-03-03 15:42:30 +0000
Version: 23.2

    testing

commit 0dfe48948b5ecaf74f1864b4cb43a991ac7ab7e4ca5bbab36ac30b6596d81799
Date:  2016-03-03 15:42:30 +0000
Version: 23.1

    testing

Notice there are 3 versions on just one branch.

In an ideal world pulp ostree would create a new unit per branch + commit version using history from ostree log on the branch.

Actions #1

Updated by bizhang about 7 years ago

  • Tracker changed from Issue to Story
  • Subject changed from Pulp OSTree does not show history on branches. to As a User I want Pulp OSTree to create a new unit for each commit version on a branch
  • % Done set to 0
Actions #4

Updated by jortel@redhat.com about 7 years ago

The problem

To start, I'm going to add more background. Each ostree branch content unit has a natural key of: {remote_id: <id>, branch: <name>, commit: <hash>} and represents a branch HEAD. For each branch that is pulled (from the remote), a unit is created when the branch (HEAD) commit has changed.

Pull-1

fedora/docker/host => commit: 0x1234 {version: 1}

Someone does a commit on branch fedora/docker/host and makes it version 2.

Pull-2

fedora/docker/host => commit: 0x1235 {version: 2}

Someone does a commit on branch fedora/docker/host and makes it version 3.
Someone does a commit on branch fedora/docker/host and makes it version 4.

Notice the user did not have pulp do a pull when the branch was at version 3. Later the user requests the pull.

Pull-3

fedora/docker/host => commit: 0x1237 {version: 4}

Note: pulp has created (3) content units:

{remote_id:1, branch: fedora/docker/host, commit: 0x1234 metadata: {version: 1}}
{remote_id:1, branch: fedora/docker/host, commit: 0x1235 metadata: {version: 2}}
{remote_id:1, branch: fedora/docker/host, commit: 0x1237 metadata: {version: 4}}

because no pulls (sync) happened while the branch HEAD was at commit: 0x1236 metadata: {version: 3}

And

{remote_id:1, branch: fedora/docker/host, commit: 0x1236 metadata: {version: 3}}     ** MISSING**

Is missing.

This is working as designed but as OSTree gets used in Pulp, we're getting new requirements.

The solution

In the Add[1] step - use the list the local refs to build the commit hierarchy for each ref (branch). Then, using the list of commits in the hierarchy instead of the branch HEAD, the importer should ensure a unit exists for all of those commits.

Doing this should be straight forward:

  • Add Repository.log(ref) in the library[0] using the CLI code[2] as a road map. Basically use load_variant[3] and get_parent[4] to walk up the commit tree.
  • Update the importer Add[1] step to use the new Repository.log() for each branch and use this list of commits to create the units instead of just the branch HEAD.

[0] https://github.com/pulp/pulp_ostree/blob/master/plugins/pulp_ostree/plugins/lib.py#L246
[1] https://github.com/pulp/pulp_ostree/blob/master/plugins/pulp_ostree/plugins/importers/steps.py#L237
[2] https://github.com/ostreedev/ostree/blob/master/src/ostree/ot-builtin-log.c
[3] https://github.com/ostreedev/ostree/blob/master/src/libostree/ostree-repo.c#L3536
[4] https://github.com/ostreedev/ostree/blob/master/src/libostree/ostree-repo.c#L2434

Actions #5

Updated by mhrivnak about 7 years ago

  • Groomed changed from No to Yes
Actions #7

Updated by mhrivnak almost 7 years ago

  • Sprint Candidate changed from No to Yes
Actions #8

Updated by mhrivnak almost 7 years ago

  • Sprint/Milestone set to 37
Actions #9

Updated by jortel@redhat.com almost 7 years ago

  • Assignee set to jortel@redhat.com
Actions #10

Updated by ipanova@redhat.com almost 7 years ago

  • Status changed from NEW to ASSIGNED
Actions #11

Updated by jortel@redhat.com almost 7 years ago

  • Status changed from ASSIGNED to POST
Actions #12

Updated by jortel@redhat.com almost 7 years ago

Testing:

Create an ostree repository and commit a tree using /etc/pulp as the source.

$ cd /tmp
$ mkdir test-2594
$ cd test-2594
$ ostree init --repo=. --mode=archive-z2
$ ostree commit --repo=. -b test --add-metadata-string=version=1.0  /etc/pulp
3b13a82238b8d2482b0393ed9fb0bb3b26a80b0446716c4e6ffa136ad4d8931c
$ ostree summary -u --repo=.

Create and sync a repository.

$ pulp-admin -u admin -p admin ostree repo create --depth=100 --repo-id=test-2594 --feed=file:///tmp/test-2594
$ pulp-admin -u admin -p admin ostree repo sync run --repo-id=test-2594

List content and note that a content unit was created for the branch HEAD.

$ pulp-admin -u admin -p admin ostree repo search --repo-id=test-2594
+----------------------------------------------------------------------+
                             Content Units
+----------------------------------------------------------------------+

Id:        42f16f4e-b07b-4c8a-b58f-84f502f5c99e
Created:   2017-04-25T19:27:08Z
Updated:   2017-04-25T19:27:08Z
Remote Id: 7c5796ee9f12c773a6015c1a681760165aba1a2d0caf111783d6b2b42812b17e
Branch:    test
Commit:    3b13a82238b8d2482b0393ed9fb0bb3b26a80b0446716c4e6ffa136ad4d8931c
Version:   1.0

Make a few more commits:

$ ostree commit --repo=. -b test --add-metadata-string=version=2.0  /etc/pulp
626fa1e81e1cc2389b48bd40daf83cbf154e916dc43c68378ff042b539751e55

$ ostree commit --repo=. -b test --add-metadata-string=version=3.0  /etc/pulp
f8d09aa106f5679d15a5ccd38a5b1b8d7c9762cc7c28a340eacc89a57b1fd28a

$ ostree commit --repo=. -b test --add-metadata-string=version=4.0  /etc/pulp
ae0eff3aa393254205889f2fb37e3f28c6c23f7df0e3f150a25a6e21e3b7d5c3

$ ostree commit --repo=. -b test --add-metadata-string=version=5.0  /etc/pulp
3f2cf2aac14bedd1d7ad10e5b1966f7c4ca0baa8d060a26a5dd66b32e0ff8444

$ ostree summary -u --repo=.

Sync again and list the content.

- The list should contain the first branch HEAD commit
- Intermediate commits (this is new)
- The current branch HEAD commit.

Ignore the order.

$ pulp-admin -u admin -p admin ostree repo sync run --repo-id=test-2594
$ pulp-admin -u admin -p admin ostree repo search --repo-id=test-2594
+----------------------------------------------------------------------+
                             Content Units
+----------------------------------------------------------------------+

Id:        2084b330-eeb8-46cd-803d-d02176b86620
Created:   2017-04-25T19:35:42Z
Updated:   2017-04-25T19:35:42Z
Remote Id: 7c5796ee9f12c773a6015c1a681760165aba1a2d0caf111783d6b2b42812b17e
Branch:    test
Commit:    f8d09aa106f5679d15a5ccd38a5b1b8d7c9762cc7c28a340eacc89a57b1fd28a
Version:   3.0

Id:        42f16f4e-b07b-4c8a-b58f-84f502f5c99e
Created:   2017-04-25T19:27:08Z
Updated:   2017-04-25T19:35:42Z
Remote Id: 7c5796ee9f12c773a6015c1a681760165aba1a2d0caf111783d6b2b42812b17e
Branch:    test
Commit:    3b13a82238b8d2482b0393ed9fb0bb3b26a80b0446716c4e6ffa136ad4d8931c
Version:   1.0

Id:        80f953bc-3df1-4762-8c57-01b63f12fe0d
Created:   2017-04-25T19:34:44Z
Updated:   2017-04-25T19:35:42Z
Remote Id: 7c5796ee9f12c773a6015c1a681760165aba1a2d0caf111783d6b2b42812b17e
Branch:    test
Commit:    3f2cf2aac14bedd1d7ad10e5b1966f7c4ca0baa8d060a26a5dd66b32e0ff8444
Version:   5.0

Id:        e714bd17-68ef-4f86-b6f7-2b3483d06ea0
Created:   2017-04-25T19:35:42Z
Updated:   2017-04-25T19:35:42Z
Remote Id: 7c5796ee9f12c773a6015c1a681760165aba1a2d0caf111783d6b2b42812b17e
Branch:    test
Commit:    626fa1e81e1cc2389b48bd40daf83cbf154e916dc43c68378ff042b539751e55
Version:   2.0

Id:        f05d7941-9d3e-4f34-9553-df752b73959f
Created:   2017-04-25T19:35:42Z
Updated:   2017-04-25T19:35:42Z
Remote Id: 7c5796ee9f12c773a6015c1a681760165aba1a2d0caf111783d6b2b42812b17e
Branch:    test
Commit:    ae0eff3aa393254205889f2fb37e3f28c6c23f7df0e3f150a25a6e21e3b7d5c3
Version:   4.0
Actions #13

Updated by bmbouter almost 7 years ago

This test plan looks sweet. Can it be moved into a Pulp smash issue [0] and linked with the 'Smash Test' field? QE uses that field to help know if things already have a plan or not. Thanks for writing this test plan; they are important for the code quality.

[0]: https://github.com/PulpQE/pulp-smash/issues

Actions #14

Updated by jortel@redhat.com almost 7 years ago

  • Sprint/Milestone changed from 37 to 38

Added by jortel@redhat.com almost 7 years ago

Revision b16514b9 | View on GitHub

Units created using commit history. Added support for the depth option in the CLI. closes #2594

Actions #16

Updated by jortel@redhat.com almost 7 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #17

Updated by pcreech over 6 years ago

  • Platform Release set to 2.14.0
Actions #18

Updated by pcreech over 6 years ago

  • Status changed from MODIFIED to 5
Actions #19

Updated by pcreech over 6 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #20

Updated by bmbouter about 6 years ago

  • Sprint set to Sprint 19
Actions #21

Updated by bmbouter about 6 years ago

  • Sprint/Milestone deleted (38)
Actions #22

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF