Project

Profile

Help

Issue #2723

As a user GET /crane/repositories/v2 shows v2 repositories

Added by Ichimonji10 almost 3 years ago. Updated 12 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Sprint/Milestone:
-
Start date:
Due date:
Severity:
3. High
Version - Crane:
Platform Release:
2.14.0
Blocks Release:
Target Release - Crane:
OS:
Backwards Incompatible:
No
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 20

Description

Let's say that you've correctly installed the pulp_docker plugin and Crane. Let's say that you've also created, synced and published some Docker repositories. (v1 or v2 - it doesn't matter) You can now issue an HTTP GET request to /crane/repositories and get a report stating which repositories are available and the contents of each of those repositories. The report that's returned should look like this:

{
    "pulpdemo-busybox": {
        "image_ids": [
            "2982ec56c8d910121e7594ca7890b062f6d37fadf7575f6a6f3adbabbafac9f5",
            "2aed48a4e41d3931167146e9b7492aa5639e7f6478be9eac584726ecec6824ed",
            "492dad4279bae5bb73648efe9bf467b2cfa8bab1d593595226e3e7a95d9f6c35",
            "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125",
            "511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158",
            "618b1fc306b06d11e192812ede4c685dcbf886d2a0189e9a552c550fd7663df0",
            "df7546f9f060a2268024c8a230d8639878585defcc1bc6f79d2728a13957871b",
            "e8a999563c473139dc74d02eefb7b13ffea63799bc05b8936b9ad7119b37742f",
            "ea13149945cb6b1e746bf28032f02e9b5a793523481a0a18645fc77ad53c4ea2",
            "f6169d24347d30de48e4493836bec15c78a34f08cc7f17d6a45a19d68dc283ac"
        ],
        "protected": false,
        "tags": {
            "buildroot-2013.08.1": "2982ec56c8d910121e7594ca7890b062f6d37fadf7575f6a6f3adbabbafac9f5",
            "buildroot-2014.02": "2aed48a4e41d3931167146e9b7492aa5639e7f6478be9eac584726ecec6824ed",
            "latest": "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125",
            "ubuntu-12.04": "492dad4279bae5bb73648efe9bf467b2cfa8bab1d593595226e3e7a95d9f6c35",
            "ubuntu-14.04": "f6169d24347d30de48e4493836bec15c78a34f08cc7f17d6a45a19d68dc283ac"
        }
    },
    "pulpdemo-busybox2": {
        "image_ids": [
            "2982ec56c8d910121e7594ca7890b062f6d37fadf7575f6a6f3adbabbafac9f5",
            "2aed48a4e41d3931167146e9b7492aa5639e7f6478be9eac584726ecec6824ed",
            "492dad4279bae5bb73648efe9bf467b2cfa8bab1d593595226e3e7a95d9f6c35",
            "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125",
            "511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158",
            "618b1fc306b06d11e192812ede4c685dcbf886d2a0189e9a552c550fd7663df0",
            "df7546f9f060a2268024c8a230d8639878585defcc1bc6f79d2728a13957871b",
            "e8a999563c473139dc74d02eefb7b13ffea63799bc05b8936b9ad7119b37742f",
            "ea13149945cb6b1e746bf28032f02e9b5a793523481a0a18645fc77ad53c4ea2",
            "f6169d24347d30de48e4493836bec15c78a34f08cc7f17d6a45a19d68dc283ac"
        ],
        "protected": false,
        "tags": {
            "buildroot-2013.08.1": "2a4d48a4e51d39a1167146e9b7492aa5639e7f6478be9eac584726ecec6824ed",
            "latest": "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125",
            "ubuntu-12.04": "492dad4279bae5bb73648efe9bf467b2cfa8bab1d593595226e3e7a95d9f6c35",
            "ubuntu-14.04": "f6169d24347d30de48e4493836bec15c78a34f08cc7f17d6a45a19d68dc283ac"
        }
    }
}

Instead, the reports look like this:

{
    "pulpdemo-busybox": {
        "image_ids": [],
        "protected": false,
        "tags": {}
    },
    "pulpdemo-busybox2": {
        "image_ids": [],
        "protected": false,
        "tags": {}
    }
}

That's not right. Here's a script to reproduce the issue:

#!/usr/bin/env bash
pulp-admin docker repo create --repo-id foo --feed https://index.docker.io --upstream-name library/busybox
pulp-admin docker repo sync run --repo-id foo
pulp-admin docker repo publish run --repo-id foo

pulp-admin docker repo create --repo-id bar --feed https://registry-1.docker.io --upstream-name library/busybox
pulp-admin docker repo sync run --repo-id bar
pulp-admin docker repo publish run --repo-id bar

systemctl restart httpd  # make crane read metadata
curl -H 'accept:application/json' http://localhost:5000/crane/repositories | python -m json.tool

Here's the tail end of its output:

{
    "bar": {
        "image_ids": [],
        "protected": false,
        "tags": {}
    },
    "foo": {
        "image_ids": [],
        "protected": false,
        "tags": {}
    }
}

I've tested this issue against a matrix of nine VMs, where one axis is nightly builds of Pulp 2.12, 2.13 and 2.14, and the other axis is F24, F25 and RHEL7.

Associated revisions

Revision 71b7707f View on GitHub
Added by ipanova@redhat.com almost 3 years ago

Add endpoint /crane/repositories/v2 to display v2 repositories.

closes #2723 https://pulp.plan.io/issues/2723

Revision a6c6e764 View on GitHub
Added by ipanova@redhat.com almost 3 years ago

Add endpoint /crane/repositories/v2 to display v2 repositories.

closes #2723 https://pulp.plan.io/issues/2723

History

#1 Updated by ipanova@redhat.com almost 3 years ago

can you please show your crane.conf? specifically to what points the data_dir

#2 Updated by Ichimonji10 almost 3 years ago

/etc/crane.conf:

# See:
#
# * https://docs.pulpproject.org/plugins/pulp_docker/user-guide/recipes.html
# * https://docs.pulpproject.org/plugins/crane/index.html
# * https://pulp.plan.io/issues/2700
# * https://pulp.plan.io/issues/2717
#
[general]
data_dir: /var/lib/pulp/published/docker
debug: false

/etc/httpd/conf.d/pulp_crane.conf:

# Place this config in /etc/httpd/conf.d/. Use with Apache 2.4+. See:
#
# * https://docs.pulpproject.org/plugins/crane/index.html
# * https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html
#
Listen 5000
<VirtualHost *:5000>
    WSGIScriptAlias / /usr/share/crane/crane.wsgi
    <Directory /usr/share/crane/>
        Require all granted
    </Directory>
</VirtualHost>

By the way, docker works with these published repositories. Commands like docker pull localhost:5000/repo-name succeed.

#3 Updated by ipanova@redhat.com almost 3 years ago

when you created repo foo you needed to disable v2 and enable v1 like mentioned in recipes https://docs.pulpproject.org/plugins/pulp_docker/user-guide/recipes.html#sync
basically both repos are v2 content. that's why you don't see the v1 content.

if you do like in docs eventually you will see:

{
    "synctest": {
        "image_ids": [],
        "protected": false,
        "tags": {}
    },
    "v1synctest": {
        "image_ids": [
            "29017da868fba2d82fca0c416f5a91d3744bc8269a8b8ad0968a31837bf018bd",
            "4db501ca0880e5791368fa81e339ebb86ec2f2f6354abfe6b44f4b0304fadb00",
            "ff8f955d1fed83a6239675b9a767fc9502db9934922a2f69ac6bf4cad8a7d7be",
            "f1e4b055fb652a75379f41aee7b23a23a5f6dcc3b07e61f181d542ebb04f2ac9",
            "437595becdebaaaf3a4fc3db02c59a980f955dee825c153308c670610bb694e1",
            "2b8777f8541c6d03f7995f22506ecf4e4b3aaf73dc30a227844ab7c4cca16927",
            "d7057cb020844f245031d27b76cb18af05db1cc3a96a29fa7777af75f5ac91a3",
            "1c677c871a78a34d8af0f934762c121cb67c5ca9eb7f7aa82e031ad4f46d7b49",
            "bf0f46991aed1fa45508683e740601df23a2395c47f793883e4b0160ab906c1e",
            "79722f6accc3345ac95c95dd5dd4af5563cd68a04691e728df4724fc8e8ad581",
            "71571befa96b398b6fcb5640c22e118c52e4cd822e44dcf2241174c1f45818fa",
            "920777304d1d5e337bc59877253e946f224df5aae64c72538672eb74637b3c9e",
            "77608b59d92f23a988df653ad660e91e04c72d20bfaf1a7f06ae6a6a79331482",
            "c058d2db34b192789daa4c7c304f87f491b5602ca150328f726035e31efb3550",
            "633fcd11259e8d6bccfbb59a4086b95b0d0fb44edfc3912000ef1f70e8a7bfc6",
            "b65c936b5fb601d680ed656b1ccf8ab857c0e5cb521043a005405c194e9a69f3",
            "b84f77d69dcdac49a32315fde9274b726472505cd5edb0bdf112eadb64de060c",
            "2f0c79238b2fd6eed3ca4bea9b3b22388f9bc1d9882a18121fba9c2e4cee300a",
            "cfa753dfea5e68a24366dfba16e6edf573daa447abf65bc11619c1a98a3aff54",
            "c2ddbea624bdf70a9c1b1e843d48065509f4a631d8723aa347b401aebbf7ec98",
            "0f864637f229eee9da53fd5591fb58138b6bfb0196f0ee4fd9417d3655fb3d28",
            "79e69b372961d11d555c12fe0e2f0e8893abffb38ef43ef3eb479a3b654ae265",
            "cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff",
            "118a2fc47f99753bb828e50ef6d586c9013ade6f8739b9b64650ad1b3ecd0eb2",
            "3d5bcd78e074f6f77b820bf4c6db0e05d522e24c855f3c2a3bbf3b1c8f967ba8",
            "5f7820582aae34a153bd540c057ba80f62e94ad5cfab23a1f9533df984167748",
            "a6dbc8d6ddbb9e905518a9df65f414efce038de5f253a081b1205c6cea4bac17"
        ],
        "protected": false,
        "tags": {
            "1-ubuntu": "a6dbc8d6ddbb9e905518a9df65f414efce038de5f253a081b1205c6cea4bac17",
            "1.21-ubuntu": "a6dbc8d6ddbb9e905518a9df65f414efce038de5f253a081b1205c6cea4bac17",
            "1.21.0-ubuntu": "a6dbc8d6ddbb9e905518a9df65f414efce038de5f253a081b1205c6cea4bac17",
            "1.23": "d7057cb020844f245031d27b76cb18af05db1cc3a96a29fa7777af75f5ac91a3",
            "1.23.2": "d7057cb020844f245031d27b76cb18af05db1cc3a96a29fa7777af75f5ac91a3",
            "1.24": "437595becdebaaaf3a4fc3db02c59a980f955dee825c153308c670610bb694e1",
            "1.24-glibc": "c058d2db34b192789daa4c7c304f87f491b5602ca150328f726035e31efb3550",
            "1.24-musl": "71571befa96b398b6fcb5640c22e118c52e4cd822e44dcf2241174c1f45818fa",
            "1.24-uclibc": "437595becdebaaaf3a4fc3db02c59a980f955dee825c153308c670610bb694e1",
            "1.24.0": "3d5bcd78e074f6f77b820bf4c6db0e05d522e24c855f3c2a3bbf3b1c8f967ba8",
            "1.24.1": "f1e4b055fb652a75379f41aee7b23a23a5f6dcc3b07e61f181d542ebb04f2ac9",
            "1.24.1-glibc": "118a2fc47f99753bb828e50ef6d586c9013ade6f8739b9b64650ad1b3ecd0eb2",
            "1.24.1-musl": "29017da868fba2d82fca0c416f5a91d3744bc8269a8b8ad0968a31837bf018bd",
            "1.24.1-uclibc": "f1e4b055fb652a75379f41aee7b23a23a5f6dcc3b07e61f181d542ebb04f2ac9",
            "1.24.2": "437595becdebaaaf3a4fc3db02c59a980f955dee825c153308c670610bb694e1",
            "1.24.2-glibc": "c058d2db34b192789daa4c7c304f87f491b5602ca150328f726035e31efb3550",
            "1.24.2-uclibc": "437595becdebaaaf3a4fc3db02c59a980f955dee825c153308c670610bb694e1",
            "buildroot-2013.08.1": "1c677c871a78a34d8af0f934762c121cb67c5ca9eb7f7aa82e031ad4f46d7b49",
            "buildroot-2014.02": "0f864637f229eee9da53fd5591fb58138b6bfb0196f0ee4fd9417d3655fb3d28",
            "ubuntu": "a6dbc8d6ddbb9e905518a9df65f414efce038de5f253a081b1205c6cea4bac17",
            "ubuntu-12.04": "ff8f955d1fed83a6239675b9a767fc9502db9934922a2f69ac6bf4cad8a7d7be",
            "ubuntu-14.04": "633fcd11259e8d6bccfbb59a4086b95b0d0fb44edfc3912000ef1f70e8a7bfc6"
        }
    }
}

But the issues still persists here, the code needs to be updated, to parse the v2_data, seems like this was not working since we added v2 api support

https://github.com/pulp/crane/blob/master/crane/app_util.py#L186

#4 Updated by Ichimonji10 almost 3 years ago

Gotcha. pulp-admin docker repo create --help states that v1 content is synced by default. Is this a bug?

[root@fedora-24-pulp-2-12 ~]# pulp-admin docker repo create --help
Command: create
Description: creates a new repository

Available Arguments:

[snip!]
  --enable-v1        - Enable sync of v1 API. defaults to "true"
  --enable-v2        - Enable sync of v2 API. defaults to "true"

[snip!]

EDIT: I see the same output on a Pulp 2.14 system too.

#5 Updated by ipanova@redhat.com almost 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com
  • Sprint/Milestone set to 37

#6 Updated by ipanova@redhat.com almost 3 years ago

  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High

#7 Updated by ipanova@redhat.com almost 3 years ago

  • Triaged changed from No to Yes

#8 Updated by Ichimonji10 almost 3 years ago

  • Subject changed from /crane/repositories incorrectly states that repositories are empty to GET /crane/repositories doesn't report content of v2 repositories

After some discussion with the developers, it turns out that HTTP GET requests to /crane/repositories will reveal v1 content, but not v2 content. I've updated the issue title accordingly.

#9 Updated by jortel@redhat.com almost 3 years ago

  • Sprint/Milestone changed from 37 to 38

#10 Updated by ipanova@redhat.com almost 3 years ago

  • Status changed from ASSIGNED to MODIFIED

#11 Updated by ipanova@redhat.com almost 3 years ago

  • Subject changed from GET /crane/repositories doesn't report content of v2 repositories to As a user GET /crane/repositories/v2 shows v2 repositories
  • Status changed from MODIFIED to POST

#12 Updated by ipanova@redhat.com almost 3 years ago

  • Status changed from POST to MODIFIED

#13 Updated by Ichimonji10 almost 3 years ago

  • Status changed from MODIFIED to ASSIGNED

This issue doesn't appear to have been fixed. The relevant tests are failing in Jenkins, and I can reproduce the issue 100% of the time. To reproduce this issue, I spun up a matrix of nine VMs, where the axes are:

  • Pulp 2.13 beta, 2.13 nightly, 2.14 nightly
  • Fedora 24, Fedora 25, RHEL 7

I then executed the automated tests which target this issue:

python -m unittest \
  pulp_smash.tests.docker.cli.test_sync_publish.RepoRegistryIdTestCase \
  pulp_smash.tests.docker.cli.test_sync_publish.SyncNonNamespacedV2TestCase \
  pulp_smash.tests.docker.cli.test_sync_publish.SyncPublishV2TestCase

All tests failed. For debugging purposes, I used some pprint() statements to print out the responses to GET /crane/repositories. Here's the output of one of those statements, where the JSON has been encoded as Python:

{'b83a36a1-4393-48a4-a722-6fde0d5c368d': {'image_ids': [],
                                          'protected': False,
                                          'tags': {}}}

#14 Updated by bmbouter almost 3 years ago

  • Sprint/Milestone changed from 38 to 39

#15 Updated by ipanova@redhat.com almost 3 years ago

I just checked, it looks like it is in working state.
1) did you check that repo 'b83a36a1-4393-48a4-a722-6fde0d5c368d' actually has the v1 content?
2) Also please mind this note https://github.com/pulp/crane/pull/72#issue-229358800
3) And this https://github.com/pulp/crane/pull/72/files#diff-caf2a6b8f4947d018f68893c695b5202R173

#16 Updated by ipanova@redhat.com almost 3 years ago

@ Ichimonji10 also would be good to check not just api representation but html as well.

#17 Updated by Ichimonji10 almost 3 years ago

1) did you check that repo 'b83a36a1-4393-48a4-a722-6fde0d5c368d' actually has the v1 content?

It has no v1 content.

2) Also please mind this note https://github.com/pulp/crane/pull/72#issue-229358800

Ahh. The automated test was wrong-ish. Thanks.

#18 Updated by Ichimonji10 almost 3 years ago

  • Status changed from ASSIGNED to MODIFIED

So, the fix is a little bit different than I expected: only the "protected" property is visible for v2 repositories. It works as described for the nightly builds of Pulp 2.14 and above.

#19 Updated by pcreech over 2 years ago

  • Platform Release set to 2.14.0

#20 Updated by pcreech over 2 years ago

  • Status changed from MODIFIED to ON_QA

#21 Updated by pcreech over 2 years ago

  • Status changed from ON_QA to CLOSED - CURRENTRELEASE

#22 Updated by bmbouter about 2 years ago

  • Sprint set to Sprint 20

#23 Updated by bmbouter about 2 years ago

  • Sprint/Milestone deleted (39)

#24 Updated by bmbouter 12 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF