Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #2723

closed

As a user GET /crane/repositories/v2 shows v2 repositories

Added by Ichimonji10 almost 6 years ago. Updated almost 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
ipanova@redhat.com
Start date:
Due date:
Estimated time:
Severity:
3. High
Version - Crane:
Platform Release:
2.14.0
Target Release - Crane:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 20
Quarter:

Description

Let's say that you've correctly installed the pulp_docker plugin and Crane. Let's say that you've also created, synced and published some Docker repositories. (v1 or v2 - it doesn't matter) You can now issue an HTTP GET request to /crane/repositories and get a report stating which repositories are available and the contents of each of those repositories. The report that's returned should look like this:

{
    "pulpdemo-busybox": {
        "image_ids": [
            "2982ec56c8d910121e7594ca7890b062f6d37fadf7575f6a6f3adbabbafac9f5",
            "2aed48a4e41d3931167146e9b7492aa5639e7f6478be9eac584726ecec6824ed",
            "492dad4279bae5bb73648efe9bf467b2cfa8bab1d593595226e3e7a95d9f6c35",
            "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125",
            "511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158",
            "618b1fc306b06d11e192812ede4c685dcbf886d2a0189e9a552c550fd7663df0",
            "df7546f9f060a2268024c8a230d8639878585defcc1bc6f79d2728a13957871b",
            "e8a999563c473139dc74d02eefb7b13ffea63799bc05b8936b9ad7119b37742f",
            "ea13149945cb6b1e746bf28032f02e9b5a793523481a0a18645fc77ad53c4ea2",
            "f6169d24347d30de48e4493836bec15c78a34f08cc7f17d6a45a19d68dc283ac"
        ],
        "protected": false,
        "tags": {
            "buildroot-2013.08.1": "2982ec56c8d910121e7594ca7890b062f6d37fadf7575f6a6f3adbabbafac9f5",
            "buildroot-2014.02": "2aed48a4e41d3931167146e9b7492aa5639e7f6478be9eac584726ecec6824ed",
            "latest": "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125",
            "ubuntu-12.04": "492dad4279bae5bb73648efe9bf467b2cfa8bab1d593595226e3e7a95d9f6c35",
            "ubuntu-14.04": "f6169d24347d30de48e4493836bec15c78a34f08cc7f17d6a45a19d68dc283ac"
        }
    },
    "pulpdemo-busybox2": {
        "image_ids": [
            "2982ec56c8d910121e7594ca7890b062f6d37fadf7575f6a6f3adbabbafac9f5",
            "2aed48a4e41d3931167146e9b7492aa5639e7f6478be9eac584726ecec6824ed",
            "492dad4279bae5bb73648efe9bf467b2cfa8bab1d593595226e3e7a95d9f6c35",
            "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125",
            "511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158",
            "618b1fc306b06d11e192812ede4c685dcbf886d2a0189e9a552c550fd7663df0",
            "df7546f9f060a2268024c8a230d8639878585defcc1bc6f79d2728a13957871b",
            "e8a999563c473139dc74d02eefb7b13ffea63799bc05b8936b9ad7119b37742f",
            "ea13149945cb6b1e746bf28032f02e9b5a793523481a0a18645fc77ad53c4ea2",
            "f6169d24347d30de48e4493836bec15c78a34f08cc7f17d6a45a19d68dc283ac"
        ],
        "protected": false,
        "tags": {
            "buildroot-2013.08.1": "2a4d48a4e51d39a1167146e9b7492aa5639e7f6478be9eac584726ecec6824ed",
            "latest": "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125",
            "ubuntu-12.04": "492dad4279bae5bb73648efe9bf467b2cfa8bab1d593595226e3e7a95d9f6c35",
            "ubuntu-14.04": "f6169d24347d30de48e4493836bec15c78a34f08cc7f17d6a45a19d68dc283ac"
        }
    }
}

Instead, the reports look like this:

{
    "pulpdemo-busybox": {
        "image_ids": [],
        "protected": false,
        "tags": {}
    },
    "pulpdemo-busybox2": {
        "image_ids": [],
        "protected": false,
        "tags": {}
    }
}

That's not right. Here's a script to reproduce the issue:

#!/usr/bin/env bash
pulp-admin docker repo create --repo-id foo --feed https://index.docker.io --upstream-name library/busybox
pulp-admin docker repo sync run --repo-id foo
pulp-admin docker repo publish run --repo-id foo

pulp-admin docker repo create --repo-id bar --feed https://registry-1.docker.io --upstream-name library/busybox
pulp-admin docker repo sync run --repo-id bar
pulp-admin docker repo publish run --repo-id bar

systemctl restart httpd  # make crane read metadata
curl -H 'accept:application/json' http://localhost:5000/crane/repositories | python -m json.tool

Here's the tail end of its output:

{
    "bar": {
        "image_ids": [],
        "protected": false,
        "tags": {}
    },
    "foo": {
        "image_ids": [],
        "protected": false,
        "tags": {}
    }
}

I've tested this issue against a matrix of nine VMs, where one axis is nightly builds of Pulp 2.12, 2.13 and 2.14, and the other axis is F24, F25 and RHEL7.

Actions
Copy link
 #1

Updated by ipanova@redhat.com almost 6 years ago

can you please show your crane.conf? specifically to what points the data_dir

Actions
Copy link
 #2

Updated by Ichimonji10 almost 6 years ago

/etc/crane.conf:

# See:
#
# * https://docs.pulpproject.org/plugins/pulp_docker/user-guide/recipes.html
# * https://docs.pulpproject.org/plugins/crane/index.html
# * https://pulp.plan.io/issues/2700
# * https://pulp.plan.io/issues/2717
#
[general]
data_dir: /var/lib/pulp/published/docker
debug: false

/etc/httpd/conf.d/pulp_crane.conf:

# Place this config in /etc/httpd/conf.d/. Use with Apache 2.4+. See:
#
# * https://docs.pulpproject.org/plugins/crane/index.html
# * https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html
#
Listen 5000
<VirtualHost *:5000>
    WSGIScriptAlias / /usr/share/crane/crane.wsgi
    <Directory /usr/share/crane/>
        Require all granted
    </Directory>
</VirtualHost>

By the way, docker works with these published repositories. Commands like docker pull localhost:5000/repo-name succeed.

Actions
Copy link
 #3

Updated by ipanova@redhat.com almost 6 years ago

when you created repo foo you needed to disable v2 and enable v1 like mentioned in recipes https://docs.pulpproject.org/plugins/pulp_docker/user-guide/recipes.html#sync
basically both repos are v2 content. that's why you don't see the v1 content.

if you do like in docs eventually you will see:

{
    "synctest": {
        "image_ids": [],
        "protected": false,
        "tags": {}
    },
    "v1synctest": {
        "image_ids": [
            "29017da868fba2d82fca0c416f5a91d3744bc8269a8b8ad0968a31837bf018bd",
            "4db501ca0880e5791368fa81e339ebb86ec2f2f6354abfe6b44f4b0304fadb00",
            "ff8f955d1fed83a6239675b9a767fc9502db9934922a2f69ac6bf4cad8a7d7be",
            "f1e4b055fb652a75379f41aee7b23a23a5f6dcc3b07e61f181d542ebb04f2ac9",
            "437595becdebaaaf3a4fc3db02c59a980f955dee825c153308c670610bb694e1",
            "2b8777f8541c6d03f7995f22506ecf4e4b3aaf73dc30a227844ab7c4cca16927",
            "d7057cb020844f245031d27b76cb18af05db1cc3a96a29fa7777af75f5ac91a3",
            "1c677c871a78a34d8af0f934762c121cb67c5ca9eb7f7aa82e031ad4f46d7b49",
            "bf0f46991aed1fa45508683e740601df23a2395c47f793883e4b0160ab906c1e",
            "79722f6accc3345ac95c95dd5dd4af5563cd68a04691e728df4724fc8e8ad581",
            "71571befa96b398b6fcb5640c22e118c52e4cd822e44dcf2241174c1f45818fa",
            "920777304d1d5e337bc59877253e946f224df5aae64c72538672eb74637b3c9e",
            "77608b59d92f23a988df653ad660e91e04c72d20bfaf1a7f06ae6a6a79331482",
            "c058d2db34b192789daa4c7c304f87f491b5602ca150328f726035e31efb3550",
            "633fcd11259e8d6bccfbb59a4086b95b0d0fb44edfc3912000ef1f70e8a7bfc6",
            "b65c936b5fb601d680ed656b1ccf8ab857c0e5cb521043a005405c194e9a69f3",
            "b84f77d69dcdac49a32315fde9274b726472505cd5edb0bdf112eadb64de060c",
            "2f0c79238b2fd6eed3ca4bea9b3b22388f9bc1d9882a18121fba9c2e4cee300a",
            "cfa753dfea5e68a24366dfba16e6edf573daa447abf65bc11619c1a98a3aff54",
            "c2ddbea624bdf70a9c1b1e843d48065509f4a631d8723aa347b401aebbf7ec98",
            "0f864637f229eee9da53fd5591fb58138b6bfb0196f0ee4fd9417d3655fb3d28",
            "79e69b372961d11d555c12fe0e2f0e8893abffb38ef43ef3eb479a3b654ae265",
            "cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff",
            "118a2fc47f99753bb828e50ef6d586c9013ade6f8739b9b64650ad1b3ecd0eb2",
            "3d5bcd78e074f6f77b820bf4c6db0e05d522e24c855f3c2a3bbf3b1c8f967ba8",
            "5f7820582aae34a153bd540c057ba80f62e94ad5cfab23a1f9533df984167748",
            "a6dbc8d6ddbb9e905518a9df65f414efce038de5f253a081b1205c6cea4bac17"
        ],
        "protected": false,
        "tags": {
            "1-ubuntu": "a6dbc8d6ddbb9e905518a9df65f414efce038de5f253a081b1205c6cea4bac17",
            "1.21-ubuntu": "a6dbc8d6ddbb9e905518a9df65f414efce038de5f253a081b1205c6cea4bac17",
            "1.21.0-ubuntu": "a6dbc8d6ddbb9e905518a9df65f414efce038de5f253a081b1205c6cea4bac17",
            "1.23": "d7057cb020844f245031d27b76cb18af05db1cc3a96a29fa7777af75f5ac91a3",
            "1.23.2": "d7057cb020844f245031d27b76cb18af05db1cc3a96a29fa7777af75f5ac91a3",
            "1.24": "437595becdebaaaf3a4fc3db02c59a980f955dee825c153308c670610bb694e1",
            "1.24-glibc": "c058d2db34b192789daa4c7c304f87f491b5602ca150328f726035e31efb3550",
            "1.24-musl": "71571befa96b398b6fcb5640c22e118c52e4cd822e44dcf2241174c1f45818fa",
            "1.24-uclibc": "437595becdebaaaf3a4fc3db02c59a980f955dee825c153308c670610bb694e1",
            "1.24.0": "3d5bcd78e074f6f77b820bf4c6db0e05d522e24c855f3c2a3bbf3b1c8f967ba8",
            "1.24.1": "f1e4b055fb652a75379f41aee7b23a23a5f6dcc3b07e61f181d542ebb04f2ac9",
            "1.24.1-glibc": "118a2fc47f99753bb828e50ef6d586c9013ade6f8739b9b64650ad1b3ecd0eb2",
            "1.24.1-musl": "29017da868fba2d82fca0c416f5a91d3744bc8269a8b8ad0968a31837bf018bd",
            "1.24.1-uclibc": "f1e4b055fb652a75379f41aee7b23a23a5f6dcc3b07e61f181d542ebb04f2ac9",
            "1.24.2": "437595becdebaaaf3a4fc3db02c59a980f955dee825c153308c670610bb694e1",
            "1.24.2-glibc": "c058d2db34b192789daa4c7c304f87f491b5602ca150328f726035e31efb3550",
            "1.24.2-uclibc": "437595becdebaaaf3a4fc3db02c59a980f955dee825c153308c670610bb694e1",
            "buildroot-2013.08.1": "1c677c871a78a34d8af0f934762c121cb67c5ca9eb7f7aa82e031ad4f46d7b49",
            "buildroot-2014.02": "0f864637f229eee9da53fd5591fb58138b6bfb0196f0ee4fd9417d3655fb3d28",
            "ubuntu": "a6dbc8d6ddbb9e905518a9df65f414efce038de5f253a081b1205c6cea4bac17",
            "ubuntu-12.04": "ff8f955d1fed83a6239675b9a767fc9502db9934922a2f69ac6bf4cad8a7d7be",
            "ubuntu-14.04": "633fcd11259e8d6bccfbb59a4086b95b0d0fb44edfc3912000ef1f70e8a7bfc6"
        }
    }
}

But the issues still persists here, the code needs to be updated, to parse the v2_data, seems like this was not working since we added v2 api support

https://github.com/pulp/crane/blob/master/crane/app_util.py#L186

Actions
Copy link
 #4

Updated by Ichimonji10 almost 6 years ago

Gotcha. pulp-admin docker repo create --help states that v1 content is synced by default. Is this a bug?

[root@fedora-24-pulp-2-12 ~]# pulp-admin docker repo create --help
Command: create
Description: creates a new repository

Available Arguments:

[snip!]
  --enable-v1        - Enable sync of v1 API. defaults to "true"
  --enable-v2        - Enable sync of v2 API. defaults to "true"

[snip!]

EDIT: I see the same output on a Pulp 2.14 system too.

Actions
Copy link
 #5

Updated by ipanova@redhat.com almost 6 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com
  • Sprint/Milestone set to 37
Actions
Copy link
 #6

Updated by ipanova@redhat.com almost 6 years ago

  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
Actions
Copy link
 #7

Updated by ipanova@redhat.com almost 6 years ago

  • Triaged changed from No to Yes
Actions
Copy link
 #8

Updated by Ichimonji10 almost 6 years ago

  • Subject changed from /crane/repositories incorrectly states that repositories are empty to GET /crane/repositories doesn't report content of v2 repositories

After some discussion with the developers, it turns out that HTTP GET requests to /crane/repositories will reveal v1 content, but not v2 content. I've updated the issue title accordingly.

Actions
Copy link
 #9

Updated by jortel@redhat.com almost 6 years ago

  • Sprint/Milestone changed from 37 to 38

Added by ipanova@redhat.com almost 6 years ago

Revision 71b7707f

Add endpoint /crane/repositories/v2 to display v2 repositories.

closes #2723 https://pulp.plan.io/issues/2723

Actions
Copy link
 #10

Updated by ipanova@redhat.com almost 6 years ago

  • Status changed from ASSIGNED to MODIFIED
Actions
Copy link
 #11

Updated by ipanova@redhat.com almost 6 years ago

  • Subject changed from GET /crane/repositories doesn't report content of v2 repositories to As a user GET /crane/repositories/v2 shows v2 repositories
  • Status changed from MODIFIED to POST

Added by ipanova@redhat.com almost 6 years ago

Revision a6c6e764

Add endpoint /crane/repositories/v2 to display v2 repositories.

closes #2723 https://pulp.plan.io/issues/2723

Actions
Copy link
 #12

Updated by ipanova@redhat.com almost 6 years ago

  • Status changed from POST to MODIFIED
Actions
Copy link
 #13

Updated by Ichimonji10 almost 6 years ago

  • Status changed from MODIFIED to ASSIGNED

This issue doesn't appear to have been fixed. The relevant tests are failing in Jenkins, and I can reproduce the issue 100% of the time. To reproduce this issue, I spun up a matrix of nine VMs, where the axes are:

  • Pulp 2.13 beta, 2.13 nightly, 2.14 nightly
  • Fedora 24, Fedora 25, RHEL 7

I then executed the automated tests which target this issue:

python -m unittest \
  pulp_smash.tests.docker.cli.test_sync_publish.RepoRegistryIdTestCase \
  pulp_smash.tests.docker.cli.test_sync_publish.SyncNonNamespacedV2TestCase \
  pulp_smash.tests.docker.cli.test_sync_publish.SyncPublishV2TestCase

All tests failed. For debugging purposes, I used some pprint() statements to print out the responses to GET /crane/repositories. Here's the output of one of those statements, where the JSON has been encoded as Python:

{'b83a36a1-4393-48a4-a722-6fde0d5c368d': {'image_ids': [],
                                          'protected': False,
                                          'tags': {}}}
Actions
Copy link
 #14

Updated by bmbouter almost 6 years ago

  • Sprint/Milestone changed from 38 to 39
Actions
Copy link
 #15

Updated by ipanova@redhat.com almost 6 years ago

I just checked, it looks like it is in working state.
1) did you check that repo 'b83a36a1-4393-48a4-a722-6fde0d5c368d' actually has the v1 content?
2) Also please mind this note https://github.com/pulp/crane/pull/72#issue-229358800
3) And this https://github.com/pulp/crane/pull/72/files#diff-caf2a6b8f4947d018f68893c695b5202R173

Actions
Copy link
 #16

Updated by ipanova@redhat.com almost 6 years ago

@ Ichimonji10 also would be good to check not just api representation but html as well.

Actions
Copy link
 #17

Updated by Ichimonji10 almost 6 years ago

1) did you check that repo 'b83a36a1-4393-48a4-a722-6fde0d5c368d' actually has the v1 content?

It has no v1 content.

2) Also please mind this note https://github.com/pulp/crane/pull/72#issue-229358800

Ahh. The automated test was wrong-ish. Thanks.

Actions
Copy link
 #18

Updated by Ichimonji10 almost 6 years ago

  • Status changed from ASSIGNED to MODIFIED

So, the fix is a little bit different than I expected: only the "protected" property is visible for v2 repositories. It works as described for the nightly builds of Pulp 2.14 and above.

Actions
Copy link
 #19

Updated by pcreech over 5 years ago

  • Platform Release set to 2.14.0
Actions
Copy link
 #20

Updated by pcreech over 5 years ago

  • Status changed from MODIFIED to 5
Actions
Copy link
 #21

Updated by pcreech over 5 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions
Copy link
 #22

Updated by bmbouter about 5 years ago

  • Sprint set to Sprint 20
Actions
Copy link
 #23

Updated by bmbouter about 5 years ago

  • Sprint/Milestone deleted (39)
Actions
Copy link
 #24

Updated by bmbouter almost 4 years ago

  • Tags Pulp 2 added
Actions
Send by e-mail Copy link

Also available in: Atom PDF