Docker Support

Verified Release¶

Platform Version: 2.19a1

Status¶

Works as Expected

Verification Log/Snippet¶

By moving the destination of the symlink blob on the master_node, a 403 sync issue was created with crane.

Pulp successfully prevents a corrupted repo, as noted by the well documented source RM #2849.

Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:e7277075c9a84b0f55bcdd09f24116677c8209bb25107366c6cd3153e1a186bd.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:56e295030a62a3f50bd00ffde14f90232cb019cacaa9b63e5647b36c596c9dd0.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download failed: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed with code 403: Forbidden
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:e1e5da6c811515844589523b3f8f77db1f5fcc98f3f7360a62c1f800e8203f70 was cancelled
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:87213c40360b3c4ebd5f410429b643dba098e51c42c505655bf72e4ec7937c9b was cancelled
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:9500078affc13405feaacbd641347e3b0524305176bafd2773113a969672dfad was cancelled
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:224a21997e8ca8514d42eb2ed98b19a7ee2537bce0b3a26b8dff510ab637f15c.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: pulp.server.async.tasks:INFO: [69368909] Task failed : [69368909-4bbe-4988-b98d-303f085b9ba9] : Image download(s) from http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed. Sync task has failed to prevent a corrupted repository.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: celery.app.trace:INFO: [69368909] Task pulp.server.managers.repo.sync.sync[69368909-4bbe-4988-b98d-303f085b9ba9] raised expected: PulpCodedException()
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: celery.app.trace:INFO: [15f40f3b] Task pulp.server.async.tasks._release_resource[15f40f3b-5733-4afd-8272-82a545884a40] succeeded in 0.00527241999953s: None

Recreation Notes¶

Since a lot of the information about setting up the test harness was non-obvious, the notes for each section here is noted.

Hopefully this can aid as a reference in the future if anyone requires these steps.

High Level Test Design¶

• Remote Docker Repo <---> Pulp Docker Repo Master_Node <----> Pulp Docker Slave_Node

The goal is to test the sync of `Pulp Docker Node` with `Pulp Docker Repo Master Node` when the master node is in a state that causes a 403 or 404 sync error. Ideally, it would be ANY error and verification that Pulp handles it sanely.

Test Caveats and Scope¶

The scope for test is manually tested regression verification, in this instance.

Ideally, the Remote Docker Repo would able to be manipulated making the harness simpler.

Also, the current test harness does not easily handle multiple pulp instances for automated test execution. This will be improved in the future so test instances such as this can be fully automated looking for regressions.

General Test Scenario¶

1. Load a remote docker repo. In this instance. docker.io was used

   pulp-admin -u admin -p admin docker repo create --repo-id test --feed https://index.docker.io --upstream-name library/busybox
   

2. Verify the state of Master_Node

   pulp-admin -u admin -p admin repo list
   

3. Configure CRANE on the Master_Node for docker pull request redirection - /etc/crane.conf

   [general]
   data_dir: /var/lib/pulp/published/docker
   debug: true
   endpoint: kvm-04-guest09.rhts.eng.bos.redhat.com:5000
   

4. Turn-off SELinux and firewalld or correctly deal with any issues that would have these ports blocked.

   setenforce 0
   systemctl stop firewalld
   

5. Verify Slave_Node and synch with the Master_Node. An additional or test verification of a `docker pull` against the Master_Node can also verify functionality of Crane on the Master_Node

   pulp-admin -u admin -p admin docker repo create --repo-id fixture --upstream-name test --feed http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000
   
   pulp-admin -u admin -p admin docker repo sync run --repo-id fixture
   
   ## If there are errors, see if curl works for secure and insecure
   curl https://kvm-04-guest09.rhts.eng.bos.redhat.com/pulp/docker/v2/test/tags/list
   
   curl --insecure https://kvm-04-guest09.rhts.eng.bos.redhat.com/pulp/docker/v2/test/tags/list
   

6. Modify the source of a symbolic link on the Master_Node in /var/lib/pulp/published/docker/v2/web/<repo>/blobs

   cd /var/lib/pulp/published/docker/v2/web/test/blobs
   mv /var/lib/pulp/content/units/docker_blob/63/09d7d49789d2dd03ae713c55be95d3fe2fabd7cb011ea3dbedd001646fe857/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 /var/lib/pulp/content/units/docker_blob/63/09d7d49789d2dd03ae713c55be95d3fe2fabd7cb011ea3dbedd001646fe857/sha256-moved:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04
   

7. Drop the Tables and restart the workers on the Slave_Node for a clean state without cached blobs. To do this simply, pulp/devl/ had some bash aliases in the `2-master` branch that was used to drop the Mongo Tables and clean Pulp2

   cd /root
   git clone https://github.com/pulp/devel.git
   cp -p devel/ansible/roles/dev/files/drop_database.js .
   source devel/ansible/roles/dev/files/bashrc
   pclean
   

8. Add the remote repo with `--force-full` that has the expected error

   pulp-admin -u admin -p admin docker repo create --repo-id fixture --upstream-name test --feed http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000
   pulp-admin -u admin -p admin docker repo sync run --repo-id fixture --force-full
   

9. Expect the failure on sync

   +----------------------------------------------------------------------+
                      Synchronizing Repository [fixture]
   +----------------------------------------------------------------------+
   
   This command may be exited via ctrl+c without affecting the request.
   
   Downloading manifests
   [/]
   ... completed
   
   Copying units already in pulp
   [-]
   ... completed
   
   Copying units already in pulp
   [-]
   ... completed
   
   Downloading remote files
   [===============================                   ] 62%
   235 of 375 items
   ... failed
   
   Task Failed
   
   Image download(s) from
   http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d
   3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed. Sync task has
   failed to prevent a corrupted repository.
   
   [root@kvm-04-guest08 ~]#
   

10. Expect the 403/404 error in journalctl, depending on the type of 4xx error produced. This is an example of 403.

    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:56e295030a62a3f50bd00ffde14f90232cb019cacaa9b63e5647b36c596c9dd0.
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download failed: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed with code 403: Forbidden
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:e1e5da6c811515844589523b3f8f77db1f5fcc98f3f7360a62c1f800e8203f70 was cancelled
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:87213c40360b3c4ebd5f410429b643dba098e51c42c505655bf72e4ec7937c9b was cancelled
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:9500078affc13405feaacbd641347e3b0524305176bafd2773113a969672dfad was cancelled
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:224a21997e8ca8514d42eb2ed98b19a7ee2537bce0b3a26b8dff510ab637f15c.
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: pulp.server.async.tasks:INFO: [69368909] Task failed : [69368909-4bbe-4988-b98d-303f085b9ba9] : Image download(s) from http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed. Sync task has failed to prevent a corrupted repository.
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: celery.app.trace:INFO: [69368909] Task pulp.server.managers.repo.sync.sync[69368909-4bbe-4988-b98d-303f085b9ba9] raised expected: PulpCodedException()
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: celery.app.trace:INFO: [15f40f3b] Task pulp.server.async.tasks._release_resource[15f40f3b-5733-4afd-8272-82a545884a40] succeeded in 0.00527241999953s: None
    ~ 
    

Other Hacks¶

Edit Pulp to not to SSL Validation¶

Ideally, SSL configuration is done correctly on the Master_Node. However, if it is not, this will cause issues with the pulp <-> pulp sync.

To get around this in a test-only scenario for recreation, SSL validation can be `turned off`.

Hacks:
- Edit: /usr/lib/python2.7/site-packages/pulp/plugins/util/nectar_config.py
    101     download_config.ssl_validation = False

Reset the worker on the Node and retest connectivty

#Look for the workers and restart each one. There is a single worker in this example
systemctl | grep pulp 
systemctl restart pulp_worker-0

#Restart HTTPD, just to be sure.
systemctl restart httpd
systemctl status httpd

References¶

[0] - https://pulp.plan.io/issues/4129
[1] - https://pulp.plan.io/issues/4143
[2] - https://github.com/pulp/pulp-ci/tree/master/ci/ansible/roles/pulp-crane/tasks
[3] - https://docs.docker.com/registry/insecure/#deploy-a-plain-http-registry
[4] - https://github.com/pulp/devel.git
[5] - https://github.com/pulp/pulp-ci/blob/master/ci/ansible/pulp_server.yaml
[6] - https://github.com/pulp/crane
[7] - https://docs.pulpproject.org/plugins/pulp_docker/user-guide/recipes.html
[8] - https://docs.pulpproject.org/plugins/crane/index.html
[9] - https://mojo.redhat.com/docs/DOC-1059499