Project

Profile

Help

Test #4129

Test sync of a repository that returns a 403 response for a download request

Added by amacdona@redhat.com about 1 year ago. Updated 8 months ago.

Status:
CLOSED - COMPLETE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Platform Release:
Tags:
Pulp 2
Sprint:

Description

This test will require the creation of a new fixture for pulp_docker that is a corrupted repository. This fixture is primarily intended to test https://pulp.plan.io/issues/2966 (pulp 2) but could also be useful for testing pulp 3.

From 2966, it appears that a 403 can be caused by a missing symlink in the published repository.


Related issues

Related to Docker Support - Issue #2966: Unable to sync docker repo because worker dies CLOSED - CURRENTRELEASE Actions
Related to Pulp - Test #4259: As a user I can sync from a docker registry CLOSED - COMPLETE Actions

History

#1 Updated by amacdona@redhat.com about 1 year ago

  • Related to Issue #2966: Unable to sync docker repo because worker dies added

#2 Updated by amacdona@redhat.com about 1 year ago

I was able to reproduce this with a pulp-pulp sync.

https://pulp.plan.io/issues/2849#note-14
https://pulp.plan.io/issues/2966#note-20

These notes explain how it was done, and I confirmed that a broken symlink raises the 403.

#3 Updated by bherring about 1 year ago

  • Assignee set to bherring

#4 Updated by bherring about 1 year ago

  • Status changed from NEW to ASSIGNED

#5 Updated by amacdona@redhat.com about 1 year ago

Unfortunately, whatever we use as a fixture must be able to return specific headers, which are used by the sync code. This (AFAIK) rules out fedorapeople.

#6 Updated by bherring about 1 year ago

Thanks for taking the time to talk to me about our current testing of this,

Just so the more verbose version is captured here:

Testing comes down to is that docker sync relies on certain headers coming down from the repository. As far as we can tell, fedorapeople doesn't give us the option to control the headers. Therefore with our current test harness, this appears to not be a candidate for automation.

The intention is to still manually test the current fix manually to ensure the current master for beta/GA has the fix. Also, this Trackers will serve to document any test setup and references for future use, mostly as document centralization and re-link.

When QE moves to Pulp3 and using Travis, it may make sense to implement more robust testing, such as spinning-up multiple containers as part of the test or test harness for instances such as this in the correct part of the test cycle.

Once it is clear where `Tracker: Test Story` trackers for Story or Epics to consider in the future, I will link that work to any additional testing done in this Tracker.

#7 Updated by bherring about 1 year ago

  • Status changed from ASSIGNED to CLOSED - COMPLETE

Verified Release

Platform Version: 2.19a1

Status

Works as Expected

Verification Log/Snippet

By moving the destination of the symlink blob on the master_node, a 403 sync issue was created with crane.

Pulp successfully prevents a corrupted repo, as noted by the well documented source RM #2849.

Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:e7277075c9a84b0f55bcdd09f24116677c8209bb25107366c6cd3153e1a186bd.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:56e295030a62a3f50bd00ffde14f90232cb019cacaa9b63e5647b36c596c9dd0.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download failed: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed with code 403: Forbidden
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:e1e5da6c811515844589523b3f8f77db1f5fcc98f3f7360a62c1f800e8203f70 was cancelled
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:87213c40360b3c4ebd5f410429b643dba098e51c42c505655bf72e4ec7937c9b was cancelled
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:9500078affc13405feaacbd641347e3b0524305176bafd2773113a969672dfad was cancelled
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:224a21997e8ca8514d42eb2ed98b19a7ee2537bce0b3a26b8dff510ab637f15c.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: pulp.server.async.tasks:INFO: [69368909] Task failed : [69368909-4bbe-4988-b98d-303f085b9ba9] : Image download(s) from http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed. Sync task has failed to prevent a corrupted repository.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: celery.app.trace:INFO: [69368909] Task pulp.server.managers.repo.sync.sync[69368909-4bbe-4988-b98d-303f085b9ba9] raised expected: PulpCodedException()
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: celery.app.trace:INFO: [15f40f3b] Task pulp.server.async.tasks._release_resource[15f40f3b-5733-4afd-8272-82a545884a40] succeeded in 0.00527241999953s: None

Recreation Notes

Since a lot of the information about setting up the test harness was non-obvious, the notes for each section here is noted.

Hopefully this can aid as a reference in the future if anyone requires these steps.

High Level Test Design

  • Remote Docker Repo <---> Pulp Docker Repo Master_Node <----> Pulp Docker Slave_Node

The goal is to test the sync of `Pulp Docker Node` with `Pulp Docker Repo Master Node` when the master node is in a state that causes a 403 or 404 sync error. Ideally, it would be ANY error and verification that Pulp handles it sanely.

Test Caveats and Scope

The scope for test is manually tested regression verification, in this instance.

Ideally, the Remote Docker Repo would able to be manipulated making the harness simpler.

Also, the current test harness does not easily handle multiple pulp instances for automated test execution. This will be improved in the future so test instances such as this can be fully automated looking for regressions.

General Test Scenario

  1. Load a remote docker repo. In this instance. docker.io was used
    pulp-admin -u admin -p admin docker repo create --repo-id test --feed https://index.docker.io --upstream-name library/busybox
    
  2. Verify the state of Master_Node
    pulp-admin -u admin -p admin repo list
    
  3. Configure CRANE on the Master_Node for docker pull request redirection - /etc/crane.conf
    [general]
    data_dir: /var/lib/pulp/published/docker
    debug: true
    endpoint: kvm-04-guest09.rhts.eng.bos.redhat.com:5000
    
  4. Turn-off SELinux and firewalld or correctly deal with any issues that would have these ports blocked.
    setenforce 0
    systemctl stop firewalld
    
  5. Verify Slave_Node and synch with the Master_Node. An additional or test verification of a `docker pull` against the Master_Node can also verify functionality of Crane on the Master_Node
    pulp-admin -u admin -p admin docker repo create --repo-id fixture --upstream-name test --feed http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000
    
    pulp-admin -u admin -p admin docker repo sync run --repo-id fixture
    
    ## If there are errors, see if curl works for secure and insecure
    curl https://kvm-04-guest09.rhts.eng.bos.redhat.com/pulp/docker/v2/test/tags/list
    
    curl --insecure https://kvm-04-guest09.rhts.eng.bos.redhat.com/pulp/docker/v2/test/tags/list
    
  6. Modify the source of a symbolic link on the Master_Node in /var/lib/pulp/published/docker/v2/web/<repo>/blobs
    cd /var/lib/pulp/published/docker/v2/web/test/blobs
    mv /var/lib/pulp/content/units/docker_blob/63/09d7d49789d2dd03ae713c55be95d3fe2fabd7cb011ea3dbedd001646fe857/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 /var/lib/pulp/content/units/docker_blob/63/09d7d49789d2dd03ae713c55be95d3fe2fabd7cb011ea3dbedd001646fe857/sha256-moved:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04
    
  7. Drop the Tables and restart the workers on the Slave_Node for a clean state without cached blobs. To do this simply, pulp/devl/ had some bash aliases in the `2-master` branch that was used to drop the Mongo Tables and clean Pulp2
    cd /root
    git clone https://github.com/pulp/devel.git
    cp -p devel/ansible/roles/dev/files/drop_database.js .
    source devel/ansible/roles/dev/files/bashrc
    pclean
    
  8. Add the remote repo with `--force-full` that has the expected error
    pulp-admin -u admin -p admin docker repo create --repo-id fixture --upstream-name test --feed http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000
    pulp-admin -u admin -p admin docker repo sync run --repo-id fixture --force-full
    
  9. Expect the failure on sync
    +----------------------------------------------------------------------+
                       Synchronizing Repository [fixture]
    +----------------------------------------------------------------------+
    
    This command may be exited via ctrl+c without affecting the request.
    
    Downloading manifests
    [/]
    ... completed
    
    Copying units already in pulp
    [-]
    ... completed
    
    Copying units already in pulp
    [-]
    ... completed
    
    Downloading remote files
    [===============================                   ] 62%
    235 of 375 items
    ... failed
    
    Task Failed
    
    Image download(s) from
    http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d
    3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed. Sync task has
    failed to prevent a corrupted repository.
    
    [root@kvm-04-guest08 ~]#
    
  10. Expect the 403/404 error in journalctl, depending on the type of 4xx error produced. This is an example of 403.
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:56e295030a62a3f50bd00ffde14f90232cb019cacaa9b63e5647b36c596c9dd0.
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download failed: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed with code 403: Forbidden
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:e1e5da6c811515844589523b3f8f77db1f5fcc98f3f7360a62c1f800e8203f70 was cancelled
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:87213c40360b3c4ebd5f410429b643dba098e51c42c505655bf72e4ec7937c9b was cancelled
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:9500078affc13405feaacbd641347e3b0524305176bafd2773113a969672dfad was cancelled
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:224a21997e8ca8514d42eb2ed98b19a7ee2537bce0b3a26b8dff510ab637f15c.
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: pulp.server.async.tasks:INFO: [69368909] Task failed : [69368909-4bbe-4988-b98d-303f085b9ba9] : Image download(s) from http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed. Sync task has failed to prevent a corrupted repository.
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: celery.app.trace:INFO: [69368909] Task pulp.server.managers.repo.sync.sync[69368909-4bbe-4988-b98d-303f085b9ba9] raised expected: PulpCodedException()
    Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: celery.app.trace:INFO: [15f40f3b] Task pulp.server.async.tasks._release_resource[15f40f3b-5733-4afd-8272-82a545884a40] succeeded in 0.00527241999953s: None
    ~ 
    

Other Hacks

Edit Pulp to not to SSL Validation

Ideally, SSL configuration is done correctly on the Master_Node. However, if it is not, this will cause issues with the pulp <-> pulp sync.

To get around this in a test-only scenario for recreation, SSL validation can be `turned off`.

Hacks:
- Edit: /usr/lib/python2.7/site-packages/pulp/plugins/util/nectar_config.py
    101     download_config.ssl_validation = False

Reset the worker on the Node and retest connectivty

#Look for the workers and restart each one. There is a single worker in this example
systemctl | grep pulp 
systemctl restart pulp_worker-0

#Restart HTTPD, just to be sure.
systemctl restart httpd
systemctl status httpd

References

[0] - https://pulp.plan.io/issues/4129
[1] - https://pulp.plan.io/issues/4143
[2] - https://github.com/pulp/pulp-ci/tree/master/ci/ansible/roles/pulp-crane/tasks
[3] - https://docs.docker.com/registry/insecure/#deploy-a-plain-http-registry
[4] - https://github.com/pulp/devel.git
[5] - https://github.com/pulp/pulp-ci/blob/master/ci/ansible/pulp_server.yaml
[6] - https://github.com/pulp/crane
[7] - https://docs.pulpproject.org/plugins/pulp_docker/user-guide/recipes.html
[8] - https://docs.pulpproject.org/plugins/crane/index.html
[9] - https://mojo.redhat.com/docs/DOC-1059499

#8 Updated by bherring about 1 year ago

  • Related to Test #4259: As a user I can sync from a docker registry added

#9 Updated by bmbouter 8 months ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF