Test #4129
closed
Test sync of a repository that returns a 403 response for a download request
Added by amacdona@redhat.com over 5 years ago. Updated about 5 years ago.
Description
This test will require the creation of a new fixture for pulp_docker that is a corrupted repository. This fixture is primarily intended to test https://pulp.plan.io/issues/2966 (pulp 2) but could also be useful for testing pulp 3.
From 2966, it appears that a 403 can be caused by a missing symlink in the published repository.
Related issues
Updated by amacdona@redhat.com over 5 years ago
- Related to Issue #2966: Unable to sync docker repo because worker dies added
Updated by amacdona@redhat.com over 5 years ago
I was able to reproduce this with a pulp-pulp sync.
https://pulp.plan.io/issues/2849#note-14
https://pulp.plan.io/issues/2966#note-20
These notes explain how it was done, and I confirmed that a broken symlink raises the 403.
Updated by amacdona@redhat.com over 5 years ago
Unfortunately, whatever we use as a fixture must be able to return specific headers, which are used by the sync code. This (AFAIK) rules out fedorapeople.
Updated by bherring over 5 years ago
Thanks for taking the time to talk to me about our current testing of this, amacdona@redhat.com
Just so the more verbose version is captured here:
Testing comes down to is that docker sync relies on certain headers coming down from the repository. As far as we can tell, fedorapeople doesn't give us the option to control the headers. Therefore with our current test harness, this appears to not be a candidate for automation.
The intention is to still manually test the current fix manually to ensure the current master for beta/GA has the fix. Also, this Trackers will serve to document any test setup and references for future use, mostly as document centralization and re-link.
When QE moves to Pulp3 and using Travis, it may make sense to implement more robust testing, such as spinning-up multiple containers as part of the test or test harness for instances such as this in the correct part of the test cycle.
Once it is clear where `Tracker: Test Story` trackers for Story or Epics to consider in the future, I will link that work to any additional testing done in this Tracker.
Updated by bherring over 5 years ago
- Status changed from ASSIGNED to CLOSED - COMPLETE
Verified Release¶
Platform Version: 2.19a1
Status¶
Works as Expected
Verification Log/Snippet¶
By moving the destination of the symlink blob on the master_node, a 403 sync issue was created with crane.
Pulp successfully prevents a corrupted repo, as noted by the well documented source RM #2849.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:e7277075c9a84b0f55bcdd09f24116677c8209bb25107366c6cd3153e1a186bd.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:56e295030a62a3f50bd00ffde14f90232cb019cacaa9b63e5647b36c596c9dd0.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download failed: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed with code 403: Forbidden
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:e1e5da6c811515844589523b3f8f77db1f5fcc98f3f7360a62c1f800e8203f70 was cancelled
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:87213c40360b3c4ebd5f410429b643dba098e51c42c505655bf72e4ec7937c9b was cancelled
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:9500078affc13405feaacbd641347e3b0524305176bafd2773113a969672dfad was cancelled
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:224a21997e8ca8514d42eb2ed98b19a7ee2537bce0b3a26b8dff510ab637f15c.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: pulp.server.async.tasks:INFO: [69368909] Task failed : [69368909-4bbe-4988-b98d-303f085b9ba9] : Image download(s) from http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed. Sync task has failed to prevent a corrupted repository.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: celery.app.trace:INFO: [69368909] Task pulp.server.managers.repo.sync.sync[69368909-4bbe-4988-b98d-303f085b9ba9] raised expected: PulpCodedException()
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: celery.app.trace:INFO: [15f40f3b] Task pulp.server.async.tasks._release_resource[15f40f3b-5733-4afd-8272-82a545884a40] succeeded in 0.00527241999953s: None
Recreation Notes¶
Since a lot of the information about setting up the test harness was non-obvious, the notes for each section here is noted.
Hopefully this can aid as a reference in the future if anyone requires these steps.
High Level Test Design¶
- Remote Docker Repo <---> Pulp Docker Repo Master_Node <----> Pulp Docker Slave_Node
The goal is to test the sync of `Pulp Docker Node` with `Pulp Docker Repo Master Node` when the master node is in a state that causes a 403 or 404 sync error. Ideally, it would be ANY error and verification that Pulp handles it sanely.
Test Caveats and Scope¶
The scope for test is manually tested regression verification, in this instance.
Ideally, the Remote Docker Repo would able to be manipulated making the harness simpler.
Also, the current test harness does not easily handle multiple pulp instances for automated test execution. This will be improved in the future so test instances such as this can be fully automated looking for regressions.
General Test Scenario¶
-
Load a remote docker repo. In this instance. docker.io was used
pulp-admin -u admin -p admin docker repo create --repo-id test --feed https://index.docker.io --upstream-name library/busybox -
Verify the state of Master_Node
pulp-admin -u admin -p admin repo list -
Configure CRANE on the Master_Node for docker pull request redirection - /etc/crane.conf
[general] data_dir: /var/lib/pulp/published/docker debug: true endpoint: kvm-04-guest09.rhts.eng.bos.redhat.com:5000 -
Turn-off SELinux and firewalld or correctly deal with any issues that would have these ports blocked.
setenforce 0 systemctl stop firewalld -
Verify Slave_Node and synch with the Master_Node. An additional or test verification of a `docker pull` against the Master_Node can also verify functionality of Crane on the Master_Node
pulp-admin -u admin -p admin docker repo create --repo-id fixture --upstream-name test --feed http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000 pulp-admin -u admin -p admin docker repo sync run --repo-id fixture ## If there are errors, see if curl works for secure and insecure curl https://kvm-04-guest09.rhts.eng.bos.redhat.com/pulp/docker/v2/test/tags/list curl --insecure https://kvm-04-guest09.rhts.eng.bos.redhat.com/pulp/docker/v2/test/tags/list -
Modify the source of a symbolic link on the Master_Node in /var/lib/pulp/published/docker/v2/web/<repo>/blobs
cd /var/lib/pulp/published/docker/v2/web/test/blobs mv /var/lib/pulp/content/units/docker_blob/63/09d7d49789d2dd03ae713c55be95d3fe2fabd7cb011ea3dbedd001646fe857/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 /var/lib/pulp/content/units/docker_blob/63/09d7d49789d2dd03ae713c55be95d3fe2fabd7cb011ea3dbedd001646fe857/sha256-moved:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 -
Drop the Tables and restart the workers on the Slave_Node for a clean state without cached blobs. To do this simply, pulp/devl/ had some bash aliases in the `2-master` branch that was used to drop the Mongo Tables and clean Pulp2
cd /root git clone https://github.com/pulp/devel.git cp -p devel/ansible/roles/dev/files/drop_database.js . source devel/ansible/roles/dev/files/bashrc pclean -
Add the remote repo with `--force-full` that has the expected error
pulp-admin -u admin -p admin docker repo create --repo-id fixture --upstream-name test --feed http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000 pulp-admin -u admin -p admin docker repo sync run --repo-id fixture --force-full -
Expect the failure on sync
+----------------------------------------------------------------------+ Synchronizing Repository [fixture] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading manifests [/] ... completed Copying units already in pulp [-] ... completed Copying units already in pulp [-] ... completed Downloading remote files [=============================== ] 62% 235 of 375 items ... failed Task Failed Image download(s) from http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d 3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed. Sync task has failed to prevent a corrupted repository. [root@kvm-04-guest08 ~]# -
Expect the 403/404 error in journalctl, depending on the type of 4xx error produced. This is an example of 403.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:56e295030a62a3f50bd00ffde14f90232cb019cacaa9b63e5647b36c596c9dd0. Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download failed: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed with code 403: Forbidden Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:e1e5da6c811515844589523b3f8f77db1f5fcc98f3f7360a62c1f800e8203f70 was cancelled Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:87213c40360b3c4ebd5f410429b643dba098e51c42c505655bf72e4ec7937c9b was cancelled Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:9500078affc13405feaacbd641347e3b0524305176bafd2773113a969672dfad was cancelled Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:224a21997e8ca8514d42eb2ed98b19a7ee2537bce0b3a26b8dff510ab637f15c. Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: pulp.server.async.tasks:INFO: [69368909] Task failed : [69368909-4bbe-4988-b98d-303f085b9ba9] : Image download(s) from http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed. Sync task has failed to prevent a corrupted repository. Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: celery.app.trace:INFO: [69368909] Task pulp.server.managers.repo.sync.sync[69368909-4bbe-4988-b98d-303f085b9ba9] raised expected: PulpCodedException() Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: celery.app.trace:INFO: [15f40f3b] Task pulp.server.async.tasks._release_resource[15f40f3b-5733-4afd-8272-82a545884a40] succeeded in 0.00527241999953s: None ~
Other Hacks¶
Edit Pulp to not to SSL Validation¶
Ideally, SSL configuration is done correctly on the Master_Node. However, if it is not, this will cause issues with the pulp <-> pulp sync.
To get around this in a test-only scenario for recreation, SSL validation can be `turned off`.
Hacks:
- Edit: /usr/lib/python2.7/site-packages/pulp/plugins/util/nectar_config.py
101 download_config.ssl_validation = False
Reset the worker on the Node and retest connectivty
#Look for the workers and restart each one. There is a single worker in this example
systemctl | grep pulp
systemctl restart pulp_worker-0
#Restart HTTPD, just to be sure.
systemctl restart httpd
systemctl status httpd
References¶
[0] - https://pulp.plan.io/issues/4129
[1] - https://pulp.plan.io/issues/4143
[2] - https://github.com/pulp/pulp-ci/tree/master/ci/ansible/roles/pulp-crane/tasks
[3] - https://docs.docker.com/registry/insecure/#deploy-a-plain-http-registry
[4] - https://github.com/pulp/devel.git
[5] - https://github.com/pulp/pulp-ci/blob/master/ci/ansible/pulp_server.yaml
[6] - https://github.com/pulp/crane
[7] - https://docs.pulpproject.org/plugins/pulp_docker/user-guide/recipes.html
[8] - https://docs.pulpproject.org/plugins/crane/index.html
[9] - https://mojo.redhat.com/docs/DOC-1059499
Updated by bherring over 5 years ago
- Related to Test #4259: As a user I can sync from a docker registry added
.