https://pulp.plan.io/https://pulp.plan.io/favicon.ico2018-11-06T19:56:34ZPulpDocker Support - Test #4129: Test sync of a repository that returns a 403 response for a download requesthttps://pulp.plan.io/issues/4129?journal_id=317402018-11-06T19:56:34Zamacdona@redhat.comaustin@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-11 priority-6 priority-default closed" href="/issues/2966">Issue #2966</a>: Unable to sync docker repo because worker dies</i> added</li></ul> Docker Support - Test #4129: Test sync of a repository that returns a 403 response for a download requesthttps://pulp.plan.io/issues/4129?journal_id=318322018-11-12T16:16:50Zamacdona@redhat.comaustin@redhat.com
<ul></ul><p>I was able to reproduce this with a pulp-pulp sync.</p>
<p><a href="https://pulp.plan.io/issues/2849#note-14" class="external">https://pulp.plan.io/issues/2849#note-14</a><br>
<a href="https://pulp.plan.io/issues/2966#note-20" class="external">https://pulp.plan.io/issues/2966#note-20</a></p>
<p>These notes explain how it was done, and I confirmed that a broken symlink raises the 403.</p> Docker Support - Test #4129: Test sync of a repository that returns a 403 response for a download requesthttps://pulp.plan.io/issues/4129?journal_id=320132018-11-19T18:11:19Zbherring
<ul><li><strong>Assignee</strong> set to <i>bherring</i></li></ul> Docker Support - Test #4129: Test sync of a repository that returns a 403 response for a download requesthttps://pulp.plan.io/issues/4129?journal_id=320142018-11-19T18:11:23Zbherring
<ul><li><strong>Status</strong> changed from <i>NEW</i> to <i>ASSIGNED</i></li></ul> Docker Support - Test #4129: Test sync of a repository that returns a 403 response for a download requesthttps://pulp.plan.io/issues/4129?journal_id=320152018-11-19T18:33:18Zamacdona@redhat.comaustin@redhat.com
<ul></ul><p>Unfortunately, whatever we use as a fixture must be able to return specific headers, which are used by the sync code. This (AFAIK) rules out fedorapeople.</p> Docker Support - Test #4129: Test sync of a repository that returns a 403 response for a download requesthttps://pulp.plan.io/issues/4129?journal_id=320162018-11-19T19:07:26Zbherring
<ul></ul><p>Thanks for taking the time to talk to me about our current testing of this, <a href="mailto:amacdona@redhat.com" class="email">amacdona@redhat.com</a></p>
<p>Just so the more verbose version is captured here:</p>
<blockquote>
<p>Testing comes down to is that docker sync relies on certain headers coming down from the repository. As far as we can tell, fedorapeople doesn't give us the option to control the headers. Therefore with our current test harness, this appears to not be a candidate for automation.</p>
</blockquote>
<p>The intention is to still manually test the current fix manually to ensure the current master for beta/GA has the fix. Also, this Trackers will serve to document any test setup and references for future use, mostly as document centralization and re-link.</p>
<p>When QE moves to Pulp3 and using Travis, it may make sense to implement more robust testing, such as spinning-up multiple containers as part of the test or test harness for instances such as this in the correct part of the test cycle.</p>
<p>Once it is clear where `Tracker: Test Story` trackers for Story or Epics to consider in the future, I will link that work to any additional testing done in this Tracker.</p> Docker Support - Test #4129: Test sync of a repository that returns a 403 response for a download requesthttps://pulp.plan.io/issues/4129?journal_id=324412018-12-03T20:49:26Zbherring
<ul><li><strong>Status</strong> changed from <i>ASSIGNED</i> to <i>CLOSED - COMPLETE</i></li></ul><a name="Verified-Release"></a>
<h2 >Verified Release<a href="#Verified-Release" class="wiki-anchor">¶</a></h2>
<p>Platform Version: 2.19a1</p>
<a name="Status"></a>
<h2 >Status<a href="#Status" class="wiki-anchor">¶</a></h2>
<p><strong>Works as Expected</strong></p>
<a name="Verification-LogSnippet"></a>
<h2 >Verification Log/Snippet<a href="#Verification-LogSnippet" class="wiki-anchor">¶</a></h2>
<p>By moving the destination of the symlink blob on the master_node, a 403 sync issue was created with crane.</p>
<p>Pulp successfully prevents a corrupted repo, as noted by the well documented source RM <a class="issue tracker-1 status-11 priority-7 priority-high2 closed" title="Issue: Worker terminates abnormally while processing task (CLOSED - CURRENTRELEASE)" href="https://pulp.plan.io/issues/2849">#2849</a>.</p>
<pre><code>Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:e7277075c9a84b0f55bcdd09f24116677c8209bb25107366c6cd3153e1a186bd.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:56e295030a62a3f50bd00ffde14f90232cb019cacaa9b63e5647b36c596c9dd0.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download failed: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed with code 403: Forbidden
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:e1e5da6c811515844589523b3f8f77db1f5fcc98f3f7360a62c1f800e8203f70 was cancelled
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:87213c40360b3c4ebd5f410429b643dba098e51c42c505655bf72e4ec7937c9b was cancelled
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:9500078affc13405feaacbd641347e3b0524305176bafd2773113a969672dfad was cancelled
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:224a21997e8ca8514d42eb2ed98b19a7ee2537bce0b3a26b8dff510ab637f15c.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: pulp.server.async.tasks:INFO: [69368909] Task failed : [69368909-4bbe-4988-b98d-303f085b9ba9] : Image download(s) from http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed. Sync task has failed to prevent a corrupted repository.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: celery.app.trace:INFO: [69368909] Task pulp.server.managers.repo.sync.sync[69368909-4bbe-4988-b98d-303f085b9ba9] raised expected: PulpCodedException()
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: celery.app.trace:INFO: [15f40f3b] Task pulp.server.async.tasks._release_resource[15f40f3b-5733-4afd-8272-82a545884a40] succeeded in 0.00527241999953s: None
</code></pre>
<a name="Recreation-Notes"></a>
<h2 >Recreation Notes<a href="#Recreation-Notes" class="wiki-anchor">¶</a></h2>
<p>Since a lot of the information about setting up the test harness was non-obvious, the notes for each section here is noted.</p>
<p>Hopefully this can aid as a reference in the future if anyone requires these steps.</p>
<a name="High-Level-Test-Design"></a>
<h3 >High Level Test Design<a href="#High-Level-Test-Design" class="wiki-anchor">¶</a></h3>
<ul>
<li>Remote Docker Repo <---> Pulp Docker Repo Master_Node <----> Pulp Docker Slave_Node</li>
</ul>
<p>The goal is to test the sync of `Pulp Docker Node` with `Pulp Docker Repo Master Node` when the master node is in a state that causes a 403 or 404 sync error. Ideally, it would be ANY error and verification that Pulp handles it sanely.</p>
<a name="Test-Caveats-and-Scope"></a>
<h4 >Test Caveats and Scope<a href="#Test-Caveats-and-Scope" class="wiki-anchor">¶</a></h4>
<p>The scope for test is manually tested regression verification, in this instance.</p>
<p>Ideally, the Remote Docker Repo would able to be manipulated making the harness simpler.</p>
<p>Also, the current test harness does not easily handle multiple pulp instances for automated test execution. This will be improved in the future so test instances such as this can be fully automated looking for regressions.</p>
<a name="General-Test-Scenario"></a>
<h3 >General Test Scenario<a href="#General-Test-Scenario" class="wiki-anchor">¶</a></h3>
<ol>
<li>
<p>Load a remote docker repo. In this instance. docker.io was used</p>
<pre><code>pulp-admin -u admin -p admin docker repo create --repo-id test --feed https://index.docker.io --upstream-name library/busybox
</code></pre>
</li>
<li>
<p>Verify the state of Master_Node</p>
<pre><code>pulp-admin -u admin -p admin repo list
</code></pre>
</li>
<li>
<p>Configure CRANE on the Master_Node for docker pull request redirection - /etc/crane.conf</p>
<pre><code>[general]
data_dir: /var/lib/pulp/published/docker
debug: true
endpoint: kvm-04-guest09.rhts.eng.bos.redhat.com:5000
</code></pre>
</li>
<li>
<p>Turn-off SELinux and firewalld or correctly deal with any issues that would have these ports blocked.</p>
<pre><code>setenforce 0
systemctl stop firewalld
</code></pre>
</li>
<li>
<p>Verify Slave_Node and synch with the Master_Node. An additional or test verification of a `docker pull` against the Master_Node can also verify functionality of Crane on the Master_Node</p>
<pre><code>pulp-admin -u admin -p admin docker repo create --repo-id fixture --upstream-name test --feed http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000
pulp-admin -u admin -p admin docker repo sync run --repo-id fixture
## If there are errors, see if curl works for secure and insecure
curl https://kvm-04-guest09.rhts.eng.bos.redhat.com/pulp/docker/v2/test/tags/list
curl --insecure https://kvm-04-guest09.rhts.eng.bos.redhat.com/pulp/docker/v2/test/tags/list
</code></pre>
</li>
<li>
<p>Modify the source of a symbolic link on the Master_Node in /var/lib/pulp/published/docker/v2/web/<repo>/blobs</p>
<pre><code>cd /var/lib/pulp/published/docker/v2/web/test/blobs
mv /var/lib/pulp/content/units/docker_blob/63/09d7d49789d2dd03ae713c55be95d3fe2fabd7cb011ea3dbedd001646fe857/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 /var/lib/pulp/content/units/docker_blob/63/09d7d49789d2dd03ae713c55be95d3fe2fabd7cb011ea3dbedd001646fe857/sha256-moved:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04
</code></pre>
</li>
<li>
<p>Drop the Tables and restart the workers on the Slave_Node for a clean state without cached blobs. To do this simply, pulp/devl/ had some bash aliases in the `2-master` branch that was used to drop the Mongo Tables and clean Pulp2</p>
<pre><code>cd /root
git clone https://github.com/pulp/devel.git
cp -p devel/ansible/roles/dev/files/drop_database.js .
source devel/ansible/roles/dev/files/bashrc
pclean
</code></pre>
</li>
<li>
<p>Add the remote repo with `--force-full` that has the expected error</p>
<pre><code>pulp-admin -u admin -p admin docker repo create --repo-id fixture --upstream-name test --feed http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000
pulp-admin -u admin -p admin docker repo sync run --repo-id fixture --force-full
</code></pre>
</li>
<li>
<p>Expect the failure on sync</p>
<pre><code>+----------------------------------------------------------------------+
Synchronizing Repository [fixture]
+----------------------------------------------------------------------+
This command may be exited via ctrl+c without affecting the request.
Downloading manifests
[/]
... completed
Copying units already in pulp
[-]
... completed
Copying units already in pulp
[-]
... completed
Downloading remote files
[=============================== ] 62%
235 of 375 items
... failed
Task Failed
Image download(s) from
http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d
3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed. Sync task has
failed to prevent a corrupted repository.
[root@kvm-04-guest08 ~]#
</code></pre>
</li>
<li>
<p>Expect the 403/404 error in journalctl, depending on the type of 4xx error produced. This is an example of 403.</p>
<pre><code>Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:56e295030a62a3f50bd00ffde14f90232cb019cacaa9b63e5647b36c596c9dd0.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download failed: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed with code 403: Forbidden
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:e1e5da6c811515844589523b3f8f77db1f5fcc98f3f7360a62c1f800e8203f70 was cancelled
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:87213c40360b3c4ebd5f410429b643dba098e51c42c505655bf72e4ec7937c9b was cancelled
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download of http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:9500078affc13405feaacbd641347e3b0524305176bafd2773113a969672dfad was cancelled
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: nectar.downloaders.threaded:INFO: Download succeeded: http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:224a21997e8ca8514d42eb2ed98b19a7ee2537bce0b3a26b8dff510ab637f15c.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: pulp.server.async.tasks:INFO: [69368909] Task failed : [69368909-4bbe-4988-b98d-303f085b9ba9] : Image download(s) from http://kvm-04-guest09.rhts.eng.bos.redhat.com:5000/v2/test/blobs/sha256:ffc8a12d3678ba8f82b54c3a9ca8260f56ce4be47748743658d89d8f39e80a04 failed. Sync task has failed to prevent a corrupted repository.
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: celery.app.trace:INFO: [69368909] Task pulp.server.managers.repo.sync.sync[69368909-4bbe-4988-b98d-303f085b9ba9] raised expected: PulpCodedException()
Dec 03 14:25:05 kvm-04-guest08.rhts.eng.bos.redhat.com pulp[24698]: celery.app.trace:INFO: [15f40f3b] Task pulp.server.async.tasks._release_resource[15f40f3b-5733-4afd-8272-82a545884a40] succeeded in 0.00527241999953s: None
~
</code></pre>
</li>
</ol>
<a name="Other-Hacks"></a>
<h4 >Other Hacks<a href="#Other-Hacks" class="wiki-anchor">¶</a></h4>
<a name="Edit-Pulp-to-not-to-SSL-Validation"></a>
<h5 >Edit Pulp to not to SSL Validation<a href="#Edit-Pulp-to-not-to-SSL-Validation" class="wiki-anchor">¶</a></h5>
<p>Ideally, SSL configuration is done correctly on the Master_Node. However, if it is not, this will cause issues with the pulp <-> pulp sync.</p>
<p>To get around this in a test-only scenario for recreation, SSL validation can be `turned off`.</p>
<pre><code>Hacks:
- Edit: /usr/lib/python2.7/site-packages/pulp/plugins/util/nectar_config.py
101 download_config.ssl_validation = False
</code></pre>
<p>Reset the worker on the Node and retest connectivty</p>
<pre><code>#Look for the workers and restart each one. There is a single worker in this example
systemctl | grep pulp
systemctl restart pulp_worker-0
#Restart HTTPD, just to be sure.
systemctl restart httpd
systemctl status httpd
</code></pre>
<a name="References"></a>
<h2 >References<a href="#References" class="wiki-anchor">¶</a></h2>
<p>[0] - <a href="https://pulp.plan.io/issues/4129" class="external">https://pulp.plan.io/issues/4129</a><br>
[1] - <a href="https://pulp.plan.io/issues/4143" class="external">https://pulp.plan.io/issues/4143</a><br>
[2] - <a href="https://github.com/pulp/pulp-ci/tree/master/ci/ansible/roles/pulp-crane/tasks" class="external">https://github.com/pulp/pulp-ci/tree/master/ci/ansible/roles/pulp-crane/tasks</a><br>
[3] - <a href="https://docs.docker.com/registry/insecure/#deploy-a-plain-http-registry" class="external">https://docs.docker.com/registry/insecure/#deploy-a-plain-http-registry</a><br>
[4] - <a href="https://github.com/pulp/devel.git" class="external">https://github.com/pulp/devel.git</a><br>
[5] - <a href="https://github.com/pulp/pulp-ci/blob/master/ci/ansible/pulp_server.yaml" class="external">https://github.com/pulp/pulp-ci/blob/master/ci/ansible/pulp_server.yaml</a><br>
[6] - <a href="https://github.com/pulp/crane" class="external">https://github.com/pulp/crane</a><br>
[7] - <a href="https://docs.pulpproject.org/plugins/pulp_docker/user-guide/recipes.html" class="external">https://docs.pulpproject.org/plugins/pulp_docker/user-guide/recipes.html</a><br>
[8] - <a href="https://docs.pulpproject.org/plugins/crane/index.html" class="external">https://docs.pulpproject.org/plugins/crane/index.html</a><br>
[9] - <a href="https://mojo.redhat.com/docs/DOC-1059499" class="external">https://mojo.redhat.com/docs/DOC-1059499</a></p> Docker Support - Test #4129: Test sync of a repository that returns a 403 response for a download requesthttps://pulp.plan.io/issues/4129?journal_id=327082018-12-11T13:38:14Zbherring
<ul><li><strong>Related to</strong> <i><a class="issue tracker-5 status-13 priority-6 priority-default closed" href="/issues/4259">Test #4259</a>: As a user I can sync from a docker registry</i> added</li></ul> Docker Support - Test #4129: Test sync of a repository that returns a 403 response for a download requesthttps://pulp.plan.io/issues/4129?journal_id=377162019-04-15T20:08:15Zbmbouterbmbouter@redhat.com
<ul><li><strong>Tags</strong> <i>Pulp 2</i> added</li></ul>