Project

Profile

Help

Issue #4857

Updated by bherring almost 5 years ago

h2. Problem Summary 

 Testing of *test_iso_crud` in nightly CI in 2-master since May 16, 2019. 

 Please see the test in Pulp-2-Tests for a good description of the test. 

 Using pulp-smash shell or running the test through pulp-smash and verifying the failure through journalctl is the easiest way to recreate for this test type. 

 This test is long-standing. 

 h2. Test Synopsis 

 Do the following: 
 1. Create and sync an ISO repository. 
 2. Update the repository's feed URL, and sync it. This simulates a change in the contents of the source ISOs. 
 3. Assert that number of units remain the same, but the content has changed. 

 h2. Command Failure 

 Changing the import to the second file feed and then syncing causes the task error: 

 <pre> 
         # Step 2 
         client.put(repo['importers'][0]['_href'], { 
             'importer_config': {'feed': FILE2_FEED_URL} 
         }) 
         sync_repo(cfg, repo) 
 </pre> 

 h2. journalctl of task failure  

 <pre> 
 May 21 08:13:35 rhel76-nightly pulp[20384]: pulp.plugins.file.distributor:ERROR: [b91bfe80] (20384-05632) [(u'/var/cache/pulp/reserved_resource_worker-0@rhel76-nightly/b91bfe80-69fd-4fef-bd96-c5dc4f9f1ec6/build/3.iso', u'/var/lib/pulp/published/https/isos/870fe7fa-dae5-49b2-ad00-ca19f7075fc7/3.iso', '[Errno 17] File exists'), (u'/var/cache/pulp/reserved_resource_worker-0@rhel76-nightly/b91bfe80-69fd-4fef-bd96-c5dc4f9f1ec6/build/1.iso', u'/var/lib/pulp/published/https/isos/870fe7fa-dae5-49b2-ad00-ca19f7075fc7/1.iso', '[Errno 17] File exists'), (u'/var/cache/pulp/reserved_resource_worker-0@rhel76-nightly/b91bfe80-69fd-4fef-bd96-c5dc4f9f1ec6/build/2.iso', u'/var/lib/pulp/published/https/isos/870fe7fa-dae5-49b2-ad00-ca19f7075fc7/2.iso', '[Errno 17] File exists')] [(u'/var/cache/pulp/reserved_resource_worker-0@rhel76-nightly/b91bfe80-69fd-4fef-bd96-c5dc4f9f1ec6/build/ 
 May 21 08:13:35 rhel76-nightly pulp[20384]: pulp.plugins.file.distributor:ERROR: [b91bfe80] (20384-05632) Traceback (most recent call last): 
 May 21 08:13:35 rhel76-nightly pulp[20384]: pulp.plugins.file.distributor:ERROR: [b91bfe80] (20384-05632)     File "/usr/lib/python2.7/site-packages/pulp/plugins/file/distributor.py", line 189, in publish_repo_fast_forward publish_repo_fas 
 May 21 08:13:35 rhel76-nightly pulp[20384]: pulp.plugins.file.distributor:ERROR: [b91bfe80] (20384-05632)       copytree(build_dir, location, symlinks=True) 
 May 21 08:13:35 rhel76-nightly pulp[20384]: pulp.plugins.file.distributor:ERROR: [b91bfe80] (20384-05632)     File "/usr/lib/python2.7/site-packages/pulp/server/util.py", line 287, in copytree 
 May 21 08:13:35 rhel76-nightly pulp[20384]: pulp.plugins.file.distributor:ERROR: [b91bfe80] (20384-05632)       raise Error(errors) 
 May 21 08:13:35 rhel76-nightly pulp[20384]: pulp.plugins.file.distributor:ERROR: [b91bfe80] (20384-05632) Error: [(u'/var/cache/pulp/reserved_resource_worker-0@rhel76-nightly/b91bfe80-69fd-4fef-bd96-c5dc4f9f1ec6/build/3.iso', u'/var/lib/pulp/published/https/isos/870fe7fa-dae5-49b2-ad00-ca19f7075fc7/3.iso', '[Errno 17] File exists'), (u'/var/cache/pulp/reserved_resource_worker-0@rhel76-nightly/b91bfe80-69fd-4fef-bd96-c5dc4f9f1ec6/build/1.iso', u'/var/lib/pulp/published/https/isos/870fe7fa-dae5-49b2-ad00-ca19f7075fc7/1.iso', '[Errno 17] File exists'), (u'/var/cache/pulp/reserved_resource_worker-0@rhel76-nightly/b91bfe80-69fd-4fef-bd96-c5dc4f9f1ec6/build/2.iso', u'/var/lib/pulp/published/https/isos/870fe7fa-dae5-49b2-ad00-ca19f7075fc7/2.iso', '[Errno 17] File exists')] [(u'/var/cache/pulp/reserved_resource_worker-0@rhel76-nightly/b91bfe80-69fd-4fef-bd96-c5dc4f9f1ec6 
 May 21 08:13:35 rhel76-nightly pulp[20384]: pulp.server.controllers.repository:INFO: [b91bfe80] {'state': 'failed', 'error_message': "[(u'/var/cache/pulp/reserved_resource_worker-0@rhel76-nightly/b91bfe80-69fd-4fef-bd96-c5dc4f9f1ec6/build/3.iso', u'/var/lib/pulp/published/https/isos/870fe7fa-dae5-49b2-ad00-ca19f7075fc7/3.iso', '[Errno 17] File exists'), (u'/var/cache/pulp/reserved_resource_worker-0@rhel76-nightly/b91bfe80-69fd-4fef-bd96-c5dc4f9f1ec6/build/1.iso', u'/var/lib/pulp/published/https/isos/870fe7fa-dae5-49b2-ad00-ca19f7075fc7/1.iso', '[Errno 17] File exists'), (u'/var/cache/pulp/reserved_resource_worker-0@rhel76-nightly/b91bfe80-69fd-4fef-bd96-c5dc4f9f1ec6/build/2.iso', u'/var/lib/pulp/published/https/isos/870fe7fa-dae5-49b2-ad00-ca19f7075fc7/2.iso', '[Errno 17] File exists')]", 'traceback': 'Traceback (most recent call last):\n    File "/usr/lib/python2.7/site-packages/pulp/plugins/file/distributor.py", line 189, in publish_repo_fast_forward\n      copytree(build_dir, location, symlinks=True)\n    File "/usr/lib/python2.7/site-packages/pulp/server/util.py", line 287, in copytree\n      raise Error(errors)\nError: [(u\'/var/cache/pulp/reserved_resource_worker-0@rhel76-nightly/b91bfe80-69fd-4fef-bd96-c5dc4f9f1ec6/build/3.iso\', u\'/var/lib/pulp/published/https/isos/870fe7fa-dae5-49b2-ad00-ca19f7075fc7/3.iso\', \'[Errno 17] File exists\'), (u\'/var/cache/pulp/reserved_resource_worker-0@rhel76-nightly/b91bfe80-69fd-4fef-bd96-c5dc4f9f1ec6/build/1.iso\', u\'/var/lib/pulp/published/https/isos/870fe7fa-dae5-49b2-ad00-ca19f7075fc7/1.iso\', \'[Errno 17] File exists\'), (u\'/var/cache/pulp/reserved_resource_worker-0@rhel76-nightly/b91bfe80-69fd-4fef-bd96-c5dc4f9f1ec6/build/2.iso\', u\'/var/lib/pulp/published/https/isos/870fe7fa-dae5-49b2-ad00-ca19f7075fc7/2.iso\', \'[Errno 17] File exists\')]\n', 'state_times': {'failed': '2019-05-21T12:13:35', 'not_started': '2019-05-21T12:13:35', 'in_progress': '2019-05-21T12:13:35'}} "[(u'/var/cache/pulp/reserved_resource_worker-0@rhel76-nightly/b91bfe80-69fd-4 
 May 21 08:13:35 rhel76-nightly pulp[20384]: pulp.server.controllers.repository:ERROR: [b91bfe80] (20384-05632) Exception caught from plugin during publish for repo [870fe7fa-dae5-49b2-ad00-ca19f7075fc7] 
 May 21 08:13:35 rhel76-nightly pulp[20384]: pulp.server.controllers.repository:ERROR: [b91bfe80] (20384-05632) Traceback (most recent call last): 
 May 21 08:13:35 rhel76-nightly pulp[20384]: pulp.server.controllers.repository:ERROR: [b91bfe80] (20384-05632)     File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1265, in _do_publish _do_p 
 May 21 08:13:35 rhel76-nightly pulp[20384]: pulp.server.controllers.repository:ERROR: [b91bfe80] (20384-05632)       distributor_id=dist_id, summary=publish_report.summary 
 May 21 08:13:35 rhel76-nightly pulp[20384]: pulp.server.controllers.repository:ERROR: [b91bfe80] (20384-05632) PulpCodedException: The distributor iso_distributor indicated a failed response when publishing repository 870fe7fa-dae5-49b2-ad00-ca19f7075fc7. repo 
 May 21 08:13:35 rhel76-nightly pulp[20384]: pulp.server.async.tasks:INFO: [b91bfe80] Task failed : [b91bfe80-69fd-4fef-bd96-c5dc4f9f1ec6] : The distributor iso_distributor indicated a failed response when publishing repository 870fe7fa-dae5-49b2-ad00-ca19f7075fc7. publis 
 May 21 08:13:35 rhel76-nightly pulp[20384]: celery.app.trace:INFO: [b91bfe80] Task pulp.server.managers.repo.publish.publish[b91bfe80-69fd-4fef-bd96-c5dc4f9f1ec6] raised expected: PulpCodedException() 
 May 21 08:13:35 rhel76-nightly pulp[20384]: celery.app.trace:INFO: [80f9f51e] Task pulp.server.async.tasks._release_resource[80f9f51e-bae6-434f-834c-1c9ffde941d6] succeeded in 0.00546025601216s: None 

 
 </pre> 

 h2. Additional Information 

 * Test: rpm.api_v2.test_iso_crud.ISOUpdateTestCase.test_all 
 ** https://github.com/PulpQE/Pulp-2-Tests/blob/9867e0593ea4eb0680b4871fe734e01f91998c40/pulp_2_tests/tests/rpm/api_v2/test_iso_crud.py#L372 
 * First Failure Occurance: May 16, 2019 
 * Failing locally on 2-master (2.20) 
 * Works without an issue on 2.19.0 and 2.18.1 
 * Failed on Jenkins running subset of rpm/apiv2 
 * Jenkins Failure (latest) 

Back