Pulp

Story #5286

Updated by bmbouter about 5 years ago

h3. Problem 

 If you sync a file repository where one of the files is missing, it seems that the repository syncs as much as it can (as expected), but its reported as a fatal error, with a state of 'failed'. 

 This is especially problematic for remote repos that you cannot contact the maintainer, have the content you want, but don't have all content available (because it's an incomplete or corrupted repo). 

 Steps to reproduce: 

 1) create a file repository where one of the files is missing 
 2) create a file remote and repository and sync them 

 Actual task status (apologies its been yaml-fied): 


 <pre> 
 - _href: "/pulp/api/v3/tasks/b6f9b619-c174-4e43-b546-0bbefdfb11e7/" 
   _created: '2019-08-15T15:21:37.058+00:00' 
   state: failed 
   name: pulp_file.app.tasks.synchronizing.synchronize 
   started_at: '2019-08-15T15:21:37.177+00:00' 
   finished_at: '2019-08-15T15:21:37.382+00:00' 
   non_fatal_errors: "[]" 
   error: 
     code: '' 
     description: 404, message='Not Found' 
     traceback: |2 
         File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 822, in perform_job 
           rv = job.perform() 
         File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 605, in perform 
           self._result = self._execute() 
         File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 611, in _execute 
           return self.func(*self.args, **self.kwargs) 
         File "/usr/local/lib/pulp/src/pulp-file/pulp_file/app/tasks/synchronizing.py", line 45, in synchronize 
           dv.create() 
         File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py", line 169, in create 
           loop.run_until_complete(pipeline) 
         File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete 
           return future.result() 
         File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py", line 209, in create_pipeline 
           await asyncio.gather(*futures) 
         File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py", line 43, in __call__ 
           await self.run() 
         File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/artifact_stages.py", line 132, in run 
           pb.done += task.result()    # download_count 
         File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/artifact_stages.py", line 155, in _handle_content_unit 
           await asyncio.gather(*downloaders_for_content) 
         File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/models.py", line 78, in download 
           download_result = await downloader.run(extra_data=self.extra_data) 
         File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/download/base.py", line 212, in run 
           return await self._run(extra_data=extra_data) 
         File "/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py", line 131, in retry 
           ret = await target(*args, **kwargs) 
         File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/download/http.py", line 183, in _run 
           response.raise_for_status() 
         File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client_reqrep.py", line 942, in raise_for_status 
           headers=self.headers) 
   worker: "/pulp/api/v3/workers/df7e0085-b0dd-4073-b74d-9ab78ad27a03/" 
   spawned_tasks: [] 
   progress_reports: 
   - message: Downloading Metadata 
     state: completed 
     total: 1 
     done: 1 
   - message: Parsing Metadata Lines 
     state: completed 
     total: 2 
     done: 2 
   - message: Downloading Artifacts 
     state: failed 
     done: 0 
   - message: Associating Content 
     state: canceled 
     done: 0 
   created_resources: [] 
   reserved_resources_record: [] 
 create_version: true 
 poll_attempts: 
   total: 1 
   failed: 1 
 </pre> 


 h3. Solution 

 It would I'd expect this error to be useful in the 'non-fatal' errors attribute, and the state to have an option that causes sync to not fail on download errors, but instead continue and record the errors as non-fatal exceptions somehow. be 'failed'

Back