Story #5286
closed
As a user, a Remote should provide an option that allows download errors, e.g. 404 errors to still allow creation of a RepositoryVersion
0%
Description
Ticket moved to GitHub: "pulp/pulpcore/1843":https://github.com/pulp/pulpcore/issues/1843
Problem¶
If you sync a file repository where one of the files is missing, it seems that the repository syncs as much as it can (as expected), but its reported as a fatal error, with a state of 'failed'.
This is especially problematic for remote repos that you cannot contact the maintainer, have the content you want, but don't have all content available (because it's an incomplete or corrupted repo).
Steps to reproduce:
1) create a file repository where one of the files is missing
2) create a file remote and repository and sync them
Actual task status (apologies its been yaml-fied):
- _href: "/pulp/api/v3/tasks/b6f9b619-c174-4e43-b546-0bbefdfb11e7/"
_created: '2019-08-15T15:21:37.058+00:00'
state: failed
name: pulp_file.app.tasks.synchronizing.synchronize
started_at: '2019-08-15T15:21:37.177+00:00'
finished_at: '2019-08-15T15:21:37.382+00:00'
non_fatal_errors: "[]"
error:
code: ''
description: 404, message='Not Found'
traceback: |2
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py", line 822, in perform_job
rv = job.perform()
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 605, in perform
self._result = self._execute()
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py", line 611, in _execute
return self.func(*self.args, **self.kwargs)
File "/usr/local/lib/pulp/src/pulp-file/pulp_file/app/tasks/synchronizing.py", line 45, in synchronize
dv.create()
File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/declarative_version.py", line 169, in create
loop.run_until_complete(pipeline)
File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py", line 209, in create_pipeline
await asyncio.gather(*futures)
File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/api.py", line 43, in __call__
await self.run()
File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/artifact_stages.py", line 132, in run
pb.done += task.result() # download_count
File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/artifact_stages.py", line 155, in _handle_content_unit
await asyncio.gather(*downloaders_for_content)
File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/stages/models.py", line 78, in download
download_result = await downloader.run(extra_data=self.extra_data)
File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/download/base.py", line 212, in run
return await self._run(extra_data=extra_data)
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/backoff/_async.py", line 131, in retry
ret = await target(*args, **kwargs)
File "/usr/local/lib/pulp/src/pulpcore-plugin/pulpcore/plugin/download/http.py", line 183, in _run
response.raise_for_status()
File "/usr/local/lib/pulp/lib64/python3.6/site-packages/aiohttp/client_reqrep.py", line 942, in raise_for_status
headers=self.headers)
worker: "/pulp/api/v3/workers/df7e0085-b0dd-4073-b74d-9ab78ad27a03/"
spawned_tasks: []
progress_reports:
- message: Downloading Metadata
state: completed
total: 1
done: 1
- message: Parsing Metadata Lines
state: completed
total: 2
done: 2
- message: Downloading Artifacts
state: failed
done: 0
- message: Associating Content
state: canceled
done: 0
created_resources: []
reserved_resources_record: []
create_version: true
poll_attempts:
total: 1
failed: 1
Solution¶
It would be useful to have an option that causes sync to not fail on download errors, but instead continue and record the errors as non-fatal exceptions somehow.
Files
Related issues
.