Project

Profile

Help

Issue #2458

closed

Lazy download repo content task fails silently.

Added by jortel@redhat.com about 7 years ago. Updated almost 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
2.10.3
Platform Release:
2.12.1
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 14
Quarter:

Description

 pulp-admin repo download --repo-id lazy
+----------------------------------------------------------------------+
                     Downloading Repository [lazy]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.

Download Repository Content
[==========================                        ] 53%
17 of 32 items
... failed

Task Succeeded

This needs to be logged at INFO. Otherwise, this fails with nothing in the log. I spend 4 hours troubleshooting this.

        except (InvalidChecksumType, VerificationException, IOError), e:
            _logger.debug(_('Download of {path} failed: {reason}.').format(
                path=catalog_entry.path, reason=str(e)))
            path_entry[PATH_DOWNLOADED] = False
            self.progress_failures += 1
self.report()

[1] https://github.com/pulp/pulp/blob/master/server/pulp/server/controllers/repository.py#L1710

Actions #1

Updated by jortel@redhat.com about 7 years ago

  • Description updated (diff)
  • Version set to 2.10.3
Actions #2

Updated by jortel@redhat.com about 7 years ago

The task reports as succeeded. Does this make sense?

Actions #3

Updated by mhrivnak about 7 years ago

wrote:

The task reports as succeeded. Does this make sense?

I think yes, as long as its progress report or result communicates the one or more files failed to download. This is similar to how syncs work.

Actions #4

Updated by bizhang about 7 years ago

  • Triaged changed from No to Yes
Actions #5

Updated by bmbouter about 7 years ago

These sounds like non-fatal exceptions so I expect the final task status will be succeeded, but errors should be reported with a progress report.

Actions #6

Updated by jortel@redhat.com about 7 years ago

bmbouter wrote:

These sounds like non-fatal exceptions so I expect the final task status will be succeeded, but errors should be reported with a progress report.

By progress report I think we mean the result report that summarizes failure statistics like "400 downloads failed" and not a listing of all 400 URLs etc.

Actions #7

Updated by jortel@redhat.com about 7 years ago

wrote:

bmbouter wrote:

These sounds like non-fatal exceptions so I expect the final task status will be succeeded, but errors should be reported with a progress report.

By progress report I think we mean the result report that summarizes failure statistics like "400 downloads failed" and not a listing of all 400 URLs etc. The main thing is that the task state is not FAILED.

Actions #8

Updated by mhrivnak about 7 years ago

  • Sprint/Milestone set to 32
Actions #9

Updated by bmbouter about 7 years ago

Yes @jortel you are right. It should be on the result report not a "progress report". Also +1 to summarizing the failure.

Actions #10

Updated by jortel@redhat.com about 7 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com

Added by jortel@redhat.com about 7 years ago

Revision bb8b02cd | View on GitHub

Log VerificationException at INFO. closes #2458

Added by jortel@redhat.com about 7 years ago

Revision bb8b02cd | View on GitHub

Log VerificationException at INFO. closes #2458

Actions #11

Updated by jortel@redhat.com about 7 years ago

Looks like this task is running a LazyUnitDownloadStep which followed the flawed practice of reporting both progress and result (summary) using a progress report. And, although, the LazyUnitDownloadStep does not subclass a PublishStep, it tries to act like one and submit a progress report that can be rendered by a PublishStepStatusRenderer. This is exactly how things are set up. The DownloadRepositoryCommand IsA SyncPublishCommand and is created with the PublishStepStatusRenderer.

Given how much of this will be rewritten in pulp3 (not using PublishStep), I don't think it's worth the effort to fix this in pulp2.

[jortel@f23d pulp]$ pulp-admin tasks details --task-id=f5ad1e36-ee73-422a-9cb1-f3820f9c0e0a
+----------------------------------------------------------------------+
                              Task Details
+----------------------------------------------------------------------+

Operations:       download
Resources:        zoo (repository)
State:            Successful
Start Time:       2017-01-23T19:49:25Z
Finish Time:      2017-01-23T19:49:26Z
Result:           N/A
Task Id:          f5ad1e36-ee73-422a-9cb1-f3820f9c0e0a
Progress Report:  
  Background Download: 
    Description:   Download Repository Content
    Details:       
    Error Details: 
    Items Total:   32
    Num Failures:  1
    Num Processed: 1
    Num Success:   0
    State:         FAILED
    Step Id:       6f64950c-af41-4d20-b219-d8216c9dbaba
    Step Type:     background_download
Actions #12

Updated by jortel@redhat.com about 7 years ago

  • Status changed from ASSIGNED to POST
Actions #13

Updated by Ichimonji10 about 7 years ago

Is there a procedure for reproducing the failure described by this issue?

Actions #14

Updated by jortel@redhat.com about 7 years ago

  • Status changed from POST to MODIFIED
Actions #15

Updated by bizhang about 7 years ago

  • Platform Release set to 2.12.1
Actions #16

Updated by bizhang about 7 years ago

  • Status changed from MODIFIED to 5
Actions #17

Updated by bizhang about 7 years ago

  • Version changed from 2.10.3 to 2.12.1
Actions #18

Updated by bizhang about 7 years ago

  • Version changed from 2.12.1 to 2.10.3
Actions #19

Updated by bizhang almost 7 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #20

Updated by jortel@redhat.com over 6 years ago

To test this, I suggest:

  1. Creating a repository with a on_demand download policy.
  2. Synchronize the repository.
  3. Using mongo, update one of the URLs in the catalog entries for that importer to be incorrect.
  4. As described in the description, kick off the repository download task.
Actions #21

Updated by kersom over 6 years ago

Related to step 3. Is there another way to update the URL in catalog without use mongo?

Actions #23

Updated by Ichimonji10 over 6 years ago

If there's a way to reproduce this issue by accessing Pulp's public interface, or by performing actions like starting or stopping services, we can definitely test this. As it is, the procedure for reproducing this issue requires an uncomfortable amount of knowledge about how Pulp works internally, and I think QE should avoid creating an automated test for this issue.

Actions #24

Updated by bmbouter almost 6 years ago

  • Sprint set to Sprint 16
Actions #25

Updated by bmbouter almost 6 years ago

  • Sprint changed from Sprint 16 to Sprint 14
Actions #26

Updated by bmbouter almost 6 years ago

  • Sprint/Milestone deleted (32)
Actions #27

Updated by bmbouter almost 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF