Project

Profile

Help

Issue #2542

closed

Streamer needs to try all available catalog entries.

Added by jortel@redhat.com about 7 years ago. Updated about 5 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
Platform Release:
2.12.1
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
Yes
Tags:
Pulp 2
Sprint:
Sprint 15
Quarter:

Description

Streamer needs to try all available catalog entries. This address use cases where multiple catalog entries exist for the same path (file) but some of them are no longer valid. This also supports cases where a path (file) is available from multiple remotes but one or more of them are temporarily unavailable.

The streamer should fetch all entries sorted by (created(date), revision) to ensure it's trying the newest entry first.

https://github.com/pulp/pulp/blob/master/streamer/pulp/streamer/server.py#L172

This is related to the work done here: https://pulp.plan.io/issues/2503, because that bug happened to result in catalog entries where the remote files were no longer available.

Actions #1

Updated by mhrivnak about 7 years ago

  • Description updated (diff)
  • Status changed from NEW to ASSIGNED
  • Assignee set to jortel@redhat.com
  • Sprint/Milestone set to 32
  • Triaged changed from No to Yes
Actions #2

Updated by jortel@redhat.com about 7 years ago

Which http result code should be returned when the streamer tries multiple catalog entries? The choices seem to be:

  • always return 404
  • return the http code for the last entry tried.

Either could be a little misleading and user might have to look at the log to determine what happened. I'm leaning toward return the last code because it will likely be the most accurate in more cases. Especially for cases where only 1 catalog entry exists.

Thoughts?

Actions #3

Updated by jortel@redhat.com about 7 years ago

  • Status changed from ASSIGNED to POST
Actions #4

Updated by mhrivnak about 7 years ago

I think the response code should be chosen within the context of the current requester and responder, and should not consider context of 3rd-party interactions. A client requests a file. The streamer tries to return it, but can't find it. The client doesn't care why. Filesystem permission error, selinux, file isn't on disk where it should be, auth error to a remote source, etc. Regardless of the reason, the streamer wasn't able to find the file. It should return a 404.

Here is the official definition, which I think fits perfectly: https://tools.ietf.org/html/rfc7231#section-6.5.4

Logging reasons for failures of each catalog entry would be a great idea though.

Actions #5

Updated by jortel@redhat.com about 7 years ago

mhrivnak wrote:

I think the response code should be chosen within the context of the current requester and responder, and should not consider context of 3rd-party interactions. A client requests a file. The streamer tries to return it, but can't find it. The client doesn't care why. Filesystem permission error, selinux, file isn't on disk where it should be, auth error to a remote source, etc. Regardless of the reason, the streamer wasn't able to find the file. It should return a 404.

I do like the low-complexity and deterministic nature of this approach.

Actions #7

Updated by ipanova@redhat.com about 7 years ago

I do agree that we need to return 404 in the mentioned above situations.

Actions #8

Updated by jortel@redhat.com about 7 years ago

PR, updated.

Actions #9

Updated by mhrivnak about 7 years ago

  • Sprint/Milestone changed from 32 to 33

Added by jortel@redhat.com about 7 years ago

Revision b6d4655f | View on GitHub

Streamer tries all catalog entries. closes #2542

Added by jortel@redhat.com about 7 years ago

Revision b6d4655f | View on GitHub

Streamer tries all catalog entries. closes #2542

Actions #10

Updated by jortel@redhat.com about 7 years ago

  • Status changed from POST to MODIFIED
Actions #11

Updated by bizhang about 7 years ago

  • Platform Release set to 2.12.1
Actions #12

Updated by bizhang about 7 years ago

  • Status changed from MODIFIED to 5
Actions #13

Updated by bizhang about 7 years ago

  • Version set to 2.12.1
Actions #14

Updated by bizhang about 7 years ago

  • Version deleted (2.12.1)
Actions #15

Updated by bizhang about 7 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE
Actions #17

Updated by bmbouter about 6 years ago

  • Sprint set to Sprint 15
Actions #18

Updated by bmbouter about 6 years ago

  • Sprint/Milestone deleted (33)
Actions #19

Updated by bmbouter about 5 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF