Project

Profile

Help

Story #3907

As a plugin writer, the HttpDownloader provides auto_decompress=True like aiohttp

Added by bmbouter about 1 year ago. Updated 6 months ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Platform Release:
Blocks Release:
Backwards Incompatible:
No
Groomed:
No
Sprint Candidate:
No
Tags:
QA Contact:
Complexity:
Smash Test:
Verified:
No
Verification Required:
No
Sprint:
Sprint 41

Description

Problem Statement

aiohttp by default decompresses everything it downloads, e.g. gzip, or other compression types. This is great for typical clients which would otherwise have to decompress downloaded data, e.g. downloading a foo.tar.gz file, when reading the data from aiohttp you'regetting the actual contents of foo, not the foo.tar.gz compressed data.

Pulp needs to store data exactly as it was stored remotely. To uncompress the data and save it in Pulp's backend is incorrect because Pulp things of that file as foo.tar.gz, not foo. When it presents it to client's it's presented as foo.tar.gz. If we uncompress it, but then save it as foo.tar.gz (incorrect) then clients will in turn be receiving foo.tar.gz and go to uncompress it (a second time) and that will fail.

There are other situations though where we want the auto_decompression. When downloading temporary metadata for example, often times remote metadata is compressed, and we need to download, uncompress it, and then have our code parse it. So auto-decompression is really useful in some cases.

Solution

Enable Pulp's downloaders to have auto decompression by default, but have the option to turn it off and download binary data.

Associated revisions

Revision b1fed0c4 View on GitHub
Added by bmbouter about 1 year ago

Add auto_decompress option everywhere

https://pulp.plan.io/issues/3907
closes #3907

Revision b1fed0c4 View on GitHub
Added by bmbouter about 1 year ago

Add auto_decompress option everywhere

https://pulp.plan.io/issues/3907
closes #3907

Revision b1fed0c4 View on GitHub
Added by bmbouter about 1 year ago

Add auto_decompress option everywhere

https://pulp.plan.io/issues/3907
closes #3907

Revision 655d8e1f View on GitHub
Added by dalley about 1 year ago

Revert "Add auto_decompress option everywhere"

History

#1 Updated by dalley about 1 year ago

  • Status changed from NEW to POST
  • Assignee set to bmbouter

#2 Updated by bmbouter about 1 year ago

  • Sprint set to Sprint 41

I'm adding to the sprint because this is required to have plugins use DeclarativeVersion.

#3 Updated by bmbouter about 1 year ago

  • Description updated (diff)

#4 Updated by bmbouter about 1 year ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#5 Updated by bmbouter about 1 year ago

  • Status changed from MODIFIED to ASSIGNED

Actually after investigating the original symptoms that motivated this issue I believe we do not need to offer this feature. The Content-Encoding response header indicates the encoding of the response by the webserver just before sending it. In all cases we want to have aiohttp automatically decompress this data.

I'm going to negative commit this feature.

#6 Updated by bmbouter about 1 year ago

  • Status changed from ASSIGNED to POST

#7 Updated by bmbouter about 1 year ago

  • Status changed from POST to CLOSED - WONTFIX

closing as wontfix since we never implemented it.

#8 Updated by daviddavis 6 months ago

  • Sprint/Milestone set to 3.0

#9 Updated by bmbouter 6 months ago

  • Tags deleted (Pulp 3)

Please register to edit this issue

Also available in: Atom PDF