Project

Profile

Help

Issue #3667

Merrory error when trying to upload a big tar

Added by ipanova@redhat.com over 3 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Start date:
Due date:
Estimated time:
Severity:
3. High
Version - Docker:
Platform Release:
2.16.2
Target Release - Docker:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 38
Quarter:

Description

we make it possible to upload image manifest( tar ) and manifest lists(json) .

In order to figure out what is what we try to json.load()

looks like json.load() likes to read stuff into the memory and when we provide a 8gb tar we get an memory error

$ time ./my-test.sh 
Traceback (most recent call last):
  File "./my-test.sh", line 5, in <module>
    json.load(upload)
  File "/usr/lib64/python2.7/json/__init__.py", line 287, in load
    return loads(fp.read(),
MemoryError

real    0m0.099s
user    0m0.064s
sys    0m0.031s
$ pulp-admin -vv docker repo uploads upload --repo-id myquay --file /home/ipanova/z
Warning: path should have mode 0700 because it may contain sensitive information: /home/ipanova/.pulp/

+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

2018-05-11 14:56:22,240 - DEBUG - sending GET request to /pulp/api/v2/repositories/myquay/
2018-05-11 14:56:22,368 - INFO - GET request to /pulp/api/v2/repositories/myquay/ with parameters None
2018-05-11 14:56:22,368 - INFO - Response status : 200 

2018-05-11 14:56:22,368 - INFO - Response body :
 {
  "scratchpad": {}, 
  "display_name": null, 
  "description": null, 
  "last_unit_added": "2018-05-09T12:53:26Z", 
  "notes": {
    "_repo-type": "docker-repo"
  }, 
  "last_unit_removed": null, 
  "content_unit_counts": {
    "docker_blob": 3, 
    "docker_manifest": 1, 
    "docker_tag": 1
  }, 
  "_ns": "repos", 
  "_id": {
    "$oid": "5af2eca945ef481597ada3ae"
  }, 
  "id": "myquay", 
  "_href": "/pulp/api/v2/repositories/myquay/"
}

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: z
2018-05-11 14:56:22,369 - ERROR - Client-side exception occurred
Traceback (most recent call last):
  File "/home/ipanova/pulp_development/pulp/client_lib/pulp/client/extensions/core.py", line 474, in run
    exit_code = Cli.run(self, args)
  File "/usr/lib/python2.7/site-packages/okaara/cli.py", line 974, in run
    exit_code = command_or_section.execute(self.prompt, remaining_args)
  File "/home/ipanova/pulp_development/pulp/client_lib/pulp/client/extensions/extensions.py", line 210, in execute
    return self.method(*arg_list, **clean_kwargs)
  File "/home/ipanova/pulp_development/pulp/client_lib/pulp/client/commands/repo/upload.py", line 254, in run
    type_id = self.determine_type_id(filename, **user_input)
  File "/home/ipanova/pulp_development/pulp_docker/extensions_admin/pulp_docker/extensions/admin/upload.py", line 45, in determine_type_id
    json.loads(upload.read())
MemoryError
An unexpected error has occurred.

Associated revisions

Revision b473c3f7 View on GitHub
Added by ipanova@redhat.com over 3 years ago

Merrory error when trying to upload a big tar.

closes #3667 https://pulp.plan.io/issues/3667

Revision b473c3f7 View on GitHub
Added by ipanova@redhat.com over 3 years ago

Merrory error when trying to upload a big tar.

closes #3667 https://pulp.plan.io/issues/3667

Revision b473c3f7 View on GitHub
Added by ipanova@redhat.com over 3 years ago

Merrory error when trying to upload a big tar.

closes #3667 https://pulp.plan.io/issues/3667

Revision b473c3f7 View on GitHub
Added by ipanova@redhat.com over 3 years ago

Merrory error when trying to upload a big tar.

closes #3667 https://pulp.plan.io/issues/3667

Revision 31de1684 View on GitHub
Added by ipanova@redhat.com over 3 years ago

Merrory error when trying to upload a big tar.

closes #3667 https://pulp.plan.io/issues/3667

(cherry picked from commit b473c3f73b71f15e3353436de02f348f2882f2f4)

Revision 31de1684 View on GitHub
Added by ipanova@redhat.com over 3 years ago

Merrory error when trying to upload a big tar.

closes #3667 https://pulp.plan.io/issues/3667

(cherry picked from commit b473c3f73b71f15e3353436de02f348f2882f2f4)

History

#1 Updated by ipanova@redhat.com over 3 years ago

  • Sprint Candidate changed from No to Yes

#2 Updated by ipanova@redhat.com over 3 years ago

  • Sprint Candidate changed from Yes to No

#3 Updated by dalley over 3 years ago

  • Severity changed from 2. Medium to 3. High
  • Triaged changed from No to Yes
  • Sprint set to Sprint 37

#4 Updated by rchan over 3 years ago

  • Sprint changed from Sprint 37 to Sprint 38

#5 Updated by ipanova@redhat.com over 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ipanova@redhat.com

#6 Updated by ipanova@redhat.com over 3 years ago

  • Status changed from ASSIGNED to POST

#7 Updated by ipanova@redhat.com over 3 years ago

  • Status changed from POST to MODIFIED

#8 Updated by ipanova@redhat.com over 3 years ago

  • Platform Release set to 2.16.2

#10 Updated by ipanova@redhat.com over 3 years ago

  • Status changed from MODIFIED to 5

#11 Updated by ipanova@redhat.com over 3 years ago

  • Status changed from 5 to CLOSED - CURRENTRELEASE

#12 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF