Project

Profile

Help

Issue #9463

closed

The chunked upload test fails in the dev environment

Added by daviddavis about 3 years ago. Updated about 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
Installer - Moved to GitHub issues
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Dev Environment
Sprint:
Quarter:

Description

The chunked upload test in pulpcore/tests/functional/api/test_upload.py fails on the dev environment or when the test is run against a pulp server using nginx. This is because the max request size by default in nginx is 1M.

________________________________________________________________ ChunkedUploadTestCase.test_create_artifact_passing_checksum _________________________________________________________________
                                                         
self = <pulpcore.tests.functional.api.test_upload.ChunkedUploadTestCase testMethod=test_create_artifact_passing_checksum>
                                
    def test_create_artifact_passing_checksum(self):
        """Test creation of artifact using upload of files in chunks passing checksum."""
        upload_request = self.client.post(UPLOAD_PATH, {"size": self.size_file})                      

        for data in self.chunked_data:     
            import pdb; pdb.set_trace()                                                               
>           self.client.put(
                upload_request["pulp_href"],
                data={"sha256": hashlib.sha256(data[0]).hexdigest()},
                files={"file": data[0]},                                                                                                                                                      
                headers=data[1],
            )
                                                                                   
pulpcore/tests/functional/api/test_upload.py:78:                                                                                                                                              
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../pulp-smash/pulp_smash/api.py:597: in put                                                                                                                                                  
    return self.request("PUT", url, **kwargs)                                                                               
../pulp-smash/pulp_smash/api.py:629: in request
    response = self.response_handler(self, requests.request(method, **request_kwargs))
../pulp-smash/pulp_smash/api.py:284: in smart_handler   
    response = safe_handler(client, response)                                                                                                                                                 
../pulp-smash/pulp_smash/api.py:148: in safe_handler
    response.raise_for_status()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Response [413]>

    def raise_for_status(self):
        """Raises :class:`HTTPError`, if one occurred."""

        http_error_msg = ''
        if isinstance(self.reason, bytes):
            # We attempt to decode utf-8 first because some servers
            # choose to localize their reason strings. If the string
            # isn't utf-8, we fall back to iso-8859-1 for all other
            # encodings. (See PR #3538)
            try:
                reason = self.reason.decode('utf-8')
            except UnicodeDecodeError:
                reason = self.reason.decode('iso-8859-1')
        else:
            reason = self.reason
    
        if 400 <= self.status_code < 500:
            http_error_msg = u'%s Client Error: %s for url: %s' % (self.status_code, reason, self.url)
    
        elif 500 <= self.status_code < 600:
            http_error_msg = u'%s Server Error: %s for url: %s' % (self.status_code, reason, self.url)
    
        if http_error_msg:
>           raise HTTPError(http_error_msg, response=self)
E           requests.exceptions.HTTPError: 413 Client Error: Request Entity Too Large for url: http://pulp3-source-fedora.crake.example.com:80/pulp/api/v3/uploads/a222e7e9-2e4e-4119-8d7a-d54
2a319624c/

/usr/local/lib/pulp/lib64/python3.9/site-packages/requests/models.py:943: HTTPError
Actions #1

Updated by daviddavis about 3 years ago

  • Status changed from ASSIGNED to POST
Actions #2

Updated by daviddavis about 3 years ago

  • Sprint set to Sprint 106
Actions #3

Updated by daviddavis about 3 years ago

Looks like the test passes in the CI because the CI container sets the request limit size to 10m:

https://github.com/pulp/pulp-oci-images/blob/d7413e6f0f1f56a6b61288f86939f7261d63da54/assets/nginx.conf#L62

Actions #4

Updated by daviddavis about 3 years ago

  • Category set to Installer - Moved to GitHub issues
  • Status changed from POST to NEW
  • Assignee deleted (daviddavis)
  • Sprint deleted (Sprint 106)

It looks like updating the fixture to use a smaller size is incredibly painful. It breaks a number of tests and we'd have to backport these fixes to numerous older release branches.

I think perhaps the best way to fix this is to update the dev environment to raise the client_max_body_size to 10m?

Actions #5

Updated by fao89 about 3 years ago

Actions #6

Updated by daviddavis about 3 years ago

Nice, I wonder if we could/should default it to 10m for dev environments?

Actions #7

Updated by fao89 about 3 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to fao89
  • Tags Dev Environment added

Added by Fabricio Aguiar about 3 years ago

Revision 34d25453 | View on GitHub

Set client_max_body_size to 10m for dev environments

closes #9463

Added by Fabricio Aguiar about 3 years ago

Revision 34d25453 | View on GitHub

Set client_max_body_size to 10m for dev environments

closes #9463

Actions #8

Updated by pulpbot about 3 years ago

  • Status changed from ASSIGNED to POST
Actions #9

Updated by Anonymous about 3 years ago

  • Status changed from POST to MODIFIED
Actions #10

Updated by dkliban@redhat.com about 3 years ago

  • Sprint/Milestone set to 3.16.0
Actions #11

Updated by pulpbot about 3 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF