Issue #480
closed
Uploading an rpm unit in a clustered environment causes a 500 error
Status:
CLOSED - CURRENTRELEASE
Description
To reproduce:
0) Install a 2 node, fresh Pulp install with a localhost certificate
1) login to Pulp
2) create a repo named zoo
3) Have a valid rpm in the current working directory (ie: python-kombu-3.0.24-5.pulp.fc20.noarch.rpm)
4) Make sure that you are interacting directly with a server that is the NFS client. This is important. Now on that server, upload the RPM using
pulp-admin -u admin -p admin rpm repo uploads rpm --file ./python-kombu-3.0.24-5.pulp.fc20.noarch.rpm --repo-id zoo
5) Observe the following 500 error:
+----------------------------------------------------------------------+
Unit Upload
+----------------------------------------------------------------------+
Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: python-kombu-3.0.24-5.pulp.fc20.noarch.rpm
... completed
Creating upload requests on the server...
[==================================================] 100%
Initializing: python-kombu-3.0.24-5.pulp.fc20.noarch.rpm
... completed
Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
cancelled entirely using the cancel command.
Uploading: python-kombu-3.0.24-5.pulp.fc20.noarch.rpm
[==================================================] 100%
492792/492792 bytes
... completed
Importing into the repository...
This command may be exited via ctrl+c without affecting the request.
[-]
Running...
[\]
Running...
Task Succeeded
Deleting the upload request...
An internal error occurred on the Pulp server:
RequestException: DELETE request
on /pulp/api/v2/content/uploads/2033d7a9-85bc-4f81-a481-84e068fa44f8/ failed
with 500 - [Errno 2] No such file or directory:
'/var/lib/pulp/uploads/2033d7a9-85bc-4f81-a481-84e068fa44f8'
- Severity changed from High to 3. High
- Subject changed from Distributing http service on Multiple servers to Distributing http service across multiple servers
- Status changed from NEW to ASSIGNED
- Assignee set to bmbouter
- Platform Release set to 2.6.1
- Subject changed from Distributing http service across multiple servers to Uploading a unit causes a 500 error
- Description updated (diff)
- Version set to 2.4 Beta
I was able to reproduce this on a single server Pulp installation of the 2.6.1 beta so it's not related to the clustering issue. I'm simplifying the bug description.
- Subject changed from Uploading a unit causes a 500 error to Uploading an rpm unit causes a 500 error
- File deleted (
64f1669636dc8c9c4badb268de8388e1)
- Subject changed from Uploading an rpm unit causes a 500 error to Uploading an rpm unit in a clustered environment causes a 500 error
- Description updated (diff)
Actually my dev environment was not working correctly. I did need to reproduce this in a clustered environment.
- Description updated (diff)
The root cause of this is that the pulp_rpm importer code for upload moves the code instead of just importing it.
Then when the delete occurs later as an API action it causes a 500 traceback exception due to the NFS client having cached inode pointers which are stale. Then when Pulp code goes to delete the file it does not exist and a 500 exception occurs. On local filesystems the inode pointers are correct so Pulp doesn't take the same codepath.
It is somewhat of a gray area if uploaded files should be able to be moved by an upload importer. This is mostly due to ambiguity in the API. This traceback is being caused primarily by the implementation of DELETE of an upload request not working well on systems that cache inode data (ie: NFS). The fix is going to adjust the DELETE implementation so that it works well in both places.
- Platform Release changed from 2.6.1 to 2.6.2
- Status changed from ASSIGNED to POST
- Description updated (diff)
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
- Status changed from MODIFIED to 5
- Status changed from 5 to 6
verified
Set up clustered pulp and upload performed on the nfs client
[root@cloud-qe-4 ~]# rpm -qa pulp-server
pulp-server-2.6.2-0.2.beta.el6.noarch
[root@cloud-qe-4 ~]#
[root@cloud-qe-4 ~]# pulp-admin -u admin -p admin rpm repo uploads rpm --repo-id upload -f pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm -v
+----------------------------------------------------------------------+
Unit Upload
+----------------------------------------------------------------------+
Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm
... completed
Files to be uploaded:
pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm
Creating upload requests on the server...
[==================================================] 100%
Initializing: pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm
... completed
Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
cancelled entirely using the cancel command.
Uploading: pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm
[==================================================] 100%
2359/2359 bytes
... completed
Importing into the repository...
This command may be exited via ctrl+c without affecting the request.
[\]
Running...
Task Succeeded
Deleting the upload request...
... completed
[root@cloud-qe-4 ~]#
- Status changed from 6 to CLOSED - CURRENTRELEASE
Also available in: Atom
PDF
Adjusts the upload delete API implementation
The upload delete method used inode data to determine if a file exists before deleting it. On filesystems with cached or slow-to- respond inode data like NFS this was causing issues. The new implementation always tries to remove a file and silences only the error in the case where the file does not exist.
Also some small flake8 fixes.
closes #480
https://pulp.plan.io/issues/480