Issue #480
closed
Uploading an rpm unit in a clustered environment causes a 500 error
Description
To reproduce:
0) Install a 2 node, fresh Pulp install with a localhost certificate
1) login to Pulp
2) create a repo named zoo
3) Have a valid rpm in the current working directory (ie: python-kombu-3.0.24-5.pulp.fc20.noarch.rpm)
4) Make sure that you are interacting directly with a server that is the NFS client. This is important. Now on that server, upload the RPM using
pulp-admin -u admin -p admin rpm repo uploads rpm --file ./python-kombu-3.0.24-5.pulp.fc20.noarch.rpm --repo-id zoo
5) Observe the following 500 error:
+----------------------------------------------------------------------+
Unit Upload
+----------------------------------------------------------------------+
Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: python-kombu-3.0.24-5.pulp.fc20.noarch.rpm
... completed
Creating upload requests on the server...
[==================================================] 100%
Initializing: python-kombu-3.0.24-5.pulp.fc20.noarch.rpm
... completed
Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
cancelled entirely using the cancel command.
Uploading: python-kombu-3.0.24-5.pulp.fc20.noarch.rpm
[==================================================] 100%
492792/492792 bytes
... completed
Importing into the repository...
This command may be exited via ctrl+c without affecting the request.
[-]
Running...
[\]
Running...
Task Succeeded
Deleting the upload request...
An internal error occurred on the Pulp server:
RequestException: DELETE request
on /pulp/api/v2/content/uploads/2033d7a9-85bc-4f81-a481-84e068fa44f8/ failed
with 500 - [Errno 2] No such file or directory:
'/var/lib/pulp/uploads/2033d7a9-85bc-4f81-a481-84e068fa44f8'
Updated by rbarlow almost 9 years ago
https://github.com/pulp/pulp/pull/1097
+ This comment was cloned from Bugzilla #1126723 comment 1 +
Updated by bmbouter over 8 years ago
- Subject changed from Distributing http service on Multiple servers to Distributing http service across multiple servers
- Status changed from NEW to ASSIGNED
- Assignee set to bmbouter
Updated by bmbouter over 8 years ago
- Subject changed from Distributing http service across multiple servers to Uploading a unit causes a 500 error
- Description updated (diff)
- Version set to 2.4 Beta
I was able to reproduce this on a single server Pulp installation of the 2.6.1 beta so it's not related to the clustering issue. I'm simplifying the bug description.
Updated by bmbouter over 8 years ago
- Subject changed from Uploading a unit causes a 500 error to Uploading an rpm unit causes a 500 error
Updated by bmbouter over 8 years ago
- File deleted (
64f1669636dc8c9c4badb268de8388e1)
Updated by bmbouter over 8 years ago
- Subject changed from Uploading an rpm unit causes a 500 error to Uploading an rpm unit in a clustered environment causes a 500 error
- Description updated (diff)
Actually my dev environment was not working correctly. I did need to reproduce this in a clustered environment.
Updated by bmbouter over 8 years ago
The root cause of this is that the pulp_rpm importer code for upload moves the code instead of just importing it.
Then when the delete occurs later as an API action it causes a 500 traceback exception due to the NFS client having cached inode pointers which are stale. Then when Pulp code goes to delete the file it does not exist and a 500 exception occurs. On local filesystems the inode pointers are correct so Pulp doesn't take the same codepath.
Updated by bmbouter over 8 years ago
It is somewhat of a gray area if uploaded files should be able to be moved by an upload importer. This is mostly due to ambiguity in the API. This traceback is being caused primarily by the implementation of DELETE of an upload request not working well on systems that cache inode data (ie: NFS). The fix is going to adjust the DELETE implementation so that it works well in both places.
Updated by bmbouter over 8 years ago
- Platform Release changed from 2.6.1 to 2.6.2
Added by bmbouter over 8 years ago
Added by bmbouter over 8 years ago
Revision e5f3c3fb | View on GitHub
Adjusts the upload delete API implementation
The upload delete method used inode data to determine if a file exists before deleting it. On filesystems with cached or slow-to- respond inode data like NFS this was causing issues. The new implementation always tries to remove a file and silences only the error in the case where the file does not exist.
Also some small flake8 fixes.
closes #480
Updated by bmbouter over 8 years ago
- Status changed from ASSIGNED to POST
PR available at: https://github.com/pulp/pulp/pull/1791
Updated by bmbouter over 8 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulp|e5f3c3fb70d4a67f81cf8c4740a582f90420ca49.
Updated by dkliban@redhat.com over 8 years ago
- Status changed from MODIFIED to 5
Updated by pthomas@redhat.com over 8 years ago
- Status changed from 5 to 6
verified
Set up clustered pulp and upload performed on the nfs client
[root@cloud-qe-4 ~]# rpm -qa pulp-server
pulp-server-2.6.2-0.2.beta.el6.noarch
[root@cloud-qe-4 ~]#
[root@cloud-qe-4 ~]# pulp-admin -u admin -p admin rpm repo uploads rpm --repo-id upload -f pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm -v
+----------------------------------------------------------------------+
Unit Upload
+----------------------------------------------------------------------+
Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm
... completed
Files to be uploaded:
pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm
Creating upload requests on the server...
[==================================================] 100%
Initializing: pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm
... completed
Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
cancelled entirely using the cancel command.
Uploading: pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm
[==================================================] 100%
2359/2359 bytes
... completed
Importing into the repository...
This command may be exited via ctrl+c without affecting the request.
[\]
Running...
Task Succeeded
Deleting the upload request...
... completed
[root@cloud-qe-4 ~]#
Updated by dkliban@redhat.com about 8 years ago
- Status changed from 6 to CLOSED - CURRENTRELEASE
.
Adjusts the upload delete API implementation
The upload delete method used inode data to determine if a file exists before deleting it. On filesystems with cached or slow-to- respond inode data like NFS this was causing issues. The new implementation always tries to remove a file and silences only the error in the case where the file does not exist.
Also some small flake8 fixes.
closes #480
https://pulp.plan.io/issues/480