Project

Profile

Help

Issue #480

Uploading an rpm unit in a clustered environment causes a 500 error

Added by dhaval.joshi@nomura.com over 6 years ago. Updated over 2 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.4 Beta
Platform Release:
2.6.2
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

To reproduce:

0) Install a 2 node, fresh Pulp install with a localhost certificate
1) login to Pulp
2) create a repo named zoo
3) Have a valid rpm in the current working directory (ie: python-kombu-3.0.24-5.pulp.fc20.noarch.rpm)
4) Make sure that you are interacting directly with a server that is the NFS client. This is important. Now on that server, upload the RPM using

pulp-admin -u admin -p admin rpm repo uploads rpm --file ./python-kombu-3.0.24-5.pulp.fc20.noarch.rpm --repo-id zoo

5) Observe the following 500 error:

+----------------------------------------------------------------------+   
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: python-kombu-3.0.24-5.pulp.fc20.noarch.rpm
... completed

Creating upload requests on the server...
[==================================================] 100%
Initializing: python-kombu-3.0.24-5.pulp.fc20.noarch.rpm
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
cancelled entirely using the cancel command.

Uploading: python-kombu-3.0.24-5.pulp.fc20.noarch.rpm
[==================================================] 100%
492792/492792 bytes
... completed

Importing into the repository...
This command may be exited via ctrl+c without affecting the request.

[-]
Running...
[\]
Running...

Task Succeeded

Deleting the upload request...
An internal error occurred on the Pulp server:

RequestException: DELETE request
on /pulp/api/v2/content/uploads/2033d7a9-85bc-4f81-a481-84e068fa44f8/ failed
with 500 - [Errno 2] No such file or directory:
'/var/lib/pulp/uploads/2033d7a9-85bc-4f81-a481-84e068fa44f8'

Associated revisions

Revision e5f3c3fb View on GitHub
Added by bmbouter over 6 years ago

Adjusts the upload delete API implementation

The upload delete method used inode data to determine if a file exists before deleting it. On filesystems with cached or slow-to- respond inode data like NFS this was causing issues. The new implementation always tries to remove a file and silences only the error in the case where the file does not exist.

Also some small flake8 fixes.

closes #480

https://pulp.plan.io/issues/480

Revision e5f3c3fb View on GitHub
Added by bmbouter over 6 years ago

Adjusts the upload delete API implementation

The upload delete method used inode data to determine if a file exists before deleting it. On filesystems with cached or slow-to- respond inode data like NFS this was causing issues. The new implementation always tries to remove a file and silences only the error in the case where the file does not exist.

Also some small flake8 fixes.

closes #480

https://pulp.plan.io/issues/480

History

#2 Updated by bmbouter over 6 years ago

  • Severity changed from High to 3. High

#3 Updated by bmbouter over 6 years ago

  • Subject changed from Distributing http service on Multiple servers to Distributing http service across multiple servers
  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter

#4 Updated by bmbouter over 6 years ago

  • Platform Release set to 2.6.1

#5 Updated by bmbouter over 6 years ago

  • Subject changed from Distributing http service across multiple servers to Uploading a unit causes a 500 error
  • Description updated (diff)
  • Version set to 2.4 Beta

I was able to reproduce this on a single server Pulp installation of the 2.6.1 beta so it's not related to the clustering issue. I'm simplifying the bug description.

#6 Updated by bmbouter over 6 years ago

  • Subject changed from Uploading a unit causes a 500 error to Uploading an rpm unit causes a 500 error

#7 Updated by bmbouter over 6 years ago

  • File deleted (64f1669636dc8c9c4badb268de8388e1)

#8 Updated by bmbouter over 6 years ago

  • Subject changed from Uploading an rpm unit causes a 500 error to Uploading an rpm unit in a clustered environment causes a 500 error
  • Description updated (diff)

Actually my dev environment was not working correctly. I did need to reproduce this in a clustered environment.

#9 Updated by bmbouter over 6 years ago

  • Description updated (diff)

#10 Updated by bmbouter over 6 years ago

The root cause of this is that the pulp_rpm importer code for upload moves the code instead of just importing it.

Then when the delete occurs later as an API action it causes a 500 traceback exception due to the NFS client having cached inode pointers which are stale. Then when Pulp code goes to delete the file it does not exist and a 500 exception occurs. On local filesystems the inode pointers are correct so Pulp doesn't take the same codepath.

#11 Updated by bmbouter over 6 years ago

It is somewhat of a gray area if uploaded files should be able to be moved by an upload importer. This is mostly due to ambiguity in the API. This traceback is being caused primarily by the implementation of DELETE of an upload request not working well on systems that cache inode data (ie: NFS). The fix is going to adjust the DELETE implementation so that it works well in both places.

#12 Updated by bmbouter over 6 years ago

  • Platform Release changed from 2.6.1 to 2.6.2

#13 Updated by bmbouter over 6 years ago

  • Status changed from ASSIGNED to POST

#14 Updated by bmbouter over 6 years ago

  • Description updated (diff)

#15 Updated by bmbouter over 6 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#16 Updated by dkliban@redhat.com over 6 years ago

  • Status changed from MODIFIED to 5

#18 Updated by pthomas@redhat.com over 6 years ago

  • Status changed from 5 to 6

verified

Set up clustered pulp and upload performed on the nfs client

[root@cloud-qe-4 ~]# rpm -qa pulp-server
pulp-server-2.6.2-0.2.beta.el6.noarch
[root@cloud-qe-4 ~]# 
[root@cloud-qe-4 ~]# pulp-admin -u admin -p admin rpm repo uploads rpm --repo-id upload -f pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm -v
+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm
... completed

Files to be uploaded:
  pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm

Creating upload requests on the server...
[==================================================] 100%
Initializing: pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
cancelled entirely using the cancel command.

Uploading: pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm
[==================================================] 100%
2359/2359 bytes
... completed

Importing into the repository...
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

Deleting the upload request...
... completed

[root@cloud-qe-4 ~]# 

#19 Updated by dkliban@redhat.com about 6 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#21 Updated by bmbouter over 2 years ago

  • Tags Pulp 2 added

Please register to edit this issue

Also available in: Atom PDF