Project

Profile

Help

Issue #480

closed

Uploading an rpm unit in a clustered environment causes a 500 error

Added by dhaval.joshi@nomura.com almost 8 years ago. Updated over 3 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.4 Beta
Platform Release:
2.6.2
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Quarter:

Description

To reproduce:

0) Install a 2 node, fresh Pulp install with a localhost certificate
1) login to Pulp
2) create a repo named zoo
3) Have a valid rpm in the current working directory (ie: python-kombu-3.0.24-5.pulp.fc20.noarch.rpm)
4) Make sure that you are interacting directly with a server that is the NFS client. This is important. Now on that server, upload the RPM using

pulp-admin -u admin -p admin rpm repo uploads rpm --file ./python-kombu-3.0.24-5.pulp.fc20.noarch.rpm --repo-id zoo

5) Observe the following 500 error:

+----------------------------------------------------------------------+   
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: python-kombu-3.0.24-5.pulp.fc20.noarch.rpm
... completed

Creating upload requests on the server...
[==================================================] 100%
Initializing: python-kombu-3.0.24-5.pulp.fc20.noarch.rpm
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
cancelled entirely using the cancel command.

Uploading: python-kombu-3.0.24-5.pulp.fc20.noarch.rpm
[==================================================] 100%
492792/492792 bytes
... completed

Importing into the repository...
This command may be exited via ctrl+c without affecting the request.

[-]
Running...
[\]
Running...

Task Succeeded

Deleting the upload request...
An internal error occurred on the Pulp server:

RequestException: DELETE request
on /pulp/api/v2/content/uploads/2033d7a9-85bc-4f81-a481-84e068fa44f8/ failed
with 500 - [Errno 2] No such file or directory:
'/var/lib/pulp/uploads/2033d7a9-85bc-4f81-a481-84e068fa44f8'
Actions #2

Updated by bmbouter over 7 years ago

  • Severity changed from High to 3. High
Actions #3

Updated by bmbouter over 7 years ago

  • Subject changed from Distributing http service on Multiple servers to Distributing http service across multiple servers
  • Status changed from NEW to ASSIGNED
  • Assignee set to bmbouter
Actions #4

Updated by bmbouter over 7 years ago

  • Platform Release set to 2.6.1
Actions #5

Updated by bmbouter over 7 years ago

  • Subject changed from Distributing http service across multiple servers to Uploading a unit causes a 500 error
  • Description updated (diff)
  • Version set to 2.4 Beta

I was able to reproduce this on a single server Pulp installation of the 2.6.1 beta so it's not related to the clustering issue. I'm simplifying the bug description.

Actions #6

Updated by bmbouter over 7 years ago

  • Subject changed from Uploading a unit causes a 500 error to Uploading an rpm unit causes a 500 error
Actions #7

Updated by bmbouter over 7 years ago

  • File deleted (64f1669636dc8c9c4badb268de8388e1)
Actions #8

Updated by bmbouter over 7 years ago

  • Subject changed from Uploading an rpm unit causes a 500 error to Uploading an rpm unit in a clustered environment causes a 500 error
  • Description updated (diff)

Actually my dev environment was not working correctly. I did need to reproduce this in a clustered environment.

Actions #9

Updated by bmbouter over 7 years ago

  • Description updated (diff)
Actions #10

Updated by bmbouter over 7 years ago

The root cause of this is that the pulp_rpm importer code for upload moves the code instead of just importing it.

Then when the delete occurs later as an API action it causes a 500 traceback exception due to the NFS client having cached inode pointers which are stale. Then when Pulp code goes to delete the file it does not exist and a 500 exception occurs. On local filesystems the inode pointers are correct so Pulp doesn't take the same codepath.

Actions #11

Updated by bmbouter over 7 years ago

It is somewhat of a gray area if uploaded files should be able to be moved by an upload importer. This is mostly due to ambiguity in the API. This traceback is being caused primarily by the implementation of DELETE of an upload request not working well on systems that cache inode data (ie: NFS). The fix is going to adjust the DELETE implementation so that it works well in both places.

Actions #12

Updated by bmbouter over 7 years ago

  • Platform Release changed from 2.6.1 to 2.6.2

Added by bmbouter over 7 years ago

Revision e5f3c3fb

Adjusts the upload delete API implementation

The upload delete method used inode data to determine if a file exists before deleting it. On filesystems with cached or slow-to- respond inode data like NFS this was causing issues. The new implementation always tries to remove a file and silences only the error in the case where the file does not exist.

Also some small flake8 fixes.

closes #480

https://pulp.plan.io/issues/480

Added by bmbouter over 7 years ago

Revision e5f3c3fb

Adjusts the upload delete API implementation

The upload delete method used inode data to determine if a file exists before deleting it. On filesystems with cached or slow-to- respond inode data like NFS this was causing issues. The new implementation always tries to remove a file and silences only the error in the case where the file does not exist.

Also some small flake8 fixes.

closes #480

https://pulp.plan.io/issues/480

Actions #13

Updated by bmbouter over 7 years ago

  • Status changed from ASSIGNED to POST
Actions #14

Updated by bmbouter over 7 years ago

  • Description updated (diff)
Actions #15

Updated by bmbouter over 7 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100
Actions #16

Updated by dkliban@redhat.com over 7 years ago

  • Status changed from MODIFIED to 5
Actions #18

Updated by pthomas@redhat.com over 7 years ago

  • Status changed from 5 to 6

verified

Set up clustered pulp and upload performed on the nfs client

[root@cloud-qe-4 ~]# rpm -qa pulp-server
pulp-server-2.6.2-0.2.beta.el6.noarch
[root@cloud-qe-4 ~]# 
[root@cloud-qe-4 ~]# pulp-admin -u admin -p admin rpm repo uploads rpm --repo-id upload -f pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm -v
+----------------------------------------------------------------------+
                              Unit Upload
+----------------------------------------------------------------------+

Extracting necessary metadata for each request...
[==================================================] 100%
Analyzing: pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm
... completed

Files to be uploaded:
  pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm

Creating upload requests on the server...
[==================================================] 100%
Initializing: pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm
... completed

Starting upload of selected units. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
cancelled entirely using the cancel command.

Uploading: pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm
[==================================================] 100%
2359/2359 bytes
... completed

Importing into the repository...
This command may be exited via ctrl+c without affecting the request.

[\]
Running...

Task Succeeded

Deleting the upload request...
... completed

[root@cloud-qe-4 ~]# 
Actions #19

Updated by dkliban@redhat.com about 7 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE
Actions #21

Updated by bmbouter over 3 years ago

  • Tags Pulp 2 added

Also available in: Atom PDF