Issue #2334
closed
Python egg uploads fail
Description
Let's say one decides to upload a Python egg with pulp-admin. The following script for doing so succeeds:
wget https://repos.fedorapeople.org/pulp/pulp/fixtures/python/packages/source/s/shelf-reader/shelf-reader-0.1.tar.gz
pulp-admin python repo create --repo-id foo
pulp-admin python repo upload --repo-id foo --file shelf-reader-0.1.tar.gz
On the other hand, let's say one uploads directly via the API. Doing so is a four-step process:
1. Create an upload request.
2. Make one or more PUT requests, sending a chunk of the file each time.
3. Import the file into a repository.
4. Close the upload request.
This has worked since at least Pulp 2.8. Unfortunately, the code for accomplishing step 3 has changed in Pulp 2.11. A POST request to /{repo_href}/actions/import_upload/ with the following body will succeed in Pulp 2.8, 2.9 and 2.10:
{
"unit_key": {},
"unit_type_id": "python_package",
"upload_id": "..."
}
In Pulp 2.11, this request will instead cause Pulp to respond with the following failed task:
{'_href': '/pulp/api/v2/tasks/37de3871-e878-4abc-9c6f-0d39c86cd4bb/',
'_id': {'$oid': '57ff9b425137beaefc1f9d0b'},
'_ns': 'task_status',
'error': {'code': 'PLP0000',
'data': {},
'description': 'Pulp exception occurred: PulpExecutionException',
'sub_errors': []},
'exception': None,
'finish_time': '2016-10-13T14:33:38Z',
'id': '57ff9b425137beaefc1f9d0b',
'progress_report': {},
'queue': 'reserved_resource_worker-0@localhost.localdomain.dq',
'result': None,
'spawned_tasks': [],
'start_time': '2016-10-13T14:33:38Z',
'state': 'error',
'tags': ['pulp:repository:a962599e-2dfd-4560-8dfa-d6d391d141fa',
'pulp:action:import_upload'],
'task_id': '37de3871-e878-4abc-9c6f-0d39c86cd4bb',
'task_type': 'pulp.server.managers.content.upload.import_uploaded_unit',
'traceback': 'Traceback (most recent call last):\n'
' File "/usr/lib/python2.7/site-packages/celery/app/trace.py", '
'line 240, in trace_task\n'
' R = retval = fun(*args, **kwargs)\n'
' File '
'"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", '
'line 488, in __call__\n'
' return super(Task, self).__call__(*args, **kwargs)\n'
' File '
'"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", '
'line 103, in __call__\n'
' return super(PulpTask, self).__call__(*args, **kwargs)\n'
' File "/usr/lib/python2.7/site-packages/celery/app/trace.py", '
'line 437, in __protected_call__\n'
' return self.run(*args, **kwargs)\n'
' File '
'"/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py", '
'line 218, in import_uploaded_unit\n'
' unit_metadata, file_path, conduit, call_config)\n'
' File '
'"/usr/lib/python2.7/site-packages/pulp_python/plugins/importers/importer.py", '
'line 191, in upload_unit\n'
' new_file_path = os.path.join(working_dir, '
"unit_key['filename'])\n"
'PulpExecutionException: Pulp exception occurred: '
'PulpExecutionException\n',
'worker_name': 'reserved_resource_worker-0@localhost.localdomain'}
Here's a snippet from journalctl:
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) Error from the importer while importing uploaded unit to repository [a962599e-2dfd-4560-8dfa-d6d391d141fa]
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) Traceback (most recent call last):
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py", line 218, in import_uploaded_unit
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) unit_metadata, file_path, conduit, call_config)
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) File "/usr/lib/python2.7/site-packages/pulp_python/plugins/importers/importer.py", line 191, in upload_unit
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) new_file_path = os.path.join(working_dir, unit_key['filename'])
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) KeyError: 'filename'
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.async.tasks:INFO: Task failed : [fe698de5-9172-4994-ad37-d0c14188b505]
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) Task pulp.server.managers.content.upload.import_uploaded_unit[fe698de5-9172-4994-ad37-d0c14188b505] raised unexpected: PulpExecutionException(KeyError('filename',),)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) Traceback (most recent call last):
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) R = retval = fun(*args, **kwargs)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 488, in __call__
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) return super(Task, self).__call__(*args, **kwargs)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) return super(PulpTask, self).__call__(*args, **kwargs)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) return self.run(*args, **kwargs)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py", line 218, in import_uploaded_unit
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) unit_metadata, file_path, conduit, call_config)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) File "/usr/lib/python2.7/site-packages/pulp_python/plugins/importers/importer.py", line 191, in upload_unit
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) new_file_path = os.path.join(working_dir, unit_key['filename'])
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) PulpExecutionException: Pulp exception occurred: PulpExecutionException
The culprit seems to be that Pulp now expects the following POST request body:
{
"unit_key": {"filename": "shelf-reader-0.1.tar.gz"},
"unit_type_id": "python_package",
"upload_id": "..."
}
This change is backward-incompatible. It also seems unnecessary, as the filename can be derived from the package metadata.
If this information is needed when uploading packages, it should be documented in one of the following locations:
Updated by amacdona@redhat.com over 7 years ago
- Priority changed from Normal to High
- Severity changed from 2. Medium to 4. Urgent
- Target Release - Python set to 2.0.0
- Triaged changed from No to Yes
This does represent an API change and it brings up some valid concerns.
I am not really sure which direction to go here, but I think there is enough info to accept it. It should block the release of pulp_python 2.0
Updated by Ichimonji10 about 7 years ago
Python egg and wheel uploads succeed on the nightly builds of Pulp 2.12.0. I think this issue can be marked as VERIFIED with a target platform release of 2.12.0.
They may also succeed on earlier versions of Pulp, but additional changes need to be made to Pulp Smash before that can be determined with certainty.
Updated by amacdona@redhat.com almost 6 years ago
- Status changed from NEW to CLOSED - CURRENTRELEASE
Updated by Ichimonji10 almost 6 years ago
- Status changed from CLOSED - CURRENTRELEASE to NEW
It seems that this bug got lost in the shuffle of things.
The pulp-python plugin still requires a filename. This is true on hosts running Pulp 2.16.1 stable with pulp-python 2.0.2:
pulp-admin-client-2.16.1-1.fc26.noarch
pulp-deb-admin-extensions-1.7.0-1.fc26.noarch
pulp-deb-plugins-1.7.0-1.fc26.noarch
pulp-docker-admin-extensions-3.1.3-1.fc26.noarch
pulp-docker-plugins-3.1.3-1.fc26.noarch
pulp-ostree-admin-extensions-1.3.0-1.fc26.noarch
pulp-ostree-plugins-1.3.0-1.fc26.noarch
pulp-puppet-admin-extensions-2.16.1-1.fc26.noarch
pulp-puppet-plugins-2.16.1-1.fc26.noarch
pulp-puppet-tools-2.16.1-1.fc26.noarch
pulp-python-admin-extensions-2.0.2-1.fc26.noarch
pulp-python-plugins-2.0.2-1.fc26.noarch
pulp-rpm-admin-extensions-2.16.1-1.fc26.noarch
pulp-rpm-plugins-2.16.1-1.fc26.noarch
pulp-selinux-2.16.1-1.fc26.noarch
pulp-server-2.16.1-1.fc26.noarch
python-pulp-bindings-2.16.1-1.fc26.noarch
python-pulp-client-lib-2.16.1-1.fc26.noarch
python-pulp-common-2.16.1-1.fc26.noarch
python-pulp-deb-common-1.7.0-1.fc26.noarch
python-pulp-docker-common-3.1.3-1.fc26.noarch
python-pulp-oid_validation-2.16.1-1.fc26.noarch
python-pulp-ostree-common-1.3.0-1.fc26.noarch
python-pulp-puppet-common-2.16.1-1.fc26.noarch
python-pulp-python-common-2.0.2-1.fc26.noarch
python-pulp-repoauth-2.16.1-1.fc26.noarch
python-pulp-rpm-common-2.16.1-1.fc26.noarch
python-pulp-streamer-2.16.1-1.fc26.noarch
What should we do about this? It still seems unnecessary for a filename to be passed in, as it can be derived from the package metadata. But this change appears to have been implemented for some time in practice.
Updated by Ichimonji10 almost 6 years ago
- Status changed from NEW to CLOSED - CURRENTRELEASE
Updated by Ichimonji10 almost 6 years ago
- Status changed from CLOSED - CURRENTRELEASE to ASSIGNED
I'm setting this issue back to ASSIGNED because the bug described in the original report hasn't been fixed. Currently, Pulp requires a 'filename' attribute when Python eggs are uploaded. CLOSED - CURRENTRELEASE doesn't seem appropriate.
Updated by amacdona@redhat.com almost 6 years ago
- Status changed from ASSIGNED to NEW
- Assignee set to jason.dobies@redhat.com
I've switched this from assigned to NEW so this won't get lost.
This did represent a backward incompatible change, but was merged into pulp_python 2.0, so it is ok. The smash tests for python should target the pulp_python release, rather than core since they are not necessarily released together.
From pulp 2.11 forward, filename is expected.
Updated by amacdona@redhat.com almost 6 years ago
- Assignee deleted (
jason.dobies@redhat.com)
Updated by Ichimonji10 almost 6 years ago
- Status changed from NEW to CLOSED - NOTABUG
As per a conversation with Austin, this issue isn't a bug. Filenames are now required when uploading eggs to Pulp, and this change is intentional, as parsing metadata is a much more finicky process. Closing. The associated Pulp Smash test now reflects this state of affairs. See Pulp Smash #1020.
.