Python Support

Let's say one decides to upload a Python egg with pulp-admin. The following script for doing so succeeds:

wget https://repos.fedorapeople.org/pulp/pulp/fixtures/python/packages/source/s/shelf-reader/shelf-reader-0.1.tar.gz
pulp-admin python repo create --repo-id foo
pulp-admin python repo upload --repo-id foo --file shelf-reader-0.1.tar.gz

On the other hand, let's say one uploads directly via the API. Doing so is a four-step process:

1. Create an upload request.
2. Make one or more PUT requests, sending a chunk of the file each time.
3. Import the file into a repository.
4. Close the upload request.

This has worked since at least Pulp 2.8. Unfortunately, the code for accomplishing step 3 has changed in Pulp 2.11. A POST request to /{repo_href}/actions/import_upload/ with the following body will succeed in Pulp 2.8, 2.9 and 2.10:

{
  "unit_key": {},
  "unit_type_id": "python_package",
  "upload_id": "..."
}

In Pulp 2.11, this request will instead cause Pulp to respond with the following failed task:

{'_href': '/pulp/api/v2/tasks/37de3871-e878-4abc-9c6f-0d39c86cd4bb/',
 '_id': {'$oid': '57ff9b425137beaefc1f9d0b'},
 '_ns': 'task_status',
 'error': {'code': 'PLP0000',
           'data': {},
           'description': 'Pulp exception occurred: PulpExecutionException',
           'sub_errors': []},
 'exception': None,
 'finish_time': '2016-10-13T14:33:38Z',
 'id': '57ff9b425137beaefc1f9d0b',
 'progress_report': {},
 'queue': 'reserved_resource_worker-0@localhost.localdomain.dq',
 'result': None,
 'spawned_tasks': [],
 'start_time': '2016-10-13T14:33:38Z',
 'state': 'error',
 'tags': ['pulp:repository:a962599e-2dfd-4560-8dfa-d6d391d141fa',
          'pulp:action:import_upload'],
 'task_id': '37de3871-e878-4abc-9c6f-0d39c86cd4bb',
 'task_type': 'pulp.server.managers.content.upload.import_uploaded_unit',
 'traceback': 'Traceback (most recent call last):\n'
              '  File "/usr/lib/python2.7/site-packages/celery/app/trace.py", '
              'line 240, in trace_task\n'
              '    R = retval = fun(*args, **kwargs)\n'
              '  File '
              '"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", '
              'line 488, in __call__\n'
              '    return super(Task, self).__call__(*args, **kwargs)\n'
              '  File '
              '"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", '
              'line 103, in __call__\n'
              '    return super(PulpTask, self).__call__(*args, **kwargs)\n'
              '  File "/usr/lib/python2.7/site-packages/celery/app/trace.py", '
              'line 437, in __protected_call__\n'
              '    return self.run(*args, **kwargs)\n'
              '  File '
              '"/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py", '
              'line 218, in import_uploaded_unit\n'
              '    unit_metadata, file_path, conduit, call_config)\n'
              '  File '
              '"/usr/lib/python2.7/site-packages/pulp_python/plugins/importers/importer.py", '
              'line 191, in upload_unit\n'
              '    new_file_path = os.path.join(working_dir, '
              "unit_key['filename'])\n"
              'PulpExecutionException: Pulp exception occurred: '
              'PulpExecutionException\n',
 'worker_name': 'reserved_resource_worker-0@localhost.localdomain'}

Here's a snippet from journalctl:

Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) Error from the importer while importing uploaded unit to repository [a962599e-2dfd-4560-8dfa-d6d391d141fa]
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) Traceback (most recent call last):
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py", line 218, in import_uploaded_unit
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792)     unit_metadata, file_path, conduit, call_config)
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/importers/importer.py", line 191, in upload_unit
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792)     new_file_path = os.path.join(working_dir, unit_key['filename'])
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.managers.content.upload:ERROR: (7917-13792) KeyError: 'filename'
Oct 13 10:33:38 localhost.localdomain pulp[7917]: pulp.server.async.tasks:INFO: Task failed : [fe698de5-9172-4994-ad37-d0c14188b505]
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) Task pulp.server.managers.content.upload.import_uploaded_unit[fe698de5-9172-4994-ad37-d0c14188b505] raised unexpected: PulpExecutionException(KeyError('filename',),)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) Traceback (most recent call last):
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792)     R = retval = fun(*args, **kwargs)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 488, in __call__
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792)     return super(Task, self).__call__(*args, **kwargs)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792)     return super(PulpTask, self).__call__(*args, **kwargs)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792)     return self.run(*args, **kwargs)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py", line 218, in import_uploaded_unit
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792)     unit_metadata, file_path, conduit, call_config)
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/importers/importer.py", line 191, in upload_unit
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792)     new_file_path = os.path.join(working_dir, unit_key['filename'])
Oct 13 10:33:38 localhost.localdomain pulp[7629]: celery.worker.job:ERROR: (7629-13792) PulpExecutionException: Pulp exception occurred: PulpExecutionException

The culprit seems to be that Pulp now expects the following POST request body:

{
  "unit_key": {"filename": "shelf-reader-0.1.tar.gz"},
  "unit_type_id": "python_package",
  "upload_id": "..."
}

This change is backward-incompatible. It also seems unnecessary, as the filename can be derived from the package metadata.

If this information is needed when uploading packages, it should be documented in one of the following locations:

• http://docs.pulpproject.org/dev-guide/integration/rest-api/content/upload.html#import-into-a-repository
• http://docs.pulpproject.org/plugins/pulp_python/user-docs/getting_started.html#upload-a-python-package