Description of current upload process¶

The root of the problem is that our upload API functions this way:

1. We make an API call to initialize an upload which returns the `upload_id`. This call 
   does not require or accept any additional information. Internally Pulp creates a file
   in the working directory, filename = upload_id. `upload_id` is a randomly generated uuid
   and it contains no information about itself, specifically, there is no file extension.

2. The `upload_id` is used by additional API calls that contain the actual bits. The upload
   id is used to tell core Pulp where to put the bits. Note: Because the `upload_id` 
   parameter is included in the url, it cannot have '/' in it.

Problem 1:¶

I am using twine to pull the metadata out of the a Python package, however twine relies on
the filename in order to know how to process that metadata. Since the filename is a uuid without
an extension, twine cannot do this.

Problem 2:¶

Python packages of type wheel can be created specifically for certain architectures, so name and
version are no longer enough to be the unit key. Instead, I have chosen filename because the spec
demonstrates that filenames must be unique. Current Python upload code rebuilds the filename from
metadata. Though this continues to be possible, adding the architectures significantly increases
the complexity of building the filename and increases the risk that we will generate these
filenames differently.

Possible solutions:¶

We could add an optional parameter of the body of the API call to initialize an upload that
contains the filename. Because the first call does nothing but create a file and return the
upload id (which is just the file name), we can either:

• Keep the filename of the file we are uploading by making the upload_id/filename of the temporary file the same as the original filename.
• We could write the filename into the file so it can be read later. Once the upload is complete, we could mv or copy the file to its original filename (or just add the appropriate extension).

Unfortunately, the devil will be in the details so I need to explain implementation for each of
these possibilities.

Solution 1¶

Python filenames are unique, but I am not sure that we can say the same thing about every
plugin. If filenames are not unique, it is possible that we could simultaneously upload
bits into the same file. If it is a requirement to avoid collisions between filenames at
upload time, we could work around the problem by creating a directory (perhaps the uuid) with the
filename inside of it. If we take this approach, the upload_id will need to be the relative path
inside of the working directory, but it must be urlencoded to prevent '/' from breaking the
upload_id parameter in the url. When the upload is finished, we know can decode the path, split at
'/' and take the last part to find the file name.

There are a lot of places that upload_id is used, and we would need to really explore that to make
sure that this wouldn't break something.

Solution 2¶

If we read the filename, when it comes time to initialize a unit for the file, we can change the
filename just before it is passedd to twine. This would probably have fewer places it could break,
but seems even more hacky and this behavior would be very surprising to any developers who were
trying to fix bugs.

Conclusions¶

This is messy, and all solutions that I have explored so far are fragile. Additionally, I think
that they will be unusual solutions to common problems that rely on surprising behaviors. In my
opinion, a better path forward would be to create a new upload api as a part of the next release,
depricating our old upload API. This needs to be done anyway, but if we created new endpoints, we
could get this done before we go to Pulp 3.0. Honestly, this is ideal anyway because this allows
us to depricate before we drop. This approach would continue to conform to semantic versioning
because the only plugin that would require the new API and would no longer work with the old API
would be the Python plugin, which we are already planning on bumping to 2.0 for the next release.

https://github.com/pulp/pulp_python/pull/79

Applied in changeset 5afab5b6e6370cdfef9da7c6f1d2670a04b682f8.

Applied in changeset f63de8d20a3ba9479e99df5e2bb63dd6e1c3a00b.

This issue has been removed from the 2.12.0 release, and returned to its MODIFIED state for inclusion in a future release of Pulp.

I'm moving this back to POST to flag this bug as untestable in pulp-smash. This status is also appropriate because while the changes are merged to the 2.0-dev branch of pulp_python, that entire branch was deemed unreleasable for pulp 2.12, and the situation has not improved since then.

2.0-dev is now being included in Platform 2.13 builds again, returning this to a testable state. Note that any smash tests related to this issue should not be run prior to 2.13.0; this is tracked in pulp-smash issue https://github.com/PulpQE/pulp-smash/issues/588.