As a plugin writer, I cannot export my Publication to POSIX filesystems
Say you create a Publication with 3 PublishedMetadata objects, each with an Artifact to be served to the user for that relative_path. Note
/foo also has an Artifact.
You can't create this on a POSIX filesystem because of the file/directory conflict on
[bmbouter@localhost tmp]$ touch foo [bmbouter@localhost tmp]$ mkdir foo mkdir: cannot create directory ‘foo’: File exists
This prevents users from exporting the Artifacts from a Publication to POSIX filesystems.
#2 Updated by bmbouter over 1 year ago
I believe webservers have the same issues, users can request a url like
/foo/ which would be a directory on the filesystem
foo and expect to receive data. They do this with the 'index' convention where the webserver typically looks for a file called
index with some type of file extension.
One option is to leave Pulp as-is and just have the Exporter export with ^ convention. At export time though it would be more complicated to perform.
Another option is to find some way to prevent Pulp from making a Publication that can't be exported. Some sort of validation that ensures you don't have
/foo/bar both in a Publication.
What sort of adjustment to
PublishedMetadata would improve this situation? Maybe the DB should distinguish directories? Or is it better to leave as-is, and handle at export time?
#7 Updated by bmbouter over 1 year ago
Another case to consider is plugins that don't have publications (as plugins have the option to distribute repo versions directly like we do in pulp_ansible). Perhaps one option is to have plugins validate repo versions.
That's a good point...
#10 Updated by email@example.com over 1 year ago
I agree with bmbouter's approach.
I will offer some other suggestions/ideas:
1. A filesystem is a hierarchical database. The Windows registry (also a hierarchical database) encountered a kind-of-similar issue decades ago. Their approach applied to this situation would be to create a file named like:
That gets created whenever foo is both a file and a directory. The registry is only accessed via their APIs, so they could do that easily though.
However, that would prevent package managers from accessing the filesystem directly at the original paths and reading foo, correct? Or any 3rd party webservers (hosting for package managers) would have to interpret the filepath.
2. You could also store the real path to the "default" file in a filesystem's folder's extended attribute (user_xattr), but that would still have the same problem.
#11 Updated by gmbnomis over 1 year ago
I think we are trying to make a rather "esoteric" use case work if we map file/directory name conflicts to
.default or similar.
What should happen if there is already a file/directory named
.default? We are just shifting the problem one directory layer down. (Of course we could use another mapping in this case)
Additionally, we might want to enable directory listings in the content app in the future. Theoretically,
http:/content.app/foo/ (returning the directory listing) could behave differently than
http:/content.app/foo (returning the foo content). But that would be very surprising (usually web servers deliver a redirection foo -> foo/ and that's what I would expect the content app to do as well).
I think that we should regard this use case as invalid. We have three options where to do this:
- repo version (as proposed by daviddavis, i.e. it could be part of the upcoming repo version validation)
IMHO, we should let publication fail given the strange URL scheme such a publication exhibits when being distributed.
But what is the point of allowing a repo version that can't be published? That's why I tend to already let validation fail. And, as noted above by daviddavis, this also solves the problem for plugins that have no publication.
#12 Updated by daviddavis about 1 year ago
Last week we discussed this problem and came to the conclusion that we need to add this validation to two places:
1. Repo version creation
In order to handle the case where the plugin doesn't have publications, we need to check at repo version creation time. We also need to check at publication time though because published artifacts may clash with content relative paths. For example, suppose I create a file content with relative path PULP_MANIFEST/test.iso. That will create problems for file publications.
I think we should also just fail if a overlap is detected. And I think plugins control both of these places though so we'll probably need to create a method or some util that they can reuse.
If this all sounds right, I'll update the description and get this issue groomed by next week.
Please register to edit this issue