I feel like this issue is part of a larger issue, namely designing a plugin API. Furthermore, I imagine this 'unification' requires actually tracking individual files in Pulp with metadata requires to validate the integrity of the file (checksums/checksum types, size, maybe permissions/ownership, location, etc). Our current data models don't do this.

Since we don't have either of those things fleshed out, would it be reasonable figure out what we're doing there before trying to work on this story? Or is this about having a very high-level idea of where we'd like to be?

@jcline I agree with all of the observations you made, especially the part that requires us to know what we want out of this story.

For myself, I was thinking the latter thing you suggested ... This story is to write out "a very high-level idea of where we'd like to be".

Okay, given that, how about I outline what I'm thinking and if no one objects we can edit the description on this story.

Content Integrity in Pulp¶

As a user of Pulp, I would like to ensure the content I am serving to my clients is correct (that is to say, it is what I think it is). Content can become incorrect for several reasons. The most likely reason for incorrect content is probably a bug in either Pulp's code, or one of the libraries we use. However, these bugs are not the only reason content could be incorrect. Bit rot can occur, as can bugs in hard drive firmware[0]. Some file systems are capable of detecting bit rot of whole files and potentially repairing it[1][2][3], but many do not (like ext4 and XFS). Even if they could, we should not rely on a software layer below us for the integrity of the content we manage.

Things we want to be able to do:

• Tell Pulp to check every file's integrity (a pulp-scrub if you will)
• Tell Pulp to attempt to fix problems it finds (re-download the file or similar)
• Tell Pulp a particular file is bad and it should retrieve it again (I'm thinking about content types that don't have checksums as part of their metadata, or to recover from bugs in the two bullet points above)

How Our Data Model Must Change¶

To have any chance of providing content integrity validation and potential repair (a pulp-scrub, if you will), we must track each and every file for all content units. We don't currently do this. There are multi-file units (like Distributions, and maybe OSTree?). Information we probably want to track for each file:

• checksum
• checksum_type (although we might want to just stick with sha256 and not tie this to any potential metadata we know about the file)
• size of the file
• the origin of the file
• the storage location of the file
• access control settings?

It might not be a bad idea to also track the integrity of the this metadata by hashing it and storing the hash with it. This should happen for every file we manage, regardless of content type. Therefore, this should live in the platform and leads us to...

Plugin API¶

We need to define a plugin API with this feature in mind. It could potentially happen as part of the file retrieval. The user would provide a URL and if its available to them, the data integrity information (think RPM's primary.xml metadata file which provides locations and checksums). The platform would handle retrieving the files, validating the download went smoothly (with the provided checksums) or generating the initial checksum, creating the database record for the file, etc. This is just a thought though, and worth fleshing out.

[0] http://indico.cern.ch/event/13797/contributions/1362288/attachments/115080/163419/Data_integrity_v3.pdf
[1] https://github.com/gluster/glusterfs-specs/blob/master/done/GlusterFS%203.7/BitRot.md
[2] https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-scrub
[3] https://pthree.org/2012/12/11/zfs-administration-part-vi-scrub-and-resilver/

This is a good plan that will go well with other plans for data model improvements in 3.y.

ostree is a strange case. It may not make sense to track all of its files. Otherwise, this should work well for other content types.

Thinking of this as a whishlist, in at least some cases we want to store a gpg signature with a file.

What do you have in mind for access control settings? I don't think we have anything like that on units today.

When you say "It might not be a bad idea to also track the integrity of the this metadata by hashing it and storing the hash with it.", what are you trying to guard against? Database corruption?

ostree is a strange case. It may not make sense to track all of its files. Otherwise, this should work well for other content types.

There are, of course, types of content that provide their own integrity checks. That's fine, and whether or not we introduce additional safeguards or not really depends on the situation, but we should ensure there is a way to "get at" that feature of OSTree (or Git, or whatever) to scrub the content unit and repair it.

Thinking of this as a whishlist, in at least some cases we want to store a gpg signature with a file.

It feels like it might be a layer up, abstraction-wise, but I'm not familiar enough with how each content type that supports GPG-signing does it. It's worth investigating further, in any case.

What do you have in mind for access control settings? I don't think we have anything like that on units today.

Just trying to be forward-thinking. Suppose permissions in /var/lib/pulp/content get trashed - it'd be nice to recover from that.

When you say "It might not be a bad idea to also track the integrity of the this metadata by hashing it and storing the hash with it.", what are you trying to guard against? Database corruption?

Sure. I don't know much about the integrity checks databases provide (I'd be surprised if they didn't have some), but I also know people write bugs. It'd be good to have an additional check and a good way to recover when something inevitably goes wrong. I don't feel as strongly about this particular feature because it pales in comparison to the other problems we have, but while we're thinking about it I think it's worth looking into.

Under model changes:

the origin of the file

What is an origin and why would we track it?

jortel@redhat.com wrote:

Under model changes:

the origin of the file

What is an origin and why would we track it?

Where the file came from (a list of urls, I guess) so we can make an attempt to automatically retrieve and repair the file.

Pulp 2 is approaching maintenance mode, and this Pulp 2 ticket is not being actively worked on. As such, it is being closed as WONTFIX. Pulp 2 is still accepting contributions though, so if you want to contribute a fix for this ticket, please reopen or comment on it. If you don't have permissions to reopen this ticket, or you want to discuss an issue, please reach out via the developer mailing list.