Migration Plugin

This should cover every single content we rely on for migration to be present on a disk.

• RPM/SRPM (if downloaded=True)
• .treeinfo file for DistributionTrees
• module and module-default snippets (I think we use files with snippets to migrate it)
• yum_repo_metadata_file

Feel free to update this list if anyone recalls more.

ttereshc wrote:

This should cover every single content we rely on for migration to be present on a disk.

• RPM/SRPM (if downloaded=True)
• .treeinfo file for DistributionTrees
• module and module-default snippets (I think we use files with snippets to migrate it)

Feel free to update this list if anyone recalls more.

I think it might get a bit more complicated than this. We need to think how to handle content that has relations. In Pulp3 modulemd has relation to modular rpm, if we skip a corrupted rpm it would lead to a migrated modulemd that does not have some rpms. Same for the other content that has relations - for example docker image. It is composed out of blobs and manifest, and if one of the blobs is corrupted we risk migrating a partial image and while partial modulemd can still be usable to certain extent, that's not the case for the docker image. Should we then skip entirely migration of that image? or we migrate the way it is, issue a warning and ask user to resync repositories so missing data will be fetched.

That's a valid concern.

I see 2 options:

• we skip only content which is safe to skip, e.g. non-modular rpms, distribution trees, etc
• in theory, we can skip all corrupted content and after the migration identify and remove incomplete content, e.g. modules with only some rpms migrated, docker images with some blobs missing, etc.

I'm definitely voting for the former. The latter seems to me complicated and error-prone.

@jsherrill , do you have any thoughts about ipanova's concern and the options I listed?

Any other ideas?

ttereshc wrote:

That's a valid concern.

I see 2 options:

• we skip only content which is safe to skip, e.g. non-modular rpms, distribution trees, etc
• in theory, we can skip all corrupted content and after the migration identify and remove incomplete content, e.g. modules with only some rpms migrated, docker images with some blobs missing, etc.

I'm definitely voting for the former. The latter seems to me complicated and error-prone.

I would 100% not go with the second option, it will be hell to identify missing blobs, up to parsing each manifest json file and compare that data to the number of relations. Option 1 sounds viable.

@jsherrill , do you have any thoughts about ipanova's concern and the options I listed?

Any other ideas?

just adding here not to forget what was discussed

[17:23:16] <ttereshc> jsherrill, is it ok if we skip only some content and some is not possible to skip https://pulp.plan.io/issues/7538#note-8? E.g. we can skip regular corrupted/unavailable rpms but we don't want to skip modular ones because we'll have issues with module integrity in pulp3
[17:30:40] <jsherrill> ttereshc: assuming this is a repository with a url its syncing from (and for modular repos i think thats probably a given), and that next sync should fetch the missing items
[17:30:42] <jsherrill> i think that is perfectly fine
[17:30:59] <jsherrill> ttereshc: but i guess you're saying to error
[17:31:05] <jsherrill> instead of skipping
[[17:32:30] <ttereshc> jsherrill, yes, I want to error because client won't work properly right after the migration
[17:32:53] <ttereshc> next sync might fix it if url is a working one
[17:33:21] <ttereshc> it's a good test case to have
[17:34:30] <ttereshc> I'm also worrying that it might be hard to troubleshoot customer cases related to that
[17:35:26] <jsherrill> yeah, and what are they supposed to do if that content isn't available upstream anymore?
[17:35:55] <ttereshc> they should cleanup their pulp2
[17:36:21] <ttereshc> if upstream url is unavailable, remove that module, it's not working properly anyway
[17:37:35] <jsherrill> i'm not sure we give the user the ability to remove a module
[17:37:39] <jsherrill> let me see...
[17:38:35] <ttereshc> jsherrill, what about pulp3? if we migrate a broken module, will they be able to get rid of it in pulp 3?
[17:40:31] <jsherrill> i'll have to check, but i don't think we let you delete a module
[17:40:38] <jsherrill> we don't let you delete anything manually for RH repos
[17:40:51] <jsherrill> but you can use mirror on sync/remove missing to 'sync it up' with the upstream repo
[17:41:17] <ttereshc> jsherrill, ipanova had the same concerns for the docker content, for the corrupted blobs
[17:42:10] <ttereshc> we should either fail or not migrate the rest of the image
[17:47:10] <jsherrill> i think not migrating the rest of the image is fine
[17:47:24] <jsherrill> i'm okay with broken stuff not ended up in the resulting migrated repo
[17:48:06] <ttereshc> jsherrill, so basically you are against any failure which would require user actions on pulp 2 side, correct?
[17:49:13] <jsherrill> i guess, maybe we should give them the option
[17:49:28] <jsherrill> initial run would check all checksums and fail
[17:49:44] <jsherrill> and then they can fix it or they can just 'skip' failed/missing content
[17:50:01] <jsherrill> as a user, really all i'm going to do is try to go and remove all that content anyways
[17:50:05] <jsherrill> i would imagine
[17:50:09] <jsherrill> so if the migration can just ignore it
[17:50:15] <jsherrill> that saves a lot of effort ?
[17:51:40] <ttereshc> it's not easy to ignore corrupted content because of relations between different types and due to perf reasons all the operations being in bulk fashion
[17:51:53] <ttereshc> I'll see what I can do
[17:52:03] <ttereshc> thanks for the feedback, it's helpful
[17:59:51] <jsherrill> ttereshc: yeah, if its not possible or really really hard, we can maybe brainstorm alternative options 
[17:59:57] <jsherrill> (such as katello zapping all the content in pulp2)
[18:00:11] <jsherrill> if we know which content is corrupt
[18:16:16] <ttereshc> ty

Modular RPMs and Container Blobs if skipped, can be fixed with a subsequent sync, the relations to other content is created correctly as well. So we introduce no special handling here, just skipping the migration of corrupted content entirely.