Project

Profile

Help

Issue #5976

closed

Migrate only content from the specified repositories

Added by ipanova@redhat.com about 4 years ago. Updated almost 3 years ago.

Status:
CLOSED - WONTFIX
Priority:
Normal
Assignee:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
(Total: 0:00 h)
Severity:
2. Medium
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:

Description

Problem:
All immediate content get migrated, regardless the specified repositories

$ pulp-admin repo list
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+

Id:                  file
Display Name:        None
Description:         None
Content Unit Counts: 
  Iso: 3

Id:                  file2
Display Name:        None
Description:         None
Content Unit Counts: 
  Iso: 3

Id:                  file-many
Display Name:        None
Description:         None
Content Unit Counts: 
  Iso: 250

Id:                  file-large
Display Name:        None
Description:         None
Content Unit Counts: 
  Iso: 10

$ http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{ "type": "iso", "repositories" :  [{"name": "file","pulp2_importer_repository_id": "file", "repository_versions": [{"pulp2_repository_id": "file"}]}]}]}'
HTTP/1.1 201 Created
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 289
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:49:40 GMT
Location: /pulp/api/v3/migration-plans/ae1e5041-3cb5-4ca7-aa52-33aea081f0a8/
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "plan": {
        "plugins": [
            {
                "repositories": [
                    {
                        "name": "file",
                        "pulp2_importer_repository_id": "file",
                        "repository_versions": [
                            {
                                "pulp2_repository_id": "file"
                            }
                        ]
                    }
                ],
                "type": "iso"
            }
        ]
    },
    "pulp_created": "2020-01-13T18:49:40.893261Z",
    "pulp_href": "/pulp/api/v3/migration-plans/ae1e5041-3cb5-4ca7-aa52-33aea081f0a8/"
}

]$ http GET :24817/pulp/api/v3/pulp2repositories/
HTTP/1.1 200 OK
Allow: GET, HEAD, OPTIONS
Connection: close
Content-Length: 552
Content-Type: application/json
Date: Mon, 13 Jan 2020 18:52:55 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "is_migrated": true,
            "not_in_pulp2": false,
            "pulp2_object_id": "5e1c932dc998ac367a9b79d0",
            "pulp2_repo_id": "file",
            "pulp3_distribution_hrefs": [],
            "pulp3_publication_href": [],
            "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/3730ab1a-e0e3-458e-b479-35456b607ad7/",
            "pulp3_repository_version": "/pulp/api/v3/repositories/file/file/4788f61a-815a-42fb-9331-ec6787d08281/versions/1/",
            "pulp_created": "2020-01-13T18:50:03.589032Z",
            "pulp_href": "/pulp/api/v3/pulp2repositories/070e0fe0-fc41-477a-900e-be641e348634/"
        }
    ]
}

$ http GET :24817/pulp/api/v3/repositories/file/file/4788f61a-815a-42fb-9331-ec6787d08281/versions/1/
HTTP/1.1 200 OK
Allow: GET, DELETE, HEAD, OPTIONS
Connection: close
Content-Length: 577
Content-Type: application/json
Date: Mon, 13 Jan 2020 19:00:14 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "base_version": null,
    "content_summary": {
        "added": {
            "file.file": {
                "count": 3,
                "href": "/pulp/api/v3/content/file/files/?repository_version_added=/pulp/api/v3/repositories/file/file/4788f61a-815a-42fb-9331-ec6787d08281/versions/1/"
            }
        },
        "present": {
            "file.file": {
                "count": 3,
                "href": "/pulp/api/v3/content/file/files/?repository_version=/pulp/api/v3/repositories/file/file/4788f61a-815a-42fb-9331-ec6787d08281/versions/1/"
            }
        },
        "removed": {}
    },
    "number": 1,
    "pulp_created": "2020-01-13T18:50:08.125597Z",
    "pulp_href": "/pulp/api/v3/repositories/file/file/4788f61a-815a-42fb-9331-ec6787d08281/versions/1/"
}

(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 pulp-2to3-migration]$ django-admin shell_plus
# Shell Plus Model Imports
from django.contrib.admin.models import LogEntry
from django.contrib.auth.models import Group, Permission, User
from django.contrib.contenttypes.models import ContentType
from django.contrib.sessions.models import Session
from pulp_2to3_migration.app.models.base import MigrationPlan
from pulp_2to3_migration.app.models.content import Pulp2Content, Pulp2LazyCatalog
from pulp_2to3_migration.app.models.repository import Pulp2Distributor, Pulp2Importer, Pulp2RepoContent, Pulp2Repository
from pulp_2to3_migration.app.plugin.docker.pulp_2to3_models import Pulp2Blob, Pulp2Manifest, Pulp2ManifestList, Pulp2Tag
from pulp_2to3_migration.app.plugin.iso.pulp_2to3_models import Pulp2ISO
from pulp_container.app.models import Blob, BlobManifest, ContainerDistribution, ContainerRemote, ContainerRepository, Manifest, ManifestListManifest, Tag
from pulp_file.app.models import FileContent, FileDistribution, FileFileSystemExporter, FilePublication, FileRemote, FileRepository
from pulp_rpm.app.models.advisory import UpdateCollection, UpdateCollectionPackage, UpdateRecord, UpdateReference
from pulp_rpm.app.models.comps import PackageCategory, PackageEnvironment, PackageGroup, PackageLangpacks
from pulp_rpm.app.models.custom_metadata import RepoMetadataFile
from pulp_rpm.app.models.distribution import Addon, Checksum, DistributionTree, Image, Variant
from pulp_rpm.app.models.modulemd import Modulemd, ModulemdDefaults
from pulp_rpm.app.models.package import Package
from pulp_rpm.app.models.repository import RpmDistribution, RpmPublication, RpmRemote, RpmRepository
from pulpcore.app.models.content import Artifact, Content, ContentArtifact, RemoteArtifact
from pulpcore.app.models.exporter import FileSystemExporter
from pulpcore.app.models.progress import ProgressReport
from pulpcore.app.models.publication import BaseDistribution, ContentGuard, Publication, PublishedArtifact, PublishedMetadata
from pulpcore.app.models.repository import Remote, Repository, RepositoryContent, RepositoryVersion, RepositoryVersionContentDetails
from pulpcore.app.models.status import ContentAppStatus
from pulpcore.app.models.task import CreatedResource, ReservedResource, ReservedResourceRecord, Task, TaskReservedResource, TaskReservedResourceRecord, Worker
from pulpcore.app.models.upload import Upload, UploadChunk
# Shell Plus Django Imports
from django.core.cache import cache
from django.conf import settings
from django.contrib.auth import get_user_model
from django.db import transaction
from django.db.models import Avg, Case, Count, F, Max, Min, Prefetch, Q, Sum, When, Exists, OuterRef, Subquery
from django.utils import timezone
from django.urls import reverse
Python 3.6.8 (default, Aug  7 2019, 17:28:10) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.11.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: FileContent.objects.count()                                                                                                                                                                                
Out[1]: 13

In [2]: FileRepository.objects.count()                                                                                                                                                                             
Out[2]: 1

As you can see there have been migrated 13 isos, 10 from 'file-large' and 3 from 'file'. Rest got skipped because file2 and file-many have on_demand policy


Sub-issues 1 (0 open1 closed)

Issue #5977: Migrated Content counters are wrongCLOSED - WORKSFORMEActions
Actions #1

Updated by ipanova@redhat.com about 4 years ago

  • Description updated (diff)
Actions #2

Updated by ttereshc about 4 years ago

  • Project changed from Pulp to Migration Plugin
Actions #3

Updated by dalley about 4 years ago

edit: this comment was incorrect, disregard.

Actions #4

Updated by ttereshc about 4 years ago

I think it was intentional - to migrate all content for a plugin regardless. It allows us to perform bulk operations without granular search.

The assumption is that user wants to migrate all or almost all content. If the intention is to migrate very few repositories, maybe it's not worth a hassle to use a migration plugin and write a MP, it's just easier to create those repos in Pulp 3 from scratch.

Actions #5

Updated by ttereshc about 4 years ago

  • Triaged changed from No to Yes
Actions #6

Updated by ttereshc almost 3 years ago

  • Status changed from NEW to CLOSED - WONTFIX

To be able to migrate content only from a specified repositories would mean to give up a lot of performance improvements which are based on timestamps. It's not worth it. If you are impacted and would like to migrate very small amount of content, consider a fresh pulp 3 install and then recreate your repositories from scratch.

Also available in: Atom PDF