Project

Profile

Help

Issue #7490

closed

Migrating a large database with (300K rpms) uses too much memory

Added by jsherril@redhat.com about 4 years ago. Updated about 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 83
Quarter:

Description

on a system with ~5GB free, i tried migrating a user database with 300K rpms on it. When the migration got to the 'Pre-migrating Pulp 2 RPM content (detail info)' step it got through about 200K and then the system ran out of memory:

  • message: Pre-migrating Pulp 2 RPM content (detail info) code: premigrating.content.detail state: running total: 275349 done: 206000
    traceback: |2
        File "/usr/lib/python3.6/site-packages/rq/worker.py", line 883, in perform_job
          rv = job.perform()
        File "/usr/lib/python3.6/site-packages/rq/job.py", line 657, in perform
          self._result = self._execute()
        File "/usr/lib/python3.6/site-packages/rq/job.py", line 663, in _execute
          return self.func(*self.args, **self.kwargs)
        File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 136, in migrate_from_pulp2
          pre_migrate_all_content(plan)
        File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 66, in pre_migrate_all_content
          pre_migrate_content(content_model, mutable_type, lazy_type, premigrate_hook)
        File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/pre_migration.py", line 203, in pre_migrate_content
          content_model.pulp_2to3_detail.pre_migrate_content_detail(pulp2content_batch)
        File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/pulp_2to3_models.py", line 143, in pre_migrate_content_detail
          cls.objects.bulk_create(pulp2rpm_to_save, ignore_conflicts=True)
        File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
          return getattr(self.get_queryset(), name)(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 468, in bulk_create
          self._batched_insert(objs_with_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)
        File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1211, in _batched_insert
          self._insert(item, fields=fields, using=self.db, ignore_conflicts=ignore_conflicts)
        File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
          return query.get_compiler(using=using).execute_sql(return_id)
        File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
          cursor.execute(sql, params)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
          return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
          return executor(sql, params, many, context)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
          return self.cursor.execute(sql, params)
        File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
          raise dj_exc_value.with_traceback(traceback) from exc_value
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
          return self.cursor.execute(sql, params)
    description: |
      out of memory
      DETAIL:  Failed on request of size 1073741823 in memory context "ErrorContext".

I would expect that even the biggest migration should be doable on ~5 GB, so it seems like something is holding onto too much data somewhere.

Actions #1

Updated by ttereshc about 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc
  • Triaged changed from No to Yes
  • Sprint set to Sprint 81
Actions #2

Updated by rchan about 4 years ago

  • Sprint changed from Sprint 81 to Sprint 82
Actions #3

Updated by rchan about 4 years ago

  • Sprint changed from Sprint 82 to Sprint 83

Added by ttereshc about 4 years ago

Revision 846bbc5b | View on GitHub

Pre-migrate compressed repodata for RPMs to save some memory.

Added batch_size to all the bulk_create calls. Decrease the size of the content batch as well.

closes #7490 https://pulp.plan.io/issues/7490

Added by ttereshc about 4 years ago

Revision 846bbc5b | View on GitHub

Pre-migrate compressed repodata for RPMs to save some memory.

Added batch_size to all the bulk_create calls. Decrease the size of the content batch as well.

closes #7490 https://pulp.plan.io/issues/7490

Added by ttereshc about 4 years ago

Revision 846bbc5b | View on GitHub

Pre-migrate compressed repodata for RPMs to save some memory.

Added batch_size to all the bulk_create calls. Decrease the size of the content batch as well.

closes #7490 https://pulp.plan.io/issues/7490

Actions #4

Updated by ttereshc about 4 years ago

  • Status changed from ASSIGNED to MODIFIED
Actions #5

Updated by ttereshc about 4 years ago

  • Sprint/Milestone set to 0.4.0
Actions #6

Updated by pulpbot about 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Also available in: Atom PDF