Actions
Issue #4031
closed
MemoryError during sync of a repo with many errata
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Sprint:
Quarter:
Description
To reproduce, sync EPEL7 repo https://dl.fedoraproject.org/pub/epel/7/x86_64/
Sep 20 07:48:51 pulp3.dev rq[6934]: MemoryError
Sep 20 07:48:51 pulp3.dev rq[6934]: Traceback (most recent call last):
Sep 20 07:48:51 pulp3.dev rq[6934]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/rq/worker.py", line 793, in perform_job
Sep 20 07:48:51 pulp3.dev rq[6934]: rv = job.perform()
Sep 20 07:48:51 pulp3.dev rq[6934]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/rq/job.py", line 599, in perform
Sep 20 07:48:51 pulp3.dev rq[6934]: self._result = self._execute()
Sep 20 07:48:51 pulp3.dev rq[6934]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/rq/job.py", line 605, in _execute
Sep 20 07:48:51 pulp3.dev rq[6934]: return self.func(*self.args, **self.kwargs)
Sep 20 07:48:51 pulp3.dev rq[6934]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py", line 69, in synchronize
Sep 20 07:48:51 pulp3.dev rq[6934]: loop.run_until_complete(pipeline)
Sep 20 07:48:51 pulp3.dev rq[6934]: File "/usr/lib64/python3.6/asyncio/base_events.py", line 468, in run_until_complete
Sep 20 07:48:51 pulp3.dev rq[6934]: return future.result()
Sep 20 07:48:51 pulp3.dev rq[6934]: File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/stages/api.py", line 58, in create_pipeline
Sep 20 07:48:51 pulp3.dev rq[6934]: await asyncio.gather(*futures)
Sep 20 07:48:51 pulp3.dev rq[6934]: File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/stages/content_unit_stages.py", line 185, in __call__
Sep 20 07:48:51 pulp3.dev rq[6934]: await self._post_save(batch)
Sep 20 07:48:51 pulp3.dev rq[6934]: File "/home/vagrant/devel/pulp_rpm/pulp_rpm/app/tasks/synchronizing.py", line 324, in _post_save
Sep 20 07:48:51 pulp3.dev rq[6934]: UpdateCollectionPackage.objects.bulk_create(update_collection_packages_to_save)
Sep 20 07:48:51 pulp3.dev rq[6934]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
Sep 20 07:48:51 pulp3.dev rq[6934]: return getattr(self.get_queryset(), name)(*args, **kwargs)
Sep 20 07:48:51 pulp3.dev rq[6934]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/models/query.py", line 457, in bulk_create
Sep 20 07:48:51 pulp3.dev rq[6934]: objs_with_pk, objs_without_pk = partition(lambda o: o.pk is None, objs)
Sep 20 07:48:51 pulp3.dev rq[6934]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/utils/functional.py", line 387, in partition
Sep 20 07:48:51 pulp3.dev rq[6934]: results[predicate(item)].append(item)
Sep 20 07:48:51 pulp3.dev rq[6934]: MemoryError
Updated by ttereshc about 6 years ago
Maybe related to #4014 (relations between collections UpdateRecords are incorrect)
Updated by ttereshc about 6 years ago
- Status changed from NEW to CLOSED - NOTABUG
The problem is no longer observed after fixing #4014.
Feel free to reopen, if you hit it.
I believe one can still reproduce the issue if they put 8K+ collections in one erratum but it's unlikely to happen in real-life repositories.
Actions
.