Issue #1808
closed
exporting a sufficiently large repo with 'on_demand' policy results in BSON error
Description
If you attempt to export a repo with many RPMs, you'll get a BSON document too large error.
To repro:
- create a large repo with "on_demand" policy, such as Fedora 23 everything
- sync repo
- create repo group with said repo
- attempt to publish repo group
actual result:
BSON document too large error
expected result:
error that either a.) not all units are on-disk and repo/repo group cannot be exported, or b.) 'on_demand' repos cannot be exported
Updated by mhrivnak about 8 years ago
A patch such as this will prevent the copytree() function from accumulating an unbounded number of errors before giving up. That unbounded collection of error messages is what caused the mongodb document size limit to be exceeded.
diff --git a/server/pulp/server/util.py b/server/pulp/server/util.py
index 5411e87..98fede3 100644
--- a/server/pulp/server/util.py
+++ b/server/pulp/server/util.py
@@ -207,6 +207,8 @@ def copytree(src, dst, symlinks=False, ignore=None):
os.makedirs(dst)
errors = []
for name in names:
+ if len(errors) >= 100:
+ raise Error(errors)
if name in ignored_names:
continue
srcname = os.path.join(src, name)
Updated by mhrivnak about 8 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to mhrivnak
- Sprint/Milestone set to 19
Added by mhrivnak about 8 years ago
Added by mhrivnak about 8 years ago
Revision dd5f8c7e | View on GitHub
copytree no longer accumulates an unbounded number of errors
https://pulp.plan.io/issues/1808
fixes #1808
Updated by mhrivnak about 8 years ago
- Status changed from ASSIGNED to POST
Updated by mhrivnak about 8 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulp:pulp|dd5f8c7ea40cbdc0f57c1aa1c88bbe3f4384dfff.
Updated by pthomas@redhat.com almost 8 years ago
Verified
pre>
[root@mgmt5 ~]# pulp-admin rpm repo export run --repo-id rhel67+----------------------------------------------------------------------+
Publishing Repository [rhel67]
--------------------------------------------------------------------
This command may be exited via ctrl+c without affecting the request.
Task Failed
The repository cannot be exported because some units are not downloaded.
[root@mgmt5 ~]# pulp-admin rpm repo group export run --group-id grp1
--------------------------------------------------------------------
Exporting Repository Group [grp1]
--------------------------------------------------------------------
This command may be exited via ctrl+c without affecting the request.
Task Failed
The repository group cannot be exported because these repos have units that are
not downloaded: zoo, rhel67
[root@mgmt5 ~]#
Updated by semyers almost 8 years ago
- Status changed from 5 to CLOSED - CURRENTRELEASE
.
copytree no longer accumulates an unbounded number of errors
https://pulp.plan.io/issues/1808
fixes #1808