Issue #1808
closed
exporting a sufficiently large repo with 'on_demand' policy results in BSON error
Status:
CLOSED - CURRENTRELEASE
Description
If you attempt to export a repo with many RPMs, you'll get a BSON document too large error.
To repro:
- create a large repo with "on_demand" policy, such as Fedora 23 everything
- create repo group with said repo
- attempt to publish repo group
actual result:
BSON document too large error
expected result:
error that either a.) not all units are on-disk and repo/repo group cannot be exported, or b.) 'on_demand' repos cannot be exported
A patch such as this will prevent the copytree() function from accumulating an unbounded number of errors before giving up. That unbounded collection of error messages is what caused the mongodb document size limit to be exceeded.
diff --git a/server/pulp/server/util.py b/server/pulp/server/util.py
index 5411e87..98fede3 100644
--- a/server/pulp/server/util.py
+++ b/server/pulp/server/util.py
@@ -207,6 +207,8 @@ def copytree(src, dst, symlinks=False, ignore=None):
os.makedirs(dst)
errors = []
for name in names:
+ if len(errors) >= 100:
+ raise Error(errors)
if name in ignored_names:
continue
srcname = os.path.join(src, name)
- Status changed from NEW to ASSIGNED
- Assignee set to mhrivnak
- Sprint/Milestone set to 19
- Status changed from ASSIGNED to POST
- Platform Release set to 2.8.3
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
- Triaged changed from No to Yes
Verified
pre>
[root@mgmt5 ~]# pulp-admin rpm repo export run --repo-id rhel67+----------------------------------------------------------------------+
Publishing Repository [rhel67]
--------------------------------------------------------------------
This command may be exited via ctrl+c without affecting the request.
Task Failed
The repository cannot be exported because some units are not downloaded.
[root@mgmt5 ~]# pulp-admin rpm repo group export run --group-id grp1
--------------------------------------------------------------------
Exporting Repository Group [grp1]
--------------------------------------------------------------------
This command may be exited via ctrl+c without affecting the request.
Task Failed
The repository group cannot be exported because these repos have units that are
not downloaded: zoo, rhel67
[root@mgmt5 ~]#
- Status changed from MODIFIED to 5
- Status changed from 5 to CLOSED - CURRENTRELEASE
- Sprint/Milestone deleted (
19)
Also available in: Atom
PDF
copytree no longer accumulates an unbounded number of errors
https://pulp.plan.io/issues/1808
fixes #1808