Issue #3499
Updated by dalley over 6 years ago
Trying to sync only 4 projects (Django, Scipy, Numpy, Pulpcore) will cause fatal sqlite OperationalErrors due to large query size. This has nothing to do with the "database is locked" OperationalErrors we saw recently due to concurrency. I don't think this is specifically a python plugin problem. The changeset code is creating queries too large for sqlite to handle which is in core's domain. Even This has nothing to do with the "database is locked" OperationalErrors we were recently experiencing caused by "sqlite is for development use only, don't use in production" standards, IMO this is very limiting. As a user I wouldn't expect Pulp concurrency, which should be alleviated now due to fall over syncing only a couple of projects, even if it expands to a couple hundred packages (I haven't counted but that sounds reasonable). the timeout change. Importer settings: <pre> http POST http://localhost:8000/api/v3/importers/python/ name='bar' download_policy='immediate' sync_mode='additive' repository=$REPO_HREF feed_url='https://pypi.python.org/pypi' projects='["D jango", "pulpcore", "scipy", "numpy"]' </pre> Error output: <pre> Mar 18 01:10:40 pulp3.dev celery[25219]: [2018-03-18 01:10:40,079: ERROR/ForkPoolWorker-1] Task failed : [61cf9bd4-456b-41e7-a020-5d6984dfe5f4] Mar 18 01:10:40 pulp3.dev celery[25219]: [2018-03-18 01:10:40,090: ERROR/ForkPoolWorker-1] Task pulp_python.app.tasks.sync.sync[61cf9bd4-456b-41e7-a020-5d6984dfe5f4] raised unexpected: OperationalError('Expressi on tree is too large (maximum depth 1000)',) Mar 18 01:10:40 pulp3.dev celery[25219]: Traceback (most recent call last): Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute Mar 18 01:10:40 pulp3.dev celery[25219]: return self.cursor.execute(sql, params) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute Mar 18 01:10:40 pulp3.dev celery[25219]: return Database.Cursor.execute(self, query, params) Mar 18 01:10:40 pulp3.dev celery[25219]: sqlite3.OperationalError: Expression tree is too large (maximum depth 1000) Mar 18 01:10:40 pulp3.dev celery[25219]: The above exception was the direct cause of the following exception: Mar 18 01:10:40 pulp3.dev celery[25219]: Traceback (most recent call last): Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/celery/app/trace.py", line 374, in trace_task Mar 18 01:10:40 pulp3.dev celery[25219]: R = retval = fun(*args, **kwargs) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/devel/pulp/pulpcore/pulpcore/tasking/tasks.py", line 273, in __call__ Mar 18 01:10:40 pulp3.dev celery[25219]: return super(UserFacingTask, self).__call__(*args, **kwargs) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/celery/app/trace.py", line 629, in __protected_call__ [0/353] Mar 18 01:10:40 pulp3.dev celery[25219]: return self.run(*args, **kwargs) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/devel/pulp_python/pulp_python/app/tasks/sync.py", line 62, in sync Mar 18 01:10:40 pulp3.dev celery[25219]: changeset.apply_and_drain() Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/changeset/main.py", line 218, in apply_and_drain Mar 18 01:10:40 pulp3.dev celery[25219]: for x in self.apply(): Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/changeset/main.py", line 196, in apply Mar 18 01:10:40 pulp3.dev celery[25219]: for report in itertools.chain(self._apply_additions(), self._apply_removals()): Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/changeset/main.py", line 138, in _apply_additions Mar 18 01:10:40 pulp3.dev celery[25219]: for artifact, download in downloads: Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/changeset/iterator.py", line 283, in _iter Mar 18 01:10:40 pulp3.dev celery[25219]: for batch in BatchIterator(downloads, self.concurrent): Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/changeset/iterator.py", line 49, in __iter__ Mar 18 01:10:40 pulp3.dev celery[25219]: batch = tuple(itertools.islice(generator, 0, self.batch)) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/changeset/iterator.py", line 47, in <genexpr> Mar 18 01:10:40 pulp3.dev celery[25219]: generator = (c for c in self.iterable) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/changeset/iterator.py", line 282, in <genexpr> Mar 18 01:10:40 pulp3.dev celery[25219]: downloads = ((a, a.downloader) for a in artifacts) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/changeset/iterator.py", line 232, in _iter Mar 18 01:10:40 pulp3.dev celery[25219]: for batch in self._batch_artifacts(): Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/changeset/iterator.py", line 49, in __iter__ Mar 18 01:10:40 pulp3.dev celery[25219]: batch = tuple(itertools.islice(generator, 0, self.batch)) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/changeset/iterator.py", line 47, in <genexpr> Mar 18 01:10:40 pulp3.dev celery[25219]: generator = (c for c in self.iterable) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/changeset/iterator.py", line 186, in build Mar 18 01:10:40 pulp3.dev celery[25219]: for content in self.content: Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/changeset/iterator.py", line 129, in _iter Mar 18 01:10:40 pulp3.dev celery[25219]: for batch, fetched in self._fetch(): Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/devel/pulp/plugin/pulpcore/plugin/changeset/iterator.py", line 117, in _fetch Mar 18 01:10:40 pulp3.dev celery[25219]: fetched = {c.natural_key(): c for c in q_set} Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/models/query.py", line 250, in __iter__ Mar 18 01:10:40 pulp3.dev celery[25219]: self._fetch_all() Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/models/query.py", line 1118, in _fetch_all Mar 18 01:10:40 pulp3.dev celery[25219]: self._result_cache = list(self._iterable_class(self)) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/models/query.py", line 53, in __iter__ Mar 18 01:10:40 pulp3.dev celery[25219]: results = compiler.execute_sql(chunked_fetch=self.chunked_fetch) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 899, in execute_sql Mar 18 01:10:40 pulp3.dev celery[25219]: raise original_exception Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/models/sql/compiler.py", line 889, in execute_sql Mar 18 01:10:40 pulp3.dev celery[25219]: cursor.execute(sql, params) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 79, in execute Mar 18 01:10:40 pulp3.dev celery[25219]: return super(CursorDebugWrapper, self).execute(sql, params) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute Mar 18 01:10:40 pulp3.dev celery[25219]: return self.cursor.execute(sql, params) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/utils.py", line 94, in __exit__ Mar 18 01:10:40 pulp3.dev celery[25219]: six.reraise(dj_exc_type, dj_exc_value, traceback) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/utils/six.py", line 685, in reraise Mar 18 01:10:40 pulp3.dev celery[25219]: raise value.with_traceback(tb) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute Mar 18 01:10:40 pulp3.dev celery[25219]: return self.cursor.execute(sql, params) Mar 18 01:10:40 pulp3.dev celery[25219]: File "/home/vagrant/.virtualenvs/pulp/lib64/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute Mar 18 01:10:40 pulp3.dev celery[25219]: return Database.Cursor.execute(self, query, params) Mar 18 01:10:40 pulp3.dev celery[25219]: django.db.utils.OperationalError: Expression tree is too large (maximum depth 1000) </pre>
.