Issue #646
closed
mongoengine connection uses wrong mongoDB database
Description
2.6.0 introduces mongoengine managing the connection object. On master (2.7+) models started being converted to mongoengine, and any model that is converted when running seems to use the wrong database name in MongoDB. I expect it to to use pulp_database based on my server.conf, but instead it uses localhost:27017.
Mongoengine models should use the right database, along with the rest of Pulp. Also, this needs to be fixed in 2.6.0 because that is when mongoengine was introduced. This will avoid leaving users of 2.6.0 with a weird MongoDB.
+ This bug was cloned from Bugzilla Bug #1175512 +
Files
Updated by bmbouter about 9 years ago
PR available at: https://github.com/pulp/pulp/pull/1443
+ This comment was cloned from Bugzilla #1175512 comment 1 +
Updated by bmbouter about 9 years ago
Merged to 2.6-dev -> master
+ This comment was cloned from Bugzilla #1175512 comment 2 +
Updated by cduryee about 9 years ago
fixed in pulp 2.6.0-0.2.beta
+ This comment was cloned from Bugzilla #1175512 comment 3 +
Updated by pthomas@redhat.com about 9 years ago
How do you verify this bz?
Will verifying
https://bugzilla.redhat.com/show_bug.cgi?id=1182335 verify this bz as well?
+ This comment was cloned from Bugzilla #1175512 comment 4 +
Updated by igulina@redhat.com about 9 years ago
rpm -qa pulp-server
pulp-server-2.6.0-0.5.beta.el6.noarch
grep bind_ip /etc/mongodb.conf
bind_ip = ip-XXX.internal
#bind_ip = localhost
in /etc/pulp/server.conf
[database]
#name: pulp_database
name: fake_db
seeds: ip-XXX.internal:27017
mongo --host `hostname`
MongoDB shell version: 2.4.12
connecting to: ip-XXX.internal:27017/test
use fake_db
switched to db fake_db
db.getCollectionNames()
[
"consumer_bindings",
"consumers",
"content_types",
"event_listeners",
"migration_trackers",
"permissions",
"repo_content_units",
"repo_distributors",
"repo_importers",
"repos",
"roles",
"system.indexes",
"units_distribution",
"units_drpm",
"units_erratum",
"units_iso",
"units_node",
"units_package_category",
"units_package_environment",
"units_package_group",
"units_puppet_module",
"units_repository",
"units_rpm",
"units_srpm",
"units_yum_repo_metadata_file",
"users"
]
exit
bye
pulp-admin -u admin -p admin rpm repo create --repo-id nemo
Successfully created repository [nemo]
pulp-admin -u admin -p admin rpm repo publish run --repo-id nemo
--------------------------------------------------------------------
Publishing Repository [nemo]
--------------------------------------------------------------------
This command may be exited via ctrl+c without affecting the request.
Initializing repo metadata
[-]
... completed
Publishing Distribution files
[-]
... completed
Publishing RPMs
[-]
... completed
Publishing Delta RPMs
... skipped
Publishing Errata
[-]
... completed
Publishing Comps file
[-]
... completed
Publishing Metadata.
[-]
... completed
Closing repo metadata
[-]
... completed
Generating sqlite files
... skipped
Publishing files to web
[-]
... completed
Writing Listings File
[-]
... completed
Task Succeeded
mongo --host `hostname`
MongoDB shell version: 2.4.12
connecting to: ip-XXX.internal:27017/test
use fake_db
switched to db fake_db
db.repos
db.repos
db.repos.find()
{ "_id" : ObjectId("54d367f95a0f2b48405b9245"), "scratchpad" : { }, "display_name" : "nemo", "description" : null, "last_unit_added" : null, "notes" : { "_repo-type" : "rpm-repo" }, "last_unit_removed" : null, "content_unit_counts" : { }, "_ns" : "repos", "id" : "nemo" }
exit
bye
+ This comment was cloned from Bugzilla #1175512 comment 5 +
Updated by igulina@redhat.com about 9 years ago
In addition I run some automation tests, for example:
nosetests -vs tests/general_tests/test_01_log_in.py
test_login (tests.general_tests.test_01_log_in.SimpleTestLogin) ... ok
test_wrong_login (tests.general_tests.test_01_log_in.SimpleTestLogin) ... ok
----------------------------------------------------------------------
Ran 2 tests in 1.214s
OK
+ This comment was cloned from Bugzilla #1175512 comment 6 +
Updated by bmbouter about 9 years ago
reproducer script for 1175512
+ This comment was cloned from Bugzilla #1175512 comment 7 +
Updated by bmbouter about 9 years ago
The tricky thing about this is that the bug is present, but it only will show itself in a future version of pulp (ie: what is currently on master). The commit that introduces it on master is 39d332158ad8aa9b2d09c7f9ad8d71d7e3a8fa93
Attached is a reproducer script (reproduce_1175512.py). Here's how to use it:
1) delete all databases from mongodb except the one named 'local'. The 'local' database is used by mongodb itself. Show your databases using 'show databases'.
2) run pulp-manage-db
3) show your databases again, you should see pulp_database has been added (assuming you have a default server.conf).
4) run the attached reproducer script `python reproduce_1175512.py`
5) show the databases again. If you see any change from step 3 then fail the bug.
6) list the collections on pulp_database. You should see a collection named 'test_reproducer' was added. If that is not present fail the bug.
7) Manually delete the 'test_reproducer' collection from pulp_database to clean up from the reproducer script. Alternatively you can drop your pulp_database and rerun pulp-manage-db.
8) If you got this far then VERIFY the fix.
+ This comment was cloned from Bugzilla #1175512 comment 8 +
Updated by igulina@redhat.com about 9 years ago
stopped all pulp services
mongo
MongoDB shell version: 2.4.12
connecting to: test
show databases
fake_db 0.203125GB
local 0.078125GB
use fake_db
switched to db fake_db
db.dropDatabase()
{ "dropped" : "fake_db", "ok" : 1 }
show databases
local 0.078125GB
service qpidd start
Starting Qpid AMQP daemon: [ OK ]
sudo -u apache pulp-manage-db
Mongo database for connection is version 2.4.12
Loading content types.
Loading type descriptors [nodes.json, rpm_support.json, puppet.json, iso_support.json]
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Updating the database with types [repository, node, distribution, drpm, erratum, package_group, package_category, package_environment, rpm, srpm, yum_repo_metadata_file, puppet_module, iso]
/usr/lib/python2.6/site-packages/pulp/server/db/connection.py:162: DeprecationWarning: The safe parameter is deprecated. Please use write concern options instead.
return method(*args, **kwargs)
Content types loaded.
Ensuring the admin role and user are in place.
Mongo database for connection is version 2.4.12
Admin role and user are in place.
Beginning database migrations.
Applying pulp.server.db.migrations version 1
Migration to pulp.server.db.migrations version 1 complete.
Applying pulp.server.db.migrations version 2
Migration to pulp.server.db.migrations version 2 complete.
Applying pulp.server.db.migrations version 3
Migration to pulp.server.db.migrations version 3 complete.
Applying pulp.server.db.migrations version 4
regenerating content unit counts for 0 repositories
Migration to pulp.server.db.migrations version 4 complete.
Applying pulp.server.db.migrations version 5
Migration to pulp.server.db.migrations version 5 complete.
Applying pulp.server.db.migrations version 6
Migration to pulp.server.db.migrations version 6 complete.
Applying pulp.server.db.migrations version 7
Migration to pulp.server.db.migrations version 7 complete.
Applying pulp.server.db.migrations version 8
Migration to pulp.server.db.migrations version 8 complete.
Applying pulp.server.db.migrations version 9
Migration to pulp.server.db.migrations version 9 complete.
Applying pulp.server.db.migrations version 10
Migration to pulp.server.db.migrations version 10 complete.
Applying pulp.server.db.migrations version 11
Migration to pulp.server.db.migrations version 11 complete.
Applying pulp.server.db.migrations version 12
Migration to pulp.server.db.migrations version 12 complete.
Applying pulp_puppet.plugins.migrations version 1
Migrated puppet modules to include checksum
Migration to pulp_puppet.plugins.migrations version 1 complete.
Applying pulp_puppet.plugins.migrations version 2
Migration to pulp_puppet.plugins.migrations version 2 complete.
Applying pulp_rpm.plugins.migrations version 1
Export distributor migration for rpm repositories started
Export distributor migration for rpm repositories finished
Migration to pulp_rpm.plugins.migrations version 1 complete.
Applying pulp_rpm.plugins.migrations version 2
Migrated rpms to include rpm metadata
Migration to pulp_rpm.plugins.migrations version 2 complete.
Applying pulp_rpm.plugins.migrations version 3
Migration to pulp_rpm.plugins.migrations version 3 complete.
Applying pulp_rpm.plugins.migrations version 4
Migration to pulp_rpm.plugins.migrations version 4 complete.
Applying pulp_rpm.plugins.migrations version 5
Migrated rpms to include rpm changelog and filelist metadata
Migration to pulp_rpm.plugins.migrations version 5 complete.
Applying pulp_rpm.plugins.migrations version 6
Migration to pulp_rpm.plugins.migrations version 6 complete.
Applying pulp_rpm.plugins.migrations version 7
No yum repositories found to inventory custom metadata on
Migration to pulp_rpm.plugins.migrations version 7 complete.
Applying pulp_rpm.plugins.migrations version 8
Migration to pulp_rpm.plugins.migrations version 8 complete.
Applying pulp_rpm.plugins.migrations version 9
Migration to pulp_rpm.plugins.migrations version 9 complete.
Applying pulp_rpm.plugins.migrations version 10
Migration to pulp_rpm.plugins.migrations version 10 complete.
Applying pulp_rpm.plugins.migrations version 11
Migration to pulp_rpm.plugins.migrations version 11 complete.
Applying pulp_rpm.plugins.migrations version 12
Migration to pulp_rpm.plugins.migrations version 12 complete.
Applying pulp_rpm.plugins.migrations version 13
Migration to pulp_rpm.plugins.migrations version 13 complete.
Applying pulp_rpm.plugins.migrations version 14
Migration to pulp_rpm.plugins.migrations version 14 complete.
Applying pulp_rpm.plugins.migrations version 15
examining distribution units
Migration to pulp_rpm.plugins.migrations version 15 complete.
Applying pulp_rpm.plugins.migrations version 16
Loaded plugin yum_distributor for types: rpm,srpm,drpm,erratum,package_group,package_category,distribution,yum_repo_metadata_file
Loaded plugin export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group
Loaded plugin iso_distributor for types: iso
Loaded plugin puppet_distributor for types: puppet_module
Loaded plugin puppet_file_distributor for types: puppet_module
Loaded plugin puppet_install_distributor for types: puppet_module
Loaded plugin group_export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group
Loaded plugin nodes_http_importer for types: node,repository
Loaded plugin yum_importer for types: distribution,drpm,erratum,package_group,package_category,rpm,srpm,yum_repo_metadata_file,package_environment
Loaded plugin iso_importer for types: iso
Loaded plugin puppet_importer for types: puppet_module
Loaded plugin yum_profiler for types: rpm,erratum
Loaded plugin puppet_whole_repo_profiler for types: puppet_module
Loaded plugin yum for types: rpm
Loaded plugin rhui for types: rpm
Migration to pulp_rpm.plugins.migrations version 16 complete.
Applying pulp_rpm.plugins.migrations version 17
Migration to pulp_rpm.plugins.migrations version 17 complete.
Applying pulp_rpm.plugins.migrations version 18
Migration to pulp_rpm.plugins.migrations version 18 complete.
Database migrations complete.
mongo
MongoDB shell version: 2.4.12
connecting to: test
show databases
local 0.078125GB
pulp_database 0.203125GB
use pulp_database
switched to db pulp_database
show collections
consumer_bindings
consumers
content_types
event_listeners
migration_trackers
permissions
repo_content_units
repo_distributors
repo_importers
repos
roles
system.indexes
units_distribution
units_drpm
units_erratum
units_iso
units_node
units_package_category
units_package_environment
units_package_group
units_puppet_module
units_repository
units_rpm
units_srpm
units_yum_repo_metadata_file
users
python ./BZ1175512.py
/usr/lib64/python2.6/site-packages/pymongo/collection.py:266: DeprecationWarning: The safe parameter is deprecated. Please use write concern options instead.
return self.insert(to_save, manipulate, safe, check_keys, **kwargs)
mongo
MongoDB shell version: 2.4.12
connecting to: test
show databases
local 0.078125GB
pulp_database 0.203125GB
use pulp_database
switched to db pulp_database
show collections
consumer_bindings
consumers
content_types
event_listeners
migration_trackers
permissions
repo_content_units
repo_distributors
repo_importers
repos
roles
system.indexes
test_reproducer <----------it's here
units_distribution
units_drpm
units_erratum
units_iso
units_node
units_package_category
units_package_environment
units_package_group
units_puppet_module
units_repository
units_rpm
units_srpm
units_yum_repo_metadata_file
users
Cleaning UP
db.dropDatabase()
{ "dropped" : "pulp_database", "ok" : 1 }
exit
bye
sudo -u apache pulp-manage-db
Mongo database for connection is version 2.4.12
Loading content types.
Loading type descriptors [nodes.json, rpm_support.json, puppet.json, iso_support.json]
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Updating the database with types [repository, node, distribution, drpm, erratum, package_group, package_category, package_environment, rpm, srpm, yum_repo_metadata_file, puppet_module, iso]
/usr/lib/python2.6/site-packages/pulp/server/db/connection.py:162: DeprecationWarning: The safe parameter is deprecated. Please use write concern options instead.
return method(*args, **kwargs)
Content types loaded.
Ensuring the admin role and user are in place.
Mongo database for connection is version 2.4.12
Admin role and user are in place.
Beginning database migrations.
Applying pulp.server.db.migrations version 1
Migration to pulp.server.db.migrations version 1 complete.
Applying pulp.server.db.migrations version 2
Migration to pulp.server.db.migrations version 2 complete.
Applying pulp.server.db.migrations version 3
Migration to pulp.server.db.migrations version 3 complete.
Applying pulp.server.db.migrations version 4
regenerating content unit counts for 0 repositories
Migration to pulp.server.db.migrations version 4 complete.
Applying pulp.server.db.migrations version 5
Migration to pulp.server.db.migrations version 5 complete.
Applying pulp.server.db.migrations version 6
Migration to pulp.server.db.migrations version 6 complete.
Applying pulp.server.db.migrations version 7
Migration to pulp.server.db.migrations version 7 complete.
Applying pulp.server.db.migrations version 8
Migration to pulp.server.db.migrations version 8 complete.
Applying pulp.server.db.migrations version 9
Migration to pulp.server.db.migrations version 9 complete.
Applying pulp.server.db.migrations version 10
Migration to pulp.server.db.migrations version 10 complete.
Applying pulp.server.db.migrations version 11
Migration to pulp.server.db.migrations version 11 complete.
Applying pulp.server.db.migrations version 12
Migration to pulp.server.db.migrations version 12 complete.
Applying pulp_puppet.plugins.migrations version 1
Migrated puppet modules to include checksum
Migration to pulp_puppet.plugins.migrations version 1 complete.
Applying pulp_puppet.plugins.migrations version 2
Migration to pulp_puppet.plugins.migrations version 2 complete.
Applying pulp_rpm.plugins.migrations version 1
Export distributor migration for rpm repositories started
Export distributor migration for rpm repositories finished
Migration to pulp_rpm.plugins.migrations version 1 complete.
Applying pulp_rpm.plugins.migrations version 2
Migrated rpms to include rpm metadata
Migration to pulp_rpm.plugins.migrations version 2 complete.
Applying pulp_rpm.plugins.migrations version 3
Migration to pulp_rpm.plugins.migrations version 3 complete.
Applying pulp_rpm.plugins.migrations version 4
Migration to pulp_rpm.plugins.migrations version 4 complete.
Applying pulp_rpm.plugins.migrations version 5
Migrated rpms to include rpm changelog and filelist metadata
Migration to pulp_rpm.plugins.migrations version 5 complete.
Applying pulp_rpm.plugins.migrations version 6
Migration to pulp_rpm.plugins.migrations version 6 complete.
Applying pulp_rpm.plugins.migrations version 7
No yum repositories found to inventory custom metadata on
Migration to pulp_rpm.plugins.migrations version 7 complete.
Applying pulp_rpm.plugins.migrations version 8
Migration to pulp_rpm.plugins.migrations version 8 complete.
Applying pulp_rpm.plugins.migrations version 9
Migration to pulp_rpm.plugins.migrations version 9 complete.
Applying pulp_rpm.plugins.migrations version 10
Migration to pulp_rpm.plugins.migrations version 10 complete.
Applying pulp_rpm.plugins.migrations version 11
Migration to pulp_rpm.plugins.migrations version 11 complete.
Applying pulp_rpm.plugins.migrations version 12
Migration to pulp_rpm.plugins.migrations version 12 complete.
Applying pulp_rpm.plugins.migrations version 13
Migration to pulp_rpm.plugins.migrations version 13 complete.
Applying pulp_rpm.plugins.migrations version 14
Migration to pulp_rpm.plugins.migrations version 14 complete.
Applying pulp_rpm.plugins.migrations version 15
examining distribution units
Migration to pulp_rpm.plugins.migrations version 15 complete.
Applying pulp_rpm.plugins.migrations version 16
Loaded plugin yum_distributor for types: rpm,srpm,drpm,erratum,package_group,package_category,distribution,yum_repo_metadata_file
Loaded plugin export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group
Loaded plugin iso_distributor for types: iso
Loaded plugin puppet_distributor for types: puppet_module
Loaded plugin puppet_file_distributor for types: puppet_module
Loaded plugin puppet_install_distributor for types: puppet_module
Loaded plugin group_export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,package_group
Loaded plugin nodes_http_importer for types: node,repository
Loaded plugin yum_importer for types: distribution,drpm,erratum,package_group,package_category,rpm,srpm,yum_repo_metadata_file,package_environment
Loaded plugin iso_importer for types: iso
Loaded plugin puppet_importer for types: puppet_module
Loaded plugin yum_profiler for types: rpm,erratum
Loaded plugin puppet_whole_repo_profiler for types: puppet_module
Loaded plugin yum for types: rpm
Loaded plugin rhui for types: rpm
Migration to pulp_rpm.plugins.migrations version 16 complete.
Applying pulp_rpm.plugins.migrations version 17
Migration to pulp_rpm.plugins.migrations version 17 complete.
Applying pulp_rpm.plugins.migrations version 18
Migration to pulp_rpm.plugins.migrations version 18 complete.
Database migrations complete.
mongo
MongoDB shell version: 2.4.12
connecting to: test
show databases
local 0.078125GB
pulp_database 0.203125GB
use pulp_database
switched to db pulp_database
show collections
consumer_bindings
consumers
content_types
event_listeners
migration_trackers
permissions
repo_content_units
repo_distributors
repo_importers
repos
roles
system.indexes
units_distribution
units_drpm
units_erratum
units_iso
units_node
units_package_category
units_package_environment
units_package_group
units_puppet_module
units_repository
units_rpm
units_srpm
units_yum_repo_metadata_file
users
no test reproducer
+ This comment was cloned from Bugzilla #1175512 comment 9 +
Updated by rbarlow almost 9 years ago
- Status changed from 6 to CLOSED - CURRENTRELEASE
Updated by bmbouter almost 4 years ago
- Category deleted (
14)
We are removing the 'API' category per open floor discussion June 16, 2020.
.