Issue #1863
closedException using regex in filters against associate endpoint
Description
Hi,
When using filters, I am seeing exceptions against associate, but not search/units
[root@waterloo ~]# rpm -q pulp-server
pulp-server-2.8.3-0.1.beta.el7.noarch
[root@waterloo ~]# pulp-admin rpm repo content rpm --repo-id centos7-live-debug --fields filename
Filename: ipa-python-4.2.0-15.el7.centos.3.x86_64.rpm
Filename: kernel-debug-3.10.0-327.13.1.el7.x86_64.rpm
Filename: ipa-mincho-fonts-003.03-5.el7.noarch.rpm
Filename: ipa-server-trust-ad-4.2.0-15.0.1.el7.centos.6.x86_64.rpm
Filename: java-1.8.0-openjdk-debug-1.8.0.91-0.b14.el7_2.x86_64.rpm
Filename: ipa-server-trust-ad-4.2.0-15.0.1.el7.centos.6.1.x86_64.rpm
Filename: libblkid-2.23.2-26.el7_2.2.i686.rpm
Filename: ipa-client-4.2.0-15.0.1.el7.centos.6.1.x86_64.rpm
Filename: ipa-pgothic-fonts-003.03-5.el7.noarch.rpm
Using the following filter to avoid any "ipa" packages that are not version 4.1.0:
filters = {
"unit": {
'$or': [
{ "name": { "$not": '^(ipa.*)(?<!fonts)$' } },
{ "version": '4.1.0' }
]
}
}
It succeeds when running against repositories/x/search/units:
[root@waterloo ~]# ./test.py
kernel-debug-3.10.0-327.13.1.el7.x86_64
ipa-mincho-fonts-003.03-5.el7.noarch
java-1.8.0-openjdk-debug-1.8.0.91-0.b14.el7_2.x86_64
libblkid-2.23.2-26.el7_2.2.i686
ipa-pgothic-fonts-003.03-5.el7.noarch
But when I try and use the exact same filter with associate:
{
"exception": [
"EncodeError: <_sre.SRE_Pattern object at 0x7ff5000f0bd0> is not JSON serializable\n"
],
"traceback": [
" File \"/usr/lib/python2.7/site-packages/django/core/handlers/base.py\", line 112, in get_response\n response = wrapped_callback(request, *callback_args, **callback_kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 69, in view\n return self.dispatch(request, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 87, in dispatch\n return handler(request, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 241, in auth_decorator\n return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 195, in _verify_auth\n value = method(self, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py\", line 130, in wrapper\n return func(*args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py\", line 1170, in post\n {'criteria': criteria.to_dict(), 'import_config_override': overrides}, tags=task_tags)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 400, in apply_async_with_reservation\n queue=RESOURCE_MANAGER_QUEUE)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 95, in apply_async\n return super(PulpTask, self).apply_async(*args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/celery/app/task.py\", line 555, in apply_async\n **dict(self._get_exec_options(), **options)\n",
" File \"/usr/lib/python2.7/site-packages/celery/app/base.py\", line 351, in send_task\n reply_to=reply_to or self.oid, **options\n",
" File \"/usr/lib/python2.7/site-packages/celery/app/amqp.py\", line 305, in publish_task\n **kwargs\n",
" File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 165, in publish\n compression, headers)\n",
" File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 241, in _prepare\n body) = dumps(body, serializer=serializer)\n",
" File \"/usr/lib/python2.7/site-packages/kombu/serialization.py\", line 164, in dumps\n payload = encoder(data)\n",
" File \"/usr/lib64/python2.7/contextlib.py\", line 35, inexit\n self.gen.throw(type, value, traceback)\n",
" File \"/usr/lib/python2.7/site-packages/kombu/serialization.py\", line 59, in _reraise_errors\n reraise(wrapper, wrapper(exc), sys.exc_info()[2])\n",
" File \"/usr/lib/python2.7/site-packages/kombu/serialization.py\", line 55, in _reraise_errors\n yield\n",
" File \"/usr/lib/python2.7/site-packages/kombu/serialization.py\", line 164, in dumps\n payload = encoder(data)\n",
" File \"/usr/lib/python2.7/site-packages/anyjson/_init__.py\", line 141, in dumps\n return implementation.dumps(value)\n",
" File \"/usr/lib/python2.7/site-packages/anyjson/__init__.py\", line 87, in dumps\n return self._encode(data)\n",
" File \"/usr/lib64/python2.7/site-packages/simplejson/__init__.py\", line 354, in dumps\n return _default_encoder.encode(obj)\n",
" File \"/usr/lib64/python2.7/site-packages/simplejson/encoder.py\", line 262, in encode\n chunks = self.iterencode(o, _one_shot=True)\n",
" File \"/usr/lib64/python2.7/site-packages/simplejson/encoder.py\", line 340, in iterencode\n return _iterencode(o, 0)\n",
" File \"/usr/lib64/python2.7/site-packages/simplejson/encoder.py\", line 239, in default\n raise TypeError(repr(o) + \" is not JSON serializable\")\n"
],
"_href": "/pulp/api/v2/repositories/centos7-development-debug/actions/associate/",
"error_message": "<_sre.SRE_Pattern object at 0x7ff5000f0bd0> is not JSON serializable",
"http_request_method": "POST",
"http_status": 500
}
Please see the attached test.py use case which is equivalent to what I'm trying within our scripts
Files
Related issues
Updated by dkliban@redhat.com over 7 years ago
- Priority changed from Normal to High
- Severity changed from 2. Medium to 3. High
- Platform Release set to 2.8.4
- Triaged changed from No to Yes
Updated by semyers over 7 years ago
- Platform Release changed from 2.8.4 to 2.8.5
Updated by semyers over 7 years ago
- Platform Release changed from 2.8.5 to 2.8.6
Updated by ttereshc over 7 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to ttereshc
- Platform Release deleted (
2.8.6)
Updated by ttereshc over 7 years ago
- Status changed from ASSIGNED to NEW
- Assignee deleted (
ttereshc)
Unassign myself in case someone would like to work on it while I'm absent.
Updated by ttereshc over 7 years ago
- Status changed from NEW to ASSIGNED
- Assignee set to ttereshc
Added by ttereshc over 7 years ago
Added by ttereshc over 7 years ago
Revision ede42954 | View on GitHub
Fix regex handling for the $not
operator in filters
Updated by ttereshc over 7 years ago
- Status changed from ASSIGNED to POST
Updated by ttereshc over 7 years ago
- Status changed from POST to MODIFIED
- % Done changed from 0 to 100
Applied in changeset pulp|ede4295484da7fa2136450f15775c786baa77995.
Updated by pthomas@redhat.com over 7 years ago
- Status changed from 5 to ASSIGNED
Seems like this failed QA
[root@tigger ~]# rpm -qa pulp-server
pulp-server-2.9.2-0.2.beta.el7.noarch
[root@tigger ~]#
[root@tigger ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
RPM Repositories
+----------------------------------------------------------------------+
Id: upload
Display Name: None
Description: None
Content Unit Counts:
Id: myzoo
Display Name: None
Description: None
Content Unit Counts:
Id: zoo
Display Name: None
Description: None
Content Unit Counts:
Erratum: 4
Package Category: 1
Package Group: 2
Package Langpacks: 1
Rpm: 30
Id: centos
Display Name: None
Description: None
Content Unit Counts:
Drpm: 2151
Rpm: 700
Id: zoo1
Display Name: None
Description: None
Content Unit Counts:
[root@tigger ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
RPM Repositories
+----------------------------------------------------------------------+
Id: upload
Display Name: None
Description: None
Content Unit Counts:
Id: myzoo
Display Name: None
Description: None
Content Unit Counts:
Erratum: 4
Package Category: 1
Package Group: 2
Package Langpacks: 1
Rpm: 30
Id: zoo
Display Name: None
Description: None
Content Unit Counts:
Erratum: 4
Package Category: 1
Package Group: 2
Package Langpacks: 1
Rpm: 30
Id: centos
Display Name: None
Description: None
Content Unit Counts:
Drpm: 2151
Rpm: 700
Id: zoo1
Display Name: None
Description: None
Content Unit Counts:
[root@tigger ~]# http --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/zoo1/actions/associate/ source_repo_id=zoo criteria='{'type_ids' : ['rpm'],'filters' : {'unit' : {'$and': [{'name': {'$regex': 'c*.*'}}, {'version': {'$gt': '1.0'}}] }}} }'
/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
HTTP/1.1 500 INTERNAL SERVER ERROR
Connection: close
Content-Length: 1591
Content-Type: application/json; charset=utf-8
Date: Thu, 11 Aug 2016 17:36:31 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5
{
"_href": "/pulp/api/v2/repositories/zoo1/actions/associate/",
"error_message": "'str' object has no attribute 'pop'",
"exception": [
"AttributeError: 'str' object has no attribute 'pop'\n"
],
"http_request_method": "POST",
"http_status": 500,
"traceback": [
" File \"/usr/lib/python2.7/site-packages/django/core/handlers/base.py\", line 112, in get_response\n response = wrapped_callback(request, *callback_args, **callback_kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 69, in view\n return self.dispatch(request, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 87, in dispatch\n return handler(request, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 241, in _auth_decorator\n return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 195, in _verify_auth\n value = method(self, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py\", line 130, in wrapper\n return func(*args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py\", line 1159, in post\n criteria = UnitAssociationCriteria.from_client_input(criteria_body)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/db/model/criteria.py\", line 260, in from_client_input\n type_ids = query.pop('type_ids', None)\n"
]
}
[root@tigger ~]#
Updated by pthomas@redhat.com over 7 years ago
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: Unhandled Exception
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) 'str' object has no attribute 'pop'
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) Traceback (most recent call last):
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) response = wrapped_callback(request, *callback_args, **callback_kwargs)
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) return self.dispatch(request, *args, **kwargs)
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) return handler(request, *args, **kwargs)
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 241, in _auth_decorator
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 195, in _verify_auth
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) value = method(self, *args, **kwargs)
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py", line 130, in wrapper
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) return func(*args, **kwargs)
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py", line 1159, in post
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) criteria = UnitAssociationCriteria.from_client_input(criteria_body)
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) File "/usr/lib/python2.7/site-packages/pulp/server/db/model/criteria.py", line 260, in from_client_input
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) type_ids = query.pop('type_ids', None)
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) AttributeError: 'str' object has no attribute 'pop'
Updated by semyers over 7 years ago
There is a bug here, but it's that we should be returning 400, not 500. It's complaining because criteria is a string, not a dict, which happens because that httpie call sends up a string, not a dict:
$ http --verbose --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/zoo1/actions/associate/ source_repo_id=zoo criteria="{'type_ids': ['rpm'], 'filters': {'unit': {'$and': [{'name': {'$regex': 'c*.*'}}, {'version': {'$gt': '1.0'}}]}}}"
POST /pulp/api/v2/repositories/zoo1/actions/associate/ HTTP/1.1
(headers)
{
"criteria": "{'type_ids': ['rpm'], 'filters': {'unit': {'': [{'name': {'': 'c*.*'}}, {'version': {'': '1.0'}}]}}}",
"source_repo_id": "zoo"
}
Using the "criteria:=" syntax tells httpie to POST a json dict, rather than a string:
$ http --verbose --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/zoo1/actions/associate/ source_repo_id=zoo criteria:='{"type_ids": ["rpm"], "filters": {"unit": {"$and": [
{"name": {"$regex": "c*.*"}}, {"version": {"$gt": "1.0"}}]}}}'
POST /pulp/api/v2/repositories/zoo1/actions/associate/ HTTP/1.1
(headers)
{
"criteria": {
"filters": {
"unit": {
"$and": [
{
"name": {
"$regex": "c*.*"
}
},
{
"version": {
"$gt": "1.0"
}
}
]
}
},
"type_ids": [
"rpm"
]
},
"source_repo_id": "zoo"
}
This gets us the expects "HTTP/1.1 202 ACCEPTED" response.
Updated by pthomas@redhat.com over 7 years ago
- Status changed from ASSIGNED to 5
Moving it back to ON_QA as per the comment.
Updated by pthomas@redhat.com over 7 years ago
Verified
[root@tigger ~]# http --verbose --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/zoo1/actions/associate/ source_repo_id=zoo criteria:='{"type_ids": ["rpm"], "filters": {"unit": {"$and": [
> {"name": {"$regex": "c*.*"}}, {"version": {"$gt": "1.0"}}]}}}'
/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
POST /pulp/api/v2/repositories/zoo1/actions/associate/ HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Authorization: Basic YWRtaW46YWRtaW4=
Connection: keep-alive
Content-Length: 152
Content-Type: application/json
Host: localhost
User-Agent: HTTPie/0.9.4
{
"criteria": {
"filters": {
"unit": {
"$and": [
{
"name": {
"$regex": "c*.*"
}
},
{
"version": {
"$gt": "1.0"
}
}
]
}
},
"type_ids": [
"rpm"
]
},
"source_repo_id": "zoo"
}
HTTP/1.1 202 ACCEPTED
Connection: Keep-Alive
Content-Length: 172
Content-Type: application/json; charset=utf-8
Date: Thu, 11 Aug 2016 18:18:56 GMT
Keep-Alive: timeout=5, max=10000
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5
{
"error": null,
"result": null,
"spawned_tasks": [
{
"_href": "/pulp/api/v2/tasks/ab3506f5-54e4-4f31-b7db-ce9652b672ae/",
"task_id": "ab3506f5-54e4-4f31-b7db-ce9652b672ae"
}
]
}
[root@tigger ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
RPM Repositories
+----------------------------------------------------------------------+
Id: upload
Display Name: None
Description: None
Content Unit Counts:
Id: myzoo
Display Name: None
Description: None
Content Unit Counts:
Erratum: 4
Package Category: 1
Package Group: 2
Package Langpacks: 1
Rpm: 30
Id: zoo
Display Name: None
Description: None
Content Unit Counts:
Erratum: 4
Package Category: 1
Package Group: 2
Package Langpacks: 1
Rpm: 30
Id: centos
Display Name: None
Description: None
Content Unit Counts:
Drpm: 2151
Rpm: 700
Id: zoo1
Display Name: None
Description: None
Content Unit Counts:
Rpm: 11
Updated by pthomas@redhat.com over 7 years ago
- Copied to Issue #2170: Getting a 500 instead of 400 when using a string in criteria instead of dict added
Updated by semyers over 7 years ago
- Status changed from 6 to CLOSED - CURRENTRELEASE
Updated by bmbouter over 3 years ago
- Category deleted (
14)
We are removing the 'API' category per open floor discussion June 16, 2020.
Fix regex handling for the
$not
operator in filterscloses #1863 https://pulp.plan.io/issues/1863