Project

Profile

Help

Issue #1863

Exception using regex in filters against associate endpoint

Added by martin@hatchlane.com almost 6 years ago. Updated over 1 year ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
High
Assignee:
Category:
-
Sprint/Milestone:
-
Start date:
Due date:
Estimated time:
Severity:
3. High
Version:
2.8.2
Platform Release:
2.9.2
OS:
RHEL 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 5
Quarter:

Description

Hi,

When using filters, I am seeing exceptions against associate, but not search/units

[root@waterloo ~]# rpm -q pulp-server
pulp-server-2.8.3-0.1.beta.el7.noarch

[root@waterloo ~]# pulp-admin rpm repo content rpm --repo-id centos7-live-debug --fields filename
Filename: ipa-python-4.2.0-15.el7.centos.3.x86_64.rpm
Filename: kernel-debug-3.10.0-327.13.1.el7.x86_64.rpm
Filename: ipa-mincho-fonts-003.03-5.el7.noarch.rpm
Filename: ipa-server-trust-ad-4.2.0-15.0.1.el7.centos.6.x86_64.rpm
Filename: java-1.8.0-openjdk-debug-1.8.0.91-0.b14.el7_2.x86_64.rpm
Filename: ipa-server-trust-ad-4.2.0-15.0.1.el7.centos.6.1.x86_64.rpm
Filename: libblkid-2.23.2-26.el7_2.2.i686.rpm
Filename: ipa-client-4.2.0-15.0.1.el7.centos.6.1.x86_64.rpm
Filename: ipa-pgothic-fonts-003.03-5.el7.noarch.rpm

Using the following filter to avoid any "ipa" packages that are not version 4.1.0:

filters = {
"unit": {
'$or': [
{ "name": { "$not": '^(ipa.*)(?<!fonts)$' } },
{ "version": '4.1.0' }
]
}
}

It succeeds when running against repositories/x/search/units:

[root@waterloo ~]# ./test.py
kernel-debug-3.10.0-327.13.1.el7.x86_64
ipa-mincho-fonts-003.03-5.el7.noarch
java-1.8.0-openjdk-debug-1.8.0.91-0.b14.el7_2.x86_64
libblkid-2.23.2-26.el7_2.2.i686
ipa-pgothic-fonts-003.03-5.el7.noarch

But when I try and use the exact same filter with associate:

{
"exception": [
"EncodeError: <_sre.SRE_Pattern object at 0x7ff5000f0bd0> is not JSON serializable\n"
],
"traceback": [
" File \"/usr/lib/python2.7/site-packages/django/core/handlers/base.py\", line 112, in get_response\n response = wrapped_callback(request, *callback_args, **callback_kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 69, in view\n return self.dispatch(request, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 87, in dispatch\n return handler(request, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 241, in auth_decorator\n return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 195, in _verify_auth\n value = method(self, *args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py\", line 130, in wrapper\n return func(*args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py\", line 1170, in post\n {'criteria': criteria.to_dict(), 'import_config_override': overrides}, tags=task_tags)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 400, in apply_async_with_reservation\n queue=RESOURCE_MANAGER_QUEUE)\n",
" File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 95, in apply_async\n return super(PulpTask, self).apply_async(*args, **kwargs)\n",
" File \"/usr/lib/python2.7/site-packages/celery/app/task.py\", line 555, in apply_async\n **dict(self._get_exec_options(), **options)\n",
" File \"/usr/lib/python2.7/site-packages/celery/app/base.py\", line 351, in send_task\n reply_to=reply_to or self.oid, **options\n",
" File \"/usr/lib/python2.7/site-packages/celery/app/amqp.py\", line 305, in publish_task\n **kwargs\n",
" File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 165, in publish\n compression, headers)\n",
" File \"/usr/lib/python2.7/site-packages/kombu/messaging.py\", line 241, in _prepare\n body) = dumps(body, serializer=serializer)\n",
" File \"/usr/lib/python2.7/site-packages/kombu/serialization.py\", line 164, in dumps\n payload = encoder(data)\n",
" File \"/usr/lib64/python2.7/contextlib.py\", line 35, inexit\n self.gen.throw(type, value, traceback)\n",
" File \"/usr/lib/python2.7/site-packages/kombu/serialization.py\", line 59, in _reraise_errors\n reraise(wrapper, wrapper(exc), sys.exc_info()[2])\n",
" File \"/usr/lib/python2.7/site-packages/kombu/serialization.py\", line 55, in _reraise_errors\n yield\n",
" File \"/usr/lib/python2.7/site-packages/kombu/serialization.py\", line 164, in dumps\n payload = encoder(data)\n",
" File \"/usr/lib/python2.7/site-packages/anyjson/
_init__.py\", line 141, in dumps\n return implementation.dumps(value)\n",
" File \"/usr/lib/python2.7/site-packages/anyjson/__init__.py\", line 87, in dumps\n return self._encode(data)\n",
" File \"/usr/lib64/python2.7/site-packages/simplejson/__init__.py\", line 354, in dumps\n return _default_encoder.encode(obj)\n",
" File \"/usr/lib64/python2.7/site-packages/simplejson/encoder.py\", line 262, in encode\n chunks = self.iterencode(o, _one_shot=True)\n",
" File \"/usr/lib64/python2.7/site-packages/simplejson/encoder.py\", line 340, in iterencode\n return _iterencode(o, 0)\n",
" File \"/usr/lib64/python2.7/site-packages/simplejson/encoder.py\", line 239, in default\n raise TypeError(repr(o) + \" is not JSON serializable\")\n"
],
"_href": "/pulp/api/v2/repositories/centos7-development-debug/actions/associate/",
"error_message": "<_sre.SRE_Pattern object at 0x7ff5000f0bd0> is not JSON serializable",
"http_request_method": "POST",
"http_status": 500
}

Please see the attached test.py use case which is equivalent to what I'm trying within our scripts

test.py (1.49 KB) test.py martin@hatchlane.com, 04/25/2016 04:37 PM

Related issues

Copied to Pulp - Issue #2170: Getting a 500 instead of 400 when using a string in criteria instead of dictCLOSED - WONTFIX<a title="Actions" class="icon-only icon-actions js-contextmenu" href="#">Actions</a>

Associated revisions

Revision ede42954 View on GitHub
Added by ttereshc over 5 years ago

Fix regex handling for the $not operator in filters

closes #1863 https://pulp.plan.io/issues/1863

Revision ede42954 View on GitHub
Added by ttereshc over 5 years ago

Fix regex handling for the $not operator in filters

closes #1863 https://pulp.plan.io/issues/1863

History

#1 Updated by dkliban@redhat.com almost 6 years ago

  • Priority changed from Normal to High
  • Severity changed from 2. Medium to 3. High
  • Platform Release set to 2.8.4
  • Triaged changed from No to Yes

#2 Updated by semyers over 5 years ago

  • Platform Release changed from 2.8.4 to 2.8.5

#3 Updated by Anonymous over 5 years ago

  • Sprint/Milestone set to 22

#4 Updated by semyers over 5 years ago

  • Platform Release changed from 2.8.5 to 2.8.6

#5 Updated by ttereshc over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc
  • Platform Release deleted (2.8.6)

#6 Updated by ttereshc over 5 years ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (ttereshc)

Unassign myself in case someone would like to work on it while I'm absent.

#7 Updated by mhrivnak over 5 years ago

  • Sprint/Milestone changed from 22 to 23

#8 Updated by ttereshc over 5 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to ttereshc

#9 Updated by ttereshc over 5 years ago

  • Status changed from ASSIGNED to POST

#10 Updated by ttereshc over 5 years ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#11 Updated by ttereshc over 5 years ago

  • Platform Release set to 2.9.2

#12 Updated by semyers over 5 years ago

  • Status changed from MODIFIED to 5

#13 Updated by pthomas@redhat.com over 5 years ago

  • Status changed from 5 to ASSIGNED

Seems like this failed QA

[root@tigger ~]# rpm -qa pulp-server
pulp-server-2.9.2-0.2.beta.el7.noarch
[root@tigger ~]#

[root@tigger ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  upload
Display Name:        None
Description:         None
Content Unit Counts: 

Id:                  myzoo
Display Name:        None
Description:         None
Content Unit Counts: 

Id:                  zoo
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:           4
  Package Category:  1
  Package Group:     2
  Package Langpacks: 1
  Rpm:               30

Id:                  centos
Display Name:        None
Description:         None
Content Unit Counts: 
  Drpm: 2151
  Rpm:  700

Id:                  zoo1
Display Name:        None
Description:         None
Content Unit Counts: 

[root@tigger ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  upload
Display Name:        None
Description:         None
Content Unit Counts: 

Id:                  myzoo
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:           4
  Package Category:  1
  Package Group:     2
  Package Langpacks: 1
  Rpm:               30

Id:                  zoo
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:           4
  Package Category:  1
  Package Group:     2
  Package Langpacks: 1
  Rpm:               30

Id:                  centos
Display Name:        None
Description:         None
Content Unit Counts: 
  Drpm: 2151
  Rpm:  700

Id:                  zoo1
Display Name:        None
Description:         None
Content Unit Counts: 

[root@tigger ~]# http --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/zoo1/actions/associate/ source_repo_id=zoo criteria='{'type_ids' : ['rpm'],'filters' : {'unit' : {'$and': [{'name': {'$regex': 'c*.*'}}, {'version': {'$gt': '1.0'}}] }}} }'
/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
HTTP/1.1 500 INTERNAL SERVER ERROR
Connection: close
Content-Length: 1591
Content-Type: application/json; charset=utf-8
Date: Thu, 11 Aug 2016 17:36:31 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5

{
    "_href": "/pulp/api/v2/repositories/zoo1/actions/associate/", 
    "error_message": "'str' object has no attribute 'pop'", 
    "exception": [
        "AttributeError: 'str' object has no attribute 'pop'\n"
    ], 
    "http_request_method": "POST", 
    "http_status": 500, 
    "traceback": [
        "  File \"/usr/lib/python2.7/site-packages/django/core/handlers/base.py\", line 112, in get_response\n    response = wrapped_callback(request, *callback_args, **callback_kwargs)\n", 
        "  File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 69, in view\n    return self.dispatch(request, *args, **kwargs)\n", 
        "  File \"/usr/lib/python2.7/site-packages/django/views/generic/base.py\", line 87, in dispatch\n    return handler(request, *args, **kwargs)\n", 
        "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 241, in _auth_decorator\n    return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n", 
        "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py\", line 195, in _verify_auth\n    value = method(self, *args, **kwargs)\n", 
        "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py\", line 130, in wrapper\n    return func(*args, **kwargs)\n", 
        "  File \"/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py\", line 1159, in post\n    criteria = UnitAssociationCriteria.from_client_input(criteria_body)\n", 
        "  File \"/usr/lib/python2.7/site-packages/pulp/server/db/model/criteria.py\", line 260, in from_client_input\n    type_ids = query.pop('type_ids', None)\n"
    ]
}

[root@tigger ~]# 

#14 Updated by pthomas@redhat.com over 5 years ago

Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: Unhandled Exception
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) 'str' object has no attribute 'pop'
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) Traceback (most recent call last):
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560)   File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560)     response = wrapped_callback(request, *callback_args, **callback_kwargs)
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560)   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560)     return self.dispatch(request, *args, **kwargs)
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560)   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560)     return handler(request, *args, **kwargs)
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 241, in _auth_decorator
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560)     return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 195, in _verify_auth
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560)     value = method(self, *args, **kwargs)
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py", line 130, in wrapper
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560)     return func(*args, **kwargs)
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py", line 1159, in post
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560)     criteria = UnitAssociationCriteria.from_client_input(criteria_body)
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560)   File "/usr/lib/python2.7/site-packages/pulp/server/db/model/criteria.py", line 260, in from_client_input
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560)     type_ids = query.pop('type_ids', None)
Aug 11 13:40:51 tigger.idmqe.lab.eng.bos.redhat.com pulp[14527]: pulp.server.webservices.middleware.exception:ERROR: (14527-94560) AttributeError: 'str' object has no attribute 'pop'

#15 Updated by semyers over 5 years ago

There is a bug here, but it's that we should be returning 400, not 500. It's complaining because criteria is a string, not a dict, which happens because that httpie call sends up a string, not a dict:

$ http --verbose --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/zoo1/actions/associate/ source_repo_id=zoo criteria="{'type_ids': ['rpm'], 'filters': {'unit': {'$and': [{'name': {'$regex': 'c*.*'}}, {'version': {'$gt': '1.0'}}]}}}"
POST /pulp/api/v2/repositories/zoo1/actions/associate/ HTTP/1.1
(headers)

{
    "criteria": "{'type_ids': ['rpm'], 'filters': {'unit': {'': [{'name': {'': 'c*.*'}}, {'version': {'': '1.0'}}]}}}", 
    "source_repo_id": "zoo"
}

Using the "criteria:=" syntax tells httpie to POST a json dict, rather than a string:

$ http --verbose --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/zoo1/actions/associate/ source_repo_id=zoo criteria:='{"type_ids": ["rpm"], "filters": {"unit": {"$and": [                                                             
{"name": {"$regex": "c*.*"}}, {"version": {"$gt": "1.0"}}]}}}'
POST /pulp/api/v2/repositories/zoo1/actions/associate/ HTTP/1.1
(headers)

{
    "criteria": {
        "filters": {
            "unit": {
                "$and": [
                    {
                        "name": {
                            "$regex": "c*.*"
                        }
                    }, 
                    {
                        "version": {
                            "$gt": "1.0"
                        }
                    }
                ]
            }
        }, 
        "type_ids": [
            "rpm"
        ]
    }, 
    "source_repo_id": "zoo"
}

This gets us the expects "HTTP/1.1 202 ACCEPTED" response.

#16 Updated by pthomas@redhat.com over 5 years ago

  • Status changed from ASSIGNED to 5

Moving it back to ON_QA as per the comment.

#17 Updated by pthomas@redhat.com over 5 years ago

Verified

[root@tigger ~]# http --verbose --verify no --auth admin:admin POST https://localhost/pulp/api/v2/repositories/zoo1/actions/associate/ source_repo_id=zoo criteria:='{"type_ids": ["rpm"], "filters": {"unit": {"$and": [                                                             
> {"name": {"$regex": "c*.*"}}, {"version": {"$gt": "1.0"}}]}}}'
/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
POST /pulp/api/v2/repositories/zoo1/actions/associate/ HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Authorization: Basic YWRtaW46YWRtaW4=
Connection: keep-alive
Content-Length: 152
Content-Type: application/json
Host: localhost
User-Agent: HTTPie/0.9.4

{
    "criteria": {
        "filters": {
            "unit": {
                "$and": [
                    {
                        "name": {
                            "$regex": "c*.*"
                        }
                    }, 
                    {
                        "version": {
                            "$gt": "1.0"
                        }
                    }
                ]
            }
        }, 
        "type_ids": [
            "rpm"
        ]
    }, 
    "source_repo_id": "zoo"
}

HTTP/1.1 202 ACCEPTED
Connection: Keep-Alive
Content-Length: 172
Content-Type: application/json; charset=utf-8
Date: Thu, 11 Aug 2016 18:18:56 GMT
Keep-Alive: timeout=5, max=10000
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5

{
    "error": null, 
    "result": null, 
    "spawned_tasks": [
        {
            "_href": "/pulp/api/v2/tasks/ab3506f5-54e4-4f31-b7db-ce9652b672ae/", 
            "task_id": "ab3506f5-54e4-4f31-b7db-ce9652b672ae"
        }
    ]
}

[root@tigger ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  upload
Display Name:        None
Description:         None
Content Unit Counts: 

Id:                  myzoo
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:           4
  Package Category:  1
  Package Group:     2
  Package Langpacks: 1
  Rpm:               30

Id:                  zoo
Display Name:        None
Description:         None
Content Unit Counts: 
  Erratum:           4
  Package Category:  1
  Package Group:     2
  Package Langpacks: 1
  Rpm:               30

Id:                  centos
Display Name:        None
Description:         None
Content Unit Counts: 
  Drpm: 2151
  Rpm:  700

Id:                  zoo1
Display Name:        None
Description:         None
Content Unit Counts: 
  Rpm: 11

#18 Updated by pthomas@redhat.com over 5 years ago

  • Status changed from 5 to 6

#19 Updated by pthomas@redhat.com over 5 years ago

  • Copied to Issue #2170: Getting a 500 instead of 400 when using a string in criteria instead of dict added

#20 Updated by semyers over 5 years ago

  • Status changed from 6 to CLOSED - CURRENTRELEASE

#22 Updated by bmbouter almost 4 years ago

  • Sprint set to Sprint 5

#23 Updated by bmbouter almost 4 years ago

  • Sprint/Milestone deleted (23)

#24 Updated by bmbouter almost 3 years ago

  • Tags Pulp 2 added

#25 Updated by bmbouter over 1 year ago

  • Category deleted (14)

We are removing the 'API' category per open floor discussion June 16, 2020.

Also available in: Atom PDF