Actions
Issue #3535
closed
Error syncing Oracle EPEL repository
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Version:
Platform Release:
2.16.2
OS:
CentOS 7
Triaged:
Yes
Groomed:
No
Sprint Candidate:
No
Tags:
Pulp 2
Sprint:
Sprint 38
Quarter:
Description
When attempting to sync the official public Oracle EPEL repository I receive the following error in Foreman/Katello. As this is a pulp error I was directed over to your project. Foreman / Katello 1.16.0 / 3.5.1
PLP0000: Invalid block tag: '}gmail{*}com>'
Here is the error output I receive
{"contents_changed"=>true,
"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.sync.sync",
"_href"=>"/pulp/api/v2/tasks/57edd0fc-dcf8-453f-a5ac-2ede318d2311/",
"task_id"=>"57edd0fc-dcf8-453f-a5ac-2ede318d2311",
"tags"=>
["pulp:repository:4d73ea9e-7776-486e-811b-d914dc823105",
"pulp:action:sync"],
"finish_time"=>"2018-03-28T04:15:52Z",
"_ns"=>"task_status",
"start_time"=>"2018-03-28T04:02:38Z",
"traceback"=>nil,
"spawned_tasks"=>
[{"_href"=>"/pulp/api/v2/tasks/18a119c8-9f71-4d05-972b-d9932e635ec8/",
"task_id"=>"18a119c8-9f71-4d05-972b-d9932e635ec8"}],
"progress_report"=>
{"yum_importer"=>
{"content"=>
{"items_total"=>10,
"state"=>"FINISHED",
"error_details"=>[],
"details"=>
{"rpm_total"=>10, "rpm_done"=>10, "drpm_total"=>0, "drpm_done"=>0},
"size_total"=>15409527,
"size_left"=>0,
"items_left"=>0},
"comps"=>{"state"=>"FINISHED"},
"purge_duplicates"=>{"state"=>"FINISHED"},
"distribution"=>
{"items_total"=>0,
"state"=>"FINISHED",
"error_details"=>[],
"items_left"=>0},
"errata"=>{"state"=>"FINISHED"},
"metadata"=>{"state"=>"FINISHED"}}},
"queue"=>"reserved_resource_worker-2@ussl-foreman.usanainc.com.dq",
"state"=>"finished",
"worker_name"=>"reserved_resource_worker-2@ussl-foreman.usanainc.com",
"result"=>
{"result"=>"success",
"importer_id"=>"yum_importer",
"exception"=>nil,
"repo_id"=>"4d73ea9e-7776-486e-811b-d914dc823105",
"traceback"=>nil,
"started"=>"2018-03-28T04:02:40Z",
"_ns"=>"repo_sync_results",
"completed"=>"2018-03-28T04:15:50Z",
"importer_type_id"=>"yum_importer",
"error_message"=>nil,
"summary"=>
{"content"=>{"state"=>"FINISHED"},
"comps"=>{"state"=>"FINISHED"},
"purge_duplicates"=>{"state"=>"FINISHED"},
"distribution"=>{"state"=>"FINISHED"},
"errata"=>{"state"=>"FINISHED"},
"metadata"=>{"state"=>"FINISHED"}},
"added_count"=>10,
"removed_count"=>0,
"updated_count"=>2,
"id"=>"5abb16f7a55b78667ce16f7a",
"details"=>
{"content"=>
{"size_total"=>15409527,
"items_left"=>0,
"items_total"=>10,
"state"=>"FINISHED",
"size_left"=>0,
"details"=>
{"rpm_total"=>10, "rpm_done"=>10, "drpm_total"=>0, "drpm_done"=>0},
"error_details"=>[]},
"comps"=>{"state"=>"FINISHED"},
"purge_duplicates"=>{"state"=>"FINISHED"},
"distribution"=>
{"items_total"=>0,
"state"=>"FINISHED",
"error_details"=>[],
"items_left"=>0},
"errata"=>{"state"=>"FINISHED"},
"metadata"=>{"state"=>"FINISHED"}}},
"error"=>nil,
"_id"=>{"$oid"=>"5abb134bde37317496f9af38"},
"id"=>"5abb134bde37317496f9af38"},
{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
"_href"=>"/pulp/api/v2/tasks/18a119c8-9f71-4d05-972b-d9932e635ec8/",
"task_id"=>"18a119c8-9f71-4d05-972b-d9932e635ec8",
"tags"=>
["pulp:repository:4d73ea9e-7776-486e-811b-d914dc823105",
"pulp:action:publish"],
"finish_time"=>"2018-03-28T04:20:36Z",
"_ns"=>"task_status",
"start_time"=>"2018-03-28T04:20:02Z",
"traceback"=>
"Traceback (most recent call last):\n File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 240, in trace_task\n R = retval = fun(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 527, in __call__\n return super(Task, self).__call__(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 107, in __call__\n return super(PulpTask, self).__call__(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 438, in __protected_call__\n return self.run(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 1106, in publish\n result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)\n File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 1200, in check_publish\n result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)\n File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 1252, in _do_publish\n publish_report = publish_repo(transfer_repo, conduit, call_config)\n File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 730, in wrap_f\n return f(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py\", line 174, in publish_repo\n return self._publisher.process_lifecycle()\n File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 572, in process_lifecycle\n super(PluginStep, self).process_lifecycle()\n File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 163, in process_lifecycle\n step.process()\n File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 239, in process\n self._process_block(item=item)\n File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 301, in _process_block\n self.process_main(item=item)\n File \"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py\", line 485, in process_main\n context.add_unit_metadata(unit)\n File \"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/metadata/other.py\", line 42, in add_unit_metadata\n self.metadata_file_handle.write(unit.render_other(self.checksum_type))\n File \"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py\", line 850, in render_other\n return self._render(metadata, context)\n File \"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py\", line 879, in _render\n t = Template(template)\n File \"/usr/lib/python2.7/site-packages/django/template/base.py\", line 125, in __init__\n self.nodelist = compile_string(template_string, origin)\n File \"/usr/lib/python2.7/site-packages/django/template/base.py\", line 153, in compile_string\n return parser.parse()\n File \"/usr/lib/python2.7/site-packages/django/template/base.py\", line 276, in parse\n self.invalid_block_tag(token, command, parse_until)\n File \"/usr/lib/python2.7/site-packages/django/template/base.py\", line 332, in invalid_block_tag\n raise self.error(token, \"Invalid block tag: '%s'\" % command)\nTemplateSyntaxError: Invalid block tag: '}gmail{*}com>'\n",
"spawned_tasks"=>[],
"progress_report"=>
{"4d73ea9e-7776-486e-811b-d914dc823105"=>
[{"num_success"=>1,
"description"=>"Copying files",
"step_type"=>"save_tar",
"items_total"=>1,
"state"=>"FINISHED",
"error_details"=>[],
"details"=>"",
"num_failures"=>0,
"step_id"=>"9c354dd3-4837-4ea8-a3e1-7a03861942ff",
"num_processed"=>1},
{"num_success"=>1,
"description"=>"Initializing repo metadata",
"step_type"=>"initialize_repo_metadata",
"items_total"=>1,
"state"=>"FINISHED",
"error_details"=>[],
"details"=>"",
"num_failures"=>0,
"step_id"=>"23f96ff2-525e-451b-bfda-b558d113f17f",
"num_processed"=>1},
{"num_success"=>0,
"description"=>"Publishing Distribution files",
"step_type"=>"distribution",
"items_total"=>0,
"state"=>"FINISHED",
"error_details"=>[],
"details"=>"",
"num_failures"=>0,
"step_id"=>"848c3bd5-0fed-4a5e-bc76-05b886952c19",
"num_processed"=>0},
{"num_success"=>299,
"description"=>"Publishing RPMs",
"step_type"=>"rpms",
"items_total"=>17849,
"state"=>"FAILED",
"error_details"=>
[{"traceback"=>
" File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 239, in process\n self._process_block(item=item)\n\n File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 301, in _process_block\n self.process_main(item=item)\n\n File \"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/publish.py\", line 485, in process_main\n context.add_unit_metadata(unit)\n\n File \"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/metadata/other.py\", line 42, in add_unit_metadata\n self.metadata_file_handle.write(unit.render_other(self.checksum_type))\n\n File \"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py\", line 850, in render_other\n return self._render(metadata, context)\n\n File \"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/db/models.py\", line 879, in _render\n t = Template(template)\n\n File \"/usr/lib/python2.7/site-packages/django/template/base.py\", line 125, in __init__\n self.nodelist = compile_string(template_string, origin)\n\n File \"/usr/lib/python2.7/site-packages/django/template/base.py\", line 153, in compile_string\n return parser.parse()\n\n File \"/usr/lib/python2.7/site-packages/django/template/base.py\", line 276, in parse\n self.invalid_block_tag(token, command, parse_until)\n\n File \"/usr/lib/python2.7/site-packages/django/template/base.py\", line 332, in invalid_block_tag\n raise self.error(token, \"Invalid block tag: '%s'\" % command)\n",
"error"=>"Invalid block tag: '}gmail{*}com>'"}],
"details"=>"",
"num_failures"=>1,
"step_id"=>"c772a9bf-e9f4-43b7-bde8-302909f9dd9b",
"num_processed"=>300},
{"num_success"=>0,
"description"=>"Publishing Delta RPMs",
"step_type"=>"drpms",
"items_total"=>1,
"state"=>"NOT_STARTED",
"error_details"=>[],
"details"=>"",
"num_failures"=>0,
"step_id"=>"d1431e03-4eb4-474f-aec2-bd850013fd91",
"num_processed"=>0},
{"num_success"=>0,
"description"=>"Publishing Errata",
"step_type"=>"errata",
"items_total"=>1,
"state"=>"NOT_STARTED",
"error_details"=>[],
"details"=>"",
"num_failures"=>0,
"step_id"=>"d148e811-003c-4402-bbd7-db155bd0d486",
"num_processed"=>0},
{"num_success"=>0,
"description"=>"Publishing Comps file",
"step_type"=>"comps",
"items_total"=>1,
"state"=>"NOT_STARTED",
"error_details"=>[],
"details"=>"",
"num_failures"=>0,
"step_id"=>"6231f6da-1020-4fd4-92fd-131e57dce3e7",
"num_processed"=>0},
{"num_success"=>0,
"description"=>"Publishing Metadata.",
"step_type"=>"metadata",
"items_total"=>1,
"state"=>"NOT_STARTED",
"error_details"=>[],
"details"=>"",
"num_failures"=>0,
"step_id"=>"1eb2afec-4d4e-4273-bb61-d7280cb86e2a",
"num_processed"=>0},
{"num_success"=>0,
"description"=>"Closing repo metadata",
"step_type"=>"close_repo_metadata",
"items_total"=>1,
"state"=>"NOT_STARTED",
"error_details"=>[],
"details"=>"",
"num_failures"=>0,
"step_id"=>"ec3585fb-9eb5-4731-b14a-ab7ab3944713",
"num_processed"=>0},
{"num_success"=>0,
"description"=>"Generating sqlite files",
"step_type"=>"generate sqlite",
"items_total"=>1,
"state"=>"NOT_STARTED",
"error_details"=>[],
"details"=>"",
"num_failures"=>0,
"step_id"=>"f2f4bedc-2662-4107-b08e-91593d7ad2cb",
"num_processed"=>0},
{"num_success"=>0,
"description"=>"Generating HTML files",
"step_type"=>"repoview",
"items_total"=>1,
"state"=>"NOT_STARTED",
"error_details"=>[],
"details"=>"",
"num_failures"=>0,
"step_id"=>"2b9388b9-622a-4e91-a85b-6f9cc743e389",
"num_processed"=>0},
{"num_success"=>0,
"description"=>"Publishing files to web",
"step_type"=>"publish_directory",
"items_total"=>1,
"state"=>"NOT_STARTED",
"error_details"=>[],
"details"=>"",
"num_failures"=>0,
"step_id"=>"777f72ca-23b9-4a38-9257-1025123a0e2d",
"num_processed"=>0},
{"num_success"=>0,
"description"=>"Writing Listings File",
"step_type"=>"initialize_repo_metadata",
"items_total"=>1,
"state"=>"NOT_STARTED",
"error_details"=>[],
"details"=>"",
"num_failures"=>0,
"step_id"=>"1b2738a1-31d0-47b3-b658-f866245b97a8",
"num_processed"=>0},
{"num_success"=>0,
"description"=>"Writing Listings File",
"step_type"=>"initialize_repo_metadata",
"items_total"=>1,
"state"=>"NOT_STARTED",
"error_details"=>[],
"details"=>"",
"num_failures"=>0,
"step_id"=>"3488c45f-94cb-43ab-976c-bbcf65275c5f",
"num_processed"=>0}]},
"queue"=>"reserved_resource_worker-0@ussl-foreman.usanainc.com.dq",
"state"=>"error",
"worker_name"=>"reserved_resource_worker-0@ussl-foreman.usanainc.com",
"result"=>nil,
"error"=>
{"code"=>"PLP0000",
"data"=>{},
"description"=>"Invalid block tag: '}gmail{*}com>'",
"sub_errors"=>[]},
"_id"=>{"$oid"=>"5abb16f8de37317496f9af98"},
"id"=>"5abb16f8de37317496f9af98"}],
"poll_attempts"=>{"total"=>37, "failed"=>1}}
Here are the versions of every pulp package currently installed
python-kombu-3.0.33-8.pulp.el7.noarch
rubygem-smart_proxy_pulp-1.3.0-1.el7.noarch
python-pulp-streamer-2.13.4-1.el7.noarch
pulp-puppet-tools-2.13.4-1.el7.noarch
python-pulp-ostree-common-1.2.1-3.el7.noarch
python-pulp-puppet-common-2.13.4-1.el7.noarch
python-pulp-docker-common-2.4.1-2.el7.noarch
python-isodate-0.5.0-4.pulp.el7.noarch
python-pulp-client-lib-2.13.4-1.el7.noarch
python-pulp-common-2.13.4-1.el7.noarch
python-pulp-rpm-common-2.13.4-1.el7.noarch
pulp-selinux-2.13.4-1.el7.noarch
pulp-puppet-plugins-2.13.4-1.el7.noarch
pulp-rpm-admin-extensions-2.13.4-1.el7.noarch
python-pulp-bindings-2.13.4-1.el7.noarch
pulp-katello-1.0.2-1.el7.noarch
pulp-ostree-plugins-1.2.1-3.el7.noarch
pulp-ostree-admin-extensions-1.2.1-3.el7.noarch
python-pulp-repoauth-2.13.4-1.el7.noarch
pulp-docker-plugins-2.4.1-2.el7.noarch
pulp-client-1.0-1.noarch
pulp-rpm-plugins-2.13.4-1.el7.noarch
pulp-server-2.13.4-1.el7.noarch
python-pulp-oid_validation-2.13.4-1.el7.noarch
pulp-admin-client-2.13.4-1.el7.noarch
Actions
.
Sanitize Invalid Changelogs Too
The diff below causes the selection of input to be sanitized to be greedy instead of non-greedy. This causes it work sanitize more data in the cases of invalid entries such as Changelogs with too many sections in one rpm.
This fix is very safe for several reasons:
Escaping all data for the template layer should be safe. This is essentially user-provided data given that we got it from the internet, so it probably all should be escaped. Since this fixes this issue I'm not going to rewrite the sanitizer now to keep it low-risk.
The extra-greedyness added is still limited by the end tag matched
https://pulp.plan.io/issues/3535 closes #3535