Project

Profile

Help

Actions
Send by e-mail Copy link

Issue #7846

closed

null value in column "stream" violates not-null constraint during migration on Katello

Added by iballou about 4 years ago. Updated almost 4 years ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
dalley
Sprint/Milestone:
0.7.0
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 88
Quarter:

Description

A Katello user is experiencing errors when migrating to Pulp 3: https://community.theforeman.org/t/katello-3-17-upgrade-issues/21262

Running “foreman-maintain advanced procedure run content-prepare” produces the following error

Running ForemanMaintain::Scenario
================================================================================
Prepare content for Pulp 3:                                           [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
 Rubocop not loaded.
Migration failed, You will want to investigate: https://FQDN/foreman_tasks/tasks/b57d7a70-bbe1-46b2-beaf-fa627bcf02cd
rake aborted!
ForemanTasks::TaskError: Task b57d7a70-bbe1-46b2-beaf-fa627bcf02cd: Katello::Errors::Pulp3Error: null value in column "stream" violates not-null constraint
DETAIL:  Failing row contains (ff6ac58a-cc9e-48f4-bfde-05a6e04fdb29, 2020-11-12 22:57:58.425104+00, 2020-11-12 22:57:58.425113+00, 389-directory-server, null, {"stable": ["default"], "testing": ["default"]}, 015ae06403a7d0318bb78bdf560d3616bb11369161ffaa7bd2eb658aae5c2764, 1-Red_Hat_Enterprise_Linux_8_x86_64-v5_0-18c3ae48-ca68-4b4e-b396..., 7746a071-6587-44a3-8082-b75641a9aa64).
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/lib/katello/tasks/pulp3_migration.rake:17:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
--------------------------------------------------------------------------------
Scenario [ForemanMaintain::Scenario] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

Running the command a second time produces the following message:

Running ForemanMaintain::Scenario
================================================================================
Prepare content for Pulp 3:                                           [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
 Rubocop not loaded.
Migration failed, You will want to investigate: https://FQDN/foreman_tasks/tasks/b3afcbc8-db69-4580-97aa-148572c11851
rake aborted!
ForemanTasks::TaskError: Task b3afcbc8-db69-4580-97aa-148572c11851: Katello::Errors::Pulp3Error: __init__() got an unexpected keyword argument 'box_settings'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/lib/katello/tasks/pulp3_migration.rake:17:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
--------------------------------------------------------------------------------
Scenario [ForemanMaintain::Scenario] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"
Actions
Copy link
 #1

Updated by iballou about 4 years ago

pulp-2to3-migration (0.5.0) pulp-certguard (1.0.2) pulp-container (2.0.1) pulp-file (1.2.0) pulp-rpm (3.6.2) pulpcore (3.6.3)

Actions
Copy link
 #2

Updated by ttereshc about 4 years ago

According to the forum post, the issue is likely for the pre-migration of the modulemd-defaults. The modulemd pre-migration finished successfully.

Actions
Copy link
 #3

Updated by ttereshc almost 4 years ago

  • Project changed from Migration Plugin to RPM Support

It looks to me that pulp_rpm doesn't support null value for the stream field for module-defaults. https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/modulemd.py#L74

The spec says that the stream is optional https://github.com/fedora-modularity/libmodulemd/blob/main/yaml_specs/modulemd_defaults_v1.yaml#L15

Actions
Copy link
 #4

Updated by ttereshc almost 4 years ago

Profiles are optional as well, only module name is mandatory.

Actions
Copy link
 #5

Updated by iballou almost 4 years ago

If anyone is curious, here are the repositories that one user was syncing from when they hit this issue. It seems like the module streams related to EPEL were the problem.

-------------------------------|---------------------|--------------|--------------------------------------------------------------------------------
 NAME                          | PRODUCT             | CONTENT TYPE | URL                                                                                 
-------------------------------|---------------------|--------------|--------------------------------------------------------------------------------
 Alienvault-Agent-CentOS-7     | Binary Defense      | yum          | s://prod-usm-saas-agent-config.s3.amazonaws.com/repo/rpm/x86_64/                    
 CandlePin                     | Foreman-Katello     | yum          | s://fedorapeople.org/groups/katello/releases/yum/3.14/candlepin/el7/x86_64/
 CentOS-7-base                 | CentOS              | yum          | ://mirror.centos.org/centos/7/os/x86_64/                                            
 CentOS-7-extras               | CentOS              | yum          | ://mirror.centos.org/centos/7/extras/x86_64/                                        
 CentOS-7-SCLo-rh              | CentOS              | yum          | ://mirror.centos.org/centos/7/sclo/x86_64/rh/                                       
 CentOS-7-updates              | CentOS              | yum          | ://mirror.centos.org/centos/7/updates/x86_64/                                       
 CentOS-8-Appstream            | CentOS              | yum          | ://mirror.centos.org/centos-8/8/AppStream/x86_64/os/                                
 CentOS-8-base                 | CentOS              | yum          | ://mirror.centos.org/centos-8/8/BaseOS/x86_64/os/                                   
 CentOS-8-extras               | CentOS              | yum          | ://mirror.centos.org/centos-8/8/extras/x86_64/os/                                   
 CentOS-8-PowerTools           | CentOS              | yum          | ://mirror.centos.org/centos-8/8/PowerTools/x86_64/os/                               
 Cisco_AMP_7                   | Cisco_amp           | yum          | ://katello.cohen.cpa/pub/custom_repo/cisco_amp                                      
 Cisco Amp CentOS 7            | File Based Products | file         |                                                                                     
 Docker repo                   | Docker              | yum          | s://download.docker.com/linux/centos/7/x86_64/stable                                
 DUO_Security-CentOS-7         | DUO                 | yum          | s://pkg.duosecurity.com/CentOS/7/x86_64                                             
 DUO_Security-CentOS-8         | DUO                 | yum          | s://pkg.duosecurity.com/CentOS/8/x86_64                                             
 EPEL-7                        | EPEL                | yum          | ://dl.fedoraproject.org/pub/epel/7//x86_64/                                         
 EPEL-8                        | EPEL                | yum          | s://dl.fedoraproject.org/pub/epel/8/Everything/x86_64                               
 EPEL-8-modular                | EPEL                | yum          | s://dl.fedoraproject.org/pub/epel/8/Modular/x86_64                                  
 Foreman                       | Foreman-Katello     | yum          | s://yum.theforeman.org/releases/1.24/el7/x86_64                                     
 Foreman Client EL7            | Foreman Client      | yum          | s://yum.theforeman.org/client/1.24/el7/x86_64                                       
 Foreman-Plugins               | Foreman-Katello     | yum          | s://yum.theforeman.org/plugins/1.24/el7/x86_64                                      
 grafana                       | Grafana             | yum          | s://packages.grafana.com/oss/rpm                                                
 Jenkins-RH-Stable             | Jenkins             | yum          | ://pkg.jenkins.io/redhat-stable                                                     
 Katello                       | Foreman-Katello     | yum          | s://fedorapeople.org/groups/katello/releases/yum/3.14/katello/el7/x86_64/  
 MySQL_8_CommunityServer_el7   | MySQL               | yum          | ://repo.mysql.com/yum/mysql-8.0-community/el/7/x86_64/                              
 MySQL_CommunityConnectors_el7 | MySQL               | yum          | ://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64                        
 MySQL_CommunityTools_el7      | MySQL               | yum          | ://repo.mysql.com/yum/mysql-tools-community/el/7/x86_64/                            
 NGINX_7                       | NGINX               | yum          | ://nginx.org/packages/centos/7/x86_64/                                              
 NGINX_8                       | NGINX               | yum          | ://nginx.org/packages/centos/8/x86_64/                                              
 NodeSource_7_v10              | NodeSource          | yum          | s://rpm.nodesource.com/pub_10.x/el/7/x86_64/                                        
 NodeSource_7_v12              | NodeSource          | yum          | s://rpm.nodesource.com/pub_12.x/el/7/x86_64/                                        
 NodeSource_7_v14              | NodeSource          | yum          | s://rpm.nodesource.com/pub_14.x/el/7/x86_64/                                        
 NodeSource_8_v10              | NodeSource          | yum          | s://rpm.nodesource.com/pub_10.x/el/8/x86_64/                                        
 NodeSource_8_v12              | NodeSource          | yum          | s://rpm.nodesource.com/pub_12.x/el/8/x86_64/                                        
 NodeSource_8_v14              | NodeSource          | yum          | s://rpm.nodesource.com/pub_14.x/el/8/x86_64/                                        
 Packages-7                    | Microsoft           | yum          | s://packages.microsoft.com/rhel/7/prod/                                             
 pulp-2                        | pulp-2              | yum          | s://repos.fedorapeople.org/repos/pulp/pulp/stable/2/7/x86_64                        
 Puppet-6-CentOS-7             | Puppet              | yum          | ://yum.puppetlabs.com/puppet6/el/7/x86_64                                           
 Puppet-6-CentOS-8             | Puppet              | yum          | ://yum.puppetlabs.com/puppet/el/8/x86_64                                            
 zabbix_official_el7           | Zabbix              | yum          | ://repo.zabbix.com/zabbix/5.0/rhel/7/x86_64/
Actions
Copy link
 #6

Updated by ipanova@redhat.com almost 4 years ago 

In [7]: Pulp2ModulemdDefaults(module='test2', profiles={}, pulp2content_id=p.pk, stream=None).save()
---------------------------------------------------------------------------
NotNullViolation                          Traceback (most recent call last)
/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py in _execute(self, sql, params, *ignored_wrapper_args)
     83             else:
---> 84                 return self.cursor.execute(sql, params)
     85 

NotNullViolation: null value in column "stream" violates not-null constraint
DETAIL:  Failing row contains (bf6e5ab5-5b32-4eb1-a524-cac7266ceaa5, 2021-01-14 17:16:43.607299+00, 2021-01-14 17:16:43.607354+00, test2, null, {}, , , 0329f8c7-e75b-4eb6-b39d-542ff6be7bc8).

We need to fix this line https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/plugin/rpm/pulp_2to3_models.py#L665

As mentioned previously, we should also fix profiles, since they are optional but seems like we require them.


In [8]: Pulp2ModulemdDefaults(module='test2',pulp2content_id=p.pk).save()
---------------------------------------------------------------------------
NotNullViolation                          Traceback (most recent call last)
/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py in _execute(self, sql, params, *ignored_wrapper_args)
     83             else:
---> 84                 return self.cursor.execute(sql, params)
     85 

NotNullViolation: null value in column "profiles" violates not-null constraint
DETAIL:  Failing row contains (80f2125a-b790-4de7-8ab1-98ef7362a636, 2021-01-14 17:23:34.868795+00, 2021-01-14 17:23:34.868838+00, test2, , null, , , 0329f8c7-e75b-4eb6-b39d-542ff6be7bc8).

In pulp3 modules and defaults are saved without troubles without 'stream' unless explicitly sending stream=None

In [12]: Modulemd(name='bar1').save()

In [14]: ModulemdDefaults(stream='bar1', digest=12344).save()

In [15]:
Actions
Copy link
 #8

Updated by ipanova@redhat.com almost 4 years ago

  • Project changed from RPM Support to Migration Plugin
Actions
Copy link
 #9

Updated by dalley almost 4 years ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley
Actions
Copy link
 #10

Updated by dalley almost 4 years ago

  • Status changed from ASSIGNED to POST
  • Sprint set to Sprint 88
Actions
Copy link
 #11

Updated by ttereshc almost 4 years ago

  • Sprint/Milestone set to 0.9.0

Added by dalley almost 4 years ago

Revision e51d319a | View on GitHub

Fix null value in column "stream" error

closes: #7846 https://pulp.plan.io/issues/7846

Added by dalley almost 4 years ago

Revision e51d319a | View on GitHub

Fix null value in column "stream" error

closes: #7846 https://pulp.plan.io/issues/7846

Added by dalley almost 4 years ago

Revision e51d319a | View on GitHub

Fix null value in column "stream" error

closes: #7846 https://pulp.plan.io/issues/7846

Actions
Copy link
 #12

Updated by dalley almost 4 years ago

  • Status changed from POST to MODIFIED
Actions
Copy link
 #13

Updated by ttereshc almost 4 years ago

  • Sprint/Milestone changed from 0.9.0 to 0.7.0
Actions
Copy link
 #14

Updated by pulpbot almost 4 years ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
Actions
Send by e-mail Copy link

Also available in: Atom PDF