Project

Profile

Help

Issue #7846

null value in column "stream" violates not-null constraint during migration on Katello

Added by iballou 12 months ago. Updated 9 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Sprint/Milestone:
Start date:
Due date:
Estimated time:
Severity:
2. Medium
Platform Release:
OS:
Triaged:
No
Groomed:
No
Sprint Candidate:
No
Tags:
Katello
Sprint:
Sprint 88
Quarter:

Description

A Katello user is experiencing errors when migrating to Pulp 3: https://community.theforeman.org/t/katello-3-17-upgrade-issues/21262

Running “foreman-maintain advanced procedure run content-prepare” produces the following error

Running ForemanMaintain::Scenario
================================================================================
Prepare content for Pulp 3:                                           [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
 Rubocop not loaded.
Migration failed, You will want to investigate: https://FQDN/foreman_tasks/tasks/b57d7a70-bbe1-46b2-beaf-fa627bcf02cd
rake aborted!
ForemanTasks::TaskError: Task b57d7a70-bbe1-46b2-beaf-fa627bcf02cd: Katello::Errors::Pulp3Error: null value in column "stream" violates not-null constraint
DETAIL:  Failing row contains (ff6ac58a-cc9e-48f4-bfde-05a6e04fdb29, 2020-11-12 22:57:58.425104+00, 2020-11-12 22:57:58.425113+00, 389-directory-server, null, {"stable": ["default"], "testing": ["default"]}, 015ae06403a7d0318bb78bdf560d3616bb11369161ffaa7bd2eb658aae5c2764, 1-Red_Hat_Enterprise_Linux_8_x86_64-v5_0-18c3ae48-ca68-4b4e-b396..., 7746a071-6587-44a3-8082-b75641a9aa64).
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/lib/katello/tasks/pulp3_migration.rake:17:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
--------------------------------------------------------------------------------
Scenario [ForemanMaintain::Scenario] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

Running the command a second time produces the following message:

Running ForemanMaintain::Scenario
================================================================================
Prepare content for Pulp 3:                                           [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1:
 Rubocop not loaded.
Migration failed, You will want to investigate: https://FQDN/foreman_tasks/tasks/b3afcbc8-db69-4580-97aa-148572c11851
rake aborted!
ForemanTasks::TaskError: Task b3afcbc8-db69-4580-97aa-148572c11851: Katello::Errors::Pulp3Error: __init__() got an unexpected keyword argument 'box_settings'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0/lib/katello/tasks/pulp3_migration.rake:17:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
--------------------------------------------------------------------------------
Scenario [ForemanMaintain::Scenario] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

Associated revisions

Revision e51d319a View on GitHub
Added by dalley 9 months ago

Fix null value in column "stream" error

closes: #7846 https://pulp.plan.io/issues/7846

Revision e51d319a View on GitHub
Added by dalley 9 months ago

Fix null value in column "stream" error

closes: #7846 https://pulp.plan.io/issues/7846

Revision e51d319a View on GitHub
Added by dalley 9 months ago

Fix null value in column "stream" error

closes: #7846 https://pulp.plan.io/issues/7846

History

#1 Updated by iballou 12 months ago

pulp-2to3-migration (0.5.0) pulp-certguard (1.0.2) pulp-container (2.0.1) pulp-file (1.2.0) pulp-rpm (3.6.2) pulpcore (3.6.3)

#2 Updated by ttereshc 11 months ago

According to the forum post, the issue is likely for the pre-migration of the modulemd-defaults. The modulemd pre-migration finished successfully.

#3 Updated by ttereshc 10 months ago

  • Project changed from Migration Plugin to RPM Support

It looks to me that pulp_rpm doesn't support null value for the stream field for module-defaults. https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/modulemd.py#L74

The spec says that the stream is optional https://github.com/fedora-modularity/libmodulemd/blob/main/yaml_specs/modulemd_defaults_v1.yaml#L15

#4 Updated by ttereshc 10 months ago

Profiles are optional as well, only module name is mandatory.

#5 Updated by iballou 10 months ago

If anyone is curious, here are the repositories that one user was syncing from when they hit this issue. It seems like the module streams related to EPEL were the problem.

-------------------------------|---------------------|--------------|--------------------------------------------------------------------------------
 NAME                          | PRODUCT             | CONTENT TYPE | URL                                                                                 
-------------------------------|---------------------|--------------|--------------------------------------------------------------------------------
 Alienvault-Agent-CentOS-7     | Binary Defense      | yum          | s://prod-usm-saas-agent-config.s3.amazonaws.com/repo/rpm/x86_64/                    
 CandlePin                     | Foreman-Katello     | yum          | s://fedorapeople.org/groups/katello/releases/yum/3.14/candlepin/el7/x86_64/
 CentOS-7-base                 | CentOS              | yum          | ://mirror.centos.org/centos/7/os/x86_64/                                            
 CentOS-7-extras               | CentOS              | yum          | ://mirror.centos.org/centos/7/extras/x86_64/                                        
 CentOS-7-SCLo-rh              | CentOS              | yum          | ://mirror.centos.org/centos/7/sclo/x86_64/rh/                                       
 CentOS-7-updates              | CentOS              | yum          | ://mirror.centos.org/centos/7/updates/x86_64/                                       
 CentOS-8-Appstream            | CentOS              | yum          | ://mirror.centos.org/centos-8/8/AppStream/x86_64/os/                                
 CentOS-8-base                 | CentOS              | yum          | ://mirror.centos.org/centos-8/8/BaseOS/x86_64/os/                                   
 CentOS-8-extras               | CentOS              | yum          | ://mirror.centos.org/centos-8/8/extras/x86_64/os/                                   
 CentOS-8-PowerTools           | CentOS              | yum          | ://mirror.centos.org/centos-8/8/PowerTools/x86_64/os/                               
 Cisco_AMP_7                   | Cisco_amp           | yum          | ://katello.cohen.cpa/pub/custom_repo/cisco_amp                                      
 Cisco Amp CentOS 7            | File Based Products | file         |                                                                                     
 Docker repo                   | Docker              | yum          | s://download.docker.com/linux/centos/7/x86_64/stable                                
 DUO_Security-CentOS-7         | DUO                 | yum          | s://pkg.duosecurity.com/CentOS/7/x86_64                                             
 DUO_Security-CentOS-8         | DUO                 | yum          | s://pkg.duosecurity.com/CentOS/8/x86_64                                             
 EPEL-7                        | EPEL                | yum          | ://dl.fedoraproject.org/pub/epel/7//x86_64/                                         
 EPEL-8                        | EPEL                | yum          | s://dl.fedoraproject.org/pub/epel/8/Everything/x86_64                               
 EPEL-8-modular                | EPEL                | yum          | s://dl.fedoraproject.org/pub/epel/8/Modular/x86_64                                  
 Foreman                       | Foreman-Katello     | yum          | s://yum.theforeman.org/releases/1.24/el7/x86_64                                     
 Foreman Client EL7            | Foreman Client      | yum          | s://yum.theforeman.org/client/1.24/el7/x86_64                                       
 Foreman-Plugins               | Foreman-Katello     | yum          | s://yum.theforeman.org/plugins/1.24/el7/x86_64                                      
 grafana                       | Grafana             | yum          | s://packages.grafana.com/oss/rpm                                                
 Jenkins-RH-Stable             | Jenkins             | yum          | ://pkg.jenkins.io/redhat-stable                                                     
 Katello                       | Foreman-Katello     | yum          | s://fedorapeople.org/groups/katello/releases/yum/3.14/katello/el7/x86_64/  
 MySQL_8_CommunityServer_el7   | MySQL               | yum          | ://repo.mysql.com/yum/mysql-8.0-community/el/7/x86_64/                              
 MySQL_CommunityConnectors_el7 | MySQL               | yum          | ://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64                        
 MySQL_CommunityTools_el7      | MySQL               | yum          | ://repo.mysql.com/yum/mysql-tools-community/el/7/x86_64/                            
 NGINX_7                       | NGINX               | yum          | ://nginx.org/packages/centos/7/x86_64/                                              
 NGINX_8                       | NGINX               | yum          | ://nginx.org/packages/centos/8/x86_64/                                              
 NodeSource_7_v10              | NodeSource          | yum          | s://rpm.nodesource.com/pub_10.x/el/7/x86_64/                                        
 NodeSource_7_v12              | NodeSource          | yum          | s://rpm.nodesource.com/pub_12.x/el/7/x86_64/                                        
 NodeSource_7_v14              | NodeSource          | yum          | s://rpm.nodesource.com/pub_14.x/el/7/x86_64/                                        
 NodeSource_8_v10              | NodeSource          | yum          | s://rpm.nodesource.com/pub_10.x/el/8/x86_64/                                        
 NodeSource_8_v12              | NodeSource          | yum          | s://rpm.nodesource.com/pub_12.x/el/8/x86_64/                                        
 NodeSource_8_v14              | NodeSource          | yum          | s://rpm.nodesource.com/pub_14.x/el/8/x86_64/                                        
 Packages-7                    | Microsoft           | yum          | s://packages.microsoft.com/rhel/7/prod/                                             
 pulp-2                        | pulp-2              | yum          | s://repos.fedorapeople.org/repos/pulp/pulp/stable/2/7/x86_64                        
 Puppet-6-CentOS-7             | Puppet              | yum          | ://yum.puppetlabs.com/puppet6/el/7/x86_64                                           
 Puppet-6-CentOS-8             | Puppet              | yum          | ://yum.puppetlabs.com/puppet/el/8/x86_64                                            
 zabbix_official_el7           | Zabbix              | yum          | ://repo.zabbix.com/zabbix/5.0/rhel/7/x86_64/

#6 Updated by ipanova@redhat.com 10 months ago

In [7]: Pulp2ModulemdDefaults(module='test2', profiles={}, pulp2content_id=p.pk, stream=None).save()
---------------------------------------------------------------------------
NotNullViolation                          Traceback (most recent call last)
/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py in _execute(self, sql, params, *ignored_wrapper_args)
     83             else:
---> 84                 return self.cursor.execute(sql, params)
     85 

NotNullViolation: null value in column "stream" violates not-null constraint
DETAIL:  Failing row contains (bf6e5ab5-5b32-4eb1-a524-cac7266ceaa5, 2021-01-14 17:16:43.607299+00, 2021-01-14 17:16:43.607354+00, test2, null, {}, , , 0329f8c7-e75b-4eb6-b39d-542ff6be7bc8).


We need to fix this line https://github.com/pulp/pulp-2to3-migration/blob/master/pulp_2to3_migration/app/plugin/rpm/pulp_2to3_models.py#L665

As mentioned previously, we should also fix profiles, since they are optional but seems like we require them.


In [8]: Pulp2ModulemdDefaults(module='test2',pulp2content_id=p.pk).save()
---------------------------------------------------------------------------
NotNullViolation                          Traceback (most recent call last)
/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/backends/utils.py in _execute(self, sql, params, *ignored_wrapper_args)
     83             else:
---> 84                 return self.cursor.execute(sql, params)
     85 

NotNullViolation: null value in column "profiles" violates not-null constraint
DETAIL:  Failing row contains (80f2125a-b790-4de7-8ab1-98ef7362a636, 2021-01-14 17:23:34.868795+00, 2021-01-14 17:23:34.868838+00, test2, , null, , , 0329f8c7-e75b-4eb6-b39d-542ff6be7bc8).

In pulp3 modules and defaults are saved without troubles without 'stream' unless explicitly sending stream=None

In [12]: Modulemd(name='bar1').save()

In [14]: ModulemdDefaults(stream='bar1', digest=12344).save()

In [15]: 


#8 Updated by ipanova@redhat.com 10 months ago

  • Project changed from RPM Support to Migration Plugin

#9 Updated by dalley 9 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to dalley

#10 Updated by dalley 9 months ago

  • Status changed from ASSIGNED to POST
  • Sprint set to Sprint 88

#11 Updated by ttereshc 9 months ago

  • Sprint/Milestone set to 0.9.0

#12 Updated by dalley 9 months ago

  • Status changed from POST to MODIFIED

#13 Updated by ttereshc 9 months ago

  • Sprint/Milestone changed from 0.9.0 to 0.7.0

#14 Updated by pulpbot 9 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE

Please register to edit this issue

Also available in: Atom PDF