Project

Profile

Help

Task #6155

Task #5961: [EPIC] Test pulpcore against S3

Get our tests running against minio or localstack

Added by daviddavis 6 months ago. Updated 4 months ago.

Status:
CLOSED - CURRENTRELEASE
Priority:
Normal
Assignee:
Category:
-
Sprint/Milestone:
Start date:
Due date:
% Done:

100%

Estimated time:
Platform Release:
Groomed:
Yes
Sprint Candidate:
Yes
Tags:
Sprint:
Sprint 68

History

#2 Updated by fao89 6 months ago

#!/usr/bin/env sh
export MINIO_ACCESS_KEY=AKIAIT2Z5TDYPX3ARJBA
export MINIO_SECRET_KEY=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
sudo mkdir /data
podman run -d -p 9000:9000 -e MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY -e MINIO_SECRET_KEY=$MINIO_SECRET_KEY minio/minio server /data
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
sudo mv mc /usr/local/bin
sudo dnf install -y nc
while ! nc -z localhost 9000; do echo 'Wait minio to startup...' && sleep 0.1; done;
sleep 5
mc config host add s3 http://localhost:9000 AKIAIT2Z5TDYPX3ARJBA fqRvjWaPU5o0fCqQuUWbj9Fainj2pVZtBCiDiieS --api S3v4
mc config host rm local
mc mb s3/pulp3 --region eu-central-1
mc tree s3
workon pulp
sudo bash -c "echo AWS_ACCESS_KEY_ID = \'AKIAIT2Z5TDYPX3ARJBA\' >> /etc/pulp/settings.py"
sudo bash -c "echo AWS_SECRET_ACCESS_KEY = \'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\' >> /etc/pulp/settings.py"
sudo bash -c "echo AWS_STORAGE_BUCKET_NAME = \'pulp3\' >> /etc/pulp/settings.py"
sudo bash -c "echo AWS_DEFAULT_ACL = None >> /etc/pulp/settings.py"
sudo bash -c "echo S3_USE_SIGV4 = True >> /etc/pulp/settings.py"
sudo bash -c "echo AWS_S3_SIGNATURE_VERSION = \'s3v4\' >> /etc/pulp/settings.py"
sudo bash -c "echo AWS_S3_ADDRESSING_STYLE = \'path\' >> /etc/pulp/settings.py"
sudo bash -c "echo AWS_S3_REGION_NAME = \'eu-central-1\' >> /etc/pulp/settings.py"
sudo bash -c "echo DEFAULT_FILE_STORAGE = \'storages.backends.s3boto3.S3Boto3Storage\' >> /etc/pulp/settings.py"
sudo bash -c "echo MEDIA_ROOT = \'\' >> /etc/pulp/settings.py"
sudo bash -c "echo AWS_S3_ENDPOINT_URL = \'http:\/\/localhost:9000\' >> /etc/pulp/settings.py"
pip install django-storages[boto3] pytest
prestart
cat /etc/pulp/settings.py
cd devel/pulp_file/docs/_scripts
./docs_check_sync_publish.sh
mc tree s3/pulp3
cd ~/devel/pulp_file/
pytest pulp_file/tests/functional

#3 Updated by fao89 6 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to fao89

#4 Updated by fao89 6 months ago

  • Status changed from ASSIGNED to NEW

#5 Updated by fao89 6 months ago

  • Assignee deleted (fao89)

#6 Updated by mdepaulo@redhat.com 6 months ago

Our regular CI currently runs against containers / pulp-operator

It is TBD whether it would be quicker to run against containers / pulp-operator, or to try to use ansible-pulp CI somehow.

#7 Updated by dkliban@redhat.com 6 months ago

  • Groomed changed from No to Yes
  • Sprint Candidate changed from No to Yes

#8 Updated by rchan 6 months ago

  • Sprint set to Sprint 67

#9 Updated by fao89 5 months ago

#!/usr/bin/env sh
sudo dnf install -y docker
sudo groupadd docker && sudo gpasswd -a ${USER} docker && sudo systemctl restart docker
newgrp docker
workon pulp
pip install awscli django-storages[boto3] localstack pytest
aws configure set aws_access_key_id AKIAIT2Z5TDYPX3ARJBA
aws configure set aws_secret_access_key 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
aws configure set default.region eu-central-1
aws configure set s3.signature_version s3v4
aws configure set default.s3.addressing_style path
mkdir -p /tmp/localstack/data
export LOCALSTACK_SERVICES=s3
export DEFAULT_REGION="eu-central-1"
export DATA_DIR=/tmp/localstack/data
ENTRYPOINT=-d localstack start --docker
sleep 10
sudo bash -c "echo AWS_ACCESS_KEY_ID = \'AKIAIT2Z5TDYPX3ARJBA\' >> /etc/pulp/settings.py"
sudo bash -c "echo AWS_SECRET_ACCESS_KEY = \'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\' >> /etc/pulp/settings.py"
sudo bash -c "echo AWS_STORAGE_BUCKET_NAME = \'pulp3\' >> /etc/pulp/settings.py"
sudo bash -c "echo AWS_DEFAULT_ACL = None >> /etc/pulp/settings.py"
sudo bash -c "echo S3_USE_SIGV4 = True >> /etc/pulp/settings.py"
sudo bash -c "echo AWS_S3_SIGNATURE_VERSION = \'s3v4\' >> /etc/pulp/settings.py"
sudo bash -c "echo AWS_S3_ADDRESSING_STYLE = \'path\' >> /etc/pulp/settings.py"
sudo bash -c "echo AWS_S3_REGION_NAME = \'eu-central-1\' >> /etc/pulp/settings.py"
sudo bash -c "echo DEFAULT_FILE_STORAGE = \'storages.backends.s3boto3.S3Boto3Storage\' >> /etc/pulp/settings.py"
sudo bash -c "echo MEDIA_ROOT = \'\' >> /etc/pulp/settings.py"
sudo bash -c "echo AWS_S3_ENDPOINT_URL = \'http:\/\/localhost:4572\' >> /etc/pulp/settings.py"
prestart
aws --endpoint-url=http://localhost:4572 s3 mb s3://pulp3
aws --endpoint-url=http://localhost:4572 s3api list-buckets
cd devel/pulp_file/docs/_scripts
./docs_check_sync_publish.sh
aws --endpoint-url=http://localhost:4572 s3 ls s3://pulp3 --recursive
ls /tmp/localstack/data
cd ~/devel/pulp_file/
pytest pulp_file/tests/functional

#10 Updated by fao89 5 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to fao89

#11 Updated by fao89 5 months ago

  • Status changed from ASSIGNED to NEW
  • Assignee deleted (fao89)

#12 Updated by fao89 5 months ago

https://github.com/ceph/cn cn is very limited, I could not find a way to set the region, signature version and addressing style. Even secret and access key we have to change in a hack-ish way: https://github.com/ceph/cn#note-that-the-object-gateway-tab-in-the-dashboard-is-not-enabled-yet-so-run-the-following-to-enable-rgw-dashboard

#13 Updated by fao89 5 months ago

  • Status changed from NEW to ASSIGNED
  • Assignee set to fao89

#15 Updated by rchan 5 months ago

  • Sprint changed from Sprint 67 to Sprint 68

#16 Updated by Anonymous 5 months ago

  • Status changed from POST to MODIFIED
  • % Done changed from 0 to 100

#17 Updated by ttereshc 4 months ago

  • Status changed from MODIFIED to CLOSED - CURRENTRELEASE
  • Sprint/Milestone set to 3.3.0

Please register to edit this issue

Also available in: Atom PDF