On the SES cluster:
ceph osd crush tunables hammer
ceph osd set-require-min-compat-client hammer
This step is not required when migrating object storage only.
If HOS is using RHEL or SLES compute nodes, make sure the Ceph packages in the HOS repos are up to date: follow the HOS5 documentation in the section “Setting up a yum repo on Lifecycle Manager Node for Hosting the Ceph Client Packages for RHEL 7.2” on how to add packages to the HOS repos using the third party integration framework. For SLES, the packages required are:
Make sure to run the osconfig-run.yml playbook against the affected compute nodes (using the –limit option) after importing the updated packages.
ceph osd pool create volumes 256
ceph osd pool create images 256
ceph osd pool create backups 256
ceph osd pool create vms 256
The number of PGs in the commands above (256) are just an example and should be changed depending on the environment: see the official ceph docs around placement gorups.
On the SES cluster:
ceph auth get-or-create client.glance-ses mon 'profile rbd' osd 'profile rbd pool=images'
ceph auth get-or-create client.cinder-ses mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images'
ceph auth get-or-create client.cinder-backup-ses mon 'profile rbd' osd 'profile rbd pool=backups'
The keys generated here will be used in the next step.
Log into the HOS deployer node as the stack user and clone/copy this repo
to the user’s home directory.
In the ceph-client-setup directory, create a vars.yml file using
example-vars.yml as a template and fill in the keys from the previous step:
---
keyring:
client.cinder: AQB21QtbHwznAxAAz7tS3m6SHzGWppCBpYgjGg==
client.glance: AQCQ1Qtb2OW0DhAAQSztXBeWfwdx/qXOvdO5bA==
client.cinder-backup: AQCg1Qtb16TzKRAAM76vnJIjWSl5hbFfi+oqpw==
libvirt_secret_uuid: 993b98b1-7c4f-4c23-95ff-1c14fb3e5e1c
cluster_name: ses
The variable libvirt_secret_uuid should be set to a random UUID: it can be
generated using the uuidgen command.
Copy the /etc/ceph/ceph.conf file from the SES cluster in the same directory
as the vars.yml file, rename it to ceph.conf.j2 and make any required change.
Make sure to set cluster_name to ses to avoid clashes with the existing
Ceph configuration files.
Run the playbook:
ansible-playbook ceph-client-deploy.yml
This will make sure the ceph configuration file in place and enable the ceph libvirt backend on the compute nodes.
From a controller or compute node, check that the client can talk to the ceph cluster:
sudo ceph -c /etc/ceph/ses.conf --id cinder-ses -s
Edit ~/helion/my_cloud/config/cinder/cinder.conf.j2 and add a new section
for the ceph backend, for example:
[ses_ceph]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ses_ceph
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ses.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
rbd_user = cinder-ses
rbd_secret_uuid = 457eb676-33da-42ec-9a8c-9293d545c337
Make sure to replace 457eb676-33da-42ec-9a8c-9293d545c337 in
rbd_secret_uuid with the value used for libvirt_secret_uuid in vars.yml
and that the new backend is listed in enabled_backends in the
DEFAULTS section, for example:
enabled_backends=ceph1,ses_ceph
Edit ~/helion/my_cloud/config/nova/kvm-hypervisor.conf.j2 and make sure
the libvirt section does not contain rbd_user or rbd_secret_uid.
This is needed because of nova bug #1635008.
rbd_user and rbd_secret_uid are strictly required only when using Ceph
for ephemeral disk, which is not a supported configuration in HOS5: for
regular cinder volumes, Ceph details from cinder will be used when attaching
volumes.
On the SES cluster:
ceph auth get-or-create client.glance mon 'profile rbd' osd 'profile rbd pool=images'
ceph auth get-or-create client.cinder mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images'
ceph auth get-or-create client.cinder-backup mon 'profile rbd' osd 'profile rbd pool=backups'
The keys generated here will be used in the next step.
Log into the HOS deployer node as the stack user and clone/copy this repo
to the user’s home directory.
In the ceph-client-setup directory, create a vars.yml file using
example-vars.yml as a template and fill in the keys from the previous step:
---
keyring:
client.cinder: AQB21QtbHwznAxAAz7tS3m6SHzGWppCBpYgjGg==
client.glance: AQCQ1Qtb2OW0DhAAQSztXBeWfwdx/qXOvdO5bA==
client.cinder-backup: AQCg1Qtb16TzKRAAM76vnJIjWSl5hbFfi+oqpw==
libvirt_secret_uuid: 993b98b1-7c4f-4c23-95ff-1c14fb3e5e1c
cluster_name: ceph
The variable libvirt_secret_uuid should be set to a random UUID: it can be
generated using the uuidgen command.
Copy the /etc/ceph/ceph.conf file from the SES cluster in the same directory
as the vars.yml file, rename it to ceph.conf.j2 and make any required change.
Run the playbook:
ansible-playbook ceph-client-deploy.yml
This will make sure the ceph client packages are installed on the nodes, put the configuration file in place and enable the ceph libvirt backend on the compute nodes.
From a controller or compute node, check that the client can talk to the ceph cluster:
sudo ceph --id cinder -s
Edit ~/helion/my_cloud/config/cinder/cinder.conf.j2 and add a new section
for the ceph backend, for example:
[ses_ceph]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ses_ceph
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
rbd_user = cinder
rbd_secret_uuid = 457eb676-33da-42ec-9a8c-9293d545c337
Make sure to replace 457eb676-33da-42ec-9a8c-9293d545c337 in
rbd_secret_uuid with the value used for libvirt_secret_uuid in vars.yml
and that the new backend is listed in enabled_backends in the
DEFAULTS section, for example:
enabled_backends=vsa-1,ses_ceph
Commit the configuration changes and verify the input model is still consistent:
cd ~/helion/hos/ansible/
git commit -am "Enable SES"
ansible-playbook -i hosts/localhost config-processor-run.yml
ansible-playbook -i hosts/localhost ready-deployment.yml
Reconfigure the services:
cd ~/scratch/ansible/next/hos/ansible/
ansible-playbook cinder-reconfigure.yml
. ~/service.osrc
cinder type-create ses
cinder type-key ses set volume_backend_name=ses_ceph
Make sure volume_backend_name matches the volume_backend_name in the SES
section added to cinder.conf.j2.
Before migrating a volume, it must be detached from its instance and all of its snapshots must be removed.
To migrate a volume to the SES backend, the cinder retype command can be used:
cinder retype --migration-policy on-demand 3b7cd205-ddbb-42c5-87db-20ce306a2c57 ses
Where 3b7cd205-ddbb-42c5-87db-20ce306a2c57 is the ID of the volume and ses is
the name of the volume-type we created in the previous step.
The status of the migration can be checked with the cinder show command.
The volume can be re-attached to the instance after the migration is complete.
Currently cinder does not support detaching the boot volume from its instance.
A workaround involves manipulating the nova block_device_mapping table
contents to trick cinder into thinking the volume is not the boot volume:
update block_device_mapping set boot_index=999 where deleted=0 and volume_id='ca166060-ba4b-4950-8308-1cf0395c934f' and boot_index=0;
Now the volume can be detached and migrated. After reattaching the volumes to the instance, the database entry can be changed back:
update block_device_mapping set boot_index=0 where deleted=0 and volume_id='ca166060-ba4b-4950-8308-1cf0395c934f' and boot_index is NULL;
The migration_planner.py script can be used to build a detailed list of
steps to execute in order to migrate the existing volumes. The output can
be executed verbatim in a shell but it is not meant to be used as a shell
script.
The script expects one or more arguments in the form oldtype=newtype where
oldtype is the original volume type and newtype is the volume type we want
to migrate to.
It is meant to be run on a controller node in the openstack-client virtualenv.
This is the state of the environment we want to migrate:
stack@hos5to8-cp1-c1-m1-mgmt:~$ . ~/service.osrc
stack@hos5to8-cp1-c1-m1-mgmt:~$ openstack volume list --all-projects --long
+--------------------------------------+--------------+-----------+------+--------+----------+------------------------------+--------------------------------------+
| ID | Display Name | Status | Size | Type | Bootable | Attached to | Properties |
+--------------------------------------+--------------+-----------+------+--------+----------+------------------------------+--------------------------------------+
| 1b199df7-3273-40eb-b705-8902e8cca727 | unused | available | 2 | VSA-r1 | false | | |
| 3fb9d86b-cea7-4508-ba44-6099e6c088e0 | vm2-disk1 | in-use | 2 | VSA-r5 | false | Attached to vm2 on /dev/vdb | attached_mode='rw', readonly='False' |
| a5de12db-7bef-469c-aac6-d1b2c109191e | vm1-disk3 | in-use | 2 | VSA-r5 | false | Attached to vm1 on /dev/vdd | attached_mode='rw', readonly='False' |
| 4986d4a8-e4ae-4691-b522-16cf34458d33 | vm1-disk2 | in-use | 2 | ceph | false | Attached to vm1 on /dev/vdc | attached_mode='rw', readonly='False' |
| a554a4ed-4893-4f7c-b19d-e6feff7683b7 | vm1-disk1 | in-use | 2 | VSA-r5 | false | Attached to vm1 on /dev/vdb | attached_mode='rw', readonly='False' |
| 83b0dad4-fd8d-4847-a50c-3efc310384d9 | vm1-boot | in-use | 8 | VSA-r1 | true | Attached to vm1 on /dev/vda | attached_mode='rw', readonly='False' |
+--------------------------------------+--------------+-----------+------+--------+----------+------------------------------+--------------------------------------+
stack@hos5to8-cp1-c1-m1-mgmt:~$ openstack snapshot list --all-projects --long
+--------------------------------------+-----------------+-------------+-----------+------+----------------------------+-----------+------------+
| ID | Name | Description | Status | Size | Created At | Volume | Properties |
+--------------------------------------+-----------------+-------------+-----------+------+----------------------------+-----------+------------+
| 222cb290-3a2e-4f94-97ec-f18d814b9944 | vm1-disk3-snap1 | None | available | 2 | 2018-05-31T15:28:13.000000 | vm1-disk3 | |
+--------------------------------------+-----------------+-------------+-----------+------+----------------------------+-----------+------------+
stack@hos5to8-cp1-c1-m1-mgmt:~$ openstack server list --all-projects --long
+--------------------------------------+------+--------+------------+-------------+------------------------------+---------------------+--------------------------------------+-------------------+---------------------------+------------+
| ID | Name | Status | Task State | Power State | Networks | Image Name | Image ID | Availability Zone | Host | Properties |
+--------------------------------------+------+--------+------------+-------------+------------------------------+---------------------+--------------------------------------+-------------------+---------------------------+------------+
| 1c9b3acb-58f5-4973-8f5d-b088c726fd6b | vm2 | ACTIVE | None | Running | lan=10.10.10.5, 172.16.122.4 | cirros-0.3.4-x86_64 | 033e6158-7813-4af8-9796-3f954b001303 | nova | hos5to8-cp1-comp0001-mgmt | |
| baff0a89-995c-4fcf-88ee-d39206a36035 | vm1 | ACTIVE | None | Running | lan=10.10.10.8, 172.16.122.3 | | | nova | hos5to8-cp1-comp0001-mgmt | |
+--------------------------------------+------+--------+------------+-------------+------------------------------+---------------------+--------------------------------------+-------------------+---------------------------+------------+
stack@hos5to8-cp1-c1-m1-mgmt:~$
We activate the openstackclient virtualenv and run the planner:
stack@hos5to8-cp1-c1-m1-mgmt:~$ . /opt/stack/service/openstackclient/venv/bin/activate
(openstackclient-20180403T122416Z) stack@hos5to8-cp1-c1-m1-mgmt:~$ python ~/hos-ceph-migration/migration_planner.py VSA-r1=ses VSA-r5=ses
# Migration plan for the following volume types:
# VSA-r1 -> ses
# VSA-r5 -> ses
# Migrating detached volumes
cinder retype --migration-policy on-demand 1b199df7-3273-40eb-b705-8902e8cca727 ses
# Monitor migration process using: openstack volume show 1b199df7-3273-40eb-b705-8902e8cca727 | grep migration_status
# WARNING: volume a5de12db-7bef-469c-aac6-d1b2c109191e has snapshot(s): removing
openstack snapshot delete 222cb290-3a2e-4f94-97ec-f18d814b9944
# Migrating attached volumes
# Migrating volumes attached to instance baff0a89-995c-4fcf-88ee-d39206a36035 (vm1)
# WARNING: instance baff0a89-995c-4fcf-88ee-d39206a36035 is running: shutting down
openstack server stop baff0a89-995c-4fcf-88ee-d39206a36035
# WARNING: patching boot volume info for volume 83b0dad4-fd8d-4847-a50c-3efc310384d9
echo "update block_device_mapping set boot_index=999 where deleted=0 and volume_id='83b0dad4-fd8d-4847-a50c-3efc310384d9' and boot_index=0" | sudo mysql nova
openstack server remove volume baff0a89-995c-4fcf-88ee-d39206a36035 83b0dad4-fd8d-4847-a50c-3efc310384d9
openstack server remove volume baff0a89-995c-4fcf-88ee-d39206a36035 a554a4ed-4893-4f7c-b19d-e6feff7683b7
openstack server remove volume baff0a89-995c-4fcf-88ee-d39206a36035 4986d4a8-e4ae-4691-b522-16cf34458d33
openstack server remove volume baff0a89-995c-4fcf-88ee-d39206a36035 a5de12db-7bef-469c-aac6-d1b2c109191e
cinder retype --migration-policy on-demand 83b0dad4-fd8d-4847-a50c-3efc310384d9 ses
# Monitor migration process using: openstack volume show 83b0dad4-fd8d-4847-a50c-3efc310384d9 | grep migration_status
cinder retype --migration-policy on-demand a554a4ed-4893-4f7c-b19d-e6feff7683b7 ses
# Monitor migration process using: openstack volume show a554a4ed-4893-4f7c-b19d-e6feff7683b7 | grep migration_status
cinder retype --migration-policy on-demand a5de12db-7bef-469c-aac6-d1b2c109191e ses
# Monitor migration process using: openstack volume show a5de12db-7bef-469c-aac6-d1b2c109191e | grep migration_status
openstack server add volume baff0a89-995c-4fcf-88ee-d39206a36035 83b0dad4-fd8d-4847-a50c-3efc310384d9
openstack server add volume baff0a89-995c-4fcf-88ee-d39206a36035 a554a4ed-4893-4f7c-b19d-e6feff7683b7
openstack server add volume baff0a89-995c-4fcf-88ee-d39206a36035 4986d4a8-e4ae-4691-b522-16cf34458d33
openstack server add volume baff0a89-995c-4fcf-88ee-d39206a36035 a5de12db-7bef-469c-aac6-d1b2c109191e
echo "update block_device_mapping set boot_index=0 where deleted=0 and volume_id='83b0dad4-fd8d-4847-a50c-3efc310384d9' and boot_index is NULL" | sudo mysql nova
openstack server start baff0a89-995c-4fcf-88ee-d39206a36035
# Migrating volumes attached to instance 1c9b3acb-58f5-4973-8f5d-b088c726fd6b (vm2)
# WARNING: instance 1c9b3acb-58f5-4973-8f5d-b088c726fd6b is running: shutting down
openstack server stop 1c9b3acb-58f5-4973-8f5d-b088c726fd6b
openstack server remove volume 1c9b3acb-58f5-4973-8f5d-b088c726fd6b 3fb9d86b-cea7-4508-ba44-6099e6c088e0
cinder retype --migration-policy on-demand 3fb9d86b-cea7-4508-ba44-6099e6c088e0 ses
# Monitor migration process using: openstack volume show 3fb9d86b-cea7-4508-ba44-6099e6c088e0 | grep migration_status
openstack server add volume 1c9b3acb-58f5-4973-8f5d-b088c726fd6b 3fb9d86b-cea7-4508-ba44-6099e6c088e0
openstack server start 1c9b3acb-58f5-4973-8f5d-b088c726fd6b
# Migration completed: 5 volumes, 16GB data, 2 instances
(openstackclient-20180403T122416Z) stack@hos5to8-cp1-c1-m1-mgmt:~$
After following the plan, this is the state of the environment:
(openstackclient-20180403T122416Z) stack@hos5to8-cp1-c1-m1-mgmt:~$ openstack volume list --all-projects --long
+--------------------------------------+--------------+-----------+------+------+----------+------------------------------+--------------------------------------+
| ID | Display Name | Status | Size | Type | Bootable | Attached to | Properties |
+--------------------------------------+--------------+-----------+------+------+----------+------------------------------+--------------------------------------+
| 1b199df7-3273-40eb-b705-8902e8cca727 | unused | available | 2 | ses | false | | |
| 3fb9d86b-cea7-4508-ba44-6099e6c088e0 | vm2-disk1 | in-use | 2 | ses | false | Attached to vm2 on /dev/vdb | attached_mode='rw', readonly='False' |
| a5de12db-7bef-469c-aac6-d1b2c109191e | vm1-disk3 | in-use | 2 | ses | false | Attached to vm1 on /dev/vde | attached_mode='rw', readonly='False' |
| 4986d4a8-e4ae-4691-b522-16cf34458d33 | vm1-disk2 | in-use | 2 | ceph | false | Attached to vm1 on /dev/vdd | attached_mode='rw', readonly='False' |
| a554a4ed-4893-4f7c-b19d-e6feff7683b7 | vm1-disk1 | in-use | 2 | ses | false | Attached to vm1 on /dev/vdc | attached_mode='rw', readonly='False' |
| 83b0dad4-fd8d-4847-a50c-3efc310384d9 | vm1-boot | in-use | 8 | ses | true | Attached to vm1 on /dev/vdb | attached_mode='rw', readonly='False' |
+--------------------------------------+--------------+-----------+------+------+----------+------------------------------+--------------------------------------+
(openstackclient-20180403T122416Z) stack@hos5to8-cp1-c1-m1-mgmt:~$ openstack snapshot list --all-projects --long
(openstackclient-20180403T122416Z) stack@hos5to8-cp1-c1-m1-mgmt:~$ openstack server list --all-projects --long
+--------------------------------------+------+--------+------------+-------------+------------------------------+---------------------+--------------------------------------+-------------------+---------------------------+------------+
| ID | Name | Status | Task State | Power State | Networks | Image Name | Image ID | Availability Zone | Host | Properties |
+--------------------------------------+------+--------+------------+-------------+------------------------------+---------------------+--------------------------------------+-------------------+---------------------------+------------+
| 1c9b3acb-58f5-4973-8f5d-b088c726fd6b | vm2 | ACTIVE | None | Running | lan=10.10.10.5, 172.16.122.4 | cirros-0.3.4-x86_64 | 033e6158-7813-4af8-9796-3f954b001303 | nova | hos5to8-cp1-comp0001-mgmt | |
| baff0a89-995c-4fcf-88ee-d39206a36035 | vm1 | ACTIVE | None | Running | lan=10.10.10.8, 172.16.122.3 | | | nova | hos5to8-cp1-comp0001-mgmt | |
+--------------------------------------+------+--------+------------+-------------+------------------------------+---------------------+--------------------------------------+-------------------+---------------------------+------------+
(openstackclient-20180403T122416Z) stack@hos5to8-cp1-c1-m1-mgmt:~$
Running the script again, will confirm the migration was successful:
(openstackclient-20180403T122416Z) stack@hos5to8-cp1-c1-m1-mgmt:~$ python ~/hos-ceph-migration/migration_planner.py VSA-r1=ses VSA-r5=ses
# Migration plan for the following volume types:
# VSA-r1 -> ses
# VSA-r5 -> ses
# Migrating detached volumes
# Migrating attached volumes
# Migration completed: 0 volumes, 0GB data, 0 instances
(openstackclient-20180403T122416Z) stack@hos5to8-cp1-c1-m1-mgmt:~$
Glance images stored on Ceph can be migrated to SES by exporting the contents
of the images pool and reimporting in the same pool in the SES cluster.
A simple script is provided. It takes source and destination cluster and moves
the rbd images across, for example:
sudo ./migrate-images.sh ceph ses
The script requires root privileges to access keyring files and to manipulate the glance mysql database.
Cinder volume backups stored on Ceph can be migrated to SES by exporting the
contents of the backups pool and reimporting in the same pool in the SES
cluster. A simple script is provided. It takes source and destination cluster
and moves the rbd images across, for example:
sudo ./migrate-backups.sh ceph ses
The script requires root privileges to access keyring files.
On both the HOS5 ceph and SES clusters, run:
radosgw-admin user create --system --uid=migration --display-name="HOS5-HOS8 migration"
It will print a JSON structure containing the details of the newly created
user. Take note of the access_key and secret_key values from the keys
field.
On one of the HOS5 radosgw nodes run:
iptables -I INPUT -m tcp -p tcp --dport 7480 -j ACCEPT
The object migration script depends on a python module which is not included in any of the HOS5 virtualenvs. To create a suitable virtualenv:
virtualenv ~/radosgw-venv
. ~/radosgw-venv/bin/activate
pip install radosgw-admin python-swiftclient boto click
From the same HOS5 radosgw node run the migration script:
./rgw-migrate.py ip_from:port_from:access_key_from:secret_key_from ip_to:port_to:access_key_to:secret_key_to
Where:
- ip_from is the IP of the management interface of the HOS5 radosgw node
- port_from is the TCP port where the source radosgw is listening (7480)
- access_key_from and secret_key_from are the credentials for the radosgw
admin user on the HOS5 Ceph cluster
- ip_to is the IP of the SES radosgw node
- port_to is the TCP port where the destination radosgw is listening (80 by
default)
- access_key_to and secret_key_to are the credentials for the radosgw
admin user on the SES cluster
Copy the rgw lines from the HOS5’s ceph config file and paste them in the SES’ ceph config file in each of the sections reserved for the radosgw daemon. This is an example of the fields:
rgw keystone url = https://hos5to8-cp1-vip-KEY-API-mgmt:5000
rgw keystone accepted roles = admin, _member_
rgw keystone admin user = radosgw
rgw keystone admin password = I18xNEJD
rgw keystone admin tenant = services
rgw s3 auth use keystone = False
Make sure the SES nodes can resolve the keystone endpoint address, for example
by adding a new entry in /etc/hosts.
TLS certificates from HOS5 must also be configured on the SES cluster for
keystone authentication to work: copy /etc/ssl/certs/helion*.pem from one of
the HOS5 controllers to /etc/pki/trust/anchors/ on the SES nodes and, as
root, run:
update-ca-certificates
. ~/keystone.osrc
openstack endpoint list|grep -i ceph-object-store | awk '{print $2}' | xargs openstack endpoint delete
for ep in admin internal public ; do openstack endpoint create --region region1 ceph-rgw $ep http://172.16.110.11:80/swift/v1 ; done
Where 172.16.110.11:80 must be replaced with the IP and port where the object gateway service from the SES cluster is exposed.
On both the HOS5 ceph and SES clusters, run:
radosgw-admin user rm --uid=migration
On the HOS5 radosgw node we used for the migration run:
iptables -D INPUT -m tcp -p tcp --dport 7480 -j ACCEPT
Once all volumes have been successfully migrated, the old backend can be
removed from the cinder.conf.j2. Make sure default_volume_type is
not pointing to the old backend.
When migrating from a HLM-deployed backend like VSA or Ceph, the storage nodes must be removed from the control plane before starting the HOS upgrade process.
Also make sure to remove the ceph-monitor and ceph-radosgw service
components from the control plane and disable and stop the monitor and gateway
services if they are running on controller nodes.
Once the input model has been updated, the final configuration can be deployed:
cd ~/helion/hos/ansible/
git commit -am "Remove old cinder backend"
ansible-playbook -i hosts/localhost config-processor-run.yml
ansible-playbook -i hosts/localhost ready-deployment.yml
cd ~/scratch/ansible/next/hos/ansible/
ansible-playbook cinder-reconfigure.yml # or hlm-reconfigure.yml if the control plane was changed