Error deleting stack and disk remain
- By Lehbot
- Wed 02 August 2017
In this case i tried to delete a stack and on deletion disks got stuck, while all instances have been cleared.
+--------------------------------------+------------------+-----------+------+---------------------------------------------------------------+
| ID | Display Name | Status | Size | Attached to |
+--------------------------------------+------------------+-----------+------+---------------------------------------------------------------+
| 4abd0cc4-2c39-fooo-93c3-01f8bad02498 | Disk-Server4 | in-use | 50 | Attached to a010fa48-ff6f-46b8-bda8-d0c7fe7304b5 on /dev/vdb |
| fc7f63fa-c95b-fooo-b571-efa73aa49e69 | Disk-Server2 | available | 50 | |
| 5e239dbc-de6b-fooo-be6f-535d7556c8e0 | Disk-Server3 | deleting | 50 | |
| 8a5c867e-dc19-fooo-8bf7-533a5dff5557 | Disk-Server1 | deleting | 50 | |
| 7136ac76-f14f-fooo-90d6-cebe40f82451 | Disk-Server5 | available | 50 | |
+--------------------------------------+------------------+-----------+------+---------------------------------------------------------------+
As you can see we have two disks pending in deleting state and one disk attached to an instance which does not exist anymore.
openstack stack list
+--------------------------------------+-----------------+-----------------+---------------------+---------------------+
| ID | Stack Name | Stack Status | Creation Time | Updated Time |
+--------------------------------------+-----------------+-----------------+---------------------+---------------------+
| 8bb09339-4792-fooo-9e28-30d3fd7a9619 | ergrehg | DELETE_FAILED | 2017-01-11T10:14:44 | 2017-01-12T10:58:40 |
+--------------------------------------+-----------------+-----------------+---------------------+---------------------+
openstack stack resource list ergrehg -n 5
+--------------------------+--------------------------------------+--------------------------+-----------------+---------------------+---------------------------------------+
| resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name |
+--------------------------+--------------------------------------+--------------------------+-----------------+---------------------+---------------------------------------+
| enable_docker | | OS::Heat::SoftwareConfig | DELETE_COMPLETE | 2017-01-12T10:58:40 | ergrehg |
| configure_sudoers | | OS::Heat::SoftwareConfig | DELETE_COMPLETE | 2017-01-12T10:58:40 | ergrehg |
| server_init | | OS::Heat::MultipartMime | DELETE_COMPLETE | 2017-01-12T10:58:40 | ergrehg |
| subscription_manager | | OS::Heat::SoftwareConfig | DELETE_COMPLETE | 2017-01-12T10:58:40 | ergrehg |
| BasicNodePortRSG | | OS::Heat::ResourceGroup | DELETE_COMPLETE | 2017-01-12T10:58:40 | ergrehg |
| configure_docker_storage | | OS::Heat::SoftwareConfig | DELETE_COMPLETE | 2017-01-12T10:58:40 | ergrehg |
| applications | | OS::Heat::SoftwareConfig | DELETE_COMPLETE | 2017-01-12T10:58:40 | ergrehg |
| BasicNodeDiskRSG | 916edf60-4092-fooo-b4aa-9ac3c9b3c979 | OS::Heat::ResourceGroup | DELETE_FAILED | 2017-01-12T10:58:42 | ergrehg |
| 1 | 8a5c867e-dc19-fooo-8bf7-533a5dff5557 | OS::Cinder::Volume | DELETE_FAILED | 2017-01-11T17:01:27 | ergrehg-BasicNodeDiskRSG-ipczrfokpdlg |
| 0 | 311f81ca-e106-fooo-87ce-c86262173d02 | OS::Cinder::Volume | DELETE_FAILED | 2017-01-11T17:01:27 | ergrehg-BasicNodeDiskRSG-ipczrfokpdlg |
| 3 | 5e239dbc-de6b-fooo-be6f-535d7556c8e0 | OS::Cinder::Volume | DELETE_FAILED | 2017-01-11T17:01:27 | ergrehg-BasicNodeDiskRSG-ipczrfokpdlg |
| 2 | fc7f63fa-c95b-fooo-b571-efa73aa49e69 | OS::Cinder::Volume | DELETE_FAILED | 2017-01-11T17:01:27 | ergrehg-BasicNodeDiskRSG-ipczrfokpdlg |
| 5 | 7136ac76-f14f-fooo-90d6-cebe40f82451 | OS::Cinder::Volume | DELETE_FAILED | 2017-01-11T17:01:27 | ergrehg-BasicNodeDiskRSG-ipczrfokpdlg |
| 4 | 4abd0cc4-2c39-fooo-93c3-01f8bad02498 | OS::Cinder::Volume | DELETE_FAILED | 2017-01-11T17:01:27 | ergrehg-BasicNodeDiskRSG-ipczrfokpdlg |
| 6 | f22824bc-71b2-fooo-8891-88feb485dabb | OS::Cinder::Volume | DELETE_FAILED | 2017-01-11T17:01:27 | ergrehg-BasicNodeDiskRSG-ipczrfokpdlg |
| BasicNodeServerRSG | | OS::Heat::ResourceGroup | DELETE_COMPLETE | 2017-01-12T10:58:40 | ergrehg |
| OS_basic_config | | OS::Heat::SoftwareConfig | DELETE_COMPLETE | 2017-01-12T10:58:40 | ergrehg |
+--------------------------+--------------------------------------+--------------------------+-----------------+---------------------+---------------------------------------+
cinder reset-state --state available --attach-status detached 8a5c867e-dc19-fooo-8bf7-533a5dff5557
cinder reset-state --state available --attach-status detached 4abd0cc4-2c39-fooo-93c3-01f8bad02498
cinder reset-state --state available --attach-status detached 5e239dbc-de6b-fooo-be6f-535d7556c8e0
If you now try to delete these disks and come again into this status then consider doing the following.
!!! Make sure just the disks from this stack are remaining. !!!
Scenario: Disks are stored on a storage system via NFS protocol.
Log on to your controller Node and figure out your mountpoint by issuing mount
On my system it is: /var/lib/cinder/mnt/a652d6730031a8fooooo0b84fb31a898
List your disks and you will get a list with the corresponding Disk ID in its name.
Now delete them !
Now we can delete the entries from the database:
First take a look at the values:
mysql -e "select * from cinder.volumes where id='4abd0cc4-2c39-fooo-93c3-01f8bad02498'\G;"
Then issue the command and replace the id property with the id from your disk like done before:
mysql -u root -e "update cinder.volumes set deleted='1',status='deleted',attach_status='detached',deleted_at=now(),provider_location=NULL,volume_type_id=NULL where id='4abd0cc4-2c39-fooo-93c3-01f8bad02498'";
After deleting the remaining disk entries issue the command: openstack stack delete ergrehg --yes
Et voila the stack should be gone !
Even if this situation doesn't suite you, maybe you will find it useful for your problem and will get an idea how Openstack is build inside.