Docker Swarm HA with GlusterFS

A good design pattern for highly available applications is to deploy the application as a container on a Docker Swarm cluster with persistent storage provided by GlusterFS. GlusterFS is a fast shared filesystem that can keep the container volume in sync between multiple VMs running the Docker Swarm cluster. This pattern ensures high availability for your containerised application. In the event a VM dies, Docker Swarm will spin up the container on another VM. GlusterFS will ensure the container has access to the same data when it comes up.

In this tutorial, we’ll look at setting up GlusterFS on 3 VMs and create a replicated volume with a replication factor of 3. Later we’ll install Docker Swarm over these three VMs. Goal is to use GlusterFS to provide persistent storage to your application container, and docker swarm for high availability.

1. Plan and setup the infrastructure

For the setup, first we’ll need three Ubuntu 16.04 gluster VMs, each with 2 disks attached. We’ll use the first disk to run the OS, and the second as the GlusterFS replicated volume. Create three VMs with two disks. In my case, my VMs had the root volume on /dev/vda and the second disk on /dev/vdc. Create three VMs and let’s assume the private IPs of these VMs are 192.168.2.100, 192.168.2.101, 192.168.2.102, and their hostnames are gluster1, gluster2, gluster3.

Note: All commands are being executed as root user (hence the # at the beginning)

# lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    253:0    0  30G  0 disk
└─vda1 253:1    0  30G  0 part /
vdb    253:16   0  64M  0 disk 
vdc    253:32   0  10G  0 disk

Update the /etc/hosts files on each VM to reflect the private IPs of each VM. This is important for GlusterFS, and you may encounter bugs or issues if you give private IPs directly to Gluster volumes. After editing the files should look like:

(gluster1)# cat /etc/hosts
127.0.0.1       localhost
192.168.2.100   gluster1
192.168.2.101   gluster2
192.168.2.102   gluster3(gluster2)# cat /etc/hosts
127.0.0.1       localhost
192.168.2.100   gluster1
192.168.2.101   gluster2
192.168.2.102   gluster3(gluster3)# cat /etc/hosts
127.0.0.1       localhost
192.168.2.100   gluster1
192.168.2.101   gluster2
192.168.2.102   gluster3

Format the disks with xfs filesystem on each VM in case you haven’t already. You can also use ext4 if you prefer.

# mkfs.xfs /dev/vdc

2. Create directories for GlusterFS storage

Setup the glusterFS directories where the gluster “bricks” will reside. Better to name them differently so it’s easy to identify on which node the replicated volumes reside. Also add an entry to your /etc/fstab file on each VM so that our brick gets mounted when the operating system boots or restarts.

(gluster1)# mkdir -p /gluster/bricks/1
(gluster1)# echo '/dev/vdc /gluster/bricks/1 xfs defaults 0 0' >> /etc/fstab
(gluster1)# mount -a
(gluster1)# mkdir /gluster/bricks/1/brick(gluster2)# mkdir -p /gluster/bricks/2
(gluster2)# echo '/dev/vdc /gluster/bricks/2 xfs defaults 0 0' >> /etc/fstab
(gluster2)# mount -a
(gluster2)# mkdir /gluster/bricks/2/brick(gluster3)# mkdir -p /gluster/bricks/3
(gluster3)# echo '/dev/vdc /gluster/bricks/3 xfs defaults 0 0' >> /etc/fstab
(gluster3)# mount -a
(gluster3)# mkdir /gluster/bricks/3/brick

3. Install GlusterFS

Install GlusterFS on all VMs by executing following commands on each VM:

# apt-get -y update && apt-get -y upgrade
# apt-get install -y software-properties-common
# add-apt-repository ppa:gluster/glusterfs-6 && apt-get update # Use the latest glusterFS version instead of 6, which was the latest at the time of writing this tutorial
# apt-get install -y glusterfs-server
# systemctl enable glusterd # automatically start glusterfs on boot
# systemctl start glusterd # start glusterfs right now
# systemctl status glusterd # Should show status active

4. Peer with other Gluster VMs

Now peer with other nodes from gluster1:

(gluster1)# gluster peer probe gluster2
peer probe: success.
(gluster1)# gluster peer probe gluster3
peer probe: success.
(gluster1)# gluster peer status
Number of Peers: 2Hostname: gluster2
Uuid: 60861905-6adc-4841-8f82-216c661f9fe2
State: Peer in Cluster (Connected)Hostname: gluster3
Uuid: 572fed90-61de-40dd-97a6-4255ed8744ce
State: Peer in Cluster (Connected)

5. Setup the Gluster “replicated volume”

GlusterFS has multiple volume types. For our HA architecture, we want to setup a “replicated” volume that stores the files created on each of the 3 VMs and hence the file is available to any app or container running on these VMs. Create the replicated volume named “gfs” with 3 replicas:

(gluster1)# gluster volume create gfs \
replica 3 \
gluster1:/gluster/bricks/1/brick \
gluster2:/gluster/bricks/2/brick \
gluster3:/gluster/bricks/3/brickvolume create: gfs: success: please start the volume to access data(gluster1)# gluster volume start gfs
(gluster1)# gluster volume status gfsStatus of volume: gfs
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick gluster1:/gluster/bricks/1/brick      49152     0          Y       4619
Brick gluster2:/gluster/bricks/2/brick      49152     0          Y       4504
Brick gluster3:/gluster/bricks/3/brick      49152     0          Y       4306
Self-heal Daemon on localhost               N/A       N/A        Y       4641
Self-heal Daemon on gluster2                N/A       N/A        Y       4526
Self-heal Daemon on gluster3                N/A       N/A        Y       4328Task Status of Volume gfs
------------------------------------------------------------------------------
There are no active volume tasks(gluster1)# gluster volume info gfsVolume Name: gfs
Type: Replicate
Volume ID: 703e46cb-a637-4620-adfa-6b292a15e0d5
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: gluster1:/gluster/bricks/1/brick
Brick2: gluster2:/gluster/bricks/2/brick
Brick3: gluster3:/gluster/bricks/3/brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

6. Setup security and authentication for this volume

GlusterFS will allow any clients to connect to volumes by default. However, you will need to authorize the three infra VMs running GlusterFS to connect to the GlusterFS Volumes on each node. You can do it by authorizing the private IPs of each VM to connect to the volume. This will allow replication to happen. Execute:

(gluster1)# gluster volume set gfs auth.allow 192.168.2.100,192.168.2.101,192.168.2.102

7. Mount the glusterFS volume where applications can access the files

We’ll mount the volume onto /mnt on each VM, and also append it to our /etc/fstab file so that it mounts on boot:

(gluster1)# echo 'localhost:/gfs /mnt glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab
(gluster1)# mount.glusterfs localhost:/gfs /mnt(gluster2)# echo 'localhost:/gfs /mnt glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab
(gluster2)# mount.glusterfs localhost:/gfs /mnt(gluster3)# echo 'localhost:/gfs /mnt glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab
(gluster3)# mount.glusterfs localhost:/gfs /mnt

8. Verify

Verify mounted glusterfs volume:

# df -Th
Filesystem     Type            Size  Used Avail Use% Mounted on
udev           devtmpfs        7.9G     0  7.9G   0% /dev
tmpfs          tmpfs           1.6G   17M  1.6G   2% /run
/dev/vda1      ext4             30G  2.1G   27G   8% /
tmpfs          tmpfs           7.9G   12K  7.9G   1% /dev/shm
tmpfs          tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs          tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
tmpfs          tmpfs           1.6G     0  1.6G   0% /run/user/1001
/dev/vdb       xfs              10G   33M   10G   1% /gluster/bricks/1
localhost:/gfs fuse.glusterfs   10G  135M   10G   2% /mnt

The total space available on the volume comes up as 10G even though we have 3 disks of 10G each connected to GlusterFS. This is due to our replication factor of 3. Total volume size is 30G, but with a replication factor or 3 for each file only 10G is available to us.

Test GlusterFS replication:

(gluster1)# echo "Hello World!" | sudo tee /mnt/test.txt(gluster2)# cat /mnt/test.txt
Hello World!(gluster3)# cat /mnt/test.txt
Hello World!

Part 2: Setup Docker Swarm

Now let’s setup the Docker Swarm cluster with the gluster VMs (gluster1/2/3) as the workers, and a new VM (swarm-manger) as the Swarm manager. We’ll use our gluster replicated volume to achieve High Availability for our stateful containerized application. We’ll test with WordPress.

All commands executed as root.

1. Setup Docker community edition on all VMs

Install docker-ce on all four VMs (swarm-manager, gluster1/2/3) using the instructions given here: https://docs.docker.com/install/linux/docker-ce/ubuntu/ (I feel it’s redundant to repeat the standard instructions).

However, after the installation, please do verify if Docker is installed properly by running following command on all VMs:

# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete
Digest: sha256:92695bc579f31df7a63da6922075d0666e565ceccad16b59c3374d2cf4e8e50e
Status: Downloaded newer image for hello-world:latestHello from Docker!
This message shows that your installation appears to be working correctly.

2. Initialize Docker swarm from the swarm-manager

We’ll use the swarm-manager’s private IP as the “advertised address”.

swarm-manager:~# docker swarm init --advertise-addr 192.168.2.99
Swarm initialized: current node (sz42o1yjz08t3x98aj82z33pe) is now a manager.To add a worker to this swarm, run the following command:docker swarm join --token SWMTKN-1-3gi2wi4o22nyiqij3io055na7wt0201oamaegykllea0t5vi5k-2qjld08v7ouzax6gzw15dw2ab 192.168.2.99:2377To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

3. Add the three gluster VMs as swarm workers

gluster1:~# docker swarm join --token SWMTKN-1-3gi2wi4o22nyiqij3io055na7wt0201oamaegykllea0t5vi5k-2qjld08v7ouzax6gzw15dw2ab 192.168.2.99:2377
This node joined a swarm as a worker.gluster2:~# docker swarm join --token SWMTKN-1-3gi2wi4o22nyiqij3io055na7wt0201oamaegykllea0t5vi5k-2qjld08v7ouzax6gzw15dw2ab 192.168.2.99:2377
This node joined a swarm as a worker.gluster3:~# docker swarm join --token SWMTKN-1-3gi2wi4o22nyiqij3io055na7wt0201oamaegykllea0t5vi5k-2qjld08v7ouzax6gzw15dw2ab 192.168.2.99:2377
This node joined a swarm as a worker.swarm-manager:~# docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
qjmuz0n8n770ryougk2tsb37x     gluster1            Ready               Active                                  18.09.5
kcwsavrtzhvy038357p51lwl2     gluster2            Ready               Active                                  18.09.5
ifnzgpk25p27y19vslee4v74x     gluster3            Ready               Active                                  18.09.5
sz42o1yjz08t3x98aj82z33pe *   swarm-manager       Ready               Active              Leader              18.09.5

Part 3: Test the High Availability Setup

We’ll use docker stack to setup a single container WordPress backed by a single container of MySQL, and then test if this setup is resilient to VM failure.

1. Create replicated directories for wordpress and mysql in glusterFS

gluster1:~# mkdir /mnt/wp-content
gluster1:~# mkdir /mnt/mysql

2. Create the wordpress-stack.yml file

This stack file exposes wordpress on port 8080 on all swarm nodes, even the swarm-manager node. It mounts the directories created for wp-content and mysql as volumes on the containers.

swarm-manager:~# cat wordpress-stack.yml
# wordpress-stack.yml
version: '3.1'services:  wordpress:
    image: wordpress
    restart: always
    ports:
      - 8080:80
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
    volumes:
      - "/mnt/wp-content:/var/www/html/wp-content"
    deploy:
      placement:
        constraints: [node.role == worker]  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_RANDOM_ROOT_PASSWORD: '1'
    volumes:
      - "/mnt/mysql:/var/lib/mysql"
    deploy:
      placement:
        constraints: [node.role == worker]

3. Use docker stack to deploy WordPress and MySQL

swarm-manager:~# docker stack deploy -c wordpress-stack.yml wordpress
Ignoring unsupported options: restartCreating network wordpress_default
Creating service wordpress_db
Creating service wordpress_wordpressswarm-manager:~# docker stack ps wordpress
ID                  NAME                    IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR               PORTS
x5vvrt6ohko2        wordpress_db.1          mysql:5.7           gluster2            Running             Running 5 minutes ago
idree9r7qlxb        wordpress_wordpress.1   wordpress:latest    gluster1            Running             Running 5 minutes ago

Check if WordPress is up by entering http://<any-worker-external-ip&gt;:8080/ in the browser.

Note: 10.147.106.32 was one of my gluster worker VM’s (gluster3) external IP

Go through the install process, choose an admin username and password, and create your first post.

4. Test High Availability by shutting down a VM

Check on which VM the WordPress and MySQL containers are running. We’ll shutdown each VM to understand whether HA is working properly. In my case, the WordPress container was running on gluster1 and MySQL was running on gluster2.

swarm-manager:~# docker stack ps wordpress
ID                  NAME                    IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
x5vvrt6ohko2        wordpress_db.1          mysql:5.7           gluster2            Running             Running 24 minutes ago
idree9r7qlxb        wordpress_wordpress.1   wordpress:latest    gluster1            Running             Running 24 minutes ago

Shutdown gluster1 and check what happens. You’ll find that docker swarm starts a new container on a new worker VM. The website will continue to work, your data will still be stored, but you’ll have to login again as the session data is lost with the previous container.

swarm-manager:~# docker stack ps wordpress
ID                  NAME                    IMAGE               NODE                DESIRED STATE       CURRENT STATE                ERROR               PORTS
u8s93kowj2mx        wordpress_wordpress.1   wordpress:latest    gluster3            Running             Running 3 seconds ago
x5vvrt6ohko2        wordpress_db.1          mysql:5.7           gluster2            Running             Running 28 minutes ago
idree9r7qlxb        wordpress_wordpress.1   wordpress:latest    gluster1            Shutdown            Running about a minute ago

Start the gluster1 VM again and let’s repeat the HA test with MySQL host gluster2. Shutdown gluster2 which was running the MySQL container. After shutdown, you’ll find docker swarm has scheduled MySQL on another worker VM.

swarm-manager:~# docker stack ps wordpress
ID                  NAME                    IMAGE               NODE                DESIRED STATE       CURRENT STATE              ERROR               PORTS
px90rs5q22ei        wordpress_db.1          mysql:5.7           gluster1            Running             Preparing 41 seconds ago
u8s93kowj2mx        wordpress_wordpress.1   wordpress:latest    gluster3            Running             Running 6 minutes ago
x5vvrt6ohko2        wordpress_db.1          mysql:5.7           gluster2            Shutdown            Running 50 seconds ago
idree9r7qlxb        wordpress_wordpress.1   wordpress:latest    gluster1            Shutdown            Shutdown 3 minutes ago

The website will continue to work without any data loss as the MySQL container would have found the replicated volume under the same path (/mnt/mysql).

Add the three worker VM IPs with port behind a Load Balancer (like AWS ELB) and voilà, A Highly Available stateful deployment on Docker Swarm using GlusterFS.

Upgrade Kubernetes Cluster to v1.16 using Kubeadm

Kubernetes has released latest version 1.16 on Oct 22, 2019. Refer release notes v1.16

In this article we will see how safely upgrade step by step using Kubeadm

 

Before you begin

  1. Backup Kubernetes cluster using Heptio Velero explained in my previous blog
  2. Backup app-level state stored in database, kubeadm-upgrade doesn’t touch workloads

Upgrade Kubernetes Control Plane node (Master)

Common steps followed for upgrade like drain, plan and apply

  1. Install latest kubeadm v1.16
# replace x in 1.16.x-00 with the latest patch version
apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm=1.16.x-00 && \
apt-mark hold kubeadm

 

2. Verify latest version

kubeadm version

3. Drain the control plane node

kubectl drain $MASTER --ignore-daemonsets

4. Plan for upgrading control plane node

sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.15.2
[upgrade/versions] kubeadm version: v1.16.0

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       AVAILABLE
Kubelet     1 x v1.15.2   v1.16.0

Upgrade to the latest version in the v1.16 series:

COMPONENT            CURRENT   AVAILABLE
API Server           v1.15.2   v1.16.0
Controller Manager   v1.15.2   v1.16.0
Scheduler            v1.15.2   v1.16.0
Kube Proxy           v1.15.2   v1.16.0
CoreDNS              1.3.1     1.6.2
Etcd                 3.3.10    3.3.15

Note: It will upgrade appropriate etcd version

5. Apply upgrade

sudo kubeadm upgrade apply v1.16.0
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.16.0". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

6. Start Scheduler on control plane node

kubectl uncordon $MASTER

Upgrade Kubelet and Kubectl

# replace x in 1.16.x-00 with the latest patch version
apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet=1.16.x-00 kubectl=1.16.x-00 && \
apt-mark hold kubelet kubectl

Restart kubelet

sudo systemctl restart kubelet

Upgrade Worker Nodes

The upgrade procedure on worker nodes should be executed one node at a time or few nodes at a time, without compromising the minimum required capacity for running your workloads

  • Upgrade Kubeadm as we did on Master Node
# replace x in 1.16.x-00 with the latest patch version
apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm=1.16.x-00 && \
apt-mark hold kubeadm
  • Drain node for maintenance by marking it unschedulable and evicting the workloads
kubectl drain $NODE1 --ignore-daemonsets

You should see output similar to this:

node1 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-dj7d7, kube-system/weave-net-z65qx
node1 drained
  •   Upgrade Kubeadm on worker node

sudo kubeadm upgrade node

Upgrade the kubelet and kubectl on all worker nodes:

# replace x in 1.16.x-00 with the latest patch version
apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet=1.16.x-00 kubectl=1.16.x-00 && \
apt-mark hold kubelet kubectl
  • Restart kubelet service
sudo systemctl restart kubelet
  • Uncordon the node
kubectl uncordon $NODE1

Repeat the same process on all worker nodes one by one and verify the status of cluster

kubectl get nodes

The STATUS column should show Ready for all your nodes, and the version number should be updated

 

Recovering from failure state

If kubeadm upgrade fails and does not roll back, for example because of an unexpected shutdown during execution, you can run kubeadm upgrade again. This command is idempotent and eventually makes sure that the actual state is the desired state you declare.

To recover from a bad state, you can also run kubeadm upgrade --force without changing the version that your cluster is running

How it works

kubeadm upgrade apply does the following:

  • Checks that your cluster is in an upgradeable state:
    • The API server is reachable
    • All nodes are in the Ready state
    • The control plane is healthy
  • Enforces the version skew policies.
  • Makes sure the control plane images are available or available to pull to the machine.
  • Upgrades the control plane components or rollbacks if any of them fails to come up.
  • Applies the new kube-dns and kube-proxy manifests and makes sure that all necessary RBAC rules are created.
  • Creates new certificate and key files of the API server and backs up old files if they’re about to expire in 180 days.

kubeadm upgrade node does the following on additional control plane nodes:

  • Fetches the kubeadm ClusterConfiguration from the cluster.
  • Optionally backups the kube-apiserver certificate.
  • Upgrades the static Pod manifests for the control plane components.
  • Upgrades the kubelet configuration for this node.

kubeadm upgrade node does the following on worker nodes:

  • Fetches the kubeadm ClusterConfiguration from the cluster.
  • Upgrades the kubelet configuration for this node

 

 

Backup and Restore of Kubernetes Applications using Velero -Openstack/Cinder

In this blog post we will be using Openstack/Cinder as our underlying storage provider and Heptio’s Velero for backup and restore of our application

Restic Plugin

Starting with 0.9 version thanks to Restic support, Velero now supports taking backup of almost any type of Kubernetes volume regardless of the underlying storage provider.

Note: Unfortunately it’s not supporting Openstack/Cinder storage class. Let’s see in this blog how to fix

Velero Architecture

Work flow

When you run velero backup create test-backup:

  1. The Velero client makes a call to the Kubernetes API server to create a Backup object.
  2. The BackupController notices the new Backupobject and performs validation.
  3. The BackupController begins the backup process. It collects the data to back up by querying the API server for resources.
  4. The BackupController makes a call to the object storage service – for example, AWS S3 – to upload the backup file.

How does Restic works with Velero?

Three more Custom Resource Definitions and their associated controllers are introduced for Restic support.

  • Restic Repository
  • PodVolumeBackup
  • PodVolumeRestore

Let’s start Velero !!

Kubernetes Environment Pre-Requisites

  1. Helm
  2. Ingress
  3. Persistent storage ( default storage class set)

Kubernetes Clusters

  1. Cluster Z: Minio Cluster ( Kubernetes cluster for hosting minio, object storage for backup storage)
  2. Cluster A: Old Cluster ( Application to migrate from )
  3. Cluster B: New Cluster ( Application to migrate to )

 

Deploying Object based storage – minio on Cluster Z


helm install --name minio --namespace minio --set accessKey=minio,secretKey=minio123,persistence.size=100Gi,service.type=NodePort stable/minio

1. Login to minio with access key minio & secret key minio123

2. Create a bucket by name kubernetes

Install velero client on Cluster A

wget https://github.com/heptio/velero/releases/download/v1.0.0/velero-v1.0.0-linux-amd64.tar.gz
tar -xvf velero-v1.0.0-linux-amd64.tar.gz 
cp velero-v1.0.0-linux-amd64/velero /usr/bin

Create velero credentials-velero file ( with minio access key & secret key) on Cluster A

vim credentials-velero 
[default]
aws_access_key_id = minio
aws_secret_access_key = minio123

Install velero server on Cluster A

velero install  --provider aws --bucket kubernetes --secret-file credentials-velero --use-volume-snapshots=true --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://10.157.249.168:32270 --snapshot-location-config region=minio,s3ForcePathStyle="true",s3Url=http://10.157.249.168:32270 --use-restic

Install Sample application, i will be deploying wordpress on Cluster A

helm install --name wordpress --namespace wordpress --set ingress.enabled=true,ingress.hosts[0].name=wordpress.jaws.jio.com  stable/wordpress

Annotate volume to be backuped, since by default cinder storage class doesn’t support snapshotting on Cluster A

kubectl -n wordpress describe pod/wordpress-mariadb-0
...
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-wordpress-mariadb-0
    ReadOnly:   false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      wordpress-mariadb
    Optional:  false
  default-token-r6rpc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-r6rpc
    Optional:    false
...
kubectl -n wordpress annotate pod/wordpress-mariadb-0 backup.velero.io/backup-volumes=data,config
kubectl -n wordpress describe pod/wordpress-557589bfbc-7pzqb
...
Volumes:
  wordpress-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  wordpress
    ReadOnly:   false
  default-token-r6rpc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-r6rpc
    Optional:    false
...
kubectl -n wordpress annotate pod/wordpress-557589bfbc-7pzqb backup.velero.io/backup-volumes=wordpress-data

Create a backup on Cluster A

 velero backup create  wp-backup --snapshot-volumes --include-namespaces wordpress

Install velero client on Cluster B

wget https://github.com/heptio/velero/releases/download/v1.0.0/velero-v1.0.0-linux-amd64.tar.gz
tar -xvf velero-v1.0.0-linux-amd64.tar.gz 
cp velero-v1.0.0-linux-amd64/velero /usr/bin

Create velero credentials-velero file ( with minio access key & secret key) on Cluster B

 vim credentials-velero 
[default]
aws_access_key_id = minio
aws_secret_access_key = minio123

Install velero server on Cluster B

velero install  --provider aws --bucket kubernetes --secret-file credentials-velero --use-volume-snapshots=true --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://10.157.249.168:32270 --snapshot-location-config region=minio,s3ForcePathStyle="true",s3Url=http://10.157.249.168:32270 --use-restic

Restore from backup on Cluster B

velero restore create wordpress-restore --from-backup wp-backup --restore-volumes=true

Wait & Verify Restore from backup on Cluster B

kubectl -n wordpress get pods -w
kubectl -n wordpress get pods
NAME                         READY   STATUS    RESTARTS   AGE
wordpress-68cd5f85c6-gr5vp   1/1     Running   0          2m29s
wordpress-mariadb-0          1/1     Running   0          2m24s

Scheduled Backups

Taking a backup manually happens only in an emergency situation or for educational purposes. The real essence of a backup and disaster recovery plan is to have scheduled backups. Ark provides that support in a rather simple manner.

$ velero schedule create daily-wordpress-backup–schedule=”0 10 * * *” –include-namespaces wordpress
Schedule “wordpress-backup” created successfully.

Troubleshooting

In case you are facing any issues regarding the setting up of the kubernetes cluster. Please make sure you have enough physical resources to spin up 3 VM’s. If not you can modify the Vagrantfile as mentioned in the README for the repository to increase/decrease the number of nodes.

For issues related to velero , there are a few commands that may be helpful

$ velero backup describe <backupName>
$ velero backup logs <backupName>
$ velero restore describe <restoreName>
$ velero restore logs <restoreName>

For comprehensive troubleshooting regarding velero, please follow this link.

Cleanup

If you don’t need the cluster anymore, you can go ahead and destroy the cluster

$ cd $HOME/ark-rook-tutorial/k8s-bkp-restore
$ vagrant destroy -f
$ rm -rf $HOME/ark-rook-tutorial

Reference

https://github.com/heptio/velero

How To Set Up an NFS Mount on Ubuntu 16.04

In this article, we will learn how to install NFS on Ubuntu 16.04Network File System (NFS) protocol and a filesystem which allows you to access the shared folders from the remote system or server and also allows you to mount as a remote directory on the servers. This allows you to share the storage space between different clients in different locations. NFS has been always the easiest way to access the remote storages over a network.

To accomplish this demo, we need two systems which Ubuntu installed and user with sudo permissions with a private network.

 

Installing the Packages on Server

We will install the ‘nfs-kernel’, which will be allowed us to share the directories on the server to share the files and folders. Below is the command to install the nfs package.

$ sudo apt-get update
$ sudo apt-get install nfs-kernel-server -y

Installing the Packages on Client Side

We have to install the nfs packages on the client in general nfs-common ie., the package which provides the access to the NFS share folders from the server.

$ sudo apt-get update
$ sudo apt-get install nfs-common -y

Note: on client make sure ‘rpcbind’ is running or not, else run “systemctl restart” rpcbind”

 

Enabling and Creating the Share Directories on the Server

Here for demo purpose, we are going to share two folders to differentiate the configuration, setting, First one is with superuser permission and other with trusted users on the client system.

Exporting a General Mount

In this example, we will create a general NFS mount that with the default configuration which is difficult for a user without any permissions on the client machine they can access and this can be used to create a shared space to store the project files in the folders.

Creating a Shared Folder for General Purpose

$ sudo mkdir /usr/nfs/common –p

Change the folder permission, so that anybody can write in the folder

$ sudo chown nobody:nogroup /usr/nfs/common

And now try to access the folder from a client with the below command

Before we mount any shared folder on the client, we needed to create a mount point on the client machine

$ sudo mkdir /mnt/nfs/common
$ sudo mount 192.168.1.25:/usr/nfs/common /mnt/nfs/common

This will mount the NFS share on the server 192.168.1.25 on the client machine with /mnt/nfs/common is mounted at /mnt/nfs/common on the client machine and it can be accessed as a local folder.

Creating a Shared folder with Home Directory

We will share user home directories stored on the server to access from the client, the access needed to conveniently manage the users.

Configuring the NFS Settings on the Server

As we are seeing 2 types of NFS  Share, let”s see how to configure the setting to match our requirements.

Open /etc/exports file with an editor

$ sudo vi /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
# to NFS clients. See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check)

Add below lines to the configuration file –

/usr/nfs/common                192.168.1.100(rw,sync,no_subtree_check)
/home                          192.168.1.100(rw,sync,no_root_squash,no_subtree_check)

Below is the explanation for each option we used in the above  commands which we used.

Rw -> This will allow client computers to read and write to the share.

Sync -> This will allow the data to be written in the NFS before it applies to queries and It also increase consistent environment and will be stable.

Nosubtreecheck -> This will prevent subtree checking, where if we enable this option, it will cause many problems if the client has opened the file.

Norootsquash -> This will makes the NFS translation request from the root user for the client into a not –privileged users of the server, where it will also prevent the root account on the client from using the file system of the server as root.

$ sudo systemctl restart nfs-kernel-server

Mounting the Directories on the Client

Before, we mount the share folders on the client, we needed to create a mount point and we will link the share folder from the NFS server to the local folders (mount points).

$ mkdir /mnt/common
$ mkdir /mnt/home
$ sudo mount 192.168.1.100:/usr/nfs/common /mnt/common
$ sudo mount 192.168.1.100:/home /mnt/home

After we run the commands we will not verify that the NFS share folders are mounted correctly or not

$ df –h
Filesystem                Size  Used Avail Use% Mounted on
udev                      538M     0  538M   0% /dev
tmpfs                     249M  628K   249M   2% /run
/dev/vda1                 100G  10G   90G  10% /
tmpfs                     445M     0  445M   0% /dev/shm
tmpfs                     10.0M    0  10.0M   0% /run/lock
tmpfs                     245M     0  245M   0% /sys/fs/cgroup
tmpfs                     249M     0   249M   0% /run/user/0
192.168.1.100:/home      124G  11.28G   118.8G   9% /mnt/home
192.168.1.100:/usr/nfs/common   124G  11.28G   118.8G   9% /mnt/common

As we can see that they both share are mounted and we can see them at the bottom, as they are mounted from the same server so we can see the same disk usage.

Mounting the NFS Share at the Boot Time

We can mount the NFS share at the time of boot so that if we needed to connect the NFS share folders, we can directly access the folders at the mount points

Open the /etc/fstab file and add the below lines.

$ sudo vi /etc/fstab

Add the below line at the bottom of the files

. . .
192.168.1.100:/usr/nfs/common    /mnt/general   nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0
192.168.1.100:/home              /mnt/home      nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0

Unmounting the NFS Share Folders

As if we do not want to use the folders, we can unmount the NFS share folders using the below commands

$ sudo umount /mnt/common
$ sudo umount /mnt/home

How to Install and Configure Docker Swarm on Ubuntu

Docker Swarm is a clustering tool that turns a group of Docker hosts into a single virtual server. Docker Swarm ensures availability and high performance for your application by distributing it over the number of Docker hosts inside a cluster. Docker Swarm also allows you to increase the number of container instance for the same application.

Clustering is an important feature of container technology for redundancy and high availability. You can manage and control clusters through a swarm manager. The swarm manager allows you to create a primary manager instance and multiple replica instances in case the primary instance fails. Docker Swarm exposes standard Docker API, meaning that any tool that you used to communicate with Docker (Docker CLI, Docker Compose, Krane, and Dokku) can work equally well with Docker Swarm.

Features of Swarm Mode:

  • Provides Docker Engine CLI to create a swarm of Docker Engines where you can deploy application services. You don’t need any additional software tool to create or manage a swarm.
  • Swarm manager assigns a unique DNS name to each service in the swarm. So you can easily query every container running in the swarm through a DNS name.
  • Each node in the swarm enforces TLS mutual authentication and encryption to secure communications between itself and all other nodes. You can also use self-signed root certificates or certificates from a custom root CA.
  • Allows you to apply service updates to nodes incrementally.
  • Allows you to specify an overlay network for your services and how to distribute service containers between nodes.

In this post, we will go through how to install and configure Docker Swarm mode on an Ubuntu 16.04 server. We will:

  • Install one of the service discovery tools and run the swarm container on all nodes.
  • Install Docker and configure the swarm manager.
  • Add all the nodes to the Manager node (more on nodes in the next section).

To get the most out of this post, you should have:

  • Basic knowledge of Ubuntu and Docker.
  • Two nodes with ubuntu 16.04 installed.
  • A non-root user with sudo privileges setup on both nodes.
  • A static IP address configured on Manager Node and on Worker Node. Here, we will use IP 192.168.0.103 for Manager Node and IP 192.168.0.104 for Worker Node.

Let’s start by looking at nodes.

Manager Nodes and Worker Nodes

Docker Swarm is made up of two main components:

  • Manager Nodes
  • Worker Nodes

Manager Nodes

Manager nodes are used to handle cluster management tasks such as maintaining cluster state, scheduling services, and serving swarm mode HTTP API endpoints. One of the most important features of Docker in Swarm Mode is the manager quorum. The manager quorum stores information about the cluster, and the consistency of information is achieved through consensus via the Raft consensus algorithm. If any Manager node dies unexpectedly, other one can pick up the tasks and restore the services to a stable state.

Raft tolerates up to (N-1)/2 failures and requires a majority or quorum of (N/2)+1 members to agree on values proposed to the cluster. This means that in a cluster of 5 Managers running Raft, if 3 nodes are unavailable, the system cannot process any more requests to schedule additional tasks. The existing tasks keep running but the scheduler cannot rebalance tasks to cope with failures if the manager set is not healthy.

Worker Nodes

Worker nodes are used to execute containers. Worker nodes don’t participate in the Raft distributed state and don’t make scheduling decisions. You can create a swarm of one Manager node, but you cannot have a Worker node without at least one Manager node. You can also promote a worker node to be a Manager when you take a Manager node offline for maintenance.

As mentioned in the introduction, we use two nodes in this post — one will act as a Manager node and other as a Worker node.

Getting Started

Before starting, you should update your system repository with the latest version. You can update it with the following command:

sudo apt-get update -y && sudo apt-get upgrade -y

Once the repository is updated, restart the system to apply all the updates.

Install Docker

You will also need to install the Docker engine on both nodes. By default, Docker is not available in the Ubuntu 16.04 repository. So you will need to set up a Docker repository first.

You can start by installing the required packages with the following command:

sudo apt-get install apt-transport-https software-properties-common ca-certificates -y

Next, add the GPG key for Docker:

wget https://download.docker.com/linux/ubuntu/gpg && sudo apt-key add gpg

Then, add the Docker repository and update the package cache:

sudo echo "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable" >> /etc/apt/sources.list
sudo apt-get update -y

Finally, install the Docker engine using the following command:

sudo apt-get install docker-ce -y

Once the Docker is installed, start the Docker service and enable it to start on boot time:

sudo systemctl start docker && sudo systemctl enable docker

By default, Docker daemon always runs as the root user and other users can only access it using sudo. If you want to run docker command without using sudo, then create a Unix group called docker and add users to it. You can do this by running the following command:

sudo groupadd docker && sudo usermod -aG docker dockeruser

Next, log out and log back to your system with dockeruser so that your group membership is re-evaluated.

Note: Remember to run the above commands on both nodes.

Configure firewall

You will need to configure firewall rules for a swarm cluster to work properly on both nodes.

Allow the ports 7946, 4789, 2376, 2376, 2377, and 80 using the UFW firewall with the following command:

sudo ufw allow 2376/tcp && sudo ufw allow 7946/udp && sudo ufw allow 7946/tcp && sudo ufw allow 80/tcp && sudo ufw allow 2377/tcp && sudo ufw allow 4789/udp

Next, reload the UFW firewall and enable it to start on boot:

sudo ufw reload && sudo ufw enable

Restart the Docker service to affect the Docker rules:

sudo systemctl restart docker

Create Docker Swarm cluster

First, you will need to initialize the cluster with the IP address, so your node acts as a Manager node.

On the Manager Node, run the following command for advertising IP address:

docker swarm init --advertise-addr 192.168.0.103

You should see the following output:

Swarm initialized: current node (iwjtf6u951g7rpx6ugkty3ksa) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-5p5f6p6tv1cmjzq9ntx3zmck9kpgt355qq0uaqoj2ple629dl4-5880qso8jio78djpx5mzbqcfu 192.168.0.103:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

The token shown in the above output will be used to add worker nodes to the cluster in next step. The Docker Engine joins the swarm depending on the join-token you provide to the docker swarm join command. The node only uses the token at join time. If you subsequently rotate the token, it doesn’t affect existing swarm nodes.

Now, check the status of the Manager Node with the following command:

docker node ls

If everything is fine, you should see the following output:

ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS
iwjtf6u951g7rpx6ugkty3ksa *   Manager-Node        Ready               Active              Leader

You can also check the status of the Docker Swarm Cluster:

docker info

You should see the following output:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: iwjtf6u951g7rpx6ugkty3ksa
 Is Manager: true
 ClusterID: fo24c1dvp7ent771rhrjhplnu
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 192.168.0.103
 Manager Addresses:
  192.168.0.103:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-45-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 992.5MiB
Name: Manager-Node
ID: R5H4:JL3F:OXVI:NLNY:76MV:5FJU:XMVM:SCJG:VIL5:ISG4:YSDZ:KUV4
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Add Worker Node to swarm cluster

Manager Node is now configured properly, it’s time to add Worker Node to the Swarm Cluster.

First, copy the output of the “swarm init” command from the previous step, then paste that output on the Worker Node to join the Swarm Cluster:

docker swarm join --token SWMTKN-1-5p5f6p6tv1cmjzq9ntx3zmck9kpgt355qq0uaqoj2ple629dl4-5880qso8jio78djpx5mzbqcfu 192.168.0.103:2377

You should see the following output:

This node joined a swarm as a worker.

Now, on the Manager Node, run the following command to list the Worker Node:

docker node ls

You should see the Worker Node in the following output:

ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS
iwjtf6u951g7rpx6ugkty3ksa *   Manager-Node        Ready               Active              Leader
snrfyhi8pcleagnbs08g6nnmp     Worker-Node         Ready               Active              

Launch web service in Docker Swarm

Docker Swarm Cluster is now up and running, it’s time to launch the web service inside Docker Swarm Mode.

On the Manager Node, run the following command to deploy a web server service:

docker service create --name webserver -p 80:80 httpd

The above command will create an Apache web server container and map it to port 80, so you can access Apache web server from the remote system.

Now, you can check the running service with the following command:

docker service ls

You should see the following output:

ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
nnt7i1lipo0h        webserver           replicated          0/1                 apache:latest       *:80->80/tcp

Next, scale the web server service across two containers with the following command:

docker service scale webserver=2

Then, check the status of web server service with the following command:

docker service ps webserver

You should see the following output:

ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE                  ERROR               PORTS
7roily9zpjvq        webserver.1         httpd:latest        Worker-Node         Running             Preparing about a minute ago                       
r7nzo325cu73        webserver.2         httpd:latest        Manager-Node        Running             Preparing 58 seconds ago                           

Test Docker Swarm

Apache web server is now running on Manager Node. You can now access web server by pointing your web browser to the Manager Node IP http://192.168.0.103 or Worker Node IP http://192.168.0.104 as shown below:

Screenshot-of-docker-swarm-apache

Apache web server service is now distributed across both nodes.

Docker Swarm also provides high availability for your service. If the web server goes down on the Worker Node, then the new container will be launched on the Manager Node.

To test high availability, just stop the Docker service on the Worker Node:

sudo systemctl stop docker

On the Manager Node, run the web server service status with the following command:

docker service ps webserver

You should see that a new container is launched on Manager Node:

ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ia2qc8a5f5n4        webserver.1         httpd:latest        Manager-Node        Ready               Ready 1 second ago                           
7roily9zpjvq         \_ webserver.1     httpd:latest        Worker-Node         Shutdown            Running 15 seconds ago                       
r7nzo325cu73        webserver.2         httpd:latest        Manager-Node        Running             Running 23 minutes ago   

Conclusion

Congratulations! you have successfully installed and configured a Docker Swarm cluster on Ubuntu 16.04.

Now that you have a basic cluster set up, head over to the Docker Swarm documentation to learn more about Swarm. You’ll want to look into configuring your cluster with more than just one Manager Node, according to your organisation’s high-availability requirements.

Openstack – Delete Cinder Stuck Volume

Recently, I was using Devstack/Ocata version of OpenStack and was trying to  attach/detach volumes to an instance. Every once in a while, volumes would go to an   ‘in-use’ state even after the instance was destroyed.

In fact, even in other releases, I have seen cinder volumes stuck in in-use or error state and sometimes not being able to delete those.

If the volume is in ‘in-use’ status, you first have to change to an available status though before you can issue a delete:

cinder reset-state –state available $VOLUME_ID

cinder delete $VOLUME_ID

If ‘cinder delete’ doesn’t work and you have admin privileges, you can try force-delete.

cinder force-delete $VOLUME_ID

But may be that will fix it. May be it will not. If the volume is still stuck, try going to the database and setting the status of the volume to a detached state:

update volume_attachment setattach_status="detached"where id="<attachment_id>";
update volumes setattach_status="detached"where id="<volume_id>";
Once I did that, I was able to delete or force-delete any stuck volumes. 🙂

Best way to install Openstack – Ocata ( Devstack) Quick Tips

Here we are going to see how to install devstack with some tricks to resolve unknown errors in devstack and local.conf file with stable branch version

Pre-requisites

  • Ubuntu 16.04 ( 14.04 is having compatibility issues with devstack requirements)
  • 8 GB RAM (minimum)

 

Steps:

  1. Clone devstack

git clone https://git.openstack.org/openstack-dev/devstack -b stable/ocata

2. Set permissions

sudo chown -R <username> devstack

sudo chmod 770 devstack

3. cd devstack, create local.conf file and enter details as below

[[local|localrc]]
ADMIN_PASSWORD=admin
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD

DEFAULT_VOLUME_GROUP_NAME=stack-volumes-default
PIP_UPGRADE=True
RECLONE=False
DEFAULT_INSTANCE_TYPE=m1.tiny

HOST_IP=<Localhost IP>

#Enable SENLIN
enable_plugin senlin https://git.openstack.org/openstack/senlin
Enable senlin-dashboard
enable_plugin senlin-dashboard https://git.openstack.org/openstack/senlin-dashboard

#Enable HEAT
enable_plugin heat https://git.openstack.org/openstack/heat stable/ocata

#Enable Aodh and ceilometer
CEILOMETER_BACKEND=mongodb
enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer stable/ocata
enable_plugin aodh https://git.openstack.org/openstack/aodh stable/ocata

# Enable Gnocchi
enable_plugin gnocchi https://github.com/gnocchixyz/gnocchi stable/4.0
enable_service gnocchi-grafana,gnocchi-api,gnocchi-metricd

 

#Enable LBAAS V2
enable_plugin neutron-lbaas https://github.com/openstack/neutron-lbaas stable/ocata
NEUTRON_LBAAS_SERVICE_PROVIDERV2=”LOADBALANCERV2:Haproxy:neutron_lbaas.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default”
enable_service q-lbaasv2

#Enable LBAAS V2 Dashboard
enable_plugin neutron-lbaas-dashboard https://git.openstack.org/openstack/neutron-lbaas-dashboard stable/ocata

#Enable Ocatavia LBAAS v2 Driver
enable_plugin octavia https://git.openstack.org/openstack/octavia stable/ocata
ENABLED_SERVICES+=,octavia,o-cw,o-hk,o-hm,o-api

# Enable Logging

LOGFILE=/opt/stack/logs/stack.sh.log
LOGDAYS=2
VERBOSE=True
LOG_COLOR=True
SCREEN_LOGDIR=/opt/stack/logs

4. Create file dev_set.sh and add entry as below

 

                     #!/bin/bash
git config –global url.”https://&#8221;.insteadOf git://
export no_proxy=127.0.0.1, <localhost IP>

save and run source dev_set.sh

 

Note: no_proxy will avoid keystone authentication with localhost IP, few might come across this issue.

 

Common Devstack unknown Errors and tricks to solve

  1. For keystone credentials, run dev_set.sh as mentioned above
  2. For “Could not satisfy constraints for ‘horizon’: installation from path or url cannot be constrained to a version” try “git reset –hard /opt/stack/requirements”, it occurs if do second+ ./stack.sh, easy way.
  3. While Enabling Gnocchi with stable version make sure “uuidgen” is installed else do apt-get install uuid-runtime
  4. To connect to screen, run screen -ls , you can see screen number, now to connect enter screen -r <screen number> and restart services if any changes needed
  5. For Error “[ERROR] /home/pandy/devstack/stackrc:747 Could not determine host ip address. See local.conf for suggestions on setting HOST_IP” open stackrc file and add HOST_IP=<localhost_IP> manually and run. it’s sounds crazy, but works.

Now do stack.sh and enjoy devstack installation, Pooh !!