Cloud Infrastructure

OpenStack Cloud Deployment

Building a private cloud service provider using Kolla-Ansible on Google Cloud Platform

February 2021

Infrastructure as Code

Network Virtualization

Building a complete private cloud service provider with OpenStack and Kolla-Ansible

Project Overview

As enterprises continue to move more workloads to the cloud, the need for private cloud services that can be deployed on-premises or in a hybrid cloud environment becomes increasingly important. For organizations that want the flexibility and control of a private cloud, but don't want to invest in the hardware and infrastructure required to build and maintain one, Kolla-Ansible and OpenStack offer an appealing solution.

Kolla-Ansible is a tool for deploying OpenStack services on top of an existing infrastructure. It is designed to be easy to use and operate, and can be deployed on a wide variety of platforms, including bare-metal, virtualized, and containerized environments. OpenStack is a popular open source platform for building private and public clouds, composed of modular services that can be deployed individually or together to provide a complete cloud solution.

This project demonstrates how to use Kolla-Ansible to deploy a private cloud service using OpenStack on Google Cloud Platform, creating a fully functional cloud service provider with capabilities comparable to major public cloud providers while maintaining complete control over infrastructure, security, and costs.

Required tools and initial GCP infrastructure configuration

Prerequisites and Project Setup

Prerequisites

• Google Cloud Platform (GCP) account with billing enabled
• Google Cloud SDK installed on local machine
• Linux OS (tested on Ubuntu and Debian)
• Basic networking knowledge and Linux command line skills
• Python 3 virtual environment for Ansible dependencies

SSH Key Configuration

First step involves creating and configuring SSH keys for secure access to GCP instances:

# Generate SSH key pair
ssh-keygen -t rsa -f <ssh_key_name> -C <username>
chmod 600 <ssh_key_name>

# Prepare metadata for GCP
echo -n <username>:$(cat <ssh_key_name>.pub) > metadata.txt

# Upload public key to GCP
gcloud compute project-info add-metadata --metadata-from-file ssh-keys=metadata.txt

Automated Infrastructure Script

A comprehensive bash script automates the entire GCP infrastructure setup, including:

• Two VPC networks with custom subnet configurations
• Custom disk images with nested virtualization licenses
• Three VM instances (1 controller, 2 compute nodes)
• Firewall rules for inter-node communication

# Network IP ranges
IPRANGE1='172.28.0.0/24'     # Management network
IPRANGE1_2='10.1.0.0/16'     # Tenant network range
IPRANGE2='172.24.0.0/24'     # Provider network

# Create VPC networks
gcloud compute networks create cc-network1 --subnet-mode=custom
gcloud compute networks create cc-network2 --subnet-mode=custom

# Create subnets
gcloud compute networks subnets create cc-subnet1 \
  --range ${IPRANGE1} --secondary-range range1=${IPRANGE1_2} \
  --network=cc-network1 --region=europe-west3

Multi-node OpenStack deployment with nested virtualization

System Architecture

Infrastructure Topology

• Controller Node: Manages API endpoints, database, and message queue
• Compute Node 1: Runs Nova compute service and hypervisor
• Compute Node 2: Additional compute capacity for VM instances
• Nested Virtualization: VMX enabled for optimal resource usage
• Instance Specs: n2-standard-2 (2 vCPUs, 8GB RAM, 100GB disk)

Network Architecture

• Management Network: 172.28.0.0/24 for OpenStack services
• Provider Network: 172.24.0.0/24 for external connectivity
• Tenant Networks: 10.1.0.0/16 for VM instance communication
• Dual NIC Setup: Each node has two network interfaces
• Firewall Rules: Custom rules for TCP, UDP, and ICMP traffic

Comprehensive OpenStack deployment using Kolla-Ansible automation

Implementation Process

Infrastructure Verification

Before OpenStack deployment, comprehensive verification ensures all prerequisites are met:

• Nested Virtualization: Verify VMX support with grep -cw vmx /proc/cpuinfo
• Network Interfaces: Confirm dual NIC configuration on all nodes
• SSH Connectivity: Test SSH access to all VMs from local machine
• Inter-node Communication: Verify internal IP connectivity using netcat
• Firewall Rules: Ensure TCP, UDP, and ICMP traffic flows correctly

# Verify nested virtualization
grep -cw vmx /proc/cpuinfo

# Test internal connectivity
nc -z -v <internal_ip> 22

# Check network interfaces
ip link show

Kolla-Ansible Configuration

Detailed preparation of Kolla-Ansible deployment configuration and inventory files:

• Python Environment: Create isolated virtual environment for dependencies
• Ansible Installation: Install Ansible and Kolla-Ansible via pip
• Inventory Setup: Configure multinode inventory with IP addresses
• Global Configuration: Set deployment parameters in globals.yml
• Password Generation: Auto-generate service passwords for security

# Example globals.yml configuration
kolla_base_distro: "ubuntu"
kolla_internal_vip_address: "172.28.0.100"
network_interface: "ens4"
neutron_external_interface: "ens5"
nova_compute_virt_type: "kvm"
enable_neutron_provider_networks: "yes"

OpenStack Deployment

Automated deployment process using Kolla-Ansible playbooks with comprehensive validation:

• Connectivity Test: Verify Ansible can reach all nodes
• Bootstrap Phase: Prepare nodes with Docker and dependencies
• Pre-checks: Validate system requirements and configuration
• Deployment: Deploy containerized OpenStack services
• Post-deployment: Generate admin credentials and verify services

# Deployment sequence
ansible -m ping all -i ./multinode
kolla-ansible -i ./multinode bootstrap-servers
kolla-ansible -i ./multinode prechecks
kolla-ansible -i ./multinode deploy
kolla-ansible -i ./multinode post-deploy

OpenStack Service Configuration

Post-deployment configuration to create a fully operational cloud environment:

• Admin Environment: Source admin credentials for CLI access
• External Networks: Create provider networks for floating IPs
• Security Groups: Configure firewall rules for VM instances
• SSH Keypairs: Import public keys for secure VM access
• VM Images: Upload Ubuntu 16.04 images for instance creation
• NAT Configuration: Enable external connectivity for VMs

# Create security group with open rules
openstack security group create open-all
openstack security group rule create open-all --ingress --protocol tcp --dst-port 1:65525

# Create VM instance
openstack server create --flavor m1.medium --image ubuntu-16.04 \
  --nic net-id=admin-net --security-group open-all \
  --key-name openstack_keypair new_vm_instance

Solving connectivity challenges and implementing NAT for external access

Network Troubleshooting & Validation

Floating IP Connectivity Challenge

A critical challenge emerged when VM instances with floating IPs couldn't reach external networks. The issue stemmed from missing NAT configuration between the OpenStack tenant networks and the external provider network.

• Problem: VMs assigned floating IPs couldn't ping external addresses
• Root Cause: Missing iptables NAT rules on the controller node
• Solution: Custom iptables script to enable source NAT (SNAT)
• Validation: Successful external connectivity and metadata service access

NAT Configuration Implementation

The iptables-magic.sh script resolves connectivity issues by implementing proper NAT rules:

#!/bin/bash
# iptables-magic.sh - Enable NAT for OpenStack floating IPs

# Enable IP forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward

# Add SNAT rule for floating IP network
iptables -t nat -A POSTROUTING -s 172.24.0.0/24 -o ens4 -j MASQUERADE

# Add forwarding rules
iptables -A FORWARD -i br-ex -o ens4 -j ACCEPT
iptables -A FORWARD -i ens4 -o br-ex -j ACCEPT

Connectivity Validation Tests

• Internal Ping Test: Verify controller can reach VM floating IPs
• SSH Access Test: Confirm secure shell access to VM instances
• External Connectivity: Test internet access from VM instances
• Metadata Service: Validate OpenStack metadata API accessibility
• Network Data Retrieval: Access instance network configuration via API

# Validation commands
ping <floating_ip>                    # From controller
ssh ubuntu@<floating_ip>             # SSH access test
ping 8.8.8.8                        # External connectivity
wget 169.254.169.254/openstack/...   # Metadata service

Key insights from building a production-ready private cloud

Project Outcomes & Lessons Learned

Successful Cloud Service Provider

Successfully deployed a fully functional private cloud service provider with OpenStack, demonstrating enterprise-grade capabilities including:

• Complete IaaS Platform: Compute, networking, and storage services
• Multi-tenancy Support: Isolated environments for different users/projects
• API Compatibility: RESTful APIs compatible with public cloud services
• Web Dashboard: Horizon dashboard for graphical management
• CLI Tools: Command-line interface for automation and scripting

Technical Achievements

Infrastructure Automation

• Fully automated GCP infrastructure provisioning
• Containerized OpenStack deployment
• Infrastructure as Code principles
• Reproducible deployment process

Performance & Scalability

• Competitive performance vs. public clouds
• Multi-node compute scaling
• Efficient resource utilization
• Network performance optimization

Critical Lessons Learned

Networking Complexity

OpenStack networking requires deep understanding of virtual networking concepts:

• Provider vs. tenant network separation
• Floating IP and NAT configuration
• Security group rule management
• Neutron service troubleshooting

Deployment Challenges

Key challenges encountered during deployment:

• Nested virtualization performance overhead
• Container orchestration complexity
• Service dependency management
• Configuration parameter tuning

Business Value and Use Cases

Cost Benefits

• Reduced cloud infrastructure costs
• Predictable operational expenses
• No vendor lock-in concerns
• Open-source license advantages

Enterprise Applications

• Hybrid cloud strategies
• Data sovereignty compliance
• Development/testing environments
• Legacy application migration

Comprehensive benchmarking against major public cloud providers

Performance Evaluation

To validate the performance of our OpenStack deployment, we conducted comprehensive benchmarking tests using the same methodology as our previousGCP vs AWS performance evaluation. The benchmarks focused on four key areas: CPU performance, memory bandwidth, sequential disk I/O, and random disk access patterns.

CPU Performance Analysis

Our OpenStack deployment achieved 85% of GCP's CPU performance for compute-intensive workloads. The 15% performance gap is primarily attributed to the overhead of nested virtualization, which adds an additional hypervisor layer between the guest OS and the physical hardware.

Memory Performance Excellence

Memory benchmarks revealed exceptional performance, with our OpenStack deployment actually outperforming both GCP and AWS in memory-intensive operations by 8-12%. Sequential memory access patterns showed particularly strong results, indicating efficient memory subsystem configuration.

Disk I/O Performance

Disk performance showed mixed results. Sequential read/write operations achieved 88% of public cloud performance, while random access patterns exhibited higher latency (78% performance ratio). This indicates opportunities for storage subsystem optimization through advanced caching strategies.

Performance Variability

We observed higher volatility in benchmark results compared to dedicated public cloud resources. This variability is expected in nested virtualization environments where resources are shared with the host system and other workloads on the same physical infrastructure.

CPU benchmark results showing OpenStack achieving 85% of GCP performance

Memory performance benchmarks demonstrating superior bandwidth capabilities

Sequential disk I/O performance achieving 88% of public cloud providers

Random disk access patterns with higher latency due to nested virtualization

Key Performance Insights

Strengths

• Superior memory bandwidth performance
• Competitive sequential I/O operations
• Excellent internal network latency
• Cost-effective compared to public clouds

Areas for Optimization

• CPU overhead from nested virtualization
• Random disk access latency
• Performance variability under load
• External network throughput optimization

Project Details

Date

February 2021