Edit Page

Virtualization: Building Virtual Machines

Install, configure, and manage a custom virtual machine image for multiple hypervisors.

You will install and configure a custom virtual machine image locally. You will use a tool called Packer, which allows you to build and create a custom virtual machine image from one json file for multiple providers (e.g., Azure, AWS, Digital Ocean), hypervisors (e.g., Virtual Box), or tools designed for managing virtual machine environments (e.g., Vagrant). You will also manage your virtual machine environment using Vagrant, a tool for building virtual machines and managing the lifecycle of VMs.

In this activity, we will learn how to create a custom VM image (Debian) for VirtualBox that can be used with Vagrant. The VM image will have the following tools and systems installed:

  • Apache httpd web server version 2.x
  • PHP version 8.x
  • MariaDB 11.x

Setup/Prerequisites

  • Desktop computer running Windows, macOS, or Linux.
  • A minimum of 15GB of available disk storage.
  • Fast Internet connection to download an iso image. It’s recommended not to use a limited data plan like 4G/5G for this activity.
  • You will need to have the following tools installed on your local machine:

Step 0: Install Virtual Box, Packer, and Vagrant

Install all tools listed above in the prerequisites section.

Step 1: Generate SSH keys for SSH key-based authentication

  • We need to authenticate with the custom image using key-based SSH authentication. Create an ssh key-pair using ssh-keygen on your local machine:
ssh-keygen -t rsa -f ./vagrant-key

Step 2: Configure Packer and add builders

Packer uses a template file in JSON to create a virtual machine. The template file contains a set of properties and values. The main properties are: builders, provisioners, and post-processors.

  • builders are tasks that produce an image for a single platform. It can be for example virtualBox, AWS, or Azure.
  • provisioners are sections in Packer for running multiple scripts before launching the VM image (e.g., custom bash scripts to install or configure software and tools).
  • post-processors are sections in Packer for running multiple scripts after the machine image has been created (e.g., converting a VirtualBox image into a suitable image or box for Vagrant).

Packer supports two builders for building an image for VirtualBox:

  • virtualbox-iso - This builder is useful when we want to start from an existing ISO file. It creates a new VM image from the ISO file for VirtualBox, provisions the VM with our software and tools, and exports the VM to an image.
  • virtualbox-ovf - Takes an input file in the Open Virtualization Format (ovf or ova) and runs provisioners on top of that VM, and exports that machine to create an image. In order to use this builder, we need to export the existing VM in our hypervisor into an open virtualization format (.ovf) archive file. To export a VM in VirtualBox, go to the File menu in VirtualBox, select Export Appliance, select the VM to export (e.g., CentOS), and export the VM file as .ova. In this activity, will use the former builder, virtualbox-iso.

Step 2.1: Add Builders

  • If you are using VirtualBox as the hypervisor, then create a file named: virtualbox-debian-config.pkr.hcl with the following content:
packer {
    required_plugins {
        virtualbox = {
          version = "~> 1"
          source  = "github.com/hashicorp/virtualbox"
        }
        vagrant = {
          version = "~> 1"
          source = "github.com/hashicorp/vagrant"
        }
    }
}
  • If you are using Vmware as the hypervisor, then create a file named: vmware-debian-config.pkr.hcl with the following content:
packer {
  required_plugins {
    vmware = {
      version = "~> 1"
      source = "github.com/hashicorp/vmware"
    }
    vagrant = {
      version = "~> 1"
      source = "github.com/hashicorp/vagrant"
    }
  }
}

Step 2.2: Initialize the Packer configuration

  • To initialize the packer configuration and download the plugin you’ve defined in the .pkr.hcl config file, run:

  • VirtualBox

packer init virtualbox-debian-config.pkr.hcl
  • VMware
packer init vmware-debian-config.pkr.hcl
  • To check the installed packer plugins, run:
packer plugins installed

You should see the two installed plugins: packer-plugin-vagrant and one of the hypervisor specific plugins: packer-plugin-vmware or packer-plugin-virtualbox.

Step 3: Create a Packer template and provisioning scripts

Step 3.1: Create the Packer template file

VirtualBox
TODO
  • We specified the disk size to create for the virtual machine in the disk_size key with a value of 15360 in megabytes (15GB).
  • We select the amount of CPU and memory for this virtual machine as an array of commands to the VirtualBox’s vboxmanage command
  • This template has a section called provisioners where we have one provisioner of type shell that executes a set of commands to install packages.
  • Move to the next step (3.2) since you do not need to use VMware.
VMware
  • We specified the disk size to create for the virtual machine in the disk_size key with a value of 15360 in megabytes (15GB).
  • We select the amount of CPU and memory for this virtual machine as an array of commands to the VirtualBox’s vboxmanage command

Step 3.2: Create a preseed for the Debian installer for Unattended Installation

The preseed file is a configuration file that the OS installer uses to answer the questions it normally asks during the installation process. This allows the installation process to be fully automated, unattended, and with no manual intervention is needed to select options when installing the guest OS. This preseed file is not a shell script but rather a config file specific to each OS installers. When creating a preseed file, we should start from a good example as the one listed on debian.org,like the default preseed file, or this sample preseed.cfg file on GitHub.

  • The packer template file has a key named http_directory, which contains the path to a directory that will be served using an HTTP server. The files in this directory will be available over HTTP and can be requested from the virtual machine Packer will create. We need to create a directory and inside it we create preseed.cfg file. This will allow us to install Debian automatically and answer the installer’s questions without any user interaction.
  • On your local machine, create a directory called http and inside it create file name preseed.cfg for the Debian installer in your local environment. Copy and paste the content form https://github.com/kaucpit490/configs/blob/main/preseed.cfg. You may modify the preseed configuration file to meet your requirements and preferences.

Step 3.3: Adding Provisioning Scripts

We need to perform actions before launching the image. We want to upload our public key that we created in step 1 into the virtual machine image. We also want to execute a shell script to install tools on our vm image.

This template has a section called provisioners where we have one provisioner of type shell that executes a set of commands to configure the OS and install packages.

  • Open the Packer template file and add the following provisioners after the end of the builders array:
    "provisioners": [
        {
            "type": "file",
            "source": "{{user `public_key`}}",
            "destination": "{{user `key_destination`}}"
        },
        {
            "type": "shell",
            "script": "./installer.sh",
            "execute_command": "chmod +x '{{ .Path }}'; echo '{{user `password`}}' | sudo -S sh '{{ .Path }}'"
        },
        {
            "type": "shell",
            "script": "./set_ssh.sh",
            "execute_command": "chmod +x '{{ .Path }}'; echo '{{user `password`}}' | sudo -S sh '{{ .Path }}' {{user `user`}} {{user `key_destination`}}"
        }
    ]
  • We will create a shell script file that contains the commands to install and configure software in the VM image. Create a file named installer.sh in the same directory that your Packer template file is in. Edit it in your text editor and add the following content:
#!/bin/bash
apt update
apt upgrade -y
apt install -y nano
# Install PHP 8.3, we will add Ondrej Sury's PPA into the system
apt install ca-certificates apt-transport-https software-properties-common
add-apt-repository ppa:ondrej/php
apt update
apt install -y php8.3
apt install -y php8.3-{cli,bz2,mysql,intl,xml,zip,gd,mbstring,curl,xmlrpc,soap,fpm}
apt install -y apache2
apt install -y mariadb-server
  • Create another file named set_ssh.sh in the same directory that your Packer template file is in. Open it up in your text editor and add the following content:
#!/bin/bash
# Exit immediately if any command exits with a non-zero exit status.
set -e
# usage
if [[ $# -ne 2 ]]; then
    echo "Error: Usage $0 user_name public_key"
    exit 1
fi
# Disable SSH password-based authentication
sed -n 'H;${x;s/\#PasswordAuthentication yes/PasswordAuthentication no/;p;}' /etc/ssh/sshd_config >new_sshd_config
cat new_sshd_config >/etc/ssh/sshd_config
rm new_sshd_config
# move the public key
mkdir -p /home/$1/.ssh
mv $2 /home/$1/.ssh/authorized_keys
chmod 700 /home/$1/.ssh
chown -R $1:$1 /home/$1/.ssh
chmod 644 /home/$1/.ssh/authorized_keys

Step 4: Adding post-processors, runtime variables, and verifying the Packer Template

Step 4.1: Adding post-processors

We need to perform actions once the image has been created. We want to convert our virtual machine image into a Vagrant box, the format that Vagrant uses for creating VM images.

  • Open the Packer template file and add the following post-processors after the end of the provisioners array:
"post-processors": [
    {
        "type": "vagrant",
        "output": "builds/debian-12-{{.Provider}}.box"
    }
]

Step 4.2: Adding sensitive user variables

We want our template to use some common variables and receive sensitive data such as public key, passwords, and other user data dynamically from the command line.

  • Add the following user variables at the beginning of the template file:
"variables": {
  "key_destination": "/tmp/vagrant_key.pub"
},
"sensitive-variables": [
  "user",
  "password",
  "public_key",
  "private_key"
]

Step 4.3: Validating the Packer template

If you have followed the previous steps, you should end up with a packer template file that looks like the one below:

VirtualBox
VMware

Note: This packer template file is for VMWare Fusion on macOS (arm64). If you’re on x86, then change the iso_url and iso_checksum.

{
    "variables": {
        "key_destination": "/tmp/vagrant_key.pub"
    },
    "sensitive-variables": [
        "user",
        "password",
        "public_key"
    ],
    "builders": [
        {
            "type": "vmware-iso",
            "guest_os_type": "Debian_64",
            "iso_url": "https://cdimage.debian.org/debian-cd/current/arm64/iso-cd/debian-12.5.0-arm64-netinst.iso",
            "iso_checksum": "sha512:14c2ca243ee7f6e447cc4466296d974ee36645c06d72043236c3fbea78f1948d3af88d65139105a475288f270e4b636e6885143d01bdf69462620d1825e470ae",
            "ssh_username": "{{user `user`}}",
            "ssh_password": "{{user `password`}}",
            "ssh_timeout": "20m",
            "disk_size": "15360",
            "vm_name": "debian-12",
            "http_directory": "http",
            "boot_wait": "5s",
            "boot_command": [
                "<down><wait>",
                "<enter><wait>",
                "fb=true auto=true url=http://{{.HTTPIP}}:{{.HTTPPort}}/preseed.cfg hostname={{.Name}} domain=local <enter>"
            ],
            "shutdown_command": "echo '{{user `password`}}' | sudo -S shutdown -P now"
        }
    ],
    "provisioners": [
        {
            "type": "file",
            "source": "{{user `public_key`}}",
            "destination": "{{user `key_destination`}}"
        },
        {
            "type": "shell",
            "script": "./installer.sh",
            "execute_command": "chmod +x '{{ .Path }}'; echo '{{user `password`}}' | sudo -S sh '{{ .Path }}'"
        },
        {
            "type": "shell",
            "script": "./set_ssh.sh",
            "execute_command": "chmod +x '{{ .Path }}'; echo '{{user `password`}}' | sudo -S sh '{{ .Path }}' {{user `user`}} {{user `key_destination`}}"
        }
    ],
    "post-processors": [
        {
            "type": "vagrant",
            "output": "builds/debian-12-{{.Provider}}.box"
        }
    ]
}

Before building the image, we want to check that our template is valid. The packer validate command is used to check and validate the syntax and configuration of a Packer template file. It does not actually build any images. We need to pass the user name, password, and public key into packer from the command line as user values. Please note that we assume that vagrant-key.pub and the packer template resides in the current working directory. If not, use the full path instead. Also,

packer validate -var "user=vagrant" -var "password=vagrant" -var "public_key=./vagrant-key.pub" -var "private_key=./vagrant_key" ./path_to_packer_template.json

You should get:

The configuration is valid.

Step 5: Building the image from the template

The packer build command is used to create machine images based on the configuration specified in the Packer template file. It takes the same -var arguments as the packer validate command to pass user variables to the template.

packer build -var "user=vagrant" -var "password=vagrant" -var "public_key=./vagrant-key.pub" -var "private_key=./vagrant_key" ./path_to_packer_template.json

This may take a few minutes or more to build the image, start the kickstart script, and install the provisioning scripts. The action defined in the post-processor section of the template will take the build and convert it into a Vagrant box stored at builds/debian-12-virtualbox.box or builds/debian-12-vmware.box.

Step 6: Start and provision the Vagrant box

Now, we should have a Vagrant Box built by Packer. Vagrant Box is the package format for Vagrant environments that include the base image and the additional tools installed on top of the image. This box can be used by anyone to create an identical virtual environment for the same provider. It can be also shared on the public Vagrant box repository.

Step 6.1: Create a Vagrantfile

  • We need to create a file named Vagrantfile at the same working directory that we created the Packer template at.
  • Add the following content to the Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :
vagrant_box_file = ENV['vagrant_box_file'] or 'builds/debian-12-virtualbox.box'
user_name = ENV['vagrant_user'] or 'vagrant'
private_key_path = ENV['vagrant_private_key']
public_key_path = ENV['vagrant_public_key']
puts "Vagrant box file: #{vagrant_box_file}"
puts "Vagrant user: #{user_name}"
puts "Vagrant private key: #{private_key_path}"
puts "Vagrant public key: #{public_key_path}"

Vagrant.configure("2") do |config|
  config.vm.box_check_update = true
  config.vm.box = "debian-12-vagrant"
  config.vm.box_url = "file://" + vagrant_box_file.to_s
  config.vm.hostname = "debian-server"
  config.ssh.host = "127.0.0.1"
  config.ssh.port = 2222
  config.ssh.private_key_path = [private_key_path]
  config.ssh.insert_key = false
  config.vm.network "forwarded_port", guest: 22, host: 2222, host_ip: "127.0.0.1", id: 'ssh'
  config.vm.provision "shell", path: "./set_ssh.sh", args: "#{user_name} #{public_key_path}"

  config.vm.provider "virtualbox" do |vb|
    vb.gui = false
    vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
    vb.customize ["modifyvm", :id, "--ioapic", "on"]
    vb.name = "debian-12-vagrant"
    vb.memory = "1024"
  end

end

Step 6.2: Create, start, and access the vagrant box

  • Store the path to the public and private keys, the path to the packer’s generated vagrant box file in environment variables: vagrant_public_key, vagrant_private_key, and vagrant_box_file as defined in the Vagrantfile.

From the command line, export the following environment variables:

  • On a Unix-like system such as macos or Linux, run:
export vagrant_user=vagrant
export vagrant_private_key=/path/to/private/key
export vagrant_public_key=/path/to/public/key
export vagrant_box_file=/path/to/the/vagrant/box
  • On Windows, export may not work, so run:
set vagrant_user=vagrant
set vagrant_private_key=/path/to/private/key
set vagrant_public_key=/path/to/public/key
set vagrant_box_file=/path/to/the/vagrant/box
  • Validate the Vagrantfile
vagrant validate

vagrant up
  • You will be prompted to enter the passphrase you chose in the first step. Vagrant needs this to provision the created box.

If you are on Windows and have encountered an error message that says “unknown encoding name”, try to change the system locale on your system, then go to Control Panel » Region » Administrative tab » change system locale to English (United States).

Step 6.3: Connect to the vagrant box via SSH and manage it using vagrant

  • Log in to your VM instances using SSH
vagrant ssh

  • You will be prompted to enter the passphrase you chose in step 1.

  • Check the installed software on your vagrant box

php -v
mysql -v
  • Stop the vagrant machine. You need to exit from the connected VM and then execute vagrant halt.
exit
vagrant halt

Assignment

Build a custom virtual machine for your cloud provider (e.g., Amazon AMI, Azure VM Image, GCP Compute Engine Images, etc.) . The custom VM should match your current development environment and include a startup shell script displaying the name and the version of the VM image as well as the set of tools that you’ve added to the VM.

Submission

Submit your answers with screenshots showing the commands you executed as a PDF file by the due date posted on Teams.