Install, configure, and manage a custom virtual machine image for multiple hypervisors.
You will install and configure a custom virtual machine image locally. You will use a tool called Packer, which allows you to build and create a custom virtual machine image from one json file for multiple providers (e.g., Azure, AWS, Digital Ocean), hypervisors (e.g., Virtual Box), or tools designed for managing virtual machine environments (e.g., Vagrant). You will also manage your virtual machine environment using Vagrant, a tool for building virtual machines and managing the lifecycle of VMs.
In this activity, we will learn how to create a custom VM image (Debian) for VirtualBox that can be used with Vagrant. The VM image will have the following tools and systems installed:
- Apache httpd web server version 2.x
- PHP version 8.x
- MariaDB 11.x
Setup/Prerequisites
- Desktop computer running Windows, macOS, or Linux.
- A minimum of 15GB of available disk storage.
- Fast Internet connection to download an iso image. It’s recommended not to use a limited data plan like 4G/5G for this activity.
- You will need to have the following tools installed on your local machine:
- VirtualBox
- You may use VMware Desktop Hypervisor products: VMware Fusion (for macOS), which works on Apple Silicon (arm64) or VMware Workstation (for Windows and Linux)
- packer
- Vagrant
Step 0: Install Virtual Box, Packer, and Vagrant
Install all tools listed above in the prerequisites section.
Step 1: Generate SSH keys for SSH key-based authentication
- We need to authenticate with the custom image using key-based SSH authentication. Create an ssh key-pair using
ssh-keygen
on your local machine:
ssh-keygen -t rsa -f ./vagrant-key
Step 2: Configure Packer and add builders
Packer uses a template file in JSON to create a virtual machine. The template file contains a set of properties and values. The main properties are: builders, provisioners, and post-processors.
- builders are tasks that produce an image for a single platform. It can be for example virtualBox, AWS, or Azure.
- provisioners are sections in Packer for running multiple scripts before launching the VM image (e.g., custom bash scripts to install or configure software and tools).
- post-processors are sections in Packer for running multiple scripts after the machine image has been created (e.g., converting a VirtualBox image into a suitable image or box for Vagrant).
Packer supports two builders for building an image for VirtualBox:
- virtualbox-iso - This builder is useful when we want to start from an existing ISO file. It creates a new VM image from the ISO file for VirtualBox, provisions the VM with our software and tools, and exports the VM to an image.
- virtualbox-ovf - Takes an input file in the Open Virtualization Format (ovf or ova) and runs provisioners on top of that VM, and exports that machine to create an image. In order to use this builder, we need to export the existing VM in our hypervisor into an open virtualization format (.ovf) archive file. To export a VM in VirtualBox, go to the File menu in VirtualBox, select Export Appliance, select the VM to export (e.g., CentOS), and export the VM file as .ova. In this activity, will use the former builder, virtualbox-iso.
Step 2.1: Add Builders
- If you are using VirtualBox as the hypervisor, then create a file named:
virtualbox-debian-config.pkr.hcl
with the following content:
packer {
required_plugins {
virtualbox = {
version = "~> 1"
source = "github.com/hashicorp/virtualbox"
}
vagrant = {
version = "~> 1"
source = "github.com/hashicorp/vagrant"
}
}
}
- If you are using Vmware as the hypervisor, then create a file named:
vmware-debian-config.pkr.hcl
with the following content:
packer {
required_plugins {
vmware = {
version = "~> 1"
source = "github.com/hashicorp/vmware"
}
vagrant = {
version = "~> 1"
source = "github.com/hashicorp/vagrant"
}
}
}
Step 2.2: Initialize the Packer configuration
To initialize the packer configuration and download the plugin you’ve defined in the .pkr.hcl config file, run:
VirtualBox
packer init virtualbox-debian-config.pkr.hcl
- VMware
packer init vmware-debian-config.pkr.hcl
- To check the installed packer plugins, run:
packer plugins installed
You should see the two installed plugins: packer-plugin-vagrant and one of the hypervisor specific plugins: packer-plugin-vmware or packer-plugin-virtualbox.
Step 3: Create a Packer template and provisioning scripts
Step 3.1: Create the Packer template file
VirtualBox
Create a file named
debian-12-virtualbox.json
in your local machine. This JSON will be considered the template for Packer:You will use the following iso_url and iso_checksum depending on the architecture of your local environment
- amd64:
- iso_url: https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.5.0-amd64-netinst.iso
- iso_checksum: 33c08e56c83d13007e4a5511b9bf2c4926c4aa12fd5dd56d493c0653aecbab380988c5bf1671dbaea75c582827797d98c4a611f7fb2b131fbde2c677d5258ec9
- arm64:
- iso_url: https://cdimage.debian.org/debian-cd/current/arm64/iso-cd/debian-12.5.0-arm64-netinst.iso
- iso_checksum: 14c2ca243ee7f6e447cc4466296d974ee36645c06d72043236c3fbea78f1948d3af88d65139105a475288f270e4b636e6885143d01bdf69462620d1825e470ae
- amd64:
Add the following content to the
debian-12-vmware.json
file after replacing iso_url and iso_checksum with the ones that matches your architecture as listed above:
TODO
- We specified the disk size to create for the virtual machine in the
disk_size
key with a value of 15360 in megabytes (15GB). - We select the amount of CPU and memory for this virtual machine as an array of commands to the VirtualBox’s vboxmanage command
- This template has a section called provisioners where we have one provisioner of type shell that executes a set of commands to install packages.
- Move to the next step (3.2) since you do not need to use VMware.
VMware
Create a file named
debian-12-vmware.json
in your local machine. This JSON will be considered the template for Packer:You will use the following iso_url and iso_checksum depending on the architecture of your local environment
- amd64:
- iso_url: https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.5.0-amd64-netinst.iso
- iso_checksum: 33c08e56c83d13007e4a5511b9bf2c4926c4aa12fd5dd56d493c0653aecbab380988c5bf1671dbaea75c582827797d98c4a611f7fb2b131fbde2c677d5258ec9
- arm64:
- iso_url: https://cdimage.debian.org/debian-cd/current/arm64/iso-cd/debian-12.5.0-arm64-netinst.iso
- iso_checksum: 14c2ca243ee7f6e447cc4466296d974ee36645c06d72043236c3fbea78f1948d3af88d65139105a475288f270e4b636e6885143d01bdf69462620d1825e470ae
- amd64:
Add the following content to the
debian-12-vmware.json
file after replacing iso_url and iso_checksum with the ones that matches your architecture as listed above:
- We specified the disk size to create for the virtual machine in the
disk_size
key with a value of 15360 in megabytes (15GB). - We select the amount of CPU and memory for this virtual machine as an array of commands to the VirtualBox’s vboxmanage command
Step 3.2: Create a preseed
for the Debian installer for Unattended Installation
The preseed file is a configuration file that the OS installer uses to answer the questions it normally asks during the installation process. This allows the installation process to be fully automated, unattended, and with no manual intervention is needed to select options when installing the guest OS. This preseed file is not a shell script but rather a config file specific to each OS installers. When creating a preseed file, we should start from a good example as the one listed on debian.org,like the default preseed file, or this sample preseed.cfg file on GitHub.
- The packer template file has a key named
http_directory
, which contains the path to a directory that will be served using an HTTP server. The files in this directory will be available over HTTP and can be requested from the virtual machine Packer will create. We need to create a directory and inside it we createpreseed.cfg
file. This will allow us to install Debian automatically and answer the installer’s questions without any user interaction. - On your local machine, create a directory called
http
and inside it create file namepreseed.cfg
for the Debian installer in your local environment. Copy and paste the content form https://github.com/kaucpit490/configs/blob/main/preseed.cfg. You may modify the preseed configuration file to meet your requirements and preferences.
Step 3.3: Adding Provisioning Scripts
We need to perform actions before launching the image. We want to upload our public key that we created in step 1 into the virtual machine image. We also want to execute a shell script to install tools on our vm image.
This template has a section called provisioners where we have one provisioner of type shell that executes a set of commands to configure the OS and install packages.
- Open the Packer template file and add the following provisioners after the end of the builders array:
"provisioners": [
{
"type": "file",
"source": "{{user `public_key`}}",
"destination": "{{user `key_destination`}}"
},
{
"type": "shell",
"script": "./installer.sh",
"execute_command": "chmod +x '{{ .Path }}'; echo '{{user `password`}}' | sudo -S sh '{{ .Path }}'"
},
{
"type": "shell",
"script": "./set_ssh.sh",
"execute_command": "chmod +x '{{ .Path }}'; echo '{{user `password`}}' | sudo -S sh '{{ .Path }}' {{user `user`}} {{user `key_destination`}}"
}
]
- We will create a shell script file that contains the commands to install and configure software in the VM image. Create a file named
installer.sh
in the same directory that your Packer template file is in. Edit it in your text editor and add the following content:
#!/bin/bash
apt update
apt upgrade -y
apt install -y nano
# Install PHP 8.3, we will add Ondrej Sury's PPA into the system
apt install ca-certificates apt-transport-https software-properties-common
add-apt-repository ppa:ondrej/php
apt update
apt install -y php8.3
apt install -y php8.3-{cli,bz2,mysql,intl,xml,zip,gd,mbstring,curl,xmlrpc,soap,fpm}
apt install -y apache2
apt install -y mariadb-server
- Create another file named
set_ssh.sh
in the same directory that your Packer template file is in. Open it up in your text editor and add the following content:
#!/bin/bash
# Exit immediately if any command exits with a non-zero exit status.
set -e
# usage
if [[ $# -ne 2 ]]; then
echo "Error: Usage $0 user_name public_key"
exit 1
fi
# Disable SSH password-based authentication
sed -n 'H;${x;s/\#PasswordAuthentication yes/PasswordAuthentication no/;p;}' /etc/ssh/sshd_config >new_sshd_config
cat new_sshd_config >/etc/ssh/sshd_config
rm new_sshd_config
# move the public key
mkdir -p /home/$1/.ssh
mv $2 /home/$1/.ssh/authorized_keys
chmod 700 /home/$1/.ssh
chown -R $1:$1 /home/$1/.ssh
chmod 644 /home/$1/.ssh/authorized_keys
Step 4: Adding post-processors, runtime variables, and verifying the Packer Template
Step 4.1: Adding post-processors
We need to perform actions once the image has been created. We want to convert our virtual machine image into a Vagrant box, the format that Vagrant uses for creating VM images.
- Open the Packer template file and add the following post-processors after the end of the provisioners array:
"post-processors": [
{
"type": "vagrant",
"output": "builds/debian-12-{{.Provider}}.box"
}
]
Step 4.2: Adding sensitive user variables
We want our template to use some common variables and receive sensitive data such as public key, passwords, and other user data dynamically from the command line.
- Add the following user variables at the beginning of the template file:
"variables": {
"key_destination": "/tmp/vagrant_key.pub"
},
"sensitive-variables": [
"user",
"password",
"public_key",
"private_key"
]
Step 4.3: Validating the Packer template
If you have followed the previous steps, you should end up with a packer template file that looks like the one below:
VirtualBox
VMware
Note: This packer template file is for VMWare Fusion on macOS (arm64). If you’re on x86, then change the iso_url and iso_checksum.
{
"variables": {
"key_destination": "/tmp/vagrant_key.pub"
},
"sensitive-variables": [
"user",
"password",
"public_key"
],
"builders": [
{
"type": "vmware-iso",
"guest_os_type": "Debian_64",
"iso_url": "https://cdimage.debian.org/debian-cd/current/arm64/iso-cd/debian-12.5.0-arm64-netinst.iso",
"iso_checksum": "sha512:14c2ca243ee7f6e447cc4466296d974ee36645c06d72043236c3fbea78f1948d3af88d65139105a475288f270e4b636e6885143d01bdf69462620d1825e470ae",
"ssh_username": "{{user `user`}}",
"ssh_password": "{{user `password`}}",
"ssh_timeout": "20m",
"disk_size": "15360",
"vm_name": "debian-12",
"http_directory": "http",
"boot_wait": "5s",
"boot_command": [
"<down><wait>",
"<enter><wait>",
"fb=true auto=true url=http://{{.HTTPIP}}:{{.HTTPPort}}/preseed.cfg hostname={{.Name}} domain=local <enter>"
],
"shutdown_command": "echo '{{user `password`}}' | sudo -S shutdown -P now"
}
],
"provisioners": [
{
"type": "file",
"source": "{{user `public_key`}}",
"destination": "{{user `key_destination`}}"
},
{
"type": "shell",
"script": "./installer.sh",
"execute_command": "chmod +x '{{ .Path }}'; echo '{{user `password`}}' | sudo -S sh '{{ .Path }}'"
},
{
"type": "shell",
"script": "./set_ssh.sh",
"execute_command": "chmod +x '{{ .Path }}'; echo '{{user `password`}}' | sudo -S sh '{{ .Path }}' {{user `user`}} {{user `key_destination`}}"
}
],
"post-processors": [
{
"type": "vagrant",
"output": "builds/debian-12-{{.Provider}}.box"
}
]
}
Before building the image, we want to check that our template is valid. The packer validate
command is used to check and validate the syntax and configuration of a Packer template file. It does not actually build any images. We need to pass the user name, password, and public key into packer from the command line as user values. Please note that we assume that vagrant-key.pub
and the packer template resides in the current working directory. If not, use the full path instead. Also,
packer validate -var "user=vagrant" -var "password=vagrant" -var "public_key=./vagrant-key.pub" -var "private_key=./vagrant_key" ./path_to_packer_template.json
You should get:
The configuration is valid.
Step 5: Building the image from the template
The packer build
command is used to create machine images based on the configuration specified in the Packer template file. It takes the same -var arguments as the packer validate command to pass user variables to the template.
packer build -var "user=vagrant" -var "password=vagrant" -var "public_key=./vagrant-key.pub" -var "private_key=./vagrant_key" ./path_to_packer_template.json
This may take a few minutes or more to build the image, start the kickstart script, and install the provisioning scripts.
The action defined in the post-processor section of the template will take the build and convert it into a Vagrant box stored at builds/debian-12-virtualbox.box
or builds/debian-12-vmware.box
.
Step 6: Start and provision the Vagrant box
Now, we should have a Vagrant Box built by Packer. Vagrant Box is the package format for Vagrant environments that include the base image and the additional tools installed on top of the image. This box can be used by anyone to create an identical virtual environment for the same provider. It can be also shared on the public Vagrant box repository.
Step 6.1: Create a Vagrantfile
- We need to create a file named
Vagrantfile
at the same working directory that we created the Packer template at. - Add the following content to the Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :
vagrant_box_file = ENV['vagrant_box_file'] or 'builds/debian-12-virtualbox.box'
user_name = ENV['vagrant_user'] or 'vagrant'
private_key_path = ENV['vagrant_private_key']
public_key_path = ENV['vagrant_public_key']
puts "Vagrant box file: #{vagrant_box_file}"
puts "Vagrant user: #{user_name}"
puts "Vagrant private key: #{private_key_path}"
puts "Vagrant public key: #{public_key_path}"
Vagrant.configure("2") do |config|
config.vm.box_check_update = true
config.vm.box = "debian-12-vagrant"
config.vm.box_url = "file://" + vagrant_box_file.to_s
config.vm.hostname = "debian-server"
config.ssh.host = "127.0.0.1"
config.ssh.port = 2222
config.ssh.private_key_path = [private_key_path]
config.ssh.insert_key = false
config.vm.network "forwarded_port", guest: 22, host: 2222, host_ip: "127.0.0.1", id: 'ssh'
config.vm.provision "shell", path: "./set_ssh.sh", args: "#{user_name} #{public_key_path}"
config.vm.provider "virtualbox" do |vb|
vb.gui = false
vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
vb.customize ["modifyvm", :id, "--ioapic", "on"]
vb.name = "debian-12-vagrant"
vb.memory = "1024"
end
end
Step 6.2: Create, start, and access the vagrant box
- Store the path to the public and private keys, the path to the packer’s generated vagrant box file in environment variables:
vagrant_public_key
,vagrant_private_key
, andvagrant_box_file
as defined in the Vagrantfile.
From the command line, export the following environment variables:
- On a Unix-like system such as macos or Linux, run:
export vagrant_user=vagrant
export vagrant_private_key=/path/to/private/key
export vagrant_public_key=/path/to/public/key
export vagrant_box_file=/path/to/the/vagrant/box
- On Windows, export may not work, so run:
set vagrant_user=vagrant
set vagrant_private_key=/path/to/private/key
set vagrant_public_key=/path/to/public/key
set vagrant_box_file=/path/to/the/vagrant/box
- Validate the Vagrantfile
vagrant validate
- Create your Vagrant Box and provision the vagrant environment in VirtualBox.
vagrant up
- You will be prompted to enter the passphrase you chose in the first step. Vagrant needs this to provision the created box.
If you are on Windows and have encountered an error message that says “unknown encoding name”, try to change the system locale on your system, then go to Control Panel » Region » Administrative tab » change system locale to English (United States).
Step 6.3: Connect to the vagrant box via SSH and manage it using vagrant
- Log in to your VM instances using SSH
vagrant ssh
You will be prompted to enter the passphrase you chose in step 1.
Check the installed software on your vagrant box
php -v
mysql -v
- Stop the vagrant machine. You need to exit from the connected VM and then execute
vagrant halt
.
exit
vagrant halt
Assignment
Build a custom virtual machine for your cloud provider (e.g., Amazon AMI, Azure VM Image, GCP Compute Engine Images, etc.) . The custom VM should match your current development environment and include a startup shell script displaying the name and the version of the VM image as well as the set of tools that you’ve added to the VM.
Submission
Submit your answers with screenshots showing the commands you executed as a PDF file by the due date posted on Teams.