Day 5: KVM and LibVirt for Linux Virtualization

{{ printf “Estimated reading time: %d minutes” .ReadingTime }}

Hey! Welcome back! Day 5 builds upon the previous days and presumes you have some homelab hardware installed with linux and have followed the virtualbox virtualization tutorial on day 3.

This is akin to 10 A4 pages and will take

Understanding hypervisors

A hypervisor is a virtual machine monitor which manages virtual machines. Effectively splicing up your physical desktop, server or laptop or resources into parts. Each Virtual machine has it’s own CPU, RAM disk and networking. MOdern x86 AMD/INTEL processors have specific CPU instructions for virtualization.

THere’s type 1 and type 2 hypervisors.

The market is full of hypervisors including vmware, kvm, xen, hyper-v. Most virtualization tools use one of these.

Why KVM? Big Cloud Providers use it

There are at least 12.788 large companies (1000+ employees) which publicly acknowledge they use KVM. ¹ Amazon AWS datacenter and cloud used to use XEN as a hypervisor but began migrating to KVM ² GCP (Google CLoud PLatform) uses Compute Engine which is a security hardened KVM based hypervisor (kernel based virtual machine) ³ Since all cloud providers use KVM underneath their tech stack, YOU gain a great advantage by leveraging KVM on your systems

Everything OPenSource is packaged as a product for the uninitiated to consume

I’m also going to talk about various alternatives which you can use instead of going on the route of a specific CLoud Provider. Most big cloud provieders provide the same software apps, with various API’s which lock you in.

The top 3 cloud providers AWS (Amazon) 28%, GCP (GOogle) 14% and Azure (Microsoft) 21% are followed by hundreds of “smaller” cloud providers such as Alibaba (4%), Oracle (3%) and even smaller ones like Linode, DigitalOcean, Vultr. It’s important to understand that ALL cloud providers use open source technology repackaged to sell you the services which you can achieve on your own! Understanding the pitfalls of vendor lock-in will help you tremendously. As if you’d go on the VPS or Baremetal route you can cut your cloud costs by 500%.

This guide presumes your hardware already has virtualization enabled in BIOS

What is libvirt qemu and how are they connected to KVM?

QEMU

QEMU stands for Quick EMulator and is an emulator and virtualizer meaning it can work directly with KVM. However it also does emulation by JIT (just in time) binary translation which means you can also emulate different CPU instruction sets such as ARM, powerPC,risc-v. Making it invaluable when developing software if you don’t have hardware available or want to make development much faster.

LibVirt

Libvirt is a cross platform (MEANING YOU CAN USE IT ON Linux, freebsd, WIndows, MaCOS) toolkit which helps you manage virtualizaton platforms. is accessible from the command line but also hbas bindings in different programming languages such as C, Python, Perl, Go and more It supports plenty of hypervisors including KVM, Hypervisor.framework, QEMU, Xen, Virtuozzo, VMware ESX, LXC, BHyve and more

The most important part is that it’s used by many applications and cloud providers under the hood..

Installing KVM and LibVirt

Install libvirt, qemu, kvm, some networking tools, guestfilesystem tools, virtual viewer and the GUI virt-manager for easier beginner management;)

sudo apt install  qemu-system   bridge-utils virtinst libvirt-daemon-system libvirt-clients virt-manager libguestfs-tools virt-viewer

In older versions you might install qemu-kvm which now selects qemu-system-x86 automatically.

Add your user to the correct groups

Depending on Linux distro and/or recent version, it’s best to add them all:

$USER=youruser
sudo adduser $USER libvirt 
sudo adduser $USER libvirt-qemu
sudo adduser $USER kvm

Apply group changes

newgrp libvirt
newgrp kvm
newgrp libvirt-qemu

Verify if your system is ready for virtualization

Libvirt daemon should start automatically. In case it’s not start and enable libvirt daemon sudo systemctl enable --now libvirtd

virt-host-validate

This will tell you if hardware virtualization is on

 QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device '/dev/kvm' exists                                 : PASS
  QEMU: Checking if device '/dev/kvm' is accessible                          : PASS
  QEMU: Checking if device '/dev/vhost-net' exists                           : PASS
  QEMU: Checking if device '/dev/net/tun' exists                             : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : WARN (Enable 'devices' in kernel Kconfig file or mount/enable cgroup controller in your system)
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : PASS
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace 'ipc'                                         : PASS
   LXC: Checking for namespace 'mnt'                                         : PASS
   LXC: Checking for namespace 'pid'                                         : PASS
   LXC: Checking for namespace 'uts'                                         : PASS
   LXC: Checking for namespace 'net'                                         : PASS
   LXC: Checking for namespace 'user'                                        : PASS
   LXC: Checking for cgroup 'cpu' controller support                         : PASS
   LXC: Checking for cgroup 'cpuacct' controller support                     : PASS
   LXC: Checking for cgroup 'cpuset' controller support                      : PASS
   LXC: Checking for cgroup 'memory' controller support                      : PASS
   LXC: Checking for cgroup 'devices' controller support                     : FAIL (Enable 'devices' in kernel Kconfig file or mount/enable cgroup controller in your system)
   LXC: Checking for cgroup 'freezer' controller support                     : FAIL (Enable 'freezer' in kernel Kconfig file or mount/enable cgroup controller in your system)
   LXC: Checking for cgroup 'blkio' controller support                       : PASS
   LXC: Checking if device '/sys/fs/fuse/connections' exists                 : PASS

Normally you should enable the specific virtualization for your CPU via BIOS, check if it’s supported.

LXC failures don’t really concern us right now. LXC was a precursor to docker which made the whole containerization possible.

However, yes, you can use libvirt for containers aswell

Providing CGROUPS to VM Guest

To fix the warnings in QEMU you need to edit some grub settings such as iommu=pt intel_iommu

https://discussion.fedoraproject.org/t/how-to-enable-devices-and-freezer-in-kconfig-file/118077/2

This really depends how deep you want to test virtualization and if you’re planning to squeeze most:)

INstalling a Virtual Machine

The first step to installing a VM is to manually download a ISO and manually go through the installation process.

While this is OK in many cases especially if you want to ensure you install and configure your desired operating system it will become bothering and boring after the 10th install. So we will atuomate this process.

There’s tools like debootstrap to automate a installation of a debian into a folder, which we can convert to a VM. The added benefit of using debootstrap is that YOU can automate AND get which packages go into your VM making sure you minimize security incidents by not relying on 3rd party “images”

THere’s also a better way, there are official cloud images of most Linux distributions including Debian which can be used on

• OpenStack providers
• Local QEMU vm’s
• Amazon EC2
• Microsoft Azure

Local QEMU is what we’re after, we have 2 different files we can use https://cloud.debian.org/images/cloud/trixie/latest/debian-13-nocloud-amd64.qcow2 https://cloud.debian.org/images/cloud/trixie/latest/debian-13-nocloud-amd64.raw

Since I manually installed ZFS on my Debian 13 Trixie pre installation I’m going to use raw, use qcow2 if you have ext4 since it’s 300MB vs 3GB for the .raw one.

Homework for ZFS using zvols

if you’re running zfs you may want to download a qcow2 and convert it to a zvol ;). This maps the VM image to a ZFS zvol. It’s similar to how overlayfs works for docker containers. qemu-img convert -O raw <infile.(vdi|vmdk|$whatever)> /dev/zvol/rpool/data/<vmid>-disk0

INstallatin process

I’m now going to show you the manual process of installing virtual machines which I still use when experimenting.

Open Virtual Machine manager on your Linux machine

Add path

Select the debian linux IMG

You can also choose “browse local”. Select Debian13 as Operating System. 2048 GB 2 CPU is OK, we can change that later

Start the VM

NOTE: Make sure you make a copy of the RAW/QCOW2 fileas that’s the disk the virtual machine will use:) and it will start writing it. .qcow2 or using zfs volumes is super helpful here as you can just copy/paste the volumes around and ensure you can snapshot disks around.

You should be greeted with login interface

Usually the username root with a blank password should work.

setting a root password and customizing some details

Sometimes this won’t work… so we go the route of setting up a password. There are multiple manual ways to do this via grub editing . However, as a DevOps you will want to use commands to automate your workflow.

Shut down your VM before attempting this. DO it via virt-manager or with virsh shutdown debian13 (or whatever name you gave it!)

THe following command will set the root password, hostname and will run ssh-keygen and start sshd;)

virt-customize -a  /home/andrei/libvirt/debian-13-generic-amd64.raw  --root-password password:MyAmazingPassword --hostname "devopsmagic" --firstboot-command 'ssh-keygen -A && systemctl restart sshd'

Reboot the VM, you should see the VM devopsmagic name changed and you can login with the password for root;).

COmmand Line Usage

First set the libvirt default uri to the system one (root). You can also use the local user’s libvirt session qemu:///session

Accessing QEMU KVM instances via SSH from other machines

Notice the format, as you deepen your DevOps and linux knowledge, use something akin to qemu://192.168.135.27/session to access the KVM qemu sessions from different hosts. Accessing remote systems from the internet and making your own cloud SSH as qemu+ssh://username@hostname/system.

Setting default libvirt URI:

export LIBVIRT_DEFAULT_URI=qemu:///system
virsh list

Because you’re a devops, you will want to use the command line to manage virtual machines or use scripts for this process. A bash script in a git or fossil repository is a first step to establishing infrastructure sa code

sudo virt-install \
--name debian13-devops \
--memory 2048 \
--vcpus 2 \
--disk /home/andrei/libvirt/debian-13-devops-amd64.raw \
--import \
--os-variant debian13 \
--noautoconsole

The –noauthconsole ensures you won’t get the console slpash which allows you to login to it directly without the need to use SSH.

Opening a windowed viewer for the machine

An auto VNC system to login to your machine which is super useful if you have a GUI.

virt-viewer debian13-devops

Opening a console in your terminal

SSH-ing into the console is the best, however virsh allows you to get access to a console within the virtual machine automatically. Meaning you can copy paste commands seamlessly from your system.

virsh consoloe debian13-devops

cloud init configuration and virtual machine install automation

Oftentimes you will use cloud init for configuration of cloud images. Meaning you can specify a lot of automation steps. If this looks similar to how ansilble or terraform works, it’s basically almost the same thing. All cloud providers use this when you purchase a VPS or a service to setup your virtual machine;)

#cloud-config

users:
  # whatever username you like
  - name: andrei
    # so our user can just sudo without any password
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    # content from $HOME/.ssh/id_rsa.pub on your host system
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC[the rest of your ssh key here]Z

The following create a username called andrei define the default shell, copy over the ssh authorized keys so I can ssh into the VM without password AND will grant me sudo without asking ffor password. The sudo part is useful for local deveopment, maybe disable it for

The command to make a KVM libvirt machine.

virt-install --name devopsmagic --memory 2048 --vcpus 4 --disk /home/andrei/libvirt/debian-13-devops-amd64.raw --cloud-init user-data=/home/andrei/libvirt/cloud-init.yaml  --network bridge=virbr0 --os-variant debian13

The previous if run in Linux will automatically open the console ;) into the machine.

Internet issues in VM?

IF running the cloudinit version internet should WORK properly, when running the non cloud init version networking MAY fail to work.

Please note that there is an issue on debian 13 resulting in broken network connections when running virt-customize which we wil do later. If you run into this issue, also install these packages sudo apt install \ systemd-resolved \ dhcpcd-base

Run this on host

sudo vim /etc/libvirt/network.conf
firewall_backend=nftables

# :wq! save & exit vim;)

sudo systemctl restart libvirtd

Check ip forwarding

cat /proc/sys/net/ipv4/ip_forward

If it returns 0, enable it: sudo sysctl -w net.ipv4.ip_forward=1

sudo virsh net-start default
sudo virsh net-autostart --network default 

If all else fails… reboot your HOST machine. I know, Linux and *BSD systems don’t need reboots like Windows, however sometimes when all else fails this can be tried.

Explore LibVirt Extra Might and Magic

You can use libvirt, virt-manager, virsh etc to connect to multiple machines running KVM/Linux THis enables failover, high availability and a plethora of other nice features like live migration of

Explore these on your own.

COnclusion

RUnning KVM locally on a Linux machine is the first and best step to get started with Virtualization for real.

You can build and sell your own cloud VPS

If you build a simple UI web interface you can now easily deploy VM’s for people and ask for money, effectively creating your own cloud infrastructure, it’s this easy. There are some extra steps but the process is pretty simple one you get the hang of it

TODO VIDEO

TODO still have to explain how to set it up ;) and make a video about it:) but this is for later

What’s next?

We’ll automate things even more with terraform for libvirt and other automations. The next days we’ll see why engineering is non linear and why it’s a good thing and we’ll start by setting up our own Linux Virtual private server with a cloud provider.