Day 5: KVM and LibVirt for Linux Virtualization

{{ printf “Estimated reading time: %d minutes” .ReadingTime }}

Understanding hypervisors

A hypervisor is a virtual machine monitor which manages virtual machines. Effectively splicing up your physical desktop, server or laptop or resources into parts. Each Virtual machine has it’s own CPU, RAM disk and networking. MOdern x86 AMD/INTEL processors have specific CPU instructions for virtualization.

THere’s type 1 and type 2 hypervisors.

Multiple hypervisors include vmware, kvm, xen, hyper-v.

Why KVM? Big Cloud Providers use it

Everything OPenSource is packaged as a product for the uninitiated to consume

I’m also going to talk about various alternatives which you can use instead of going on the route of a specific CLoud Provider. Most big cloud provieders provide the same software apps, with various API’s which lock you in.

The top 3 cloud providers AWS (Amazon) 28%, GCP (GOogle) 14% and Azure (Microsoft) 21% are followed by hundreds of “smaller” cloud providers such as Alibaba (4%), Oracle (3%) and even smaller ones like Linode, DigitalOcean, Vultr. It’s important to understand that ALL cloud providers use open source technology which you can achieve on your own. Understanding the pitfalls of vendor lock-in will help you tremendously. As if you’d go on the VPS or Baremetal route you can cut your cloud costs by 500%.

Installing KVM and LibVirt

Install libvirt, qemu, kvm, some networking tools, guestfilesystem tools, virtual viewer and the GUI virt-manager for easier beginner management;)

sudo apt install  qemu-system   bridge-utils virtinst libvirt-daemon-system libvirt-clients virt-manager libguestfs-tools virt-viewer

In older versions you might install qemu-kvm which now selects qemu-system-x86 automatically.

Please note that there is an issue on debian 13 resulting in broken network connections when running virt-customize which we wil do later. If you run into this issue, also install these packages:

sudo apt install \ systemd-resolved \ dhcpcd-base

Add your user to the correct groups

sudo adduser <youruser> libvirt 
sudo adduser <youruser> libvirt-qemu
sudo adduser <youruser> kvm

Apply group changes

newgrp libvirt
newgrp kvm
newgrp libvirt-qemu

Verify if your system is ready for virtualization

Libvirt daemon should start automatically. In case it’s not start and enable libvirt daemon sudo systemctl enable --now libvirtd

virt-host-validate

This will tell you if hardware virtualization is on

 QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device '/dev/kvm' exists                                 : PASS
  QEMU: Checking if device '/dev/kvm' is accessible                          : PASS
  QEMU: Checking if device '/dev/vhost-net' exists                           : PASS
  QEMU: Checking if device '/dev/net/tun' exists                             : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : WARN (Enable 'devices' in kernel Kconfig file or mount/enable cgroup controller in your system)
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : PASS
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace 'ipc'                                         : PASS
   LXC: Checking for namespace 'mnt'                                         : PASS
   LXC: Checking for namespace 'pid'                                         : PASS
   LXC: Checking for namespace 'uts'                                         : PASS
   LXC: Checking for namespace 'net'                                         : PASS
   LXC: Checking for namespace 'user'                                        : PASS
   LXC: Checking for cgroup 'cpu' controller support                         : PASS
   LXC: Checking for cgroup 'cpuacct' controller support                     : PASS
   LXC: Checking for cgroup 'cpuset' controller support                      : PASS
   LXC: Checking for cgroup 'memory' controller support                      : PASS
   LXC: Checking for cgroup 'devices' controller support                     : FAIL (Enable 'devices' in kernel Kconfig file or mount/enable cgroup controller in your system)
   LXC: Checking for cgroup 'freezer' controller support                     : FAIL (Enable 'freezer' in kernel Kconfig file or mount/enable cgroup controller in your system)
   LXC: Checking for cgroup 'blkio' controller support                       : PASS
   LXC: Checking if device '/sys/fs/fuse/connections' exists                 : PASS

Normally you should enable the specific virtualization for your CPU via BIOS, check if it’s supported.

LXC failures don’t really concern us right now. LXC was a precursor to docker which made the whole containerization possible.

Providing CGROUPS to VM Guest

To fix the warnings in QEMU you need to edit some grub settings such as iommu=pt intel_iommu

https://discussion.fedoraproject.org/t/how-to-enable-devices-and-freezer-in-kconfig-file/118077/2

This really depends how deep you want to nest:)

INstalling a Virtual Machine

The first step to installing a VM is to manually download a ISO and manually go through the installation process.

While this is OK in many cases especially if you want to ensure you install what you want, most of the time we want to atuomate this process.

There’s tools like debootstrap to automate a installation of a debian into a folder, which we can convert to a VM.

THere’s also a better way, there are official cloud images which can be used on

• OpenStack providers
• Local QEMU vm’s
• Amazon EC2
• Microsoft Azure

Local QEMU is what we’re after https://cloud.debian.org/images/cloud/trixie/latest/debian-13-nocloud-amd64.qcow2 https://cloud.debian.org/images/cloud/trixie/latest/debian-13-nocloud-amd64.raw

NOTE: Use .raw files if you have a filesystem with copy on write enabled such as ZFS, btrfs etc. Use qcow2 if you’re using ext4 or anything else. You CAN use qcow2 on ZFS but you’ll have double copy on write allocations for every modified data block. I haven’t found any meaningful

Since I manually installed ZFS on my Debian 13 Trixie pre installation I’m going to use raw, use qcow2 if you have ext4 since it’s 300MB vs 3GB.

Homework for ZFS using zvols

if you’re running zfs you may want to download a qcow2 and convert it to a zvol ;). This maps the VM image to a ZFS zvol. It’s similar to how overlayfs works for docker containers. qemu-img convert -O raw <infile.(vdi|vmdk|$whatever)> /dev/zvol/rpool/data/<vmid>-disk0

INstallatin process

Add path

Select IMG

You can also choose “browse local”. Select Debian13 as Operating System. 2048 GB 2 CPU is OK, we can change that later

Start VM

NOTE: Make sure you make a copy of the RAW/QCOW2 fileas that’s the disk the virtual machine will use:).

You should be greeted with login interface

Usually the username root with a blank password should work.

setting a root password and customizing some details

Sometimes this won’t work… so we go the route of setting up a password. There are multiple manual ways to do this via grub editing . However, as a DevOps you will want to use commands to automate your workflow.

Shut down your VM before attempting this. DO it via virt-manager or with virsh shutdown debian13 (or whatever name you gave it!)

THe following command will set the root password, hostname and will run ssh-keygen and start sshd;)

virt-customize -a  /home/andrei/libvirt/debian-13-generic-amd64.raw  --root-password password:MyAmazingPassword --hostname "devopsmagic" --firstboot-command 'ssh-keygen -A && systemctl restart sshd'

Reboot the VM, you should see the VM devopsmagic name changed and you can login with the password for root;).

COmmand Line Usage

First set the libvirt default uri to the system one (root). You can also use the local user’s libvirt session qemu:///session

Accessing QEMU KVM instances via SSH from other machines

Notice the format, you can, you will as you deepen your DevOps knowledge, use something akin to qemu://192.168.135.27/session to access the KVM qemu sessions from different hosts. This will usually be done via SSH as qemu+ssh://username@hostname/system.

export LIBVIRT_DEFAULT_URI=qemu:///system
virsh list

Because you’re a devops, you will want to use the ocmmand line to manage virtual machines:

sudo virt-install \
--name debian13-devops \
--memory 2048 \
--vcpus 2 \
--disk /home/andrei/libvirt/debian-13-devops-amd64.raw \
--import \
--os-variant debian13 \
--noautoconsole

Opening a windowed viewer for the machine

virt-viewer debian13-devops

Opening a console in your terminal

SSH-ing into the console is the best, however virsh allows you to get access to a console within the virtual machine automatically. Meaning you can copy paste commands.

virsh consoloe debian13-devops

cloud init configuration and virtual magine install

Oftentimes you can use cloud init for configuration of cloud images.

#cloud-config

users:
  # whatever username you like
  - name: andrei
    # so our user can just sudo without any password
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    # content from $HOME/.ssh/id_rsa.pub on your host system
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC[the rest of your ssh key here]Z

virt-install --name devopsmagic --memory 2048 --vcpus 4 --disk /home/andrei/libvirt/debian-13-devops-amd64.raw --cloud-init user-data=/home/andrei/libvirt/cloud-init.yaml  --network bridge=virbr0 --os-variant debian13

The previous if run in Linux will automatically open the console ;).

Internet issues in VM?

IF running the cloudinit version internet should WORK properly, when running the non cloud init version networking MAY fail to work.

Run this on host

sudo vim /etc/libvirt/network.conf
firewall_backend=nftables

# :wq! save & exit vim;)

sudo systemctl restart libvirtd

Check ip forwarding

cat /proc/sys/net/ipv4/ip_forward

If it returns 0, enable it: sudo sysctl -w net.ipv4.ip_forward=1

sudo virsh net-start default
sudo virsh net-autostart --network default 

If all else fails… reboot your HOST machine

Explore LibVirt Extra Might and Magic

You can use libvirt, virt-manager, virsh etc to connect to multiple machines running KVM/Linux THis enables failover, high availability and a plethora of other nice features like live migration of

Explore these on your own.

COnclusion

RUnning KVM locally

TODO

TODO still have to explain how to set it up ;) and make a video about it:) but this is for later

What’s next?

We’ll automate things even more with terraform for libvirt;)

https://medium.com/@sydasif78/automating-virtual-machine-provisioning-with-terraform-and-libvirt-c6d438a1a7d2 https://dev.to/ahmed_marzougui_4f08a0e78/building-a-virtual-cloud-lab-with-ansible-kvm-and-libvirt-48k8 https://packetroutes.qzz.io/posts/automating-vm-provisioning-terraform-libvirt/