EVE Virtual Environment Introduction

by ezri on February 01, 2024
tags: golang hypervisor eve

I have been using Proxmox for around three years now. It was a decent solution for EzriCloud at the time; I only had one hypervisor in the beginning. Now I have three hypervisors in California, and a few scattered around the east coast. The three hypervisors are in a Proxmox cluster. One of the east coast ones is running Proxmox, and the rest are running libvirt and cockpit-machine.

There are a few reasons why I am developing a Proxmox replacement and stopped deploying Proxmox. I will try my best to explain them here, however your milage may vary, and Proxmox is still a great solution for a lot of cases.

Custom kernel - Proxmox uses it’s own kernel, which was not something I am a big fan of. It is built off of a Debian system but with a weird custom Ubuntu based kernel. I’d much rather all of my Linux machines running off of the same kernels for consistency sake.
Firewalling - Proxmox also would rather handling their firewalling their own way, using their own CLIs and file permissions. It makes management for me a pain at a bigger scale.
Clustering - Proxmox cluster is just a pain. You need at least three for a quorum, and for a site with only three hypervisors, losing one would not be fun. Proxmox clustering is also needlessly complicated for my scale.
User management - There isn’t an easy way to manage users across clusters/silos without having to spin up a LDAP server.
Lack of centralized management - You can’t manage all the clusters and silos within the same web interface.

These are the main reasons that I could think of for now. They may not be deal breakers for you, but over the years they add up and make me want to switch away. I considered deploying OpenStack as it is one of the alternatives for bigger scaled operations, but EzriCloud is nowhere as big that justified a whole OpenStack deployment. An OpenStack deployment was also extremely complicated and I heard from many friends that it’d take a lot of effort to maintain.

Here comes Eve and Auto (yes, name was inspired from WALL-E). It’s my first big software engineering project, so please don’t expect greatness from it.

Auto is a daemon that runs on each Libvirt hypervisor. It communicates to Libvirt and other systems on a Linux system to manage virtual machines, firewalls, and storage. Auto exposes a REST HTTP endpoint for Eve to communicate with.

Eve is the main management interface, it communicates to Auto via mTLS (more on that later). Eve is also a REST API with both admin and user routes to manage virtual machines across all the hypervisors via Auto.

The mTLS setup looks like this. On initial Eve deployment, we will generate the certificate authority, and it will sign the Eve certificate. For each Auto deployment, the sysadmin would first generate a keypair, then a certificate signing request. The CSR would then be sent to the Eve server to be signed by the CA, then the certificate would be sent back to Auto. All of these key, CSR, and certificate generation are all done via the eve and auto binaries.

The Eve database will maintain the fingerprint for each auto certificate, along with their connection details. Auto’s configuration file will contain Eve’s certificate fingerprint. Auto will also have the Eve CA’s public key. For each request between Eve and Auto, both sides will verify each other’s fingerprint, and the validity of the chain of trust via the CA.

Eve will also support cloud-init, as that is something that most virtualization solutions should have in the year 2024. I also want to add live-migration support via libvirt, but I’m not sure yet.

Eve is simply the management API, but I will also develop a web interface for both users and admins. I think a command line interface would also be nice. Another idea I had was to have a TUI interface that users and admin can access via ssh, and authentication would be done automatically via user’s public keys.