Skip to content

Proxmox setup

Hardware & Network

  • Setup dedicated LAN in pfSense
  • Open admin console: https://192.168.1.1/
  • Configure new interface (PVE#), record gateway details
  • Setup DHCP in [100,200] range, add static IP based on MAC
  • Copy firewall rules from LAN
  • Make a flash drive with proxmox iso and install
  • F2 during startup to select USB boot
  • Test connection: https://192.168.2.10:8006/

PVE setup

  • Shell in with root, password
  • Update deb repository, src
  • nano /etc/apt/sources.list, add contrib non-free non-free-firmware to all 3 sources
  • nano /etc/apt/sources.list.d/pve-enterprise.list
#deb https://enterprise.proxmox.com/debian/pve bookworm pve-enterprise
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription

sudo wget https://enterprise.proxmox.com/debian/proxmox-release-bullseye.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bullseye.gpg
sudo apt install -y libguestfs-tools intel-microcode
  • Collect system stats to help with VM selection 
# Show which CPUs are P (performance) vs E (efficiency)
lscpu --all --extended
# Show total / free RAM
free -h
# Show disk size
lsblk

PCI passthrough

src

GPU

  • Update grub
# Check if grub or systemd-boot
efibootmgr -v
sudo vim /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt initcall_blacklist=sysfb_init"

sudo update-grub
  • Update modules
sudo vim /etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

sudo vim /etc/modprobe.d/pve-blacklist.conf

blacklist nvidiafb
blacklist nvidia
blacklist radeon
blacklist nouveau

blacklist snd_hda_intel
blacklist snd_hda_codec_hdmi
  • Update BIOS settings
  • Under CPU, confirm that VT-d and VT-x/VMX are enabled
  • Under Graphics, make the iGPU the Primary Display
  • Confirm its works and find PCI id
sudo reboot
# Confirm that IOMMU is enabled
sudo dmesg | grep -e DMAR -e IOMMU
# Confirm that remapping is enabled
sudo dmesg | grep 'remapping'
# Confirm dedicated IOMMU groups / ACS support, record GPU group #
find /sys/kernel/iommu_groups/ -type l | sort
# record GPU PCI IDs
lspci -nnv | grep VGA
lspci -s 01:00 && lspci -s 01:00 -n

iGPU

  • Same as above, ref
  • sudo vim /etc/modprobe.d/pve-blacklist.conf
blacklist i915

sudo reboot
lspci -nnv | grep VGA

Intel NIC

  • Fix crashes
    • sudo vim /etc/network/interfaces
iface eno1 inet manual
    post-up ethtool -K eno1 tso off gso off

Coral TPU

  • Update modules sudo vim /etc/modprobe.d/blacklist-apex.conf
blacklist gasket
blacklist apex
options vfio-pci ids=1ac1:089a

sudo reboot
lspci -nnv | grep TPU
  • In VM setup, uncheck "Pre-Enroll keys" in BIOS
  • If doesn't work, consider pcie_aspm=off ref1, ref2, ref3

VM management

Docs

  • Watchdog to prevent stuck VM
sudo su
src/debian/install_svcs.sh vm_watchdog
  • Other tools
cp src/pve2/get_vm_id.sh /usr/local/bin

Networking

  • Remove unnecessary services (not using HA mode)
systemctl disable --now pve-ha-crm.service
systemctl disable --now pve-ha-lrm.service
systemctl disable --now corosync.service

ufw default allow routed
ufw allow in from any to any port 22,8006,3128 proto tcp
ufw allow in from any to any port 5900:5999 proto tcp
ufw allow in from any to any port 111 proto udp
# Excludes corosync and live migration ports
ufw enable

Backups

Only installed on PVE2. Ref

  • Update deb repository
  • vim /etc/apt/sources.list.d/pbs-enterprise.list
# NOT recommended for production use
deb http://download.proxmox.com/debian/pbs bookworm pbs-no-subscription
  • Install PBS, ref
apt update
apt install -y proxmox-backup-server
ufw allow in from any to any port 8007 proto tcp
  • Connect to console: https://192.168.2.10:8007/
  • Further setup, setup2
  • Storage >> Directory >> Create: Directory

  • Datastore >> backup1 >> Prune & GC tab, options

    • Prune Jobs >> Add >> Last weekly: 3, last monthly: 3, daily
    • Garbage Collection >> Edit >> daily

  • PVE setup

  • Datacenter >> Storage >> Add >> Proxmox Backup Server
  • VM >> Backup >> Backup now

  • Datacenter >> Backup >> Add

    • Exclude fingerprint for Let's Encrypt derived PBS certs
    • For the schedule I picked Sunday at 1am (pve1), 2am (pve2)

  • PVE / PBS backups TODO: flesh this out

tar -czf "etc-backup-$(date -I).tar.gz" /etc

Monitoring

  • Install Node Exporter
adduser node_exporter --system
groupadd node_exporter
usermod -a -G node_exporter node_exporter
cd /root/homelab-rendered
src/debian/install_svcs.sh node_exporter
  • Allow access from metrics container in order to scrape node_exporter
# Use 192.168.4.20 on pve1, 192.168.2.20 on pve2
ufw allow in from 192.168.4.20 to any port 9100 proto tcp

Perform these steps after pve1, secsvcs and victoriametrics is configured. Ref - Get the metrics admin password from secsvcs /usr/local/bin/get_secret.sh victoriametrics_admin_password - Go to Datacenter >> Metric Server >> Add >> InfluxDB - Set: - server = metrics.janedoe.com - port = 443 - protocol = https - organization = proxmox - bucket = proxmox - token = admin:PASSWORD

Upgrade

Minor version

Consider pinning the kernel version - PVE 8.4 guide

sudo su
apt update
apt dist-upgrade
reboot
pveversion -v
systemctl status proxmox-backup-proxy.service proxmox-backup.service

From PVE 7 to 8 (bullseye to bookworm)

sudo su
pve7to8 --full
apt update
apt dist-upgrade
pveversion
sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list
sed -i -e 's/bullseye/bookworm/g' /etc/apt/sources.list.d/*.list
apt update
apt dist-upgrade
pve7to8 --full
[ -d /sys/firmware/efi ] && sudo apt install grub-efi-amd64
systemctl reboot

sudo su
systemctl status proxmox-backup-proxy.service proxmox-backup.service
pve7to8 --full
apt update
apt upgrade