#proxmox

4 posts.

Setting up Traefik as a Reverse Proxy in Proxmox

I wanted one place to receive requests for my homelab services and send each request to the right machine. Traefik now does that job in an LXC on Proxmox. It replaced Nginx Proxy Manager in my setup.

This is the setup that worked for me: install Traefik with the Proxmox community script, keep the main settings in one file, and add a small file for each service. I’ll use nvr.example.com pointing to an NVR at 192.168.0.155:5000 as the example. Replace those with your own domain and service address.

Before you start

For the Let’s Encrypt method in this guide, you need:

  • A domain you control, with a DNS record for the hostname pointing to your public IP.
  • Ports 80 and 443 forwarded from your router to the Traefik LXC’s IP address.
  • A service that Traefik can reach from the LXC, such as http://192.168.0.155:5000.

Let’s Encrypt must be able to reach Traefik on port 80 for the HTTP-01 challenge. If that isn’t possible in your network, use a different challenge method; the configuration below won’t issue a certificate on its own.

1. Install Traefik in an LXC

Run the command shown on the Traefik community script page in the Proxmox host shell, not inside an existing container:

var_os='debian' bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/traefik.sh)"
Proxmox community script options for Traefik

The script creates the LXC and installs Traefik. Note the LXC’s IP address. Inside the container, the main configuration file is /etc/traefik/traefik.yaml, and Traefik runs as a systemd service.

2. Set up Traefik’s main configuration

Traefik has two kinds of configuration. The static configuration tells it which ports to listen on, where to load route files, and how to request certificates. The dynamic configuration describes which hostname goes to which service.

Merge these settings into /etc/traefik/traefik.yaml inside the LXC. Keep any settings from the install script that you still need, and replace the example email address:

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

providers:
  file:
    directory: /etc/traefik/conf.d/
    watch: true

certificatesResolvers:
  letsencrypt:
    acme:
      email: "[email protected]"
      storage: /etc/traefik/ssl/acme.json
      httpChallenge:
        entryPoint: web

Create /etc/traefik/conf.d/ and /etc/traefik/ssl/ if they don’t exist. Make sure Traefik can write to the ACME storage path. The path in the configuration has to match the file Traefik uses; I once had a typo there and spent time looking for a certificate in the wrong place.

The web entry point receives HTTP traffic on port 80. The websecure entry point receives HTTPS traffic on port 443. Traefik watches conf.d so route changes can be loaded from separate files.

3. Add a route for one service

Create /etc/traefik/conf.d/nvr.yaml:

http:
  routers:
    nvr:
      rule: "Host(`nvr.example.com`)"
      entryPoints:
        - websecure
      service: nvr
      tls:
        certResolver: letsencrypt

  services:
    nvr:
      loadBalancer:
        servers:
          - url: "http://192.168.0.155:5000"

The router matches requests for nvr.example.com on HTTPS. The service is where Traefik sends them. certResolver: letsencrypt tells Traefik to request and manage a certificate for the hostname in the router rule. Defining the resolver in the main file isn’t enough; a TLS router must use it.

Repeat this pattern in another file for each service, changing the router name, hostname, service name, and backend URL.

4. Restart and test

Restart Traefik after changing its main configuration:

systemctl restart traefik
systemctl status traefik

Then visit https://nvr.example.com from outside your network. The request should reach your router, pass to the Traefik LXC, match the nvr router, and reach the NVR. Traefik requests the certificate when it needs one for the matching TLS router; an empty acme.json immediately after startup doesn’t by itself mean the setup is broken.

If the page doesn’t load, follow the request in that same order:

  1. Does the hostname resolve to your public IP?
  2. Do ports 80 and 443 reach the Traefik LXC?
  3. Can the LXC reach the backend URL directly?
  4. Did Traefik load the route file, and does its Host rule match the hostname you visited?

Read the service logs while testing:

journalctl -u traefik -f

I temporarily set log.level: DEBUG in the main configuration when I needed more detail. One useful error was HTTP challenge is not enabled, which pointed back to the missing httpChallenge setting. Remove the extra logging once you’ve found the problem.

Optional: redirect HTTP to HTTPS

I left redirects out while getting certificate issuance working so there was one less setting to debug. If you want HTTP requests to redirect to HTTPS, Traefik supports an entry point redirect:

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: ":443"

Merge that into your existing entryPoints section rather than adding a second one. I previously thought this kind of redirect broke HTTP-01. Traefik’s ACME documentation says the redirect is compatible with the challenge. The real requirements are that port 80 reaches Traefik and that the resolver has httpChallenge.entryPoint set to web.

For more detail on route files, see Traefik’s file provider documentation.

Related: Deploying Kubernetes using Terraform, Proxmox, Ansible, and K3s and Cloning a Tailscale VM: Fixing the Duplicate Node Key.

Proxmox iGPU Passthrough only works on Windows guests

/ Compute /

I wanted to passthrough UHD 630 to a DSM guest. I thought that PCIe passthrough in Proxmox has matured enough that it would be a straightforward task to assign my iGPU Intel UHD 630 to a Linux guest.

This was not the case.

I’m getting the following errors on Linux and Mac guests.

Host error log

DMAR: [DMA Write NO_PASID] Request device [c7:00.0] fault addr 0x27af3000 [fault reason 0x05] PTE Write access is not set

DSM error log

[ 4.093363] i915 0000:01:00.0: Invalid ROM contents
[ 4.095707] [drm:gen9_set_dc_state [i915]] *ERROR* DC state mismatch (0x0 -> 0x2)
[ 4.103321] [drm] Finished loading DMC firmware i915/kbl_dmc_ver1_04.bin (v1.4)
[ 5.754708] i915 0000:01:00.0: Resetting rcs0 after gpu hang
[ 5.755644] i915 0000:01:00.0: Resetting bcs0 after gpu hang
[ 5.756533] i915 0000:01:00.0: Resetting vcs0 after gpu hang
[ 5.757376] i915 0000:01:00.0: Resetting vecs0 after gpu hang
[ 7.707272] i915 0000:01:00.0: Resetting chip after gpu hang
[ 7.708617] i915 0000:01:00.0: GPU recovery failed
[ 7.716928] [drm] Initialized i915 1.6.0 20171222 for 0000:01:00.0 on minor 0
[ 8.107294] i915 0000:01:00.0: fb0: inteldrmfb frame buffer device
[ 8.702947] i915 0000:01:00.0: HDMI-A-1: EDID is invalid:
[ 42.201199] i915 0000:01:00.0: HDMI-A-2: EDID is invalid:

Things I tried

Blacklisting iGPU on boot

I followed https://3os.org/infrastructure/proxmox/gpu-passthrough/igpu-passthrough-to-vm/#proxmox-configuration-for-igpu-full-passthrough with all the modifications to the Proxmox host. Results are the same.

  • Blacklisting iGPU so the host won’t initialize it.

Use a vbios file

I tried using a vbios romfile from https://github.com/patmagauran/i915ovmfPkg to re-initiate the GPU. Results are the same.

  • Copied the rom file to /usr/share/kvm/ and added romfile=<vbios>.rom to hostpci0 in /etc/pve/qemu-server/<VMID>.conf.

SR-IOV

I briefly considered using SR-IOV https://github.com/strongtz/i915-sriov-dkms but it looks like a lot of work. Host and guest have to support it. I have limited control over DSM.

Probably in the future if I have a use-case of passing through iGPU to a guest.

Other useful resources

Related: Enable IOMMU on Proxmox, Plus Real IOMMU Groups and Xpenology DSM 7.2 on Proxmox: Boot Disk, USB Bootloader and AME Patch.

Emulate an NVMe Drive in a Proxmox VM with QEMU Args

/ Compute /

Proxmox does not yet support adding NVME drives using the web interface. It has to be added using a custom args for QEMU.

To do so, open the VM config file in a text editor: vi /etc/pve/qemu-server/100.conf

args: -drive file=/dev/zvol/rpool/data/vm-100-disk-1,if=none,id=nvme1 -device nvme,drive=nvme1,serial=nvme1

file is the location of the image in the hard drive. Since I’m using ZFS, it is in /dev/zvol/rpool/data/vm-100-disk-1.

Proxmox VM hardware configuration screen showing an unattached Unused Disk 1.

I created this image using the Web UI but it is unattached. It is created in a real NVME disk.

Proxmox UI showing a healthy 64 GB M.2 Drive 1 emulated as an NVMe device.

device is how it will show-up in the guest.

QEMU docs on NVME emulation: https://www.qemu.org/docs/master/system/devices/nvme.html

Patch for Proxmox NVME emulation that has not yet been merged: https://bugzilla.proxmox.com/show_bug.cgi?id=2255

Related: Segfault Using an Emulated NVMe as SSD Cache in Synology DSM.

Enable IOMMU on Proxmox, Plus Real IOMMU Groups

This is how I turn on IOMMU in Proxmox before trying PCIe passthrough.

Enable IOMMU in GRUB

vi /etc/default/grub
Find the line with "GRUB_CMDLINE_LINUX_DEFAULT"
Add `intel_iommu=on` or `amd_iommu=on`
Save
update-grub

PCIe ACS Override

There are motherboards that bundle peripherals together (does not support ACS).

A workaround is to override PCIe ACS by adding pcie_acs_override=downstream,multifunction in GRUB_CMDLINE_LINUX_DEFAULT.

This flags the kernel that a device can be “isolated”. It might have security and stability issues.

References:

IOMMU groups on real hardware

Whether passthrough works depends on how the board groups its devices. These are two machines I checked.

Asus Chromebox CN62

Linux device list showing IOMMU values for various hardware components on a Lenovo ThinkCentre.

It cannot passthrough the GPU

Lenovo ThinkCentre M920s

Proxmox PCI device selection menu showing various Intel Corporation devices and their IOMMU group IDs.

iGPU and SATA controller is in its own group

Related: Proxmox iGPU Passthrough only works on Windows guests and Attempted to make XFX RX 460 work on a macOS KVM.