The setup of nginx is basically the same as described here, but some additional directives have to be configured to make Websockets, used for the noVNC console, work:

cat /etc/nginx/sites-available/proxmox

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

server {
    listen [::]:443 ssl;
    server_name YOUR-FQDN-HERE;
    ssl on;
    ssl_certificate /etc/nginx/ssl/certs/cert.crt;
    ssl_certificate_key /etc/nginx/ssl/certs/cert.key;
    client_max_body_size 5g;
    proxy_read_timeout 300s;
    proxy_send_timeout 300s;
    error_log /var/log/nginx/proxmox/error.log;
    access_log /var/log/nginx/proxmox/access.log;
    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # Also proxy websocket
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        
        # proxy backend
        proxy_pass https://127.0.0.1:8006;
    }
}

The Important parts are:

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

# Also proxy websocket
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

I also increased the send and receive timeouts from default 60 seconds to 5 minutes for the console to not close when no data is transferred.
Additionally the max_body_size was increased to 5 GB to allow uploads of huge ISOs via the Web UI.

Note that this works with Cloudflare just fine, they just recently announced Websocket support for free customers.