diff --git a/README.md b/README.md
index eaf8163f..94c525c9 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,7 @@
项目提供面板程序、Python 依赖清单和可选的 [systemd service 文件](deploy/virsh-panel.service),没有安装器或反向代理配置。面板运行在管理服务器 A,默认只监听 `127.0.0.1:8000`;AlmaLinux 10 的 virsh 宿主机 B 通过已有的 root SSH 密钥连接。B 不需要安装面板、创建新账号或导入新公钥。面板使用 [libvirt 官方 SSH 连接方式](https://www.libvirt.org/uri.html)管理 `qemu:///system`,VNC 经 SSH 连接 B 的本机端口。
-这一版的持久化面板数据使用 **MariaDB**。首次打开网页会先配置数据库连接,再创建管理员和配置宿主机。数据库及有建表权限的专用账号需要你事先准备;面板不会安装 MariaDB 或自动创建数据库。旧版 `data/state.json`、`data/templates.json` 和本地宿主机配置**不会自动迁移**。虚拟机定义和虚拟磁盘仍由远端 libvirt/宿主机持有,数据库保存面板账户、登录会话、模板目录、宿主机连接资料、更新设置和更新任务等面板数据。
+这一版的持久化面板数据使用 **MariaDB**。首次打开网页会先配置数据库连接,再创建管理员和配置宿主机。数据库及有建表权限的专用账号需要你事先准备;面板不会安装 MariaDB 或自动创建数据库。旧版 `data/state.json`、`data/templates.json` 和本地宿主机配置**不会自动迁移**。虚拟机定义和虚拟磁盘仍由远端 libvirt/宿主机持有,数据库保存面板账户、登录会话、模板目录、宿主机连接资料、更新设置、更新任务及虚拟机性能历史等面板数据。
如果是在旧测试目录上替换代码,旧 `data/generations/` 中的明文 SSH 私钥不会被新版使用,也**不会自动删除**。为避免遗留敏感文件,建议先备份并核对后,在首次运行新版前使用全新的 `data/` 目录;不要未经核对直接删除仍需使用的资料。
@@ -49,6 +49,8 @@ PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -m uvicorn virsh_panel.app:create_app
虚拟机列表只展示状态、地区、系统或安装镜像、IP、规格和磁盘;点击虚拟机名称进入该机管理页,控制台、电源、重装和删除等操作都在管理页。地区取自当前宿主机设置。新建或重装的云镜像 VM 会记录所选模板名称及配置的公网 IP/CIDR;这个 IP 是**配置记录**,不是连通性探测。ISO 安装 VM 只记录安装镜像文件名,不能据此确认客体最终安装的系统;旧 VM 或面板外创建的 VM 若无可靠记录,会显示“未识别”或“未获取”,不会猜测客体系统与 IP。
+虚拟机详情页显示 CPU、客体内存、磁盘 I/O 和网络收发四张历史走势图,可切换最近 1 小时、24 小时、7 天和 30 天。面板后台定时从 [libvirt 域统计](https://libvirt.org/manpages/virsh.html#domstats)采样并写入 MariaDB,即使无人打开详情页也会继续记录。查询**只返回最近 30 天**的数据;运行中的面板会分批清理更早的数据库记录。面板停机或数据库不可用时物理清理会暂停,恢复后继续追赶,因此中断期间数据库中可能暂时留有超期记录。CPU 是按 vCPU 数归一化、最高显示 100% 的使用率,磁盘和网络显示每秒速率而不是累计量或网络带宽上限;网络收发方向以虚拟机为准。磁盘与网卡只汇总 libvirt 实际提供计数的设备,未提供计数的设备无法纳入统计。内存占用依赖客体内存气球统计,客户机不提供或统计过期时会显示无数据,不会把宿主机 QEMU 进程占用误当客体内存。初次采样、虚拟机停机、面板/宿主机不可达或计数器重置期间可能出现空白;面板停机期间不会补录历史。较长时间范围会聚合采样,短暂的停机或采样缺口可能不明显。
+
磁盘目录和 ISO 目录填写的是 **B 宿主机**上已存在的绝对路径,例如 `/home/kvm`、`/home/iso`,不是 A 面板服务器的路径;不需要提前手动创建 libvirt 存储池。点击“保存”时,面板会将这些现有目录定义为 libvirt `dir` 存储池并启动,作为内部管理机制;不会创建目录,也不会移动或删除现有文件。新建虚拟机的磁盘文件使用配置的磁盘目录下的 `<虚拟机名>.qcow2`,安装镜像使用配置的 ISO 目录下所选的 `.iso`;虚拟机 XML 通过绝对文件路径引用它们。“测试连接”不会修改宿主机,因此此时若目录尚未注册为池,可能仅显示资源警告。也可继续填写已有的 libvirt 存储池名称。若 B 的目录权限或 SELinux 策略阻止 libvirt 访问,保存会给出错误,需在 B 上修正权限或标签。
“默认虚拟网络或网桥”填写已有 libvirt 虚拟网络名称(如 `default`);若要使用 B 上已有的 Linux 网桥 `br0`,请填写 **`bridge:br0`**。单独填写 `br0` 表示名为 `br0` 的 libvirt 虚拟网络,不表示网桥。面板不会创建网桥;桥接网络的 IP、DHCP 与防火墙由你在 B 上配置。创建 VM 时,面板自动按 vCPU 数设置 virtio 网卡队列,例如 8 vCPU 对应 `driver.queues=8`;按 RHEL 10 建议最多设 16 队列。
@@ -84,4 +86,4 @@ systemctl enable --now virsh-panel.service
独立的 `virsh-panel-update.service` 会先拉取私有仓库并安装目标版本 Python 依赖,再以快进方式切换代码、重启主面板并检查本机 HTTP 与数据库健康状态。更新任务绑定点击“检查更新”时看到的两个提交编号;若本地或远端版本变化,任务会拒绝切换并要求重新检查。更新任务状态加密存于 MariaDB,页面中断后可继续查询;独立服务异常退出或超时会尝试将未完成任务标为失败。更新前请备份 **MariaDB 数据和 `data/master.key`、`data/database-config.json`**;更新失败时不要重复点按钮,应先看 `journalctl -u virsh-panel-update.service -n 100`。依赖在代码切换前安装到当前 `.venv`,因此若依赖安装成功而代码切换失败,可能需要人工检查依赖版本。更新服务会保留更新前的 Git 回滚引用,但不会自动执行覆盖性的回滚。网页更新不负责安装系统 RPM、替换 `/etc/systemd/system` 中已安装的 unit 或修改反向代理配置;这些变化如后续版本需要,仍需人工处理。由于在线更新会运行私有仓库中的代码与依赖,必须保护 Gitea 仓库写入权限和访问令牌。
-功能范围:虚拟机列表、从 ISO 创建、导入 AlmaLinux 10 云镜像模板并配置公网静态 IP 创建、云镜像重装、开机、关机、重启、强制断电/重置、删除和浏览器 VNC。当前为单管理员、单宿主机版本;不提供 ISO 上传、目录自动创建、虚拟网络创建、内网 IP 模板、异网段网关或多用户权限。真实 AlmaLinux 10 宿主机及 MariaDB 仍需现场联调。
+功能范围:虚拟机列表和详情页四项性能走势图、从 ISO 创建、导入 AlmaLinux 10 云镜像模板并配置公网静态 IP 创建、云镜像重装、开机、关机、重启、强制断电/重置、删除和浏览器 VNC。当前为单管理员、单宿主机版本;不提供 ISO 上传、目录自动创建、虚拟网络创建、内网 IP 模板、异网段网关或多用户权限。真实 AlmaLinux 10 宿主机及 MariaDB 仍需现场联调。
diff --git a/static/app.js b/static/app.js
index ab5f1cff..7d98e3c4 100644
--- a/static/app.js
+++ b/static/app.js
@@ -9,6 +9,11 @@ const state = {
vms: [],
selectedVmId: null,
vmActionInFlight: null,
+ metricsRange: "24h",
+ metricsPollTimer: null,
+ metricsRequest: null,
+ metricsPayload: null,
+ metricsResizeTimer: null,
resources: null,
templateLoad: null,
templateImportJobId: null,
@@ -88,9 +93,11 @@ async function api(path, options = {}) {
headers,
credentials: "same-origin",
cache: "no-store",
+ signal: options.signal,
body: options.body === undefined ? undefined : JSON.stringify(options.body),
});
- } catch {
+ } catch (error) {
+ if (options.signal?.aborted) throw error;
throw new Error("无法连接面板服务,请检查网络或稍后重试。");
}
@@ -121,6 +128,7 @@ function stopPolling() {
function showLogin() {
stopPolling();
+ stopVmMetrics();
closeModal();
state.updatePollToken += 1;
state.updateJobId = null;
@@ -145,6 +153,7 @@ function showLogin() {
function showDatabaseView() {
stopPolling();
+ stopVmMetrics();
closeModal();
byId("boot-screen").hidden = true;
byId("setup-view").hidden = true;
@@ -157,6 +166,7 @@ function showDatabaseView() {
function showSetup() {
stopPolling();
+ stopVmMetrics();
closeModal();
byId("unlock-passphrase").value = "";
byId("boot-screen").hidden = true;
@@ -178,6 +188,7 @@ function clearHostPrivateKey() {
function showUnlockView(hostConfig) {
stopPolling();
+ stopVmMetrics();
closeModal();
state.hostConfig = hostConfig;
clearHostPrivateKey();
@@ -212,6 +223,7 @@ function markHostTestStale() {
function showHostView(hostConfig = { configured: false }) {
stopPolling();
+ stopVmMetrics();
closeModal();
state.hostConfig = hostConfig;
clearHostPrivateKey();
@@ -298,6 +310,7 @@ async function initialize() {
}
function showVmWorkspace() {
+ stopVmMetrics();
state.selectedVmId = null;
byId("vm-workspace").hidden = false;
byId("vm-list-view").hidden = false;
@@ -340,6 +353,7 @@ function leaveUpdateWorkspace() {
}
async function showUpdateWorkspace() {
+ stopVmMetrics();
byId("vm-workspace").hidden = true;
byId("update-workspace").hidden = false;
byId("vm-navigation-button").classList.remove("active");
@@ -803,10 +817,13 @@ function openVmDetail(uuid) {
toast("虚拟机已不在当前列表中,请刷新后重试。", "error");
return;
}
+ stopVmMetrics();
state.selectedVmId = uuid;
byId("vm-list-view").hidden = true;
byId("vm-detail-view").hidden = false;
renderVmDetail(vm);
+ clearVmMetrics("正在加载监控数据…");
+ startVmMetrics();
window.scrollTo({ top: 0, behavior: "auto" });
byId("vm-detail-name").focus();
}
@@ -886,6 +903,238 @@ function renderVmDetail(vm) {
}
}
+const VM_METRICS_RANGE_SECONDS = { "1h": 3600, "24h": 86400, "7d": 604800, "30d": 2592000 };
+const VM_METRIC_CHARTS = [
+ { id: "cpu", title: "CPU 使用率", unit: "percent", series: [{ key: "cpu_percent", color: "#0f9eaa" }] },
+ { id: "memory", title: "内存用量", unit: "memory", series: [{ key: "memory_used_mib", color: "#4781cf" }, { key: "memory_total_mib", color: "#a4b7d7", dash: "6 5" }] },
+ { id: "disk", title: "磁盘 I/O", unit: "rate", series: [{ key: "disk_read_bps", color: "#477dc9" }, { key: "disk_write_bps", color: "#e59c46" }] },
+ { id: "network", title: "网络速率", unit: "rate", series: [{ key: "network_rx_bps", color: "#17a685" }, { key: "network_tx_bps", color: "#926ed0" }] },
+];
+
+function metricViewVisible() {
+ return Boolean(state.selectedVmId) && !document.hidden && !byId("dashboard-view").hidden
+ && !byId("vm-workspace").hidden && !byId("vm-detail-view").hidden;
+}
+
+function stopVmMetrics() {
+ if (state.metricsPollTimer) window.clearInterval(state.metricsPollTimer);
+ state.metricsPollTimer = null;
+ if (state.metricsResizeTimer) window.clearTimeout(state.metricsResizeTimer);
+ state.metricsResizeTimer = null;
+ state.metricsRequest?.controller.abort();
+ state.metricsRequest = null;
+}
+
+function startVmMetrics() {
+ if (!metricViewVisible()) return;
+ void loadVmMetrics();
+ if (!state.metricsPollTimer) {
+ state.metricsPollTimer = window.setInterval(() => {
+ if (metricViewVisible()) void loadVmMetrics();
+ }, 60000);
+ }
+}
+
+function setVmMetricsStatus(message, warning = false) {
+ const status = byId("vm-metrics-status");
+ status.textContent = message;
+ status.classList.toggle("warning", warning);
+}
+
+function clearVmMetrics(message) {
+ state.metricsPayload = null;
+ setVmMetricsStatus(message);
+ for (const chart of VM_METRIC_CHARTS) {
+ text(`vm-metric-${chart.id}-summary`, "—");
+ byId(`vm-metric-${chart.id}-plot`).replaceChildren(makeElement("p", "vm-metric-empty", message));
+ }
+}
+
+async function loadVmMetrics() {
+ if (!metricViewVisible()) return;
+ const vmId = state.selectedVmId;
+ const range = state.metricsRange;
+ const previous = state.metricsRequest;
+ if (previous?.vmId === vmId && previous.range === range) return;
+ previous?.controller.abort();
+ const request = { vmId, range, controller: new AbortController() };
+ state.metricsRequest = request;
+ setVmMetricsStatus("正在更新监控数据…");
+ try {
+ const payload = await api(`/api/v1/vms/${encodeURIComponent(vmId)}/metrics?range=${encodeURIComponent(range)}`, {
+ signal: request.controller.signal,
+ });
+ if (request !== state.metricsRequest || !metricViewVisible()) return;
+ if (!Array.isArray(payload?.points) || (payload.vm_id && payload.vm_id !== vmId)) {
+ throw new Error("监控数据格式无效。");
+ }
+ renderVmMetrics(payload);
+ } catch (error) {
+ if (request !== state.metricsRequest || request.controller.signal.aborted || !metricViewVisible()) return;
+ setVmMetricsStatus(`监控数据暂时无法加载:${error.message}。已显示的数据可能不是最新。`, true);
+ } finally {
+ if (request === state.metricsRequest) state.metricsRequest = null;
+ }
+}
+
+function metricNumber(value) {
+ return typeof value === "number" && Number.isFinite(value) && value >= 0 ? value : null;
+}
+
+function formatMetricRate(value) {
+ if (value === null) return "无统计";
+ const units = ["B/s", "KB/s", "MB/s", "GB/s", "TB/s"];
+ let amount = value;
+ let index = 0;
+ while (amount >= 1000 && index < units.length - 1) { amount /= 1000; index += 1; }
+ return `${amount.toFixed(index === 0 ? (amount < 1 && amount > 0 ? 2 : 0) : amount < 10 ? 2 : 1)} ${units[index]}`;
+}
+
+function formatMetricMemory(value) {
+ if (value === null) return "无统计";
+ return value >= 1024 ? `${(value / 1024).toFixed(value < 10240 ? 2 : 1)} GiB` : `${Math.round(value)} MiB`;
+}
+
+function formatMetricValue(value, unit) {
+ if (value === null) return "无统计";
+ if (unit === "percent") return `${value.toFixed(1)}%`;
+ if (unit === "memory") return formatMetricMemory(value);
+ return formatMetricRate(value);
+}
+
+function formatMetricTime(timestamp) {
+ return new Date(timestamp * 1000).toLocaleString("zh-CN", {
+ month: "numeric", day: "numeric", hour: "2-digit", minute: "2-digit",
+ });
+}
+
+function metricSummary(chart, points) {
+ const latest = [...points].reverse().find((point) => chart.series.some((series) => metricNumber(point[series.key]) !== null));
+ if (!latest) return "暂无统计";
+ const value = (key) => metricNumber(latest[key]);
+ if (chart.id === "cpu") return formatMetricValue(value("cpu_percent"), "percent");
+ if (chart.id === "memory") return `已使用(估算)${formatMetricMemory(value("memory_used_mib"))} · 总量 ${formatMetricMemory(value("memory_total_mib"))}`;
+ if (chart.id === "disk") return `读取 ${formatMetricRate(value("disk_read_bps"))} · 写入 ${formatMetricRate(value("disk_write_bps"))}`;
+ return `接收 ${formatMetricRate(value("network_rx_bps"))} · 发送 ${formatMetricRate(value("network_tx_bps"))}`;
+}
+
+function metricChartEmptyText(chart) {
+ if (chart.id === "memory") return "暂无内存统计;虚拟机可能未提供气球驱动数据。";
+ if (chart.id === "disk" || chart.id === "network") return "暂无速率数据;首次采样或停机时无法计算速率。";
+ return "当前时间范围内暂无 CPU 采样数据。";
+}
+
+function metricNiceMax(value) {
+ if (value <= 0) return 1;
+ const base = 10 ** Math.floor(Math.log10(value));
+ for (const step of [1, 1.5, 2, 2.5, 5, 10]) {
+ if (value <= step * base) return step * base;
+ }
+ return 10 * base;
+}
+
+function metricSvgElement(tag, attributes = {}) {
+ const element = document.createElementNS("http://www.w3.org/2000/svg", tag);
+ for (const [name, value] of Object.entries(attributes)) element.setAttribute(name, String(value));
+ return element;
+}
+
+function drawMetricChart(chart, points, start, end, intervalSeconds) {
+ const plot = byId(`vm-metric-${chart.id}-plot`);
+ const summary = metricSummary(chart, points);
+ text(`vm-metric-${chart.id}-summary`, summary);
+ const hasData = points.some((point) => chart.series.some((series) => metricNumber(point[series.key]) !== null));
+ if (!hasData) {
+ plot.replaceChildren(makeElement("p", "vm-metric-empty", metricChartEmptyText(chart)));
+ return;
+ }
+
+ const values = points.flatMap((point) => chart.series.map((series) => metricNumber(point[series.key]))).filter((value) => value !== null);
+ const maximum = chart.unit === "percent" ? Math.max(100, metricNiceMax(Math.max(...values))) : metricNiceMax(Math.max(...values));
+ // A fixed 640px viewBox shrinks axis text to ~4px inside a phone card.
+ const width = Math.max(240, Math.floor(plot.clientWidth || 640));
+ const height = Math.max(180, Math.floor(plot.clientHeight || 216));
+ const left = 64;
+ const right = 18;
+ const top = 14;
+ const bottom = 37;
+ const plotWidth = width - left - right;
+ const plotHeight = height - top - bottom;
+ const x = (timestamp) => left + Math.max(0, Math.min(1, (timestamp - start) / (end - start))) * plotWidth;
+ const y = (value) => top + (1 - Math.min(value, maximum) / maximum) * plotHeight;
+ const svg = metricSvgElement("svg", { viewBox: `0 0 ${width} ${height}`, role: "img", "aria-label": `${chart.title},${byId("vm-metrics-range").selectedOptions[0]?.textContent || "当前范围"};最近记录:${summary}` });
+ for (const fraction of [1, 0.5, 0]) {
+ const lineY = top + (1 - fraction) * plotHeight;
+ svg.append(metricSvgElement("line", { x1: left, x2: width - right, y1: lineY, y2: lineY, class: "vm-metric-gridline" }));
+ const label = metricSvgElement("text", { x: left - 8, y: lineY + 4, "text-anchor": "end", class: "vm-metric-axis-label" });
+ label.textContent = formatMetricValue(fraction * maximum, chart.unit);
+ svg.append(label);
+ }
+ for (const [fraction, anchor] of [[0, "start"], [0.5, "middle"], [1, "end"]]) {
+ const label = metricSvgElement("text", { x: left + fraction * plotWidth, y: height - 8, "text-anchor": anchor, class: "vm-metric-axis-label" });
+ label.textContent = formatMetricTime(start + fraction * (end - start));
+ svg.append(label);
+ }
+
+ const maxGap = intervalSeconds * 2.5;
+ for (const series of chart.series) {
+ const segments = [];
+ let segment = [];
+ for (const point of points) {
+ const value = metricNumber(point[series.key]);
+ if (value === null) {
+ if (segment.length) segments.push(segment);
+ segment = [];
+ continue;
+ }
+ if (segment.length && point.timestamp - segment[segment.length - 1].timestamp > maxGap) {
+ segments.push(segment);
+ segment = [];
+ }
+ segment.push({ timestamp: point.timestamp, value });
+ }
+ if (segment.length) segments.push(segment);
+ for (const group of segments) {
+ if (group.length > 1) {
+ const path = metricSvgElement("path", {
+ d: group.map((point, index) => `${index ? "L" : "M"}${x(point.timestamp).toFixed(2)} ${y(point.value).toFixed(2)}`).join(" "),
+ fill: "none", stroke: series.color, "stroke-width": 2.6,
+ "stroke-linecap": "round", "stroke-linejoin": "round",
+ ...(series.dash ? { "stroke-dasharray": series.dash } : {}),
+ });
+ svg.append(path);
+ }
+ const last = group[group.length - 1];
+ svg.append(metricSvgElement("circle", { cx: x(last.timestamp), cy: y(last.value), r: group.length === 1 ? 3.8 : 2.8, fill: series.color }));
+ }
+ }
+ plot.replaceChildren(svg);
+}
+
+function renderVmMetrics(payload) {
+ state.metricsPayload = payload;
+ const intervalSeconds = Math.max(1, Number(payload.interval_seconds) || 60);
+ const rangeSeconds = VM_METRICS_RANGE_SECONDS[state.metricsRange];
+ const now = Math.floor(Date.now() / 1000);
+ const points = payload.points
+ .filter((point) => Number.isFinite(Number(point?.timestamp)) && Number(point.timestamp) >= now - rangeSeconds - intervalSeconds && Number(point.timestamp) <= now + intervalSeconds)
+ .map((point) => ({ ...point, timestamp: Number(point.timestamp) }))
+ .sort((a, b) => a.timestamp - b.timestamp);
+ const end = Math.max(now, points.length ? points[points.length - 1].timestamp : now);
+ const start = end - rangeSeconds;
+ const visiblePoints = points.filter((point) => point.timestamp >= start && point.timestamp <= end);
+ for (const chart of VM_METRIC_CHARTS) drawMetricChart(chart, visiblePoints, start, end, intervalSeconds);
+ if (!visiblePoints.length) {
+ setVmMetricsStatus("当前时间范围内没有采样记录。虚拟机运行后,通常需等待下一次后台采样。", true);
+ return;
+ }
+ const latest = visiblePoints[visiblePoints.length - 1].timestamp;
+ // Bucket width grows for long ranges; freshness still follows the 60s
+ // collector cadence, not the downsampling bucket width.
+ const stale = end - latest > 60 * 2.5;
+ setVmMetricsStatus(`${visiblePoints.length} 个采样点 · ${stale ? "最近历史记录" : "最近采样"} ${formatMetricTime(latest)} · 每分钟自动刷新${stale ? "(当前无新数据)" : ""}`, stale);
+}
+
function resourceValue(item, kind) {
if (typeof item === "string") return item;
if (item && typeof item === "object") return String(kind === "iso" ? (item.path || item.name || "") : (item.name || item.id || ""));
@@ -1634,7 +1883,30 @@ byId("update-apply-button").addEventListener("click", async () => {
});
byId("refresh-button").addEventListener("click", () => void loadVMs());
-byId("vm-detail-refresh-button").addEventListener("click", () => void loadVMs());
+byId("vm-detail-refresh-button").addEventListener("click", () => {
+ void loadVMs();
+ void loadVmMetrics();
+});
+byId("vm-metrics-range").addEventListener("change", (event) => {
+ const range = event.currentTarget.value;
+ if (!Object.hasOwn(VM_METRICS_RANGE_SECONDS, range)) return;
+ state.metricsRange = range;
+ stopVmMetrics();
+ clearVmMetrics("正在加载所选时间范围的监控数据…");
+ startVmMetrics();
+});
+window.addEventListener("resize", () => {
+ if (!metricViewVisible() || !state.metricsPayload) return;
+ if (state.metricsResizeTimer) window.clearTimeout(state.metricsResizeTimer);
+ state.metricsResizeTimer = window.setTimeout(() => {
+ state.metricsResizeTimer = null;
+ if (metricViewVisible() && state.metricsPayload) renderVmMetrics(state.metricsPayload);
+ }, 120);
+});
+document.addEventListener("visibilitychange", () => {
+ if (document.hidden) stopVmMetrics();
+ else startVmMetrics();
+});
byId("vm-detail-back-button").addEventListener("click", () => {
showVmWorkspace();
byId("search-input").focus();
diff --git a/static/index.html b/static/index.html
index 36a2cc5f..4347bae8 100644
--- a/static/index.html
+++ b/static/index.html
@@ -5,8 +5,8 @@
VirtPilot · 虚拟机管理
-
-
+
+
@@ -235,6 +235,17 @@
维护与危险操作
强制操作可能丢失未保存的数据;重装或删除会清除数据。
+
+ PERFORMANCE HISTORY性能监控
每分钟采样并自动刷新,历史数据最长保留 30 天;长时间范围会聚合,较长的采样缺口留空。
+ 选择虚拟机后加载监控数据。
+
+
+
diff --git a/static/styles.css b/static/styles.css
index 92967844..f5041032 100644
--- a/static/styles.css
+++ b/static/styles.css
@@ -125,6 +125,7 @@ tbody tr:hover{background:#fcfefe}
.empty-state{text-align:center;padding:65px 20px 72px}.empty-icon{margin:0 auto 16px;width:54px;height:54px;display:flex;align-items:center;justify-content:center;border-radius:16px;background:#eaf7f7;color:#17a3aa;font-size:28px}.empty-state h3{font-size:16px;color:#34465a;margin:0 0 8px}.empty-state p{font-size:12px;color:#a0aebe;margin:0}
.footer-note{text-align:right;color:#a9b4c0;font-size:10px;margin-top:18px}
.vm-detail-back{padding-left:0;margin-bottom:22px;color:#267b86}.vm-detail-heading{align-items:center}.vm-detail-heading .heading-actions{align-items:center}.vm-detail-heading .state-pill{min-height:31px}.vm-detail-uuid{font-family:ui-monospace,SFMono-Regular,Menlo,monospace;overflow-wrap:anywhere}.vm-detail-grid{display:grid;grid-template-columns:minmax(0,1.15fr) minmax(320px,.85fr);gap:18px;align-items:start}.vm-detail-panel{min-height:0}.vm-detail-facts{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:0;margin:0;padding:6px 24px 17px}.vm-detail-facts>div{min-width:0;padding:17px 9px 17px 0;border-bottom:1px solid #edf1f5}.vm-detail-facts dt{font-size:11px;font-weight:700;color:#91a0af;margin-bottom:9px}.vm-detail-facts dd{margin:0;color:#273c52;font-size:13px;font-weight:700;line-height:1.5;overflow-wrap:anywhere}.vm-detail-operations{display:grid;gap:18px}.vm-detail-actions{display:flex;flex-wrap:wrap;gap:10px;padding:20px 24px 23px}.vm-detail-actions .row-button{min-height:39px;padding:9px 15px;border-color:#dce9ea;font-size:12px}.vm-detail-actions .row-button.primary{border-color:#cbecee}.vm-danger-panel .panel-heading h2{color:#a4434f}.vm-danger-panel .row-button.danger{border-color:#f2dfe2;background:#fff9f9}.vm-danger-panel .row-button.danger:not(:disabled):hover{background:#ffedef}
+.vm-metrics{margin-top:25px}.vm-metrics-heading{display:flex;align-items:flex-end;justify-content:space-between;gap:18px;margin-bottom:14px}.vm-metrics-heading h2{font-size:19px;margin:7px 0 6px}.vm-metrics-heading p{font-size:11px;color:#8191a1;line-height:1.6;margin:0}.vm-metrics-range-control{display:flex;align-items:center;gap:9px;flex:none;color:#64768a;font-size:11px;font-weight:700}.vm-metrics-range-control select{height:37px;padding:0 11px;border:1px solid #dbe6ec;border-radius:8px;color:#344e60;background:#fff;font-size:12px;font-weight:700}.vm-metrics-status{margin:0 0 14px;padding:9px 12px;border-radius:8px;background:#eaf7f7;color:#277b82;font-size:11px;line-height:1.5}.vm-metrics-status.warning{background:#fff6e9;color:#946620}.vm-metrics-grid{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:17px}.vm-metric-card{min-width:0;padding:20px 21px 16px;border:1px solid #e7ecf2;border-radius:13px;background:#fff;box-shadow:0 10px 30px rgba(19,42,65,.027)}.vm-metric-card-heading{display:flex;align-items:flex-start;gap:11px;min-height:62px}.vm-metric-card-heading>div{min-width:0}.vm-metric-card-heading h3{margin:1px 0 6px;font-size:13px;color:#32485b}.vm-metric-card-heading p{margin:0;color:#60758a;font-size:12px;line-height:1.45;overflow-wrap:anywhere}.vm-metric-icon{display:flex;align-items:center;justify-content:center;flex:none;width:34px;height:34px;border-radius:9px;font-size:18px;font-weight:700}.vm-metric-icon.cpu{background:#e8f8f8;color:#0f9eaa}.vm-metric-icon.memory{background:#edf3fb;color:#4781cf}.vm-metric-icon.disk{background:#eef3fa;color:#477dc9}.vm-metric-icon.network{background:#eaf8f2;color:#17a685}.vm-metric-plot{display:flex;align-items:center;justify-content:center;height:205px;margin:8px 0 6px;overflow:hidden}.vm-metric-plot svg{display:block;width:100%;height:100%;font-family:Inter,"SF Pro Display","PingFang SC","Microsoft YaHei",sans-serif}.vm-metric-gridline{stroke:#e5edf2;stroke-width:1;stroke-dasharray:3 4}.vm-metric-axis-label{fill:#9aabb9;font-size:10px}.vm-metric-empty{margin:0;padding:15px;text-align:center;color:#94a2af;font-size:11px;line-height:1.6}.vm-metric-legend{display:flex;align-items:center;flex-wrap:wrap;gap:15px;padding-left:7px;color:#728297;font-size:10px;font-weight:700}.vm-metric-legend span{display:inline-flex;align-items:center;gap:6px}.vm-metric-swatch{display:inline-block;width:16px;height:0;border-top:3px solid #0f9eaa;border-radius:2px}.vm-metric-swatch.cpu{border-color:#0f9eaa}.vm-metric-swatch.memory{border-color:#4781cf}.vm-metric-swatch.memory-total{border-color:#a4b7d7;border-top-style:dashed}.vm-metric-swatch.disk-read{border-color:#477dc9}.vm-metric-swatch.disk-write{border-color:#e59c46}.vm-metric-swatch.network-rx{border-color:#17a685}.vm-metric-swatch.network-tx{border-color:#926ed0}.vm-metrics-footnote{margin:12px 1px 0;color:#9ba8b6;font-size:10px;line-height:1.6}
.modal-backdrop{position:fixed;inset:0;z-index:30;background:rgba(11,25,40,.58);display:flex;align-items:center;justify-content:center;padding:20px;backdrop-filter:blur(3px)}
.modal{width:min(100%,610px);max-height:min(92vh,900px);overflow:auto;border-radius:14px;background:#fff;box-shadow:0 22px 70px rgba(0,0,0,.2)}.narrow-modal{width:min(100%,460px)}
.create-modal{width:min(100%,720px)}
@@ -149,6 +150,7 @@ tbody tr:hover{background:#fcfefe}
@media(max-width:960px){.host-columns{grid-template-columns:1fr}.host-side-stack{grid-template-columns:repeat(2,minmax(0,1fr));align-items:start}.setup-layout{grid-template-columns:minmax(280px,36%) 1fr}}
@media(max-width:960px){.update-grid{grid-template-columns:1fr}}
@media(max-width:960px){.vm-detail-grid{grid-template-columns:1fr}}
+@media(max-width:740px){.vm-metrics-grid{grid-template-columns:1fr}.vm-metrics-heading{align-items:flex-start;flex-direction:column}.vm-metric-plot{height:190px}}
@media(max-width:740px){.login-layout{display:block;min-height:100vh}.login-art{display:none}.login-form-side{min-height:100vh;padding:32px 25px}.login-mobile-brand{display:flex;align-items:center;gap:10px;font-size:21px;font-weight:800;margin-bottom:70px}.login-card h2{font-size:29px}.page-heading{align-items:flex-start;flex-direction:column;gap:20px}.heading-actions{width:100%}.heading-actions .button{flex:1}.overview-grid{gap:9px}.stat-card{min-height:121px;padding:16px}.stat-card>strong{font-size:25px}.stat-top{font-size:10px}.stat-icon{width:27px;height:27px;font-size:12px}.stat-card>small{font-size:9px}.panel-heading{align-items:flex-start;flex-direction:column;padding:19px}.panel-tools{width:100%}.search-box{flex:1}.search-box input{width:100%}.footer-note{text-align:left}.console-toolbar{align-items:flex-start}.console-tools{flex-wrap:wrap;justify-content:flex-end}.console-tools .button{font-size:10px}}
@media(max-width:740px){.vm-table{min-width:0}.vm-table thead{display:none}.vm-table tbody{display:grid;gap:12px;padding:12px}.vm-table tr{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));border:1px solid #e4ebf0;border-radius:10px;overflow:hidden}.vm-table td{display:block;min-width:0;height:auto;border:0;padding:12px 14px}.vm-table td:first-child{grid-column:1/-1;border-bottom:1px solid #edf1f5}.vm-table td:not(:first-child):before{content:attr(data-label);display:block;color:#9ba8b7;font-size:10px;font-weight:800;margin-bottom:7px}.vm-table .vm-metadata{max-width:none}.vm-table .vm-ip{white-space:normal}.vm-detail-facts{padding:4px 17px 12px}.vm-detail-actions{padding:16px 19px 19px}}
@media(max-width:740px){.setup-layout{display:block}.setup-intro{min-height:0;padding:28px 25px}.setup-brand{margin:0 0 38px}.setup-intro h1{font-size:31px;margin:11px 0}.setup-intro p{font-size:12px;margin-bottom:24px}.setup-steps{margin-top:0}.setup-form-side{padding:36px 25px 55px}.setup-card h2{font-size:27px}.host-side-stack{grid-template-columns:1fr}.host-heading{align-items:flex-start;flex-direction:column}.host-topbar{height:65px}.host-content{padding-top:27px}.host-card{padding:20px}}
diff --git a/tests/test_app.py b/tests/test_app.py
index 99749267..9a7b2f68 100644
--- a/tests/test_app.py
+++ b/tests/test_app.py
@@ -33,6 +33,7 @@ from virsh_panel.cloud_templates import ( # noqa: E402
)
from virsh_panel.config import Settings # noqa: E402
from virsh_panel.libvirt_service import VMError # noqa: E402
+from virsh_panel.metrics import host_scope, query_interval_seconds # noqa: E402
from virsh_panel.panel_state import PanelState # noqa: E402
from virsh_panel.remote_host import RemoteHostError # noqa: E402
@@ -997,3 +998,61 @@ def test_websocket_origin_auth_and_console_relay(web):
assert relay.await_args.kwargs["port"] == 5907
assert relay.await_args.kwargs["ssh_target"] == "vp-managed"
assert relay.await_args.kwargs["ssh_config_file"] == "/fake/config"
+
+
+def test_vm_metrics_requires_auth_validates_vm_and_isolates_host(tmp_path):
+ state = PanelState(tmp_path / "panel-data")
+ manager = FakeRemoteManager()
+ service = FakeVMService()
+
+ class FakeMetricStore:
+ def __init__(self):
+ self.calls = []
+ self.allowed_scope = None
+
+ def query_metric_samples(self, scope, vm_id, **kwargs):
+ self.calls.append((scope, vm_id, kwargs))
+ if scope != self.allowed_scope:
+ return []
+ return [{"timestamp": 2_999_900, "cpu_percent": 25,
+ "memory_used_mib": None, "memory_total_mib": None,
+ "disk_read_bps": 128, "disk_write_bps": 0,
+ "network_rx_bps": 4096, "network_tx_bps": 1024}]
+
+ metric_store = FakeMetricStore()
+ app = create_app(Settings(data_dir=state.data_dir, static_dir=STATIC_DIR),
+ panel_state=state, remote_manager=manager,
+ service_factory=lambda _profile: service,
+ metric_store=metric_store)
+ with TestClient(app, base_url=ORIGIN) as client:
+ endpoint = f"/api/v1/vms/{VM_ID}/metrics"
+ assert client.get(endpoint).status_code == 401
+ token = state.token_file.read_text(encoding="ascii").strip()
+ csrf = setup_admin(client, token)
+ configure_host(client, csrf)
+ metric_store.allowed_scope = host_scope(manager.current())
+ with patch("virsh_panel.app.time.time", return_value=3_000_000):
+ response = client.get(endpoint + "?range=30d")
+ assert response.status_code == 200
+ body = response.json()
+ assert body["vm_id"] == VM_ID
+ assert body["range"] == "30d"
+ assert body["interval_seconds"] == query_interval_seconds(30 * 24 * 3600)
+ assert len(body["points"]) == 1
+ scope, vm_id, args = metric_store.calls[-1]
+ assert scope == metric_store.allowed_scope and vm_id == VM_ID
+ assert args["since"] == 3_000_000 - 30 * 24 * 3600
+ assert args["until"] == 3_000_000 and args["limit"] == 600
+
+ assert client.get(endpoint + "?range=60d").status_code == 422
+ assert client.get("/api/v1/vms/not-a-uuid/metrics").status_code == 400
+ assert client.get(
+ "/api/v1/vms/00000000-0000-4000-8000-000000000002/metrics"
+ ).status_code == 404
+ before = len(metric_store.calls)
+ manager.profile["fingerprint"] = "SHA256:another-host"
+ with patch("virsh_panel.app.time.time", return_value=3_000_000):
+ changed_host = client.get(endpoint)
+ assert changed_host.status_code == 200
+ assert changed_host.json()["points"] == []
+ assert len(metric_store.calls) == before + 1
diff --git a/tests/test_libvirt_service.py b/tests/test_libvirt_service.py
index e6b6216b..e3d65262 100644
--- a/tests/test_libvirt_service.py
+++ b/tests/test_libvirt_service.py
@@ -1588,6 +1588,68 @@ class VMServiceTests(unittest.TestCase):
self.assertNotIn(TEMPLATE_NAME, self.backend.pools["images"].volumes)
self.assertEqual(self.backend.opened, self.backend.closed)
+ def test_metric_counters_use_bulk_libvirt_stats_for_running_domains(self):
+ constants = {
+ "VIR_DOMAIN_STATS_CPU_TOTAL": 2,
+ "VIR_DOMAIN_STATS_BALLOON": 4,
+ "VIR_DOMAIN_STATS_BLOCK": 8,
+ "VIR_DOMAIN_STATS_INTERFACE": 16,
+ "VIR_CONNECT_GET_ALL_DOMAINS_STATS_RUNNING": 32,
+ }
+ for name, value in constants.items():
+ setattr(self.service_module.libvirt, name, value)
+ domain = types.SimpleNamespace(
+ UUIDString=lambda: "00000000-0000-4000-8000-000000000001",
+ info=lambda: (1, 2048 * 1024, 2048 * 1024, 2, 0),
+ ID=lambda: 7,
+ )
+ calls = []
+
+ def get_stats(_connection, groups, flags):
+ calls.append((groups, flags))
+ return [(domain, {"cpu.time": 123, "net.0.rx.bytes": 456})]
+
+ with patch.object(FakeConnection, "getAllDomainStats", get_stats, create=True):
+ result = self.service.metric_counters()
+ self.assertEqual(calls, [(2 | 4 | 8 | 16, 32)])
+ self.assertEqual(len(result), 1)
+ self.assertEqual(result[0].generation, 7)
+ self.assertEqual(result[0].vcpus, 2)
+ self.assertEqual(result[0].values["net.0.rx.bytes"], 456)
+ self.assertEqual(self.backend.opened, self.backend.closed)
+
+ def test_metric_counters_skip_vm_disappearing_after_bulk_read(self):
+ constants = {
+ "VIR_DOMAIN_STATS_CPU_TOTAL": 2,
+ "VIR_DOMAIN_STATS_BALLOON": 4,
+ "VIR_DOMAIN_STATS_BLOCK": 8,
+ "VIR_DOMAIN_STATS_INTERFACE": 16,
+ "VIR_CONNECT_GET_ALL_DOMAINS_STATS_RUNNING": 32,
+ }
+ for name, value in constants.items():
+ setattr(self.service_module.libvirt, name, value)
+
+ def gone():
+ raise FakeLibvirtError("domain vanished")
+
+ disappearing = types.SimpleNamespace(info=gone)
+ healthy = types.SimpleNamespace(
+ UUIDString=lambda: "00000000-0000-4000-8000-000000000002",
+ info=lambda: (1, 2048 * 1024, 2048 * 1024, 2, 0),
+ ID=lambda: 8,
+ )
+ with patch.object(
+ FakeConnection, "getAllDomainStats",
+ lambda _connection, _groups, _flags: [
+ (disappearing, {"cpu.time": 1}),
+ (healthy, {"cpu.time": 2}),
+ ], create=True,
+ ):
+ result = self.service.metric_counters()
+ self.assertEqual(len(result), 1)
+ self.assertEqual(result[0].vm_id, "00000000-0000-4000-8000-000000000002")
+ self.assertEqual(self.backend.opened, self.backend.closed)
+
if __name__ == "__main__":
unittest.main()
diff --git a/tests/test_metrics.py b/tests/test_metrics.py
new file mode 100644
index 00000000..b59ba8e7
--- /dev/null
+++ b/tests/test_metrics.py
@@ -0,0 +1,424 @@
+"""VM metric math, DB persistence, and the process-local collection lifecycle."""
+
+from __future__ import annotations
+
+import threading
+
+import pytest
+
+from virsh_panel.database import DatabaseStore
+from virsh_panel.metrics import (
+ MAX_QUERY_POINTS, RANGES, MetricSampler, MetricsCollector, RawVMStats,
+ host_scope, query_interval_seconds,
+)
+
+
+VM_ID = "00000000-0000-4000-8000-000000000001"
+HOST = {"host": "hypervisor.example", "port": 2222,
+ "fingerprint": "SHA256:one", "storage_pool": "/home/kvm"}
+
+
+def raw(*, generation=12, cpu=1_000_000_000, block_read=100,
+ block_write=200, network_rx=1000, network_tx=2000,
+ memory_time=1000, extra=None):
+ values = {
+ "cpu.time": cpu,
+ "balloon.available": 2 * 1024 * 1024,
+ "balloon.usable": 512 * 1024,
+ "balloon.last-update": memory_time,
+ "block.count": 1,
+ "block.0.name": "vda",
+ "block.0.rd.bytes": block_read,
+ "block.0.wr.bytes": block_write,
+ "net.count": 1,
+ "net.0.name": "vnet1",
+ "net.0.rx.bytes": network_rx,
+ "net.0.tx.bytes": network_tx,
+ }
+ values.update(extra or {})
+ return RawVMStats(VM_ID, generation, 2, values)
+
+
+def test_host_scope_ignores_pool_but_changes_with_ssh_host_identity():
+ first = host_scope(HOST)
+ assert first == host_scope({**HOST, "storage_pool": "/different"})
+ assert first != host_scope({**HOST, "fingerprint": "SHA256:two"})
+ assert first != host_scope({**HOST, "port": 22})
+
+
+def test_interval_caps_all_ranges_at_600_epoch_aligned_buckets():
+ for seconds in RANGES.values():
+ interval = query_interval_seconds(seconds)
+ assert interval % 60 == 0
+ assert seconds // interval + 2 <= MAX_QUERY_POINTS
+
+
+def test_initial_sample_and_fresh_guest_memory_then_rates():
+ sampler = MetricSampler()
+ scope = host_scope(HOST)
+ first = sampler.observe(scope, raw(), timestamp=1000, monotonic=10)
+ assert first["cpu_percent"] is None
+ assert first["disk_read_bps"] is None
+ assert first["network_rx_bps"] is None
+ assert first["memory_used_mib"] == 1536
+ assert first["memory_total_mib"] == 2048
+
+ second = sampler.observe(
+ scope,
+ raw(cpu=61_000_000_000, block_read=700, block_write=1400,
+ network_rx=2200, network_tx=2600, memory_time=1050),
+ timestamp=1060, monotonic=70,
+ )
+ assert second["cpu_percent"] == 50
+ assert second["disk_read_bps"] == 10
+ assert second["disk_write_bps"] == 20
+ assert second["network_rx_bps"] == 20
+ assert second["network_tx_bps"] == 10
+
+
+def test_cpu_time_over_vcpu_capacity_is_displayed_as_100_percent():
+ sampler = MetricSampler()
+ scope = host_scope(HOST)
+ sampler.observe(scope, raw(), timestamp=1000, monotonic=10)
+ # 130 CPU-seconds across two vCPUs in a 60-second interval; libvirt can
+ # include QEMU auxiliary threads in cpu.time.
+ point = sampler.observe(scope, raw(cpu=131_000_000_000),
+ timestamp=1060, monotonic=70)
+ assert point["cpu_percent"] == 100
+
+
+def test_memory_missing_or_stale_balloon_stats_never_uses_rss_or_current():
+ sampler = MetricSampler()
+ scope = host_scope(HOST)
+ values = {"balloon.current": 2 * 1024 * 1024, "balloon.rss": 512 * 1024}
+ for extra in (
+ {"balloon.last-update": 1},
+ {"balloon.usable": None},
+ {"balloon.available": None},
+ ):
+ point = sampler.observe(scope, raw(extra={**values, **extra}),
+ timestamp=1000, monotonic=10)
+ assert point["memory_used_mib"] is None
+ assert point["memory_total_mib"] is None
+
+
+def test_counter_reset_reboot_device_change_and_partial_data_are_unknown():
+ sampler = MetricSampler()
+ scope = host_scope(HOST)
+ sampler.observe(scope, raw(), timestamp=1000, monotonic=10)
+ reset = sampler.observe(scope, raw(cpu=100, block_read=1, network_rx=1),
+ timestamp=1060, monotonic=70)
+ assert reset["cpu_percent"] is None
+ assert reset["disk_read_bps"] is None
+ assert reset["network_rx_bps"] is None
+
+ # Even if a second disk masks the first disk's decreasing counter in the
+ # aggregate, device change must not produce a transient spike.
+ changed = sampler.observe(scope, raw(extra={
+ "block.count": 2, "block.1.name": "vdb",
+ "block.1.rd.bytes": 10_000, "block.1.wr.bytes": 20_000,
+ }), timestamp=1120, monotonic=130)
+ assert changed["disk_read_bps"] is None
+ assert changed["disk_write_bps"] is None
+
+ partial = sampler.observe(scope, raw(extra={
+ "block.count": 2, "block.1.name": "vdb",
+ "block.1.rd.bytes": 11_000,
+ }), timestamp=1180, monotonic=190)
+ assert partial["disk_write_bps"] is None # second device missing wr.bytes
+ assert partial["disk_read_bps"] is not None
+
+ reboot = sampler.observe(scope, raw(generation=13, cpu=200_000_000_000),
+ timestamp=1240, monotonic=250)
+ assert reboot["cpu_percent"] is None
+ assert reboot["network_tx_bps"] is None
+
+
+def test_cdrom_without_byte_counters_does_not_hide_disk_rates():
+ sampler = MetricSampler()
+ scope = host_scope(HOST)
+ cdrom = {"block.count": 2, "block.1.name": "sda"}
+ sampler.observe(scope, raw(extra=cdrom), timestamp=1000, monotonic=10)
+ second = sampler.observe(
+ scope, raw(block_read=700, block_write=1400, extra={
+ **cdrom, "balloon.last-update": 1060,
+ }), timestamp=1060, monotonic=70,
+ )
+ assert second["disk_read_bps"] == 10
+ assert second["disk_write_bps"] == 20
+
+
+def test_single_device_counter_rollback_is_not_masked_by_another_device():
+ sampler = MetricSampler()
+ scope = host_scope(HOST)
+ second_disk = {"block.count": 2, "block.1.name": "vdb",
+ "block.1.rd.bytes": 1_000, "block.1.wr.bytes": 1_000}
+ sampler.observe(scope, raw(extra=second_disk), timestamp=1000, monotonic=10)
+ point = sampler.observe(scope, raw(block_read=1, block_write=1, extra={
+ **second_disk, "block.1.rd.bytes": 10_000,
+ "block.1.wr.bytes": 10_000,
+ }), timestamp=1060, monotonic=70)
+ assert point["disk_read_bps"] is None
+ assert point["disk_write_bps"] is None
+
+
+def test_unobserved_long_gap_is_not_averaged_as_current_rate():
+ sampler = MetricSampler()
+ scope = host_scope(HOST)
+ sampler.observe(scope, raw(), timestamp=1000, monotonic=10)
+ delayed = sampler.observe(scope, raw(cpu=50_000_000_000, block_read=2000),
+ timestamp=1600, monotonic=610)
+ assert delayed["cpu_percent"] is None
+ assert delayed["disk_read_bps"] is None
+
+
+class FakeCursor:
+ def __init__(self, rows=(), delete_counts=()):
+ self.rows = rows
+ self.commands = []
+ self.delete_counts = list(delete_counts)
+ self.rowcount = 0
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *_args):
+ return None
+
+ def execute(self, sql, args=None):
+ self.commands.append((sql, args))
+ self.rowcount = (self.delete_counts.pop(0)
+ if sql.startswith("DELETE FROM vm_metric_samples")
+ and self.delete_counts else 0)
+
+ def executemany(self, sql, args):
+ self.commands.append((sql, args))
+
+ def fetchall(self):
+ return self.rows
+
+
+class FakeConnection:
+ def __init__(self, rows=(), delete_counts=()):
+ self.fake_cursor = FakeCursor(rows, delete_counts)
+ self.commits = 0
+ self.closed = False
+
+ def cursor(self):
+ return self.fake_cursor
+
+ def commit(self):
+ self.commits += 1
+
+ def rollback(self):
+ raise AssertionError("unexpected rollback")
+
+ def close(self):
+ self.closed = True
+
+
+def test_metric_table_created_with_record_table():
+ connection = FakeConnection()
+ store = DatabaseStore.__new__(DatabaseStore)
+ store._connect = lambda: connection
+ store._initialize_schema()
+ assert len(connection.fake_cursor.commands) == 2
+ assert "vm_metric_samples" in connection.fake_cursor.commands[1][0]
+ assert "vm_metric_retention" in connection.fake_cursor.commands[1][0]
+ assert connection.commits == 1 and connection.closed
+
+
+def test_metric_store_prunes_physically_and_queries_only_scoped_recent_data():
+ scope = host_scope(HOST)
+ connection = FakeConnection([{
+ "timestamp": 1000,
+ "cpu_percent": 12.5,
+ "memory_used_mib": 1024,
+ "memory_total_mib": 2048,
+ "disk_read_bps": None,
+ "disk_write_bps": None,
+ "network_rx_bps": 256,
+ "network_tx_bps": 0,
+ }])
+ store = DatabaseStore.__new__(DatabaseStore)
+ store._connect = lambda: connection
+ point = {"vm_id": VM_ID, "timestamp": 1000, "cpu_percent": 12.5,
+ "memory_used_mib": 1024, "memory_total_mib": 2048,
+ "disk_read_bps": None, "disk_write_bps": None,
+ "network_rx_bps": 256, "network_tx_bps": 0}
+ store.save_metric_samples(scope, [point])
+ store.prune_metric_samples(500)
+ result = store.query_metric_samples(scope, VM_ID, since=500, until=1100,
+ interval_seconds=60, limit=600)
+ assert result[0]["cpu_percent"] == 12.5
+ assert result[0]["disk_read_bps"] is None
+ insert, delete, query = connection.fake_cursor.commands
+ assert "INSERT INTO vm_metric_samples" in insert[0]
+ assert "DELETE FROM vm_metric_samples WHERE sampled_at < %s" in delete[0]
+ assert delete[1] == (500, 5000)
+ assert "host_scope=%s AND vm_uuid=%s AND sampled_at >= %s" in query[0]
+ assert "LIMIT %s" in query[0]
+ assert query[1] == (scope, VM_ID, 500, 1100, 60, 600)
+ assert connection.commits == 2 and connection.closed
+
+
+def test_metric_prune_is_one_index_ordered_batch_per_transaction():
+ connection = FakeConnection(delete_counts=[5000] * 10)
+ store = DatabaseStore.__new__(DatabaseStore)
+ store._connect = lambda: connection
+ assert store.prune_metric_samples(1234) == 5000
+ assert len(connection.fake_cursor.commands) == 1
+ assert connection.commits == 1 and connection.closed
+ assert all("ORDER BY sampled_at LIMIT %s" in command
+ and args == (1234, 5000)
+ for command, args in connection.fake_cursor.commands)
+
+
+def test_metric_prune_reports_short_final_batch():
+ connection = FakeConnection(delete_counts=[275])
+ store = DatabaseStore.__new__(DatabaseStore)
+ store._connect = lambda: connection
+ assert store.prune_metric_samples(1234) == 275
+ assert len(connection.fake_cursor.commands) == 1
+ assert connection.commits == 1
+
+
+def test_metric_store_rejects_invalid_host_and_uuid_before_db_query():
+ store = DatabaseStore.__new__(DatabaseStore)
+ store._connect = lambda: pytest.fail("invalid scope must not query database")
+ with pytest.raises(ValueError):
+ store.query_metric_samples("bad-scope", VM_ID, since=0, until=1,
+ interval_seconds=60)
+ with pytest.raises(ValueError):
+ store.query_metric_samples(host_scope(HOST), "not-uuid", since=0, until=1,
+ interval_seconds=60)
+
+
+def test_collector_prunes_when_ssh_locked_and_starts_after_db_configuration():
+ class Store:
+ def __init__(self):
+ self.pruned = []
+ self.saved = []
+ self.pruned_event = threading.Event()
+ self.saved_event = threading.Event()
+
+ def prune_metric_samples(self, cutoff):
+ self.pruned.append(cutoff)
+ self.pruned_event.set()
+ return 0
+
+ def save_metric_samples(self, scope, points):
+ self.saved.append((scope, points))
+ self.saved_event.set()
+
+ current = {"store": None, "profile": None}
+ store = Store()
+ class Service:
+ def metric_counters(self):
+ return [raw()]
+ collector = MetricsCollector(lambda: current["store"],
+ lambda: current["profile"],
+ lambda _profile: Service(), interval_seconds=0.01)
+ collector.start()
+ current["profile"] = {**HOST, "key_locked": True}
+ current["store"] = store
+ try:
+ assert store.pruned_event.wait(timeout=2)
+ assert store.pruned and not store.saved
+ current["profile"]["key_locked"] = False
+ assert store.saved_event.wait(timeout=2)
+ assert store.saved
+ finally:
+ collector.close()
+
+
+def test_metric_cleanup_failure_does_not_discard_current_sample(caplog):
+ events = []
+ attempted = threading.Event()
+
+ class Store:
+ def save_metric_samples(self, _scope, points):
+ events.append(("saved", points))
+
+ def prune_metric_samples(self, _cutoff):
+ events.append(("prune", None))
+ attempted.set()
+ raise RuntimeError("secret database details")
+
+ class Service:
+ def metric_counters(self):
+ return [raw()]
+
+ collector = MetricsCollector(lambda: Store(), lambda: HOST,
+ lambda _profile: Service(), interval_seconds=1)
+ collector.start()
+ assert attempted.wait(timeout=2)
+ collector.close()
+ assert [event[0] for event in events] == ["saved", "prune"]
+ assert "RuntimeError" in caplog.text
+ assert "secret database details" not in caplog.text
+
+
+def test_backlog_cleanup_continues_in_other_thread_without_stalling_samples():
+ second_saved = threading.Event()
+ prune_entered = threading.Event()
+ release_prune = threading.Event()
+ backlog_drained = threading.Event()
+
+ class Store:
+ def __init__(self):
+ self.saved = 0
+ self.pruned = 0
+
+ def save_metric_samples(self, _scope, _points):
+ self.saved += 1
+ if self.saved >= 2:
+ second_saved.set()
+
+ def prune_metric_samples(self, _cutoff):
+ self.pruned += 1
+ if self.pruned == 1:
+ prune_entered.set()
+ assert release_prune.wait(timeout=2)
+ if self.pruned < 3:
+ return 5000
+ backlog_drained.set()
+ return 0
+
+ class Service:
+ def metric_counters(self):
+ return [raw()]
+
+ store = Store()
+ collector = MetricsCollector(lambda: store, lambda: HOST,
+ lambda _profile: Service(), interval_seconds=0.01)
+ collector.start()
+ try:
+ assert prune_entered.wait(timeout=2)
+ assert second_saved.wait(timeout=2)
+ release_prune.set()
+ assert backlog_drained.wait(timeout=2)
+ assert store.pruned >= 3
+ finally:
+ release_prune.set()
+ collector.close()
+
+
+def test_collector_thread_starts_before_db_configuration_and_stops():
+ ready = threading.Event()
+ current = {"store": None}
+
+ class Store:
+ def prune_metric_samples(self, _cutoff):
+ ready.set()
+ return 0
+
+ collector = MetricsCollector(lambda: current["store"], lambda: None,
+ lambda _profile: pytest.fail("no host configured"),
+ interval_seconds=0.01)
+ collector.start()
+ current["store"] = Store()
+ assert ready.wait(timeout=2)
+ collector.close()
+ assert collector._thread is not None and not collector._thread.is_alive()
+ assert collector._prune_thread is not None and not collector._prune_thread.is_alive()
diff --git a/virsh_panel/app.py b/virsh_panel/app.py
index ceaf3d02..ac6536d5 100644
--- a/virsh_panel/app.py
+++ b/virsh_panel/app.py
@@ -10,13 +10,14 @@ import logging
import secrets
import subprocess
import threading
+import time
import uuid
from concurrent.futures import ThreadPoolExecutor
from contextlib import asynccontextmanager, suppress
-from typing import Callable, Literal
+from typing import Any, Callable, Literal
from urllib.parse import urlsplit
-from fastapi import Depends, FastAPI, HTTPException, Request, WebSocket
+from fastapi import Depends, FastAPI, HTTPException, Query, Request, WebSocket
from fastapi.responses import FileResponse, JSONResponse
from fastapi.staticfiles import StaticFiles
from pydantic import BaseModel, ConfigDict, Field, SecretStr
@@ -32,6 +33,8 @@ from .console import ConsoleRelayError, relay_console
from .database import DatabaseError, make_components
from .database_bootstrap import DatabaseBootstrap, DatabaseSetupError
from .libvirt_service import VMError, VMService
+from .metrics import (MAX_QUERY_POINTS, RETENTION_SECONDS, RANGES, MetricsCollector,
+ host_scope, query_interval_seconds)
from .panel_state import PanelState, SetupError
from .remote_host import HostProfile, RemoteHostError, RemoteHostManager
from .updater import BRANCH, REPOSITORY_URL, UpdateError, Updater
@@ -238,6 +241,7 @@ def create_app(
panel_state: PanelState | None = None,
remote_manager: RemoteHostManager | None = None,
service_factory: Callable[[dict], VMService] | None = None,
+ metric_store: Any | None = None,
) -> FastAPI:
settings = settings or Settings.from_env()
settings.validate()
@@ -319,12 +323,21 @@ def create_app(
login_limiter = LoginLimiter()
setup_limiter = LoginLimiter()
unlock_limiter = LoginLimiter()
+ metric_collector = MetricsCollector(
+ lambda: data_store,
+ lambda: manager.current() if manager is not None else None,
+ lambda profile: new_service(profile),
+ ) if database_mode else None
@asynccontextmanager
async def lifespan(_app: FastAPI):
+ if metric_collector is not None:
+ metric_collector.start()
try:
yield
finally:
+ if metric_collector is not None:
+ metric_collector.close()
reinstall_executor.shutdown(wait=True, cancel_futures=False)
import_executor.shutdown(wait=True, cancel_futures=False)
close_manager = getattr(manager, "close", None) if manager is not None else None
@@ -852,6 +865,37 @@ def create_app(
profile = current_profile()
return {**new_service(profile).get_vm(vm_id), "region": profile.get("region", "")}
+ @app.get("/api/v1/vms/{vm_id}/metrics")
+ def get_vm_metrics(
+ vm_id: str,
+ range_: Literal["1h", "24h", "7d", "30d"] = Query("1h", alias="range"),
+ _session: Session = Depends(require_session),
+ ):
+ try:
+ canonical_id = str(uuid.UUID(vm_id))
+ except (TypeError, ValueError, AttributeError) as exc:
+ raise VMError("虚拟机 UUID 无效", 400) from exc
+ profile = current_profile()
+ # Never reveal another host's retained samples for a reused UUID.
+ new_service(profile).get_vm(canonical_id)
+ backend = metric_store if metric_store is not None else data_store
+ if backend is None:
+ raise HTTPException(status_code=503, detail="指标数据库尚未就绪")
+ now = int(time.time())
+ window = RANGES[range_]
+ interval = query_interval_seconds(window)
+ since = max(now - window, now - RETENTION_SECONDS)
+ try:
+ scope = host_scope(profile)
+ except (KeyError, ValueError, TypeError) as exc:
+ raise HTTPException(status_code=409, detail="宿主机身份信息不完整") from exc
+ points = backend.query_metric_samples(
+ scope, canonical_id, since=since, until=now,
+ interval_seconds=interval, limit=MAX_QUERY_POINTS,
+ )
+ return {"vm_id": canonical_id, "range": range_,
+ "interval_seconds": interval, "points": points}
+
@app.post("/api/v1/vms", status_code=201)
def create_vm(body: CreateISOBody | CreateCloudBody,
_session: Session = Depends(require_write)):
diff --git a/virsh_panel/database.py b/virsh_panel/database.py
index 670a9da7..a1c52e65 100644
--- a/virsh_panel/database.py
+++ b/virsh_panel/database.py
@@ -10,6 +10,7 @@ from __future__ import annotations
import hmac
import json
+import math
import os
import re
import secrets
@@ -17,6 +18,7 @@ import shutil
import stat
import sys
import threading
+import uuid
from pathlib import Path
from typing import Any, Mapping
from urllib.parse import quote
@@ -62,6 +64,28 @@ CREATE TABLE IF NOT EXISTS panel_records (
PRIMARY KEY (record_type, record_key)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
+_METRIC_SCHEMA = """
+CREATE TABLE IF NOT EXISTS vm_metric_samples (
+ host_scope CHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
+ vm_uuid CHAR(36) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
+ sampled_at BIGINT UNSIGNED NOT NULL,
+ cpu_percent DOUBLE NULL,
+ memory_used_mib DOUBLE NULL,
+ memory_total_mib DOUBLE NULL,
+ disk_read_bps DOUBLE NULL,
+ disk_write_bps DOUBLE NULL,
+ network_rx_bps DOUBLE NULL,
+ network_tx_bps DOUBLE NULL,
+ PRIMARY KEY (host_scope, vm_uuid, sampled_at),
+ KEY vm_metric_retention (sampled_at)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
+"""
+_METRIC_COLUMNS = (
+ "cpu_percent", "memory_used_mib", "memory_total_mib", "disk_read_bps",
+ "disk_write_bps", "network_rx_bps", "network_tx_bps",
+)
+_HOST_SCOPE = re.compile(r"[0-9a-f]{64}\Z")
+_METRIC_PRUNE_BATCH_SIZE = 5000
class DatabaseError(Exception):
@@ -139,6 +163,7 @@ class DatabaseStore:
try:
with connection.cursor() as cursor:
cursor.execute(_SCHEMA)
+ cursor.execute(_METRIC_SCHEMA)
connection.commit()
except pymysql.MySQLError as exc:
connection.rollback()
@@ -146,6 +171,105 @@ class DatabaseStore:
finally:
connection.close()
+ @staticmethod
+ def _metric_identity(host_scope: str, vm_id: str) -> str:
+ if not isinstance(host_scope, str) or not _HOST_SCOPE.fullmatch(host_scope):
+ raise ValueError("Metric host scope is invalid")
+ try:
+ canonical = str(uuid.UUID(vm_id))
+ except (TypeError, ValueError, AttributeError) as exc:
+ raise ValueError("Metric VM UUID is invalid") from exc
+ return canonical
+
+ def save_metric_samples(self, host_scope: str, points: list[dict[str, Any]]) -> None:
+ if not points:
+ return
+ rows = []
+ for point in points:
+ vm_id = self._metric_identity(host_scope, point["vm_id"])
+ timestamp = point["timestamp"]
+ if isinstance(timestamp, bool) or not isinstance(timestamp, int) or timestamp < 0:
+ raise ValueError("Metric timestamp is invalid")
+ metrics = []
+ for name in _METRIC_COLUMNS:
+ value = point.get(name)
+ if (value is not None and (isinstance(value, bool)
+ or not isinstance(value, (int, float))
+ or not math.isfinite(value) or value < 0)):
+ raise ValueError("Metric value is invalid")
+ metrics.append(value)
+ rows.append((host_scope, vm_id, timestamp, *metrics))
+ connection = self._connect()
+ try:
+ with connection.cursor() as cursor:
+ cursor.executemany(
+ "INSERT INTO vm_metric_samples (host_scope, vm_uuid, sampled_at, "
+ + ", ".join(_METRIC_COLUMNS) + ") VALUES ("
+ + ", ".join(["%s"] * (3 + len(_METRIC_COLUMNS))) + ") "
+ "ON DUPLICATE KEY UPDATE "
+ + ", ".join(f"{name}=VALUES({name})" for name in _METRIC_COLUMNS),
+ rows,
+ )
+ connection.commit()
+ except pymysql.MySQLError as exc:
+ connection.rollback()
+ raise DatabaseError("无法保存虚拟机指标") from exc
+ finally:
+ connection.close()
+
+ def prune_metric_samples(self, cutoff: int) -> int:
+ if isinstance(cutoff, bool) or not isinstance(cutoff, int) or cutoff < 0:
+ raise ValueError("Metric retention cutoff is invalid")
+ connection = self._connect()
+ try:
+ with connection.cursor() as cursor:
+ cursor.execute(
+ "DELETE FROM vm_metric_samples WHERE sampled_at < %s "
+ "ORDER BY sampled_at LIMIT %s",
+ (cutoff, _METRIC_PRUNE_BATCH_SIZE),
+ )
+ removed = cursor.rowcount
+ connection.commit()
+ return max(0, removed)
+ except pymysql.MySQLError as exc:
+ connection.rollback()
+ raise DatabaseError("无法清理过期虚拟机指标") from exc
+ finally:
+ connection.close()
+
+ def query_metric_samples(self, host_scope: str, vm_id: str, *,
+ since: int, until: int, interval_seconds: int,
+ limit: int = 600) -> list[dict[str, Any]]:
+ vm_id = self._metric_identity(host_scope, vm_id)
+ if (any(isinstance(value, bool) or not isinstance(value, int)
+ for value in (since, until, interval_seconds, limit))
+ or since < 0 or until < since or interval_seconds < 60
+ or not 1 <= limit <= 600):
+ raise ValueError("Metric query range is invalid")
+ columns = ", ".join(f"AVG({name}) AS {name}" for name in _METRIC_COLUMNS)
+ connection = self._connect()
+ try:
+ with connection.cursor() as cursor:
+ cursor.execute(
+ "SELECT MAX(sampled_at) AS timestamp, " + columns
+ + " FROM vm_metric_samples "
+ "WHERE host_scope=%s AND vm_uuid=%s AND sampled_at >= %s "
+ "AND sampled_at <= %s GROUP BY FLOOR(sampled_at / %s) "
+ "ORDER BY timestamp ASC LIMIT %s",
+ (host_scope, vm_id, since, until, interval_seconds, limit),
+ )
+ rows = cursor.fetchall()
+ return [
+ {"timestamp": int(row["timestamp"]),
+ **{name: float(row[name]) if row[name] is not None else None
+ for name in _METRIC_COLUMNS}}
+ for row in rows
+ ]
+ except pymysql.MySQLError as exc:
+ raise DatabaseError("无法读取虚拟机指标") from exc
+ finally:
+ connection.close()
+
def _encrypt(self, record_type: str, record_key: str, value: str) -> bytes:
if not isinstance(value, str):
raise TypeError("Database value must be a string")
diff --git a/virsh_panel/libvirt_service.py b/virsh_panel/libvirt_service.py
index d76c54b0..59d181d4 100644
--- a/virsh_panel/libvirt_service.py
+++ b/virsh_panel/libvirt_service.py
@@ -23,6 +23,7 @@ from typing import Any, Callable, Iterator
import libvirt
from virsh_panel.cloud_init import build_seed_iso, read_seed_network, validate_cloud_network
+from virsh_panel.metrics import RawVMStats
LOG = logging.getLogger(__name__)
@@ -329,6 +330,35 @@ class VMService:
with self._connect() as conn:
return self._vm_dict(self._domain(conn, uuid))
+ def metric_counters(self) -> list[RawVMStats]:
+ """Fetch running-domain counters in one libvirt API round trip.
+
+ getAllDomainStats(stats, flags) returns ``(domain, dict)`` pairs in
+ libvirt-python. Network rx/tx fields are from the guest's viewpoint.
+ Missing per-device values are preserved as missing by the sampler.
+ """
+ groups = (libvirt.VIR_DOMAIN_STATS_CPU_TOTAL | libvirt.VIR_DOMAIN_STATS_BALLOON
+ | libvirt.VIR_DOMAIN_STATS_BLOCK | libvirt.VIR_DOMAIN_STATS_INTERFACE)
+ running = libvirt.VIR_CONNECT_GET_ALL_DOMAINS_STATS_RUNNING
+ with self._connect() as conn:
+ rows: list[RawVMStats] = []
+ for domain, values in conn.getAllDomainStats(groups, running):
+ # A guest can stop or be undefined after bulk stats are read.
+ # Do not lose all other guests' samples for that transient race.
+ try:
+ info = domain.info()
+ generation = domain.ID()
+ vm_id = domain.UUIDString()
+ except libvirt.libvirtError:
+ continue
+ if info[0] != 1 or generation < 0:
+ continue
+ rows.append(RawVMStats(
+ vm_id=vm_id, generation=generation,
+ vcpus=info[3], values=values,
+ ))
+ return rows
+
def resources(self) -> dict[str, Any]:
with self._connect() as conn:
disk_pool = self._active_pool(conn, self.storage_pool)
diff --git a/virsh_panel/metrics.py b/virsh_panel/metrics.py
new file mode 100644
index 00000000..4f109f96
--- /dev/null
+++ b/virsh_panel/metrics.py
@@ -0,0 +1,276 @@
+"""Periodic libvirt VM measurements and safe counter-to-rate conversion.
+
+The collector runs independently of browser visits. A process restart deliberately
+loses counter baselines: the first subsequent rate sample is unknown, not zero.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import logging
+import math
+import re
+import threading
+import time
+import uuid
+from dataclasses import dataclass
+from typing import Any, Callable, Mapping
+
+
+LOG = logging.getLogger(__name__)
+SAMPLE_INTERVAL_SECONDS = 60
+RETENTION_SECONDS = 30 * 24 * 60 * 60
+MAX_QUERY_POINTS = 600
+RANGES = {"1h": 3600, "24h": 24 * 3600, "7d": 7 * 24 * 3600,
+ "30d": RETENTION_SECONDS}
+_COUNTER = re.compile(r"^(block|net)\.(\d+)\.(rd|wr|rx|tx)\.bytes$")
+_MEMORY_MAX_AGE_SECONDS = 180
+
+
+def host_scope(profile: Mapping[str, Any]) -> str:
+ """Keep VM history when pool settings change, but isolate a new SSH host key."""
+ identity = [profile[key] for key in ("host", "port", "fingerprint")]
+ if (not isinstance(identity[0], str) or not isinstance(identity[1], int)
+ or not isinstance(identity[2], str)):
+ raise ValueError("Host identity is invalid")
+ raw = json.dumps(identity, ensure_ascii=False, separators=(",", ":"))
+ return hashlib.sha256(raw.encode("utf-8")).hexdigest()
+
+
+def query_interval_seconds(window_seconds: int) -> int:
+ """Epoch-aligned buckets cannot exceed MAX_QUERY_POINTS in any range."""
+ if window_seconds <= 0:
+ raise ValueError("Metric range must be positive")
+ return max(SAMPLE_INTERVAL_SECONDS,
+ math.ceil(window_seconds / (MAX_QUERY_POINTS - 1)
+ / SAMPLE_INTERVAL_SECONDS) * SAMPLE_INTERVAL_SECONDS)
+
+
+def _nonnegative_int(value: Any) -> int | None:
+ if isinstance(value, bool) or not isinstance(value, int) or value < 0:
+ return None
+ return value
+
+
+def _device_counters(stats: Mapping[str, Any], group: str,
+ directions: tuple[str, str]) -> tuple[tuple[tuple[str, str], ...],
+ dict[str, dict[str, int] | None]]:
+ count = _nonnegative_int(stats.get(f"{group}.count"))
+ devices: dict[str, dict[str, int]] = {
+ str(number): {} for number in range(min(count, 1024))
+ } if count is not None else {}
+ for key, value in stats.items():
+ match = _COUNTER.fullmatch(key)
+ if not match or match.group(1) != group or match.group(3) not in directions:
+ continue
+ devices.setdefault(match.group(2), {})
+ valid = _nonnegative_int(value)
+ if valid is not None:
+ devices[match.group(2)][match.group(3)] = valid
+ signature = tuple(sorted(
+ (number, str(stats.get(f"{group}.{number}.name", number)))
+ for number in devices
+ ))
+ counters_by_direction: dict[str, dict[str, int] | None] = {}
+ for direction in directions:
+ counters = {number: fields[direction] for number, fields in devices.items()
+ if direction in fields}
+ # CD-ROM and unsupported devices may never expose byte counters. Use
+ # only reporting devices, but _rate() resets if that coverage changes.
+ counters_by_direction[direction] = counters or None
+ return signature, counters_by_direction
+
+
+def _memory_mib(stats: Mapping[str, Any], sampled_at: int) -> tuple[float | None,
+ float | None]:
+ available = _nonnegative_int(stats.get("balloon.available"))
+ usable = _nonnegative_int(stats.get("balloon.usable"))
+ last_update = _nonnegative_int(stats.get("balloon.last-update"))
+ # The QEMU RSS and balloon.current fields describe host process/allocation,
+ # not guest-used memory. Only guest balloon statistics can support used MiB.
+ if (available is None or usable is None or last_update is None
+ or last_update > sampled_at + SAMPLE_INTERVAL_SECONDS
+ or sampled_at - last_update > _MEMORY_MAX_AGE_SECONDS
+ or usable > available):
+ return None, None
+ return (available - usable) / 1024, available / 1024
+
+
+@dataclass(frozen=True)
+class RawVMStats:
+ vm_id: str
+ generation: int
+ vcpus: int
+ values: Mapping[str, Any]
+
+
+@dataclass(frozen=True)
+class _Baseline:
+ monotonic: float
+ generation: int
+ vcpus: int
+ cpu_time: int | None
+ disk_signature: tuple[tuple[str, str], ...]
+ disk: Mapping[str, Mapping[str, int] | None]
+ network_signature: tuple[tuple[str, str], ...]
+ network: Mapping[str, Mapping[str, int] | None]
+
+
+class MetricSampler:
+ def __init__(self) -> None:
+ self._previous: dict[tuple[str, str], _Baseline] = {}
+
+ @staticmethod
+ def _rate(current: Mapping[str, int] | None,
+ previous: Mapping[str, int] | None,
+ seconds: float, same_devices: bool) -> float | None:
+ if (not same_devices or current is None or previous is None
+ or seconds <= 0 or current.keys() != previous.keys()
+ or any(value < previous[name] for name, value in current.items())):
+ return None
+ return sum(value - previous[name] for name, value in current.items()) / seconds
+
+ def observe(self, scope: str, raw: RawVMStats, *, timestamp: int,
+ monotonic: float) -> dict[str, Any]:
+ vm_id = str(uuid.UUID(raw.vm_id))
+ values = raw.values
+ cpu_time = _nonnegative_int(values.get("cpu.time"))
+ disk_signature, disk = _device_counters(values, "block", ("rd", "wr"))
+ network_signature, network = _device_counters(values, "net", ("rx", "tx"))
+ memory_used, memory_total = _memory_mib(values, timestamp)
+ key = (scope, vm_id)
+ previous = self._previous.get(key)
+ seconds = monotonic - previous.monotonic if previous else 0
+ # A long pause means a graph gap rather than a misleading average.
+ same_run = bool(previous and previous.generation == raw.generation
+ and previous.vcpus == raw.vcpus
+ and 0 < seconds <= SAMPLE_INTERVAL_SECONDS * 3)
+ cpu_percent = None
+ if (same_run and cpu_time is not None and previous.cpu_time is not None
+ and raw.vcpus > 0 and cpu_time >= previous.cpu_time):
+ fraction = (cpu_time - previous.cpu_time) / (seconds * 1e9 * raw.vcpus)
+ # libvirt's total CPU time can include QEMU auxiliary threads, so
+ # a healthy sample can exceed the allotted vCPU time. The chart is
+ # defined as 0–100% of VM vCPU capacity.
+ cpu_percent = min(100.0, fraction * 100)
+ disk_same = same_run and bool(disk_signature) and disk_signature == previous.disk_signature
+ network_same = (same_run and bool(network_signature)
+ and network_signature == previous.network_signature)
+ point = {
+ "vm_id": vm_id,
+ "timestamp": timestamp,
+ "cpu_percent": cpu_percent,
+ "memory_used_mib": memory_used,
+ "memory_total_mib": memory_total,
+ "disk_read_bps": self._rate(disk["rd"], previous.disk["rd"] if previous else None,
+ seconds, disk_same),
+ "disk_write_bps": self._rate(disk["wr"], previous.disk["wr"] if previous else None,
+ seconds, disk_same),
+ "network_rx_bps": self._rate(network["rx"], previous.network["rx"] if previous else None,
+ seconds, network_same),
+ "network_tx_bps": self._rate(network["tx"], previous.network["tx"] if previous else None,
+ seconds, network_same),
+ }
+ self._previous[key] = _Baseline(monotonic, raw.generation, raw.vcpus, cpu_time,
+ disk_signature, disk, network_signature, network)
+ return point
+
+ def forget_unseen(self, scope: str, seen: set[str]) -> None:
+ for key in list(self._previous):
+ if key[0] == scope and key[1] not in seen:
+ del self._previous[key]
+
+
+class MetricsCollector:
+ """One daemon sampler owned by the single Uvicorn worker's lifespan."""
+
+ def __init__(self, store_getter: Callable[[], Any],
+ profile_getter: Callable[[], Mapping[str, Any] | None],
+ service_factory: Callable[[Mapping[str, Any]], Any],
+ *, interval_seconds: int = SAMPLE_INTERVAL_SECONDS) -> None:
+ self._store_getter = store_getter
+ self._profile_getter = profile_getter
+ self._service_factory = service_factory
+ self._interval_seconds = interval_seconds
+ self._sampler = MetricSampler()
+ self._stop = threading.Event()
+ self._thread: threading.Thread | None = None
+ self._prune_signal = threading.Event()
+ self._prune_thread: threading.Thread | None = None
+
+ def sample_once(self, *, timestamp: int | None = None,
+ monotonic: float | None = None) -> None:
+ store = self._store_getter()
+ if store is None or not hasattr(store, "prune_metric_samples"):
+ return
+ timestamp = int(time.time()) if timestamp is None else timestamp
+ try:
+ profile = self._profile_getter()
+ if profile is None or profile.get("key_locked"):
+ return
+ scope = host_scope(profile)
+ rows = self._service_factory(profile).metric_counters()
+ monotonic = time.monotonic() if monotonic is None else monotonic
+ points = [self._sampler.observe(scope, row, timestamp=timestamp,
+ monotonic=monotonic) for row in rows]
+ self._sampler.forget_unseen(scope, {str(uuid.UUID(row.vm_id)) for row in rows})
+ if points:
+ store.save_metric_samples(scope, points)
+ finally:
+ # Signal after sample/save (or a failed host access). The separate
+ # cleanup thread never delays the next libvirt measurement.
+ self._prune_signal.set()
+
+ def _prune_loop(self) -> None:
+ while not self._stop.is_set():
+ self._prune_signal.wait(timeout=self._interval_seconds)
+ self._prune_signal.clear()
+ if self._stop.is_set():
+ break
+ store = self._store_getter()
+ if store is None or not hasattr(store, "prune_metric_samples"):
+ continue
+ while not self._stop.is_set():
+ try:
+ removed = store.prune_metric_samples(
+ int(time.time()) - RETENTION_SECONDS)
+ except Exception as exc:
+ # Never echo database driver errors or connection details.
+ LOG.warning("VM metric retention cleanup failed: %s",
+ type(exc).__name__)
+ break
+ if not isinstance(removed, int) or removed < 5000:
+ break
+ # Sustain catch-up without occupying MariaDB continuously.
+ if self._stop.wait(0.1):
+ break
+
+ def _run(self) -> None:
+ while not self._stop.is_set():
+ started = time.monotonic()
+ try:
+ self.sample_once()
+ except Exception as exc:
+ # Never print libvirt/DB exception text: it can contain a URI,
+ # account name, or other connection details.
+ LOG.warning("VM metric collection failed: %s", type(exc).__name__)
+ self._stop.wait(max(0, self._interval_seconds - (time.monotonic() - started)))
+
+ def start(self) -> None:
+ if self._thread is not None:
+ return
+ self._prune_thread = threading.Thread(target=self._prune_loop, daemon=True,
+ name="vm-metrics-retention")
+ self._prune_thread.start()
+ self._thread = threading.Thread(target=self._run, daemon=True,
+ name="vm-metrics-collector")
+ self._thread.start()
+
+ def close(self) -> None:
+ self._stop.set()
+ self._prune_signal.set()
+ deadline = time.monotonic() + 15
+ for worker in (self._thread, self._prune_thread):
+ if worker is not None:
+ worker.join(timeout=max(0, deadline - time.monotonic()))