Add 30-day VM performance history and detail charts

main
admin 2 months ago
parent ff29462f2a
commit 0dc36cefc2

@ -2,7 +2,7 @@
项目提供面板程序、Python 依赖清单和可选的 [systemd service 文件](deploy/virsh-panel.service),没有安装器或反向代理配置。面板运行在管理服务器 A,默认只监听 `127.0.0.1:8000`;AlmaLinux 10 的 virsh 宿主机 B 通过已有的 root SSH 密钥连接。B 不需要安装面板、创建新账号或导入新公钥。面板使用 [libvirt 官方 SSH 连接方式](https://www.libvirt.org/uri.html)管理 `qemu:///system`,VNC 经 SSH 连接 B 的本机端口。
这一版的持久化面板数据使用 **MariaDB**。首次打开网页会先配置数据库连接,再创建管理员和配置宿主机。数据库及有建表权限的专用账号需要你事先准备;面板不会安装 MariaDB 或自动创建数据库。旧版 `data/state.json`、`data/templates.json` 和本地宿主机配置**不会自动迁移**。虚拟机定义和虚拟磁盘仍由远端 libvirt/宿主机持有,数据库保存面板账户、登录会话、模板目录、宿主机连接资料、更新设置和更新任务等面板数据。
这一版的持久化面板数据使用 **MariaDB**。首次打开网页会先配置数据库连接,再创建管理员和配置宿主机。数据库及有建表权限的专用账号需要你事先准备;面板不会安装 MariaDB 或自动创建数据库。旧版 `data/state.json`、`data/templates.json` 和本地宿主机配置**不会自动迁移**。虚拟机定义和虚拟磁盘仍由远端 libvirt/宿主机持有,数据库保存面板账户、登录会话、模板目录、宿主机连接资料、更新设置、更新任务及虚拟机性能历史等面板数据。
如果是在旧测试目录上替换代码,旧 `data/generations/` 中的明文 SSH 私钥不会被新版使用,也**不会自动删除**。为避免遗留敏感文件,建议先备份并核对后,在首次运行新版前使用全新的 `data/` 目录;不要未经核对直接删除仍需使用的资料。
@ -49,6 +49,8 @@ PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -m uvicorn virsh_panel.app:create_app
虚拟机列表只展示状态、地区、系统或安装镜像、IP、规格和磁盘;点击虚拟机名称进入该机管理页,控制台、电源、重装和删除等操作都在管理页。地区取自当前宿主机设置。新建或重装的云镜像 VM 会记录所选模板名称及配置的公网 IP/CIDR;这个 IP 是**配置记录**,不是连通性探测。ISO 安装 VM 只记录安装镜像文件名,不能据此确认客体最终安装的系统;旧 VM 或面板外创建的 VM 若无可靠记录,会显示“未识别”或“未获取”,不会猜测客体系统与 IP。
虚拟机详情页显示 CPU、客体内存、磁盘 I/O 和网络收发四张历史走势图,可切换最近 1 小时、24 小时、7 天和 30 天。面板后台定时从 [libvirt 域统计](https://libvirt.org/manpages/virsh.html#domstats)采样并写入 MariaDB,即使无人打开详情页也会继续记录。查询**只返回最近 30 天**的数据;运行中的面板会分批清理更早的数据库记录。面板停机或数据库不可用时物理清理会暂停,恢复后继续追赶,因此中断期间数据库中可能暂时留有超期记录。CPU 是按 vCPU 数归一化、最高显示 100% 的使用率,磁盘和网络显示每秒速率而不是累计量或网络带宽上限;网络收发方向以虚拟机为准。磁盘与网卡只汇总 libvirt 实际提供计数的设备,未提供计数的设备无法纳入统计。内存占用依赖客体内存气球统计,客户机不提供或统计过期时会显示无数据,不会把宿主机 QEMU 进程占用误当客体内存。初次采样、虚拟机停机、面板/宿主机不可达或计数器重置期间可能出现空白;面板停机期间不会补录历史。较长时间范围会聚合采样,短暂的停机或采样缺口可能不明显。
磁盘目录和 ISO 目录填写的是 **B 宿主机**上已存在的绝对路径,例如 `/home/kvm`、`/home/iso`,不是 A 面板服务器的路径;不需要提前手动创建 libvirt 存储池。点击“保存”时,面板会将这些现有目录定义为 libvirt `dir` 存储池并启动,作为内部管理机制;不会创建目录,也不会移动或删除现有文件。新建虚拟机的磁盘文件使用配置的磁盘目录下的 `<虚拟机名>.qcow2`,安装镜像使用配置的 ISO 目录下所选的 `.iso`;虚拟机 XML 通过绝对文件路径引用它们。“测试连接”不会修改宿主机,因此此时若目录尚未注册为池,可能仅显示资源警告。也可继续填写已有的 libvirt 存储池名称。若 B 的目录权限或 SELinux 策略阻止 libvirt 访问,保存会给出错误,需在 B 上修正权限或标签。
“默认虚拟网络或网桥”填写已有 libvirt 虚拟网络名称(如 `default`);若要使用 B 上已有的 Linux 网桥 `br0`,请填写 **`bridge:br0`**。单独填写 `br0` 表示名为 `br0` 的 libvirt 虚拟网络,不表示网桥。面板不会创建网桥;桥接网络的 IP、DHCP 与防火墙由你在 B 上配置。创建 VM 时,面板自动按 vCPU 数设置 virtio 网卡队列,例如 8 vCPU 对应 `driver.queues=8`;按 RHEL 10 建议最多设 16 队列。
@ -84,4 +86,4 @@ systemctl enable --now virsh-panel.service
独立的 `virsh-panel-update.service` 会先拉取私有仓库并安装目标版本 Python 依赖,再以快进方式切换代码、重启主面板并检查本机 HTTP 与数据库健康状态。更新任务绑定点击“检查更新”时看到的两个提交编号;若本地或远端版本变化,任务会拒绝切换并要求重新检查。更新任务状态加密存于 MariaDB,页面中断后可继续查询;独立服务异常退出或超时会尝试将未完成任务标为失败。更新前请备份 **MariaDB 数据和 `data/master.key`、`data/database-config.json`**;更新失败时不要重复点按钮,应先看 `journalctl -u virsh-panel-update.service -n 100`。依赖在代码切换前安装到当前 `.venv`,因此若依赖安装成功而代码切换失败,可能需要人工检查依赖版本。更新服务会保留更新前的 Git 回滚引用,但不会自动执行覆盖性的回滚。网页更新不负责安装系统 RPM、替换 `/etc/systemd/system` 中已安装的 unit 或修改反向代理配置;这些变化如后续版本需要,仍需人工处理。由于在线更新会运行私有仓库中的代码与依赖,必须保护 Gitea 仓库写入权限和访问令牌。
功能范围:虚拟机列表、从 ISO 创建、导入 AlmaLinux 10 云镜像模板并配置公网静态 IP 创建、云镜像重装、开机、关机、重启、强制断电/重置、删除和浏览器 VNC。当前为单管理员、单宿主机版本;不提供 ISO 上传、目录自动创建、虚拟网络创建、内网 IP 模板、异网段网关或多用户权限。真实 AlmaLinux 10 宿主机及 MariaDB 仍需现场联调。
功能范围:虚拟机列表和详情页四项性能走势图、从 ISO 创建、导入 AlmaLinux 10 云镜像模板并配置公网静态 IP 创建、云镜像重装、开机、关机、重启、强制断电/重置、删除和浏览器 VNC。当前为单管理员、单宿主机版本;不提供 ISO 上传、目录自动创建、虚拟网络创建、内网 IP 模板、异网段网关或多用户权限。真实 AlmaLinux 10 宿主机及 MariaDB 仍需现场联调。

@ -9,6 +9,11 @@ const state = {
vms: [],
selectedVmId: null,
vmActionInFlight: null,
metricsRange: "24h",
metricsPollTimer: null,
metricsRequest: null,
metricsPayload: null,
metricsResizeTimer: null,
resources: null,
templateLoad: null,
templateImportJobId: null,
@ -88,9 +93,11 @@ async function api(path, options = {}) {
headers,
credentials: "same-origin",
cache: "no-store",
signal: options.signal,
body: options.body === undefined ? undefined : JSON.stringify(options.body),
});
} catch {
} catch (error) {
if (options.signal?.aborted) throw error;
throw new Error("无法连接面板服务,请检查网络或稍后重试。");
}
@ -121,6 +128,7 @@ function stopPolling() {
function showLogin() {
stopPolling();
stopVmMetrics();
closeModal();
state.updatePollToken += 1;
state.updateJobId = null;
@ -145,6 +153,7 @@ function showLogin() {
function showDatabaseView() {
stopPolling();
stopVmMetrics();
closeModal();
byId("boot-screen").hidden = true;
byId("setup-view").hidden = true;
@ -157,6 +166,7 @@ function showDatabaseView() {
function showSetup() {
stopPolling();
stopVmMetrics();
closeModal();
byId("unlock-passphrase").value = "";
byId("boot-screen").hidden = true;
@ -178,6 +188,7 @@ function clearHostPrivateKey() {
function showUnlockView(hostConfig) {
stopPolling();
stopVmMetrics();
closeModal();
state.hostConfig = hostConfig;
clearHostPrivateKey();
@ -212,6 +223,7 @@ function markHostTestStale() {
function showHostView(hostConfig = { configured: false }) {
stopPolling();
stopVmMetrics();
closeModal();
state.hostConfig = hostConfig;
clearHostPrivateKey();
@ -298,6 +310,7 @@ async function initialize() {
}
function showVmWorkspace() {
stopVmMetrics();
state.selectedVmId = null;
byId("vm-workspace").hidden = false;
byId("vm-list-view").hidden = false;
@ -340,6 +353,7 @@ function leaveUpdateWorkspace() {
}
async function showUpdateWorkspace() {
stopVmMetrics();
byId("vm-workspace").hidden = true;
byId("update-workspace").hidden = false;
byId("vm-navigation-button").classList.remove("active");
@ -803,10 +817,13 @@ function openVmDetail(uuid) {
toast("虚拟机已不在当前列表中,请刷新后重试。", "error");
return;
}
stopVmMetrics();
state.selectedVmId = uuid;
byId("vm-list-view").hidden = true;
byId("vm-detail-view").hidden = false;
renderVmDetail(vm);
clearVmMetrics("正在加载监控数据…");
startVmMetrics();
window.scrollTo({ top: 0, behavior: "auto" });
byId("vm-detail-name").focus();
}
@ -886,6 +903,238 @@ function renderVmDetail(vm) {
}
}
const VM_METRICS_RANGE_SECONDS = { "1h": 3600, "24h": 86400, "7d": 604800, "30d": 2592000 };
const VM_METRIC_CHARTS = [
{ id: "cpu", title: "CPU 使用率", unit: "percent", series: [{ key: "cpu_percent", color: "#0f9eaa" }] },
{ id: "memory", title: "内存用量", unit: "memory", series: [{ key: "memory_used_mib", color: "#4781cf" }, { key: "memory_total_mib", color: "#a4b7d7", dash: "6 5" }] },
{ id: "disk", title: "磁盘 I/O", unit: "rate", series: [{ key: "disk_read_bps", color: "#477dc9" }, { key: "disk_write_bps", color: "#e59c46" }] },
{ id: "network", title: "网络速率", unit: "rate", series: [{ key: "network_rx_bps", color: "#17a685" }, { key: "network_tx_bps", color: "#926ed0" }] },
];
function metricViewVisible() {
return Boolean(state.selectedVmId) && !document.hidden && !byId("dashboard-view").hidden
&& !byId("vm-workspace").hidden && !byId("vm-detail-view").hidden;
}
function stopVmMetrics() {
if (state.metricsPollTimer) window.clearInterval(state.metricsPollTimer);
state.metricsPollTimer = null;
if (state.metricsResizeTimer) window.clearTimeout(state.metricsResizeTimer);
state.metricsResizeTimer = null;
state.metricsRequest?.controller.abort();
state.metricsRequest = null;
}
function startVmMetrics() {
if (!metricViewVisible()) return;
void loadVmMetrics();
if (!state.metricsPollTimer) {
state.metricsPollTimer = window.setInterval(() => {
if (metricViewVisible()) void loadVmMetrics();
}, 60000);
}
}
function setVmMetricsStatus(message, warning = false) {
const status = byId("vm-metrics-status");
status.textContent = message;
status.classList.toggle("warning", warning);
}
function clearVmMetrics(message) {
state.metricsPayload = null;
setVmMetricsStatus(message);
for (const chart of VM_METRIC_CHARTS) {
text(`vm-metric-${chart.id}-summary`, "—");
byId(`vm-metric-${chart.id}-plot`).replaceChildren(makeElement("p", "vm-metric-empty", message));
}
}
async function loadVmMetrics() {
if (!metricViewVisible()) return;
const vmId = state.selectedVmId;
const range = state.metricsRange;
const previous = state.metricsRequest;
if (previous?.vmId === vmId && previous.range === range) return;
previous?.controller.abort();
const request = { vmId, range, controller: new AbortController() };
state.metricsRequest = request;
setVmMetricsStatus("正在更新监控数据…");
try {
const payload = await api(`/api/v1/vms/${encodeURIComponent(vmId)}/metrics?range=${encodeURIComponent(range)}`, {
signal: request.controller.signal,
});
if (request !== state.metricsRequest || !metricViewVisible()) return;
if (!Array.isArray(payload?.points) || (payload.vm_id && payload.vm_id !== vmId)) {
throw new Error("监控数据格式无效。");
}
renderVmMetrics(payload);
} catch (error) {
if (request !== state.metricsRequest || request.controller.signal.aborted || !metricViewVisible()) return;
setVmMetricsStatus(`监控数据暂时无法加载:${error.message}。已显示的数据可能不是最新。`, true);
} finally {
if (request === state.metricsRequest) state.metricsRequest = null;
}
}
function metricNumber(value) {
return typeof value === "number" && Number.isFinite(value) && value >= 0 ? value : null;
}
function formatMetricRate(value) {
if (value === null) return "无统计";
const units = ["B/s", "KB/s", "MB/s", "GB/s", "TB/s"];
let amount = value;
let index = 0;
while (amount >= 1000 && index < units.length - 1) { amount /= 1000; index += 1; }
return `${amount.toFixed(index === 0 ? (amount < 1 && amount > 0 ? 2 : 0) : amount < 10 ? 2 : 1)} ${units[index]}`;
}
function formatMetricMemory(value) {
if (value === null) return "无统计";
return value >= 1024 ? `${(value / 1024).toFixed(value < 10240 ? 2 : 1)} GiB` : `${Math.round(value)} MiB`;
}
function formatMetricValue(value, unit) {
if (value === null) return "无统计";
if (unit === "percent") return `${value.toFixed(1)}%`;
if (unit === "memory") return formatMetricMemory(value);
return formatMetricRate(value);
}
function formatMetricTime(timestamp) {
return new Date(timestamp * 1000).toLocaleString("zh-CN", {
month: "numeric", day: "numeric", hour: "2-digit", minute: "2-digit",
});
}
function metricSummary(chart, points) {
const latest = [...points].reverse().find((point) => chart.series.some((series) => metricNumber(point[series.key]) !== null));
if (!latest) return "暂无统计";
const value = (key) => metricNumber(latest[key]);
if (chart.id === "cpu") return formatMetricValue(value("cpu_percent"), "percent");
if (chart.id === "memory") return `已使用(估算)${formatMetricMemory(value("memory_used_mib"))} · 总量 ${formatMetricMemory(value("memory_total_mib"))}`;
if (chart.id === "disk") return `读取 ${formatMetricRate(value("disk_read_bps"))} · 写入 ${formatMetricRate(value("disk_write_bps"))}`;
return `接收 ${formatMetricRate(value("network_rx_bps"))} · 发送 ${formatMetricRate(value("network_tx_bps"))}`;
}
function metricChartEmptyText(chart) {
if (chart.id === "memory") return "暂无内存统计;虚拟机可能未提供气球驱动数据。";
if (chart.id === "disk" || chart.id === "network") return "暂无速率数据;首次采样或停机时无法计算速率。";
return "当前时间范围内暂无 CPU 采样数据。";
}
function metricNiceMax(value) {
if (value <= 0) return 1;
const base = 10 ** Math.floor(Math.log10(value));
for (const step of [1, 1.5, 2, 2.5, 5, 10]) {
if (value <= step * base) return step * base;
}
return 10 * base;
}
function metricSvgElement(tag, attributes = {}) {
const element = document.createElementNS("http://www.w3.org/2000/svg", tag);
for (const [name, value] of Object.entries(attributes)) element.setAttribute(name, String(value));
return element;
}
function drawMetricChart(chart, points, start, end, intervalSeconds) {
const plot = byId(`vm-metric-${chart.id}-plot`);
const summary = metricSummary(chart, points);
text(`vm-metric-${chart.id}-summary`, summary);
const hasData = points.some((point) => chart.series.some((series) => metricNumber(point[series.key]) !== null));
if (!hasData) {
plot.replaceChildren(makeElement("p", "vm-metric-empty", metricChartEmptyText(chart)));
return;
}
const values = points.flatMap((point) => chart.series.map((series) => metricNumber(point[series.key]))).filter((value) => value !== null);
const maximum = chart.unit === "percent" ? Math.max(100, metricNiceMax(Math.max(...values))) : metricNiceMax(Math.max(...values));
// A fixed 640px viewBox shrinks axis text to ~4px inside a phone card.
const width = Math.max(240, Math.floor(plot.clientWidth || 640));
const height = Math.max(180, Math.floor(plot.clientHeight || 216));
const left = 64;
const right = 18;
const top = 14;
const bottom = 37;
const plotWidth = width - left - right;
const plotHeight = height - top - bottom;
const x = (timestamp) => left + Math.max(0, Math.min(1, (timestamp - start) / (end - start))) * plotWidth;
const y = (value) => top + (1 - Math.min(value, maximum) / maximum) * plotHeight;
const svg = metricSvgElement("svg", { viewBox: `0 0 ${width} ${height}`, role: "img", "aria-label": `${chart.title},${byId("vm-metrics-range").selectedOptions[0]?.textContent || "当前范围"};最近记录:${summary}` });
for (const fraction of [1, 0.5, 0]) {
const lineY = top + (1 - fraction) * plotHeight;
svg.append(metricSvgElement("line", { x1: left, x2: width - right, y1: lineY, y2: lineY, class: "vm-metric-gridline" }));
const label = metricSvgElement("text", { x: left - 8, y: lineY + 4, "text-anchor": "end", class: "vm-metric-axis-label" });
label.textContent = formatMetricValue(fraction * maximum, chart.unit);
svg.append(label);
}
for (const [fraction, anchor] of [[0, "start"], [0.5, "middle"], [1, "end"]]) {
const label = metricSvgElement("text", { x: left + fraction * plotWidth, y: height - 8, "text-anchor": anchor, class: "vm-metric-axis-label" });
label.textContent = formatMetricTime(start + fraction * (end - start));
svg.append(label);
}
const maxGap = intervalSeconds * 2.5;
for (const series of chart.series) {
const segments = [];
let segment = [];
for (const point of points) {
const value = metricNumber(point[series.key]);
if (value === null) {
if (segment.length) segments.push(segment);
segment = [];
continue;
}
if (segment.length && point.timestamp - segment[segment.length - 1].timestamp > maxGap) {
segments.push(segment);
segment = [];
}
segment.push({ timestamp: point.timestamp, value });
}
if (segment.length) segments.push(segment);
for (const group of segments) {
if (group.length > 1) {
const path = metricSvgElement("path", {
d: group.map((point, index) => `${index ? "L" : "M"}${x(point.timestamp).toFixed(2)} ${y(point.value).toFixed(2)}`).join(" "),
fill: "none", stroke: series.color, "stroke-width": 2.6,
"stroke-linecap": "round", "stroke-linejoin": "round",
...(series.dash ? { "stroke-dasharray": series.dash } : {}),
});
svg.append(path);
}
const last = group[group.length - 1];
svg.append(metricSvgElement("circle", { cx: x(last.timestamp), cy: y(last.value), r: group.length === 1 ? 3.8 : 2.8, fill: series.color }));
}
}
plot.replaceChildren(svg);
}
function renderVmMetrics(payload) {
state.metricsPayload = payload;
const intervalSeconds = Math.max(1, Number(payload.interval_seconds) || 60);
const rangeSeconds = VM_METRICS_RANGE_SECONDS[state.metricsRange];
const now = Math.floor(Date.now() / 1000);
const points = payload.points
.filter((point) => Number.isFinite(Number(point?.timestamp)) && Number(point.timestamp) >= now - rangeSeconds - intervalSeconds && Number(point.timestamp) <= now + intervalSeconds)
.map((point) => ({ ...point, timestamp: Number(point.timestamp) }))
.sort((a, b) => a.timestamp - b.timestamp);
const end = Math.max(now, points.length ? points[points.length - 1].timestamp : now);
const start = end - rangeSeconds;
const visiblePoints = points.filter((point) => point.timestamp >= start && point.timestamp <= end);
for (const chart of VM_METRIC_CHARTS) drawMetricChart(chart, visiblePoints, start, end, intervalSeconds);
if (!visiblePoints.length) {
setVmMetricsStatus("当前时间范围内没有采样记录。虚拟机运行后,通常需等待下一次后台采样。", true);
return;
}
const latest = visiblePoints[visiblePoints.length - 1].timestamp;
// Bucket width grows for long ranges; freshness still follows the 60s
// collector cadence, not the downsampling bucket width.
const stale = end - latest > 60 * 2.5;
setVmMetricsStatus(`${visiblePoints.length} 个采样点 · ${stale ? "最近历史记录" : "最近采样"} ${formatMetricTime(latest)} · 每分钟自动刷新${stale ? "(当前无新数据)" : ""}`, stale);
}
function resourceValue(item, kind) {
if (typeof item === "string") return item;
if (item && typeof item === "object") return String(kind === "iso" ? (item.path || item.name || "") : (item.name || item.id || ""));
@ -1634,7 +1883,30 @@ byId("update-apply-button").addEventListener("click", async () => {
});
byId("refresh-button").addEventListener("click", () => void loadVMs());
byId("vm-detail-refresh-button").addEventListener("click", () => void loadVMs());
byId("vm-detail-refresh-button").addEventListener("click", () => {
void loadVMs();
void loadVmMetrics();
});
byId("vm-metrics-range").addEventListener("change", (event) => {
const range = event.currentTarget.value;
if (!Object.hasOwn(VM_METRICS_RANGE_SECONDS, range)) return;
state.metricsRange = range;
stopVmMetrics();
clearVmMetrics("正在加载所选时间范围的监控数据…");
startVmMetrics();
});
window.addEventListener("resize", () => {
if (!metricViewVisible() || !state.metricsPayload) return;
if (state.metricsResizeTimer) window.clearTimeout(state.metricsResizeTimer);
state.metricsResizeTimer = window.setTimeout(() => {
state.metricsResizeTimer = null;
if (metricViewVisible() && state.metricsPayload) renderVmMetrics(state.metricsPayload);
}, 120);
});
document.addEventListener("visibilitychange", () => {
if (document.hidden) stopVmMetrics();
else startVmMetrics();
});
byId("vm-detail-back-button").addEventListener("click", () => {
showVmWorkspace();
byId("search-input").focus();

@ -5,8 +5,8 @@
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="theme-color" content="#101d2c" />
<title>VirtPilot · 虚拟机管理</title>
<link rel="stylesheet" href="/static/styles.css?v=vm-detail-1" />
<script type="module" src="/static/app.js?v=vm-detail-1"></script>
<link rel="stylesheet" href="/static/styles.css?v=vm-metrics-1" />
<script type="module" src="/static/app.js?v=vm-metrics-1"></script>
</head>
<body>
<div id="boot-screen" class="boot-screen" role="status" aria-live="polite">
@ -235,6 +235,17 @@
<div class="vm-detail-operations"><section class="inventory-panel vm-detail-panel" aria-labelledby="vm-detail-actions-title"><div class="panel-heading"><div><h2 id="vm-detail-actions-title">日常操作</h2><p>在这里管理这台虚拟机的电源和控制台。</p></div></div><div id="vm-detail-primary-actions" class="vm-detail-actions"></div></section>
<section class="inventory-panel vm-detail-panel vm-danger-panel" aria-labelledby="vm-detail-danger-title"><div class="panel-heading"><div><h2 id="vm-detail-danger-title">维护与危险操作</h2><p>强制操作可能丢失未保存的数据;重装或删除会清除数据。</p></div></div><div id="vm-detail-danger-actions" class="vm-detail-actions"></div></section></div>
</div>
<section class="vm-metrics" aria-labelledby="vm-metrics-title">
<div class="vm-metrics-heading"><div><span class="eyebrow">PERFORMANCE HISTORY</span><h2 id="vm-metrics-title">性能监控</h2><p>每分钟采样并自动刷新,历史数据最长保留 30 天;长时间范围会聚合,较长的采样缺口留空。</p></div><label class="vm-metrics-range-control" for="vm-metrics-range">时间范围 <select id="vm-metrics-range"><option value="1h">最近 1 小时</option><option value="24h" selected>最近 24 小时</option><option value="7d">最近 7 天</option><option value="30d">最近 30 天</option></select></label></div>
<p id="vm-metrics-status" class="vm-metrics-status" role="status">选择虚拟机后加载监控数据。</p>
<div class="vm-metrics-grid">
<section class="vm-metric-card" aria-labelledby="vm-metric-cpu-title"><div class="vm-metric-card-heading"><span class="vm-metric-icon cpu" aria-hidden="true">◴</span><div><h3 id="vm-metric-cpu-title">CPU 使用率</h3><p id="vm-metric-cpu-summary">—</p></div></div><div id="vm-metric-cpu-plot" class="vm-metric-plot"></div><div class="vm-metric-legend"><span><i class="vm-metric-swatch cpu"></i>使用率</span></div></section>
<section class="vm-metric-card" aria-labelledby="vm-metric-memory-title"><div class="vm-metric-card-heading"><span class="vm-metric-icon memory" aria-hidden="true">▤</span><div><h3 id="vm-metric-memory-title">内存用量</h3><p id="vm-metric-memory-summary">—</p></div></div><div id="vm-metric-memory-plot" class="vm-metric-plot"></div><div class="vm-metric-legend"><span><i class="vm-metric-swatch memory"></i>已使用(估算)</span><span><i class="vm-metric-swatch memory-total"></i>总量</span></div></section>
<section class="vm-metric-card" aria-labelledby="vm-metric-disk-title"><div class="vm-metric-card-heading"><span class="vm-metric-icon disk" aria-hidden="true">▥</span><div><h3 id="vm-metric-disk-title">磁盘 I/O</h3><p id="vm-metric-disk-summary">—</p></div></div><div id="vm-metric-disk-plot" class="vm-metric-plot"></div><div class="vm-metric-legend"><span><i class="vm-metric-swatch disk-read"></i>读取</span><span><i class="vm-metric-swatch disk-write"></i>写入</span></div></section>
<section class="vm-metric-card" aria-labelledby="vm-metric-network-title"><div class="vm-metric-card-heading"><span class="vm-metric-icon network" aria-hidden="true">⇄</span><div><h3 id="vm-metric-network-title">网络速率</h3><p id="vm-metric-network-summary">—</p></div></div><div id="vm-metric-network-plot" class="vm-metric-plot"></div><div class="vm-metric-legend"><span><i class="vm-metric-swatch network-rx"></i>接收</span><span><i class="vm-metric-swatch network-tx"></i>发送</span></div></section>
</div>
<p class="vm-metrics-footnote">磁盘 I/O 为每秒读写量,仅统计 libvirt 提供计数的设备;网络收发为虚拟机视角的每秒速率,不代表套餐带宽上限。内存已用量为估算值,取决于虚拟机是否提供气球驱动统计。</p>
</section>
</section>
</div>
<div id="update-workspace" hidden>

@ -125,6 +125,7 @@ tbody tr:hover{background:#fcfefe}
.empty-state{text-align:center;padding:65px 20px 72px}.empty-icon{margin:0 auto 16px;width:54px;height:54px;display:flex;align-items:center;justify-content:center;border-radius:16px;background:#eaf7f7;color:#17a3aa;font-size:28px}.empty-state h3{font-size:16px;color:#34465a;margin:0 0 8px}.empty-state p{font-size:12px;color:#a0aebe;margin:0}
.footer-note{text-align:right;color:#a9b4c0;font-size:10px;margin-top:18px}
.vm-detail-back{padding-left:0;margin-bottom:22px;color:#267b86}.vm-detail-heading{align-items:center}.vm-detail-heading .heading-actions{align-items:center}.vm-detail-heading .state-pill{min-height:31px}.vm-detail-uuid{font-family:ui-monospace,SFMono-Regular,Menlo,monospace;overflow-wrap:anywhere}.vm-detail-grid{display:grid;grid-template-columns:minmax(0,1.15fr) minmax(320px,.85fr);gap:18px;align-items:start}.vm-detail-panel{min-height:0}.vm-detail-facts{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:0;margin:0;padding:6px 24px 17px}.vm-detail-facts>div{min-width:0;padding:17px 9px 17px 0;border-bottom:1px solid #edf1f5}.vm-detail-facts dt{font-size:11px;font-weight:700;color:#91a0af;margin-bottom:9px}.vm-detail-facts dd{margin:0;color:#273c52;font-size:13px;font-weight:700;line-height:1.5;overflow-wrap:anywhere}.vm-detail-operations{display:grid;gap:18px}.vm-detail-actions{display:flex;flex-wrap:wrap;gap:10px;padding:20px 24px 23px}.vm-detail-actions .row-button{min-height:39px;padding:9px 15px;border-color:#dce9ea;font-size:12px}.vm-detail-actions .row-button.primary{border-color:#cbecee}.vm-danger-panel .panel-heading h2{color:#a4434f}.vm-danger-panel .row-button.danger{border-color:#f2dfe2;background:#fff9f9}.vm-danger-panel .row-button.danger:not(:disabled):hover{background:#ffedef}
.vm-metrics{margin-top:25px}.vm-metrics-heading{display:flex;align-items:flex-end;justify-content:space-between;gap:18px;margin-bottom:14px}.vm-metrics-heading h2{font-size:19px;margin:7px 0 6px}.vm-metrics-heading p{font-size:11px;color:#8191a1;line-height:1.6;margin:0}.vm-metrics-range-control{display:flex;align-items:center;gap:9px;flex:none;color:#64768a;font-size:11px;font-weight:700}.vm-metrics-range-control select{height:37px;padding:0 11px;border:1px solid #dbe6ec;border-radius:8px;color:#344e60;background:#fff;font-size:12px;font-weight:700}.vm-metrics-status{margin:0 0 14px;padding:9px 12px;border-radius:8px;background:#eaf7f7;color:#277b82;font-size:11px;line-height:1.5}.vm-metrics-status.warning{background:#fff6e9;color:#946620}.vm-metrics-grid{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:17px}.vm-metric-card{min-width:0;padding:20px 21px 16px;border:1px solid #e7ecf2;border-radius:13px;background:#fff;box-shadow:0 10px 30px rgba(19,42,65,.027)}.vm-metric-card-heading{display:flex;align-items:flex-start;gap:11px;min-height:62px}.vm-metric-card-heading>div{min-width:0}.vm-metric-card-heading h3{margin:1px 0 6px;font-size:13px;color:#32485b}.vm-metric-card-heading p{margin:0;color:#60758a;font-size:12px;line-height:1.45;overflow-wrap:anywhere}.vm-metric-icon{display:flex;align-items:center;justify-content:center;flex:none;width:34px;height:34px;border-radius:9px;font-size:18px;font-weight:700}.vm-metric-icon.cpu{background:#e8f8f8;color:#0f9eaa}.vm-metric-icon.memory{background:#edf3fb;color:#4781cf}.vm-metric-icon.disk{background:#eef3fa;color:#477dc9}.vm-metric-icon.network{background:#eaf8f2;color:#17a685}.vm-metric-plot{display:flex;align-items:center;justify-content:center;height:205px;margin:8px 0 6px;overflow:hidden}.vm-metric-plot svg{display:block;width:100%;height:100%;font-family:Inter,"SF Pro Display","PingFang SC","Microsoft YaHei",sans-serif}.vm-metric-gridline{stroke:#e5edf2;stroke-width:1;stroke-dasharray:3 4}.vm-metric-axis-label{fill:#9aabb9;font-size:10px}.vm-metric-empty{margin:0;padding:15px;text-align:center;color:#94a2af;font-size:11px;line-height:1.6}.vm-metric-legend{display:flex;align-items:center;flex-wrap:wrap;gap:15px;padding-left:7px;color:#728297;font-size:10px;font-weight:700}.vm-metric-legend span{display:inline-flex;align-items:center;gap:6px}.vm-metric-swatch{display:inline-block;width:16px;height:0;border-top:3px solid #0f9eaa;border-radius:2px}.vm-metric-swatch.cpu{border-color:#0f9eaa}.vm-metric-swatch.memory{border-color:#4781cf}.vm-metric-swatch.memory-total{border-color:#a4b7d7;border-top-style:dashed}.vm-metric-swatch.disk-read{border-color:#477dc9}.vm-metric-swatch.disk-write{border-color:#e59c46}.vm-metric-swatch.network-rx{border-color:#17a685}.vm-metric-swatch.network-tx{border-color:#926ed0}.vm-metrics-footnote{margin:12px 1px 0;color:#9ba8b6;font-size:10px;line-height:1.6}
.modal-backdrop{position:fixed;inset:0;z-index:30;background:rgba(11,25,40,.58);display:flex;align-items:center;justify-content:center;padding:20px;backdrop-filter:blur(3px)}
.modal{width:min(100%,610px);max-height:min(92vh,900px);overflow:auto;border-radius:14px;background:#fff;box-shadow:0 22px 70px rgba(0,0,0,.2)}.narrow-modal{width:min(100%,460px)}
.create-modal{width:min(100%,720px)}
@ -149,6 +150,7 @@ tbody tr:hover{background:#fcfefe}
@media(max-width:960px){.host-columns{grid-template-columns:1fr}.host-side-stack{grid-template-columns:repeat(2,minmax(0,1fr));align-items:start}.setup-layout{grid-template-columns:minmax(280px,36%) 1fr}}
@media(max-width:960px){.update-grid{grid-template-columns:1fr}}
@media(max-width:960px){.vm-detail-grid{grid-template-columns:1fr}}
@media(max-width:740px){.vm-metrics-grid{grid-template-columns:1fr}.vm-metrics-heading{align-items:flex-start;flex-direction:column}.vm-metric-plot{height:190px}}
@media(max-width:740px){.login-layout{display:block;min-height:100vh}.login-art{display:none}.login-form-side{min-height:100vh;padding:32px 25px}.login-mobile-brand{display:flex;align-items:center;gap:10px;font-size:21px;font-weight:800;margin-bottom:70px}.login-card h2{font-size:29px}.page-heading{align-items:flex-start;flex-direction:column;gap:20px}.heading-actions{width:100%}.heading-actions .button{flex:1}.overview-grid{gap:9px}.stat-card{min-height:121px;padding:16px}.stat-card>strong{font-size:25px}.stat-top{font-size:10px}.stat-icon{width:27px;height:27px;font-size:12px}.stat-card>small{font-size:9px}.panel-heading{align-items:flex-start;flex-direction:column;padding:19px}.panel-tools{width:100%}.search-box{flex:1}.search-box input{width:100%}.footer-note{text-align:left}.console-toolbar{align-items:flex-start}.console-tools{flex-wrap:wrap;justify-content:flex-end}.console-tools .button{font-size:10px}}
@media(max-width:740px){.vm-table{min-width:0}.vm-table thead{display:none}.vm-table tbody{display:grid;gap:12px;padding:12px}.vm-table tr{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));border:1px solid #e4ebf0;border-radius:10px;overflow:hidden}.vm-table td{display:block;min-width:0;height:auto;border:0;padding:12px 14px}.vm-table td:first-child{grid-column:1/-1;border-bottom:1px solid #edf1f5}.vm-table td:not(:first-child):before{content:attr(data-label);display:block;color:#9ba8b7;font-size:10px;font-weight:800;margin-bottom:7px}.vm-table .vm-metadata{max-width:none}.vm-table .vm-ip{white-space:normal}.vm-detail-facts{padding:4px 17px 12px}.vm-detail-actions{padding:16px 19px 19px}}
@media(max-width:740px){.setup-layout{display:block}.setup-intro{min-height:0;padding:28px 25px}.setup-brand{margin:0 0 38px}.setup-intro h1{font-size:31px;margin:11px 0}.setup-intro p{font-size:12px;margin-bottom:24px}.setup-steps{margin-top:0}.setup-form-side{padding:36px 25px 55px}.setup-card h2{font-size:27px}.host-side-stack{grid-template-columns:1fr}.host-heading{align-items:flex-start;flex-direction:column}.host-topbar{height:65px}.host-content{padding-top:27px}.host-card{padding:20px}}

@ -33,6 +33,7 @@ from virsh_panel.cloud_templates import ( # noqa: E402
)
from virsh_panel.config import Settings # noqa: E402
from virsh_panel.libvirt_service import VMError # noqa: E402
from virsh_panel.metrics import host_scope, query_interval_seconds # noqa: E402
from virsh_panel.panel_state import PanelState # noqa: E402
from virsh_panel.remote_host import RemoteHostError # noqa: E402
@ -997,3 +998,61 @@ def test_websocket_origin_auth_and_console_relay(web):
assert relay.await_args.kwargs["port"] == 5907
assert relay.await_args.kwargs["ssh_target"] == "vp-managed"
assert relay.await_args.kwargs["ssh_config_file"] == "/fake/config"
def test_vm_metrics_requires_auth_validates_vm_and_isolates_host(tmp_path):
state = PanelState(tmp_path / "panel-data")
manager = FakeRemoteManager()
service = FakeVMService()
class FakeMetricStore:
def __init__(self):
self.calls = []
self.allowed_scope = None
def query_metric_samples(self, scope, vm_id, **kwargs):
self.calls.append((scope, vm_id, kwargs))
if scope != self.allowed_scope:
return []
return [{"timestamp": 2_999_900, "cpu_percent": 25,
"memory_used_mib": None, "memory_total_mib": None,
"disk_read_bps": 128, "disk_write_bps": 0,
"network_rx_bps": 4096, "network_tx_bps": 1024}]
metric_store = FakeMetricStore()
app = create_app(Settings(data_dir=state.data_dir, static_dir=STATIC_DIR),
panel_state=state, remote_manager=manager,
service_factory=lambda _profile: service,
metric_store=metric_store)
with TestClient(app, base_url=ORIGIN) as client:
endpoint = f"/api/v1/vms/{VM_ID}/metrics"
assert client.get(endpoint).status_code == 401
token = state.token_file.read_text(encoding="ascii").strip()
csrf = setup_admin(client, token)
configure_host(client, csrf)
metric_store.allowed_scope = host_scope(manager.current())
with patch("virsh_panel.app.time.time", return_value=3_000_000):
response = client.get(endpoint + "?range=30d")
assert response.status_code == 200
body = response.json()
assert body["vm_id"] == VM_ID
assert body["range"] == "30d"
assert body["interval_seconds"] == query_interval_seconds(30 * 24 * 3600)
assert len(body["points"]) == 1
scope, vm_id, args = metric_store.calls[-1]
assert scope == metric_store.allowed_scope and vm_id == VM_ID
assert args["since"] == 3_000_000 - 30 * 24 * 3600
assert args["until"] == 3_000_000 and args["limit"] == 600
assert client.get(endpoint + "?range=60d").status_code == 422
assert client.get("/api/v1/vms/not-a-uuid/metrics").status_code == 400
assert client.get(
"/api/v1/vms/00000000-0000-4000-8000-000000000002/metrics"
).status_code == 404
before = len(metric_store.calls)
manager.profile["fingerprint"] = "SHA256:another-host"
with patch("virsh_panel.app.time.time", return_value=3_000_000):
changed_host = client.get(endpoint)
assert changed_host.status_code == 200
assert changed_host.json()["points"] == []
assert len(metric_store.calls) == before + 1

@ -1588,6 +1588,68 @@ class VMServiceTests(unittest.TestCase):
self.assertNotIn(TEMPLATE_NAME, self.backend.pools["images"].volumes)
self.assertEqual(self.backend.opened, self.backend.closed)
def test_metric_counters_use_bulk_libvirt_stats_for_running_domains(self):
constants = {
"VIR_DOMAIN_STATS_CPU_TOTAL": 2,
"VIR_DOMAIN_STATS_BALLOON": 4,
"VIR_DOMAIN_STATS_BLOCK": 8,
"VIR_DOMAIN_STATS_INTERFACE": 16,
"VIR_CONNECT_GET_ALL_DOMAINS_STATS_RUNNING": 32,
}
for name, value in constants.items():
setattr(self.service_module.libvirt, name, value)
domain = types.SimpleNamespace(
UUIDString=lambda: "00000000-0000-4000-8000-000000000001",
info=lambda: (1, 2048 * 1024, 2048 * 1024, 2, 0),
ID=lambda: 7,
)
calls = []
def get_stats(_connection, groups, flags):
calls.append((groups, flags))
return [(domain, {"cpu.time": 123, "net.0.rx.bytes": 456})]
with patch.object(FakeConnection, "getAllDomainStats", get_stats, create=True):
result = self.service.metric_counters()
self.assertEqual(calls, [(2 | 4 | 8 | 16, 32)])
self.assertEqual(len(result), 1)
self.assertEqual(result[0].generation, 7)
self.assertEqual(result[0].vcpus, 2)
self.assertEqual(result[0].values["net.0.rx.bytes"], 456)
self.assertEqual(self.backend.opened, self.backend.closed)
def test_metric_counters_skip_vm_disappearing_after_bulk_read(self):
constants = {
"VIR_DOMAIN_STATS_CPU_TOTAL": 2,
"VIR_DOMAIN_STATS_BALLOON": 4,
"VIR_DOMAIN_STATS_BLOCK": 8,
"VIR_DOMAIN_STATS_INTERFACE": 16,
"VIR_CONNECT_GET_ALL_DOMAINS_STATS_RUNNING": 32,
}
for name, value in constants.items():
setattr(self.service_module.libvirt, name, value)
def gone():
raise FakeLibvirtError("domain vanished")
disappearing = types.SimpleNamespace(info=gone)
healthy = types.SimpleNamespace(
UUIDString=lambda: "00000000-0000-4000-8000-000000000002",
info=lambda: (1, 2048 * 1024, 2048 * 1024, 2, 0),
ID=lambda: 8,
)
with patch.object(
FakeConnection, "getAllDomainStats",
lambda _connection, _groups, _flags: [
(disappearing, {"cpu.time": 1}),
(healthy, {"cpu.time": 2}),
], create=True,
):
result = self.service.metric_counters()
self.assertEqual(len(result), 1)
self.assertEqual(result[0].vm_id, "00000000-0000-4000-8000-000000000002")
self.assertEqual(self.backend.opened, self.backend.closed)
if __name__ == "__main__":
unittest.main()

@ -0,0 +1,424 @@
"""VM metric math, DB persistence, and the process-local collection lifecycle."""
from __future__ import annotations
import threading
import pytest
from virsh_panel.database import DatabaseStore
from virsh_panel.metrics import (
MAX_QUERY_POINTS, RANGES, MetricSampler, MetricsCollector, RawVMStats,
host_scope, query_interval_seconds,
)
VM_ID = "00000000-0000-4000-8000-000000000001"
HOST = {"host": "hypervisor.example", "port": 2222,
"fingerprint": "SHA256:one", "storage_pool": "/home/kvm"}
def raw(*, generation=12, cpu=1_000_000_000, block_read=100,
block_write=200, network_rx=1000, network_tx=2000,
memory_time=1000, extra=None):
values = {
"cpu.time": cpu,
"balloon.available": 2 * 1024 * 1024,
"balloon.usable": 512 * 1024,
"balloon.last-update": memory_time,
"block.count": 1,
"block.0.name": "vda",
"block.0.rd.bytes": block_read,
"block.0.wr.bytes": block_write,
"net.count": 1,
"net.0.name": "vnet1",
"net.0.rx.bytes": network_rx,
"net.0.tx.bytes": network_tx,
}
values.update(extra or {})
return RawVMStats(VM_ID, generation, 2, values)
def test_host_scope_ignores_pool_but_changes_with_ssh_host_identity():
first = host_scope(HOST)
assert first == host_scope({**HOST, "storage_pool": "/different"})
assert first != host_scope({**HOST, "fingerprint": "SHA256:two"})
assert first != host_scope({**HOST, "port": 22})
def test_interval_caps_all_ranges_at_600_epoch_aligned_buckets():
for seconds in RANGES.values():
interval = query_interval_seconds(seconds)
assert interval % 60 == 0
assert seconds // interval + 2 <= MAX_QUERY_POINTS
def test_initial_sample_and_fresh_guest_memory_then_rates():
sampler = MetricSampler()
scope = host_scope(HOST)
first = sampler.observe(scope, raw(), timestamp=1000, monotonic=10)
assert first["cpu_percent"] is None
assert first["disk_read_bps"] is None
assert first["network_rx_bps"] is None
assert first["memory_used_mib"] == 1536
assert first["memory_total_mib"] == 2048
second = sampler.observe(
scope,
raw(cpu=61_000_000_000, block_read=700, block_write=1400,
network_rx=2200, network_tx=2600, memory_time=1050),
timestamp=1060, monotonic=70,
)
assert second["cpu_percent"] == 50
assert second["disk_read_bps"] == 10
assert second["disk_write_bps"] == 20
assert second["network_rx_bps"] == 20
assert second["network_tx_bps"] == 10
def test_cpu_time_over_vcpu_capacity_is_displayed_as_100_percent():
sampler = MetricSampler()
scope = host_scope(HOST)
sampler.observe(scope, raw(), timestamp=1000, monotonic=10)
# 130 CPU-seconds across two vCPUs in a 60-second interval; libvirt can
# include QEMU auxiliary threads in cpu.time.
point = sampler.observe(scope, raw(cpu=131_000_000_000),
timestamp=1060, monotonic=70)
assert point["cpu_percent"] == 100
def test_memory_missing_or_stale_balloon_stats_never_uses_rss_or_current():
sampler = MetricSampler()
scope = host_scope(HOST)
values = {"balloon.current": 2 * 1024 * 1024, "balloon.rss": 512 * 1024}
for extra in (
{"balloon.last-update": 1},
{"balloon.usable": None},
{"balloon.available": None},
):
point = sampler.observe(scope, raw(extra={**values, **extra}),
timestamp=1000, monotonic=10)
assert point["memory_used_mib"] is None
assert point["memory_total_mib"] is None
def test_counter_reset_reboot_device_change_and_partial_data_are_unknown():
sampler = MetricSampler()
scope = host_scope(HOST)
sampler.observe(scope, raw(), timestamp=1000, monotonic=10)
reset = sampler.observe(scope, raw(cpu=100, block_read=1, network_rx=1),
timestamp=1060, monotonic=70)
assert reset["cpu_percent"] is None
assert reset["disk_read_bps"] is None
assert reset["network_rx_bps"] is None
# Even if a second disk masks the first disk's decreasing counter in the
# aggregate, device change must not produce a transient spike.
changed = sampler.observe(scope, raw(extra={
"block.count": 2, "block.1.name": "vdb",
"block.1.rd.bytes": 10_000, "block.1.wr.bytes": 20_000,
}), timestamp=1120, monotonic=130)
assert changed["disk_read_bps"] is None
assert changed["disk_write_bps"] is None
partial = sampler.observe(scope, raw(extra={
"block.count": 2, "block.1.name": "vdb",
"block.1.rd.bytes": 11_000,
}), timestamp=1180, monotonic=190)
assert partial["disk_write_bps"] is None # second device missing wr.bytes
assert partial["disk_read_bps"] is not None
reboot = sampler.observe(scope, raw(generation=13, cpu=200_000_000_000),
timestamp=1240, monotonic=250)
assert reboot["cpu_percent"] is None
assert reboot["network_tx_bps"] is None
def test_cdrom_without_byte_counters_does_not_hide_disk_rates():
sampler = MetricSampler()
scope = host_scope(HOST)
cdrom = {"block.count": 2, "block.1.name": "sda"}
sampler.observe(scope, raw(extra=cdrom), timestamp=1000, monotonic=10)
second = sampler.observe(
scope, raw(block_read=700, block_write=1400, extra={
**cdrom, "balloon.last-update": 1060,
}), timestamp=1060, monotonic=70,
)
assert second["disk_read_bps"] == 10
assert second["disk_write_bps"] == 20
def test_single_device_counter_rollback_is_not_masked_by_another_device():
sampler = MetricSampler()
scope = host_scope(HOST)
second_disk = {"block.count": 2, "block.1.name": "vdb",
"block.1.rd.bytes": 1_000, "block.1.wr.bytes": 1_000}
sampler.observe(scope, raw(extra=second_disk), timestamp=1000, monotonic=10)
point = sampler.observe(scope, raw(block_read=1, block_write=1, extra={
**second_disk, "block.1.rd.bytes": 10_000,
"block.1.wr.bytes": 10_000,
}), timestamp=1060, monotonic=70)
assert point["disk_read_bps"] is None
assert point["disk_write_bps"] is None
def test_unobserved_long_gap_is_not_averaged_as_current_rate():
sampler = MetricSampler()
scope = host_scope(HOST)
sampler.observe(scope, raw(), timestamp=1000, monotonic=10)
delayed = sampler.observe(scope, raw(cpu=50_000_000_000, block_read=2000),
timestamp=1600, monotonic=610)
assert delayed["cpu_percent"] is None
assert delayed["disk_read_bps"] is None
class FakeCursor:
def __init__(self, rows=(), delete_counts=()):
self.rows = rows
self.commands = []
self.delete_counts = list(delete_counts)
self.rowcount = 0
def __enter__(self):
return self
def __exit__(self, *_args):
return None
def execute(self, sql, args=None):
self.commands.append((sql, args))
self.rowcount = (self.delete_counts.pop(0)
if sql.startswith("DELETE FROM vm_metric_samples")
and self.delete_counts else 0)
def executemany(self, sql, args):
self.commands.append((sql, args))
def fetchall(self):
return self.rows
class FakeConnection:
def __init__(self, rows=(), delete_counts=()):
self.fake_cursor = FakeCursor(rows, delete_counts)
self.commits = 0
self.closed = False
def cursor(self):
return self.fake_cursor
def commit(self):
self.commits += 1
def rollback(self):
raise AssertionError("unexpected rollback")
def close(self):
self.closed = True
def test_metric_table_created_with_record_table():
connection = FakeConnection()
store = DatabaseStore.__new__(DatabaseStore)
store._connect = lambda: connection
store._initialize_schema()
assert len(connection.fake_cursor.commands) == 2
assert "vm_metric_samples" in connection.fake_cursor.commands[1][0]
assert "vm_metric_retention" in connection.fake_cursor.commands[1][0]
assert connection.commits == 1 and connection.closed
def test_metric_store_prunes_physically_and_queries_only_scoped_recent_data():
scope = host_scope(HOST)
connection = FakeConnection([{
"timestamp": 1000,
"cpu_percent": 12.5,
"memory_used_mib": 1024,
"memory_total_mib": 2048,
"disk_read_bps": None,
"disk_write_bps": None,
"network_rx_bps": 256,
"network_tx_bps": 0,
}])
store = DatabaseStore.__new__(DatabaseStore)
store._connect = lambda: connection
point = {"vm_id": VM_ID, "timestamp": 1000, "cpu_percent": 12.5,
"memory_used_mib": 1024, "memory_total_mib": 2048,
"disk_read_bps": None, "disk_write_bps": None,
"network_rx_bps": 256, "network_tx_bps": 0}
store.save_metric_samples(scope, [point])
store.prune_metric_samples(500)
result = store.query_metric_samples(scope, VM_ID, since=500, until=1100,
interval_seconds=60, limit=600)
assert result[0]["cpu_percent"] == 12.5
assert result[0]["disk_read_bps"] is None
insert, delete, query = connection.fake_cursor.commands
assert "INSERT INTO vm_metric_samples" in insert[0]
assert "DELETE FROM vm_metric_samples WHERE sampled_at < %s" in delete[0]
assert delete[1] == (500, 5000)
assert "host_scope=%s AND vm_uuid=%s AND sampled_at >= %s" in query[0]
assert "LIMIT %s" in query[0]
assert query[1] == (scope, VM_ID, 500, 1100, 60, 600)
assert connection.commits == 2 and connection.closed
def test_metric_prune_is_one_index_ordered_batch_per_transaction():
connection = FakeConnection(delete_counts=[5000] * 10)
store = DatabaseStore.__new__(DatabaseStore)
store._connect = lambda: connection
assert store.prune_metric_samples(1234) == 5000
assert len(connection.fake_cursor.commands) == 1
assert connection.commits == 1 and connection.closed
assert all("ORDER BY sampled_at LIMIT %s" in command
and args == (1234, 5000)
for command, args in connection.fake_cursor.commands)
def test_metric_prune_reports_short_final_batch():
connection = FakeConnection(delete_counts=[275])
store = DatabaseStore.__new__(DatabaseStore)
store._connect = lambda: connection
assert store.prune_metric_samples(1234) == 275
assert len(connection.fake_cursor.commands) == 1
assert connection.commits == 1
def test_metric_store_rejects_invalid_host_and_uuid_before_db_query():
store = DatabaseStore.__new__(DatabaseStore)
store._connect = lambda: pytest.fail("invalid scope must not query database")
with pytest.raises(ValueError):
store.query_metric_samples("bad-scope", VM_ID, since=0, until=1,
interval_seconds=60)
with pytest.raises(ValueError):
store.query_metric_samples(host_scope(HOST), "not-uuid", since=0, until=1,
interval_seconds=60)
def test_collector_prunes_when_ssh_locked_and_starts_after_db_configuration():
class Store:
def __init__(self):
self.pruned = []
self.saved = []
self.pruned_event = threading.Event()
self.saved_event = threading.Event()
def prune_metric_samples(self, cutoff):
self.pruned.append(cutoff)
self.pruned_event.set()
return 0
def save_metric_samples(self, scope, points):
self.saved.append((scope, points))
self.saved_event.set()
current = {"store": None, "profile": None}
store = Store()
class Service:
def metric_counters(self):
return [raw()]
collector = MetricsCollector(lambda: current["store"],
lambda: current["profile"],
lambda _profile: Service(), interval_seconds=0.01)
collector.start()
current["profile"] = {**HOST, "key_locked": True}
current["store"] = store
try:
assert store.pruned_event.wait(timeout=2)
assert store.pruned and not store.saved
current["profile"]["key_locked"] = False
assert store.saved_event.wait(timeout=2)
assert store.saved
finally:
collector.close()
def test_metric_cleanup_failure_does_not_discard_current_sample(caplog):
events = []
attempted = threading.Event()
class Store:
def save_metric_samples(self, _scope, points):
events.append(("saved", points))
def prune_metric_samples(self, _cutoff):
events.append(("prune", None))
attempted.set()
raise RuntimeError("secret database details")
class Service:
def metric_counters(self):
return [raw()]
collector = MetricsCollector(lambda: Store(), lambda: HOST,
lambda _profile: Service(), interval_seconds=1)
collector.start()
assert attempted.wait(timeout=2)
collector.close()
assert [event[0] for event in events] == ["saved", "prune"]
assert "RuntimeError" in caplog.text
assert "secret database details" not in caplog.text
def test_backlog_cleanup_continues_in_other_thread_without_stalling_samples():
second_saved = threading.Event()
prune_entered = threading.Event()
release_prune = threading.Event()
backlog_drained = threading.Event()
class Store:
def __init__(self):
self.saved = 0
self.pruned = 0
def save_metric_samples(self, _scope, _points):
self.saved += 1
if self.saved >= 2:
second_saved.set()
def prune_metric_samples(self, _cutoff):
self.pruned += 1
if self.pruned == 1:
prune_entered.set()
assert release_prune.wait(timeout=2)
if self.pruned < 3:
return 5000
backlog_drained.set()
return 0
class Service:
def metric_counters(self):
return [raw()]
store = Store()
collector = MetricsCollector(lambda: store, lambda: HOST,
lambda _profile: Service(), interval_seconds=0.01)
collector.start()
try:
assert prune_entered.wait(timeout=2)
assert second_saved.wait(timeout=2)
release_prune.set()
assert backlog_drained.wait(timeout=2)
assert store.pruned >= 3
finally:
release_prune.set()
collector.close()
def test_collector_thread_starts_before_db_configuration_and_stops():
ready = threading.Event()
current = {"store": None}
class Store:
def prune_metric_samples(self, _cutoff):
ready.set()
return 0
collector = MetricsCollector(lambda: current["store"], lambda: None,
lambda _profile: pytest.fail("no host configured"),
interval_seconds=0.01)
collector.start()
current["store"] = Store()
assert ready.wait(timeout=2)
collector.close()
assert collector._thread is not None and not collector._thread.is_alive()
assert collector._prune_thread is not None and not collector._prune_thread.is_alive()

@ -10,13 +10,14 @@ import logging
import secrets
import subprocess
import threading
import time
import uuid
from concurrent.futures import ThreadPoolExecutor
from contextlib import asynccontextmanager, suppress
from typing import Callable, Literal
from typing import Any, Callable, Literal
from urllib.parse import urlsplit
from fastapi import Depends, FastAPI, HTTPException, Request, WebSocket
from fastapi import Depends, FastAPI, HTTPException, Query, Request, WebSocket
from fastapi.responses import FileResponse, JSONResponse
from fastapi.staticfiles import StaticFiles
from pydantic import BaseModel, ConfigDict, Field, SecretStr
@ -32,6 +33,8 @@ from .console import ConsoleRelayError, relay_console
from .database import DatabaseError, make_components
from .database_bootstrap import DatabaseBootstrap, DatabaseSetupError
from .libvirt_service import VMError, VMService
from .metrics import (MAX_QUERY_POINTS, RETENTION_SECONDS, RANGES, MetricsCollector,
host_scope, query_interval_seconds)
from .panel_state import PanelState, SetupError
from .remote_host import HostProfile, RemoteHostError, RemoteHostManager
from .updater import BRANCH, REPOSITORY_URL, UpdateError, Updater
@ -238,6 +241,7 @@ def create_app(
panel_state: PanelState | None = None,
remote_manager: RemoteHostManager | None = None,
service_factory: Callable[[dict], VMService] | None = None,
metric_store: Any | None = None,
) -> FastAPI:
settings = settings or Settings.from_env()
settings.validate()
@ -319,12 +323,21 @@ def create_app(
login_limiter = LoginLimiter()
setup_limiter = LoginLimiter()
unlock_limiter = LoginLimiter()
metric_collector = MetricsCollector(
lambda: data_store,
lambda: manager.current() if manager is not None else None,
lambda profile: new_service(profile),
) if database_mode else None
@asynccontextmanager
async def lifespan(_app: FastAPI):
if metric_collector is not None:
metric_collector.start()
try:
yield
finally:
if metric_collector is not None:
metric_collector.close()
reinstall_executor.shutdown(wait=True, cancel_futures=False)
import_executor.shutdown(wait=True, cancel_futures=False)
close_manager = getattr(manager, "close", None) if manager is not None else None
@ -852,6 +865,37 @@ def create_app(
profile = current_profile()
return {**new_service(profile).get_vm(vm_id), "region": profile.get("region", "")}
@app.get("/api/v1/vms/{vm_id}/metrics")
def get_vm_metrics(
vm_id: str,
range_: Literal["1h", "24h", "7d", "30d"] = Query("1h", alias="range"),
_session: Session = Depends(require_session),
):
try:
canonical_id = str(uuid.UUID(vm_id))
except (TypeError, ValueError, AttributeError) as exc:
raise VMError("虚拟机 UUID 无效", 400) from exc
profile = current_profile()
# Never reveal another host's retained samples for a reused UUID.
new_service(profile).get_vm(canonical_id)
backend = metric_store if metric_store is not None else data_store
if backend is None:
raise HTTPException(status_code=503, detail="指标数据库尚未就绪")
now = int(time.time())
window = RANGES[range_]
interval = query_interval_seconds(window)
since = max(now - window, now - RETENTION_SECONDS)
try:
scope = host_scope(profile)
except (KeyError, ValueError, TypeError) as exc:
raise HTTPException(status_code=409, detail="宿主机身份信息不完整") from exc
points = backend.query_metric_samples(
scope, canonical_id, since=since, until=now,
interval_seconds=interval, limit=MAX_QUERY_POINTS,
)
return {"vm_id": canonical_id, "range": range_,
"interval_seconds": interval, "points": points}
@app.post("/api/v1/vms", status_code=201)
def create_vm(body: CreateISOBody | CreateCloudBody,
_session: Session = Depends(require_write)):

@ -10,6 +10,7 @@ from __future__ import annotations
import hmac
import json
import math
import os
import re
import secrets
@ -17,6 +18,7 @@ import shutil
import stat
import sys
import threading
import uuid
from pathlib import Path
from typing import Any, Mapping
from urllib.parse import quote
@ -62,6 +64,28 @@ CREATE TABLE IF NOT EXISTS panel_records (
PRIMARY KEY (record_type, record_key)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
_METRIC_SCHEMA = """
CREATE TABLE IF NOT EXISTS vm_metric_samples (
host_scope CHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
vm_uuid CHAR(36) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,
sampled_at BIGINT UNSIGNED NOT NULL,
cpu_percent DOUBLE NULL,
memory_used_mib DOUBLE NULL,
memory_total_mib DOUBLE NULL,
disk_read_bps DOUBLE NULL,
disk_write_bps DOUBLE NULL,
network_rx_bps DOUBLE NULL,
network_tx_bps DOUBLE NULL,
PRIMARY KEY (host_scope, vm_uuid, sampled_at),
KEY vm_metric_retention (sampled_at)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
_METRIC_COLUMNS = (
"cpu_percent", "memory_used_mib", "memory_total_mib", "disk_read_bps",
"disk_write_bps", "network_rx_bps", "network_tx_bps",
)
_HOST_SCOPE = re.compile(r"[0-9a-f]{64}\Z")
_METRIC_PRUNE_BATCH_SIZE = 5000
class DatabaseError(Exception):
@ -139,6 +163,7 @@ class DatabaseStore:
try:
with connection.cursor() as cursor:
cursor.execute(_SCHEMA)
cursor.execute(_METRIC_SCHEMA)
connection.commit()
except pymysql.MySQLError as exc:
connection.rollback()
@ -146,6 +171,105 @@ class DatabaseStore:
finally:
connection.close()
@staticmethod
def _metric_identity(host_scope: str, vm_id: str) -> str:
if not isinstance(host_scope, str) or not _HOST_SCOPE.fullmatch(host_scope):
raise ValueError("Metric host scope is invalid")
try:
canonical = str(uuid.UUID(vm_id))
except (TypeError, ValueError, AttributeError) as exc:
raise ValueError("Metric VM UUID is invalid") from exc
return canonical
def save_metric_samples(self, host_scope: str, points: list[dict[str, Any]]) -> None:
if not points:
return
rows = []
for point in points:
vm_id = self._metric_identity(host_scope, point["vm_id"])
timestamp = point["timestamp"]
if isinstance(timestamp, bool) or not isinstance(timestamp, int) or timestamp < 0:
raise ValueError("Metric timestamp is invalid")
metrics = []
for name in _METRIC_COLUMNS:
value = point.get(name)
if (value is not None and (isinstance(value, bool)
or not isinstance(value, (int, float))
or not math.isfinite(value) or value < 0)):
raise ValueError("Metric value is invalid")
metrics.append(value)
rows.append((host_scope, vm_id, timestamp, *metrics))
connection = self._connect()
try:
with connection.cursor() as cursor:
cursor.executemany(
"INSERT INTO vm_metric_samples (host_scope, vm_uuid, sampled_at, "
+ ", ".join(_METRIC_COLUMNS) + ") VALUES ("
+ ", ".join(["%s"] * (3 + len(_METRIC_COLUMNS))) + ") "
"ON DUPLICATE KEY UPDATE "
+ ", ".join(f"{name}=VALUES({name})" for name in _METRIC_COLUMNS),
rows,
)
connection.commit()
except pymysql.MySQLError as exc:
connection.rollback()
raise DatabaseError("无法保存虚拟机指标") from exc
finally:
connection.close()
def prune_metric_samples(self, cutoff: int) -> int:
if isinstance(cutoff, bool) or not isinstance(cutoff, int) or cutoff < 0:
raise ValueError("Metric retention cutoff is invalid")
connection = self._connect()
try:
with connection.cursor() as cursor:
cursor.execute(
"DELETE FROM vm_metric_samples WHERE sampled_at < %s "
"ORDER BY sampled_at LIMIT %s",
(cutoff, _METRIC_PRUNE_BATCH_SIZE),
)
removed = cursor.rowcount
connection.commit()
return max(0, removed)
except pymysql.MySQLError as exc:
connection.rollback()
raise DatabaseError("无法清理过期虚拟机指标") from exc
finally:
connection.close()
def query_metric_samples(self, host_scope: str, vm_id: str, *,
since: int, until: int, interval_seconds: int,
limit: int = 600) -> list[dict[str, Any]]:
vm_id = self._metric_identity(host_scope, vm_id)
if (any(isinstance(value, bool) or not isinstance(value, int)
for value in (since, until, interval_seconds, limit))
or since < 0 or until < since or interval_seconds < 60
or not 1 <= limit <= 600):
raise ValueError("Metric query range is invalid")
columns = ", ".join(f"AVG({name}) AS {name}" for name in _METRIC_COLUMNS)
connection = self._connect()
try:
with connection.cursor() as cursor:
cursor.execute(
"SELECT MAX(sampled_at) AS timestamp, " + columns
+ " FROM vm_metric_samples "
"WHERE host_scope=%s AND vm_uuid=%s AND sampled_at >= %s "
"AND sampled_at <= %s GROUP BY FLOOR(sampled_at / %s) "
"ORDER BY timestamp ASC LIMIT %s",
(host_scope, vm_id, since, until, interval_seconds, limit),
)
rows = cursor.fetchall()
return [
{"timestamp": int(row["timestamp"]),
**{name: float(row[name]) if row[name] is not None else None
for name in _METRIC_COLUMNS}}
for row in rows
]
except pymysql.MySQLError as exc:
raise DatabaseError("无法读取虚拟机指标") from exc
finally:
connection.close()
def _encrypt(self, record_type: str, record_key: str, value: str) -> bytes:
if not isinstance(value, str):
raise TypeError("Database value must be a string")

@ -23,6 +23,7 @@ from typing import Any, Callable, Iterator
import libvirt
from virsh_panel.cloud_init import build_seed_iso, read_seed_network, validate_cloud_network
from virsh_panel.metrics import RawVMStats
LOG = logging.getLogger(__name__)
@ -329,6 +330,35 @@ class VMService:
with self._connect() as conn:
return self._vm_dict(self._domain(conn, uuid))
def metric_counters(self) -> list[RawVMStats]:
"""Fetch running-domain counters in one libvirt API round trip.
getAllDomainStats(stats, flags) returns ``(domain, dict)`` pairs in
libvirt-python. Network rx/tx fields are from the guest's viewpoint.
Missing per-device values are preserved as missing by the sampler.
"""
groups = (libvirt.VIR_DOMAIN_STATS_CPU_TOTAL | libvirt.VIR_DOMAIN_STATS_BALLOON
| libvirt.VIR_DOMAIN_STATS_BLOCK | libvirt.VIR_DOMAIN_STATS_INTERFACE)
running = libvirt.VIR_CONNECT_GET_ALL_DOMAINS_STATS_RUNNING
with self._connect() as conn:
rows: list[RawVMStats] = []
for domain, values in conn.getAllDomainStats(groups, running):
# A guest can stop or be undefined after bulk stats are read.
# Do not lose all other guests' samples for that transient race.
try:
info = domain.info()
generation = domain.ID()
vm_id = domain.UUIDString()
except libvirt.libvirtError:
continue
if info[0] != 1 or generation < 0:
continue
rows.append(RawVMStats(
vm_id=vm_id, generation=generation,
vcpus=info[3], values=values,
))
return rows
def resources(self) -> dict[str, Any]:
with self._connect() as conn:
disk_pool = self._active_pool(conn, self.storage_pool)

@ -0,0 +1,276 @@
"""Periodic libvirt VM measurements and safe counter-to-rate conversion.
The collector runs independently of browser visits. A process restart deliberately
loses counter baselines: the first subsequent rate sample is unknown, not zero.
"""
from __future__ import annotations
import hashlib
import json
import logging
import math
import re
import threading
import time
import uuid
from dataclasses import dataclass
from typing import Any, Callable, Mapping
LOG = logging.getLogger(__name__)
SAMPLE_INTERVAL_SECONDS = 60
RETENTION_SECONDS = 30 * 24 * 60 * 60
MAX_QUERY_POINTS = 600
RANGES = {"1h": 3600, "24h": 24 * 3600, "7d": 7 * 24 * 3600,
"30d": RETENTION_SECONDS}
_COUNTER = re.compile(r"^(block|net)\.(\d+)\.(rd|wr|rx|tx)\.bytes$")
_MEMORY_MAX_AGE_SECONDS = 180
def host_scope(profile: Mapping[str, Any]) -> str:
"""Keep VM history when pool settings change, but isolate a new SSH host key."""
identity = [profile[key] for key in ("host", "port", "fingerprint")]
if (not isinstance(identity[0], str) or not isinstance(identity[1], int)
or not isinstance(identity[2], str)):
raise ValueError("Host identity is invalid")
raw = json.dumps(identity, ensure_ascii=False, separators=(",", ":"))
return hashlib.sha256(raw.encode("utf-8")).hexdigest()
def query_interval_seconds(window_seconds: int) -> int:
"""Epoch-aligned buckets cannot exceed MAX_QUERY_POINTS in any range."""
if window_seconds <= 0:
raise ValueError("Metric range must be positive")
return max(SAMPLE_INTERVAL_SECONDS,
math.ceil(window_seconds / (MAX_QUERY_POINTS - 1)
/ SAMPLE_INTERVAL_SECONDS) * SAMPLE_INTERVAL_SECONDS)
def _nonnegative_int(value: Any) -> int | None:
if isinstance(value, bool) or not isinstance(value, int) or value < 0:
return None
return value
def _device_counters(stats: Mapping[str, Any], group: str,
directions: tuple[str, str]) -> tuple[tuple[tuple[str, str], ...],
dict[str, dict[str, int] | None]]:
count = _nonnegative_int(stats.get(f"{group}.count"))
devices: dict[str, dict[str, int]] = {
str(number): {} for number in range(min(count, 1024))
} if count is not None else {}
for key, value in stats.items():
match = _COUNTER.fullmatch(key)
if not match or match.group(1) != group or match.group(3) not in directions:
continue
devices.setdefault(match.group(2), {})
valid = _nonnegative_int(value)
if valid is not None:
devices[match.group(2)][match.group(3)] = valid
signature = tuple(sorted(
(number, str(stats.get(f"{group}.{number}.name", number)))
for number in devices
))
counters_by_direction: dict[str, dict[str, int] | None] = {}
for direction in directions:
counters = {number: fields[direction] for number, fields in devices.items()
if direction in fields}
# CD-ROM and unsupported devices may never expose byte counters. Use
# only reporting devices, but _rate() resets if that coverage changes.
counters_by_direction[direction] = counters or None
return signature, counters_by_direction
def _memory_mib(stats: Mapping[str, Any], sampled_at: int) -> tuple[float | None,
float | None]:
available = _nonnegative_int(stats.get("balloon.available"))
usable = _nonnegative_int(stats.get("balloon.usable"))
last_update = _nonnegative_int(stats.get("balloon.last-update"))
# The QEMU RSS and balloon.current fields describe host process/allocation,
# not guest-used memory. Only guest balloon statistics can support used MiB.
if (available is None or usable is None or last_update is None
or last_update > sampled_at + SAMPLE_INTERVAL_SECONDS
or sampled_at - last_update > _MEMORY_MAX_AGE_SECONDS
or usable > available):
return None, None
return (available - usable) / 1024, available / 1024
@dataclass(frozen=True)
class RawVMStats:
vm_id: str
generation: int
vcpus: int
values: Mapping[str, Any]
@dataclass(frozen=True)
class _Baseline:
monotonic: float
generation: int
vcpus: int
cpu_time: int | None
disk_signature: tuple[tuple[str, str], ...]
disk: Mapping[str, Mapping[str, int] | None]
network_signature: tuple[tuple[str, str], ...]
network: Mapping[str, Mapping[str, int] | None]
class MetricSampler:
def __init__(self) -> None:
self._previous: dict[tuple[str, str], _Baseline] = {}
@staticmethod
def _rate(current: Mapping[str, int] | None,
previous: Mapping[str, int] | None,
seconds: float, same_devices: bool) -> float | None:
if (not same_devices or current is None or previous is None
or seconds <= 0 or current.keys() != previous.keys()
or any(value < previous[name] for name, value in current.items())):
return None
return sum(value - previous[name] for name, value in current.items()) / seconds
def observe(self, scope: str, raw: RawVMStats, *, timestamp: int,
monotonic: float) -> dict[str, Any]:
vm_id = str(uuid.UUID(raw.vm_id))
values = raw.values
cpu_time = _nonnegative_int(values.get("cpu.time"))
disk_signature, disk = _device_counters(values, "block", ("rd", "wr"))
network_signature, network = _device_counters(values, "net", ("rx", "tx"))
memory_used, memory_total = _memory_mib(values, timestamp)
key = (scope, vm_id)
previous = self._previous.get(key)
seconds = monotonic - previous.monotonic if previous else 0
# A long pause means a graph gap rather than a misleading average.
same_run = bool(previous and previous.generation == raw.generation
and previous.vcpus == raw.vcpus
and 0 < seconds <= SAMPLE_INTERVAL_SECONDS * 3)
cpu_percent = None
if (same_run and cpu_time is not None and previous.cpu_time is not None
and raw.vcpus > 0 and cpu_time >= previous.cpu_time):
fraction = (cpu_time - previous.cpu_time) / (seconds * 1e9 * raw.vcpus)
# libvirt's total CPU time can include QEMU auxiliary threads, so
# a healthy sample can exceed the allotted vCPU time. The chart is
# defined as 0–100% of VM vCPU capacity.
cpu_percent = min(100.0, fraction * 100)
disk_same = same_run and bool(disk_signature) and disk_signature == previous.disk_signature
network_same = (same_run and bool(network_signature)
and network_signature == previous.network_signature)
point = {
"vm_id": vm_id,
"timestamp": timestamp,
"cpu_percent": cpu_percent,
"memory_used_mib": memory_used,
"memory_total_mib": memory_total,
"disk_read_bps": self._rate(disk["rd"], previous.disk["rd"] if previous else None,
seconds, disk_same),
"disk_write_bps": self._rate(disk["wr"], previous.disk["wr"] if previous else None,
seconds, disk_same),
"network_rx_bps": self._rate(network["rx"], previous.network["rx"] if previous else None,
seconds, network_same),
"network_tx_bps": self._rate(network["tx"], previous.network["tx"] if previous else None,
seconds, network_same),
}
self._previous[key] = _Baseline(monotonic, raw.generation, raw.vcpus, cpu_time,
disk_signature, disk, network_signature, network)
return point
def forget_unseen(self, scope: str, seen: set[str]) -> None:
for key in list(self._previous):
if key[0] == scope and key[1] not in seen:
del self._previous[key]
class MetricsCollector:
"""One daemon sampler owned by the single Uvicorn worker's lifespan."""
def __init__(self, store_getter: Callable[[], Any],
profile_getter: Callable[[], Mapping[str, Any] | None],
service_factory: Callable[[Mapping[str, Any]], Any],
*, interval_seconds: int = SAMPLE_INTERVAL_SECONDS) -> None:
self._store_getter = store_getter
self._profile_getter = profile_getter
self._service_factory = service_factory
self._interval_seconds = interval_seconds
self._sampler = MetricSampler()
self._stop = threading.Event()
self._thread: threading.Thread | None = None
self._prune_signal = threading.Event()
self._prune_thread: threading.Thread | None = None
def sample_once(self, *, timestamp: int | None = None,
monotonic: float | None = None) -> None:
store = self._store_getter()
if store is None or not hasattr(store, "prune_metric_samples"):
return
timestamp = int(time.time()) if timestamp is None else timestamp
try:
profile = self._profile_getter()
if profile is None or profile.get("key_locked"):
return
scope = host_scope(profile)
rows = self._service_factory(profile).metric_counters()
monotonic = time.monotonic() if monotonic is None else monotonic
points = [self._sampler.observe(scope, row, timestamp=timestamp,
monotonic=monotonic) for row in rows]
self._sampler.forget_unseen(scope, {str(uuid.UUID(row.vm_id)) for row in rows})
if points:
store.save_metric_samples(scope, points)
finally:
# Signal after sample/save (or a failed host access). The separate
# cleanup thread never delays the next libvirt measurement.
self._prune_signal.set()
def _prune_loop(self) -> None:
while not self._stop.is_set():
self._prune_signal.wait(timeout=self._interval_seconds)
self._prune_signal.clear()
if self._stop.is_set():
break
store = self._store_getter()
if store is None or not hasattr(store, "prune_metric_samples"):
continue
while not self._stop.is_set():
try:
removed = store.prune_metric_samples(
int(time.time()) - RETENTION_SECONDS)
except Exception as exc:
# Never echo database driver errors or connection details.
LOG.warning("VM metric retention cleanup failed: %s",
type(exc).__name__)
break
if not isinstance(removed, int) or removed < 5000:
break
# Sustain catch-up without occupying MariaDB continuously.
if self._stop.wait(0.1):
break
def _run(self) -> None:
while not self._stop.is_set():
started = time.monotonic()
try:
self.sample_once()
except Exception as exc:
# Never print libvirt/DB exception text: it can contain a URI,
# account name, or other connection details.
LOG.warning("VM metric collection failed: %s", type(exc).__name__)
self._stop.wait(max(0, self._interval_seconds - (time.monotonic() - started)))
def start(self) -> None:
if self._thread is not None:
return
self._prune_thread = threading.Thread(target=self._prune_loop, daemon=True,
name="vm-metrics-retention")
self._prune_thread.start()
self._thread = threading.Thread(target=self._run, daemon=True,
name="vm-metrics-collector")
self._thread.start()
def close(self) -> None:
self._stop.set()
self._prune_signal.set()
deadline = time.monotonic() + 15
for worker in (self._thread, self._prune_thread):
if worker is not None:
worker.join(timeout=max(0, deadline - time.monotonic()))
Loading…
Cancel
Save