fix: harden VPC lifecycle and guest readiness

venix
admin 1 month ago
parent 6dc0d21c32
commit 7f34725d7e

@ -59,8 +59,9 @@ systemd-networkd 的 `.network/.netdev` 文件以及 ifupdown 的 interfaces 文
云初始化使用 `public0`/`private0` 作为 Netplan 配置键和产品角色;libvirt XML
使用符合设备契约的 `ua-public0`/`ua-private0` 用户 alias,两者不得混用。公网
静态 IPv4 保留唯一默认路由;私网只接受 RFC1918 `/32`,关闭 DHCP/RA 且不带
gateway、DNS 或默认路由。旧的单网卡域可按兼容规则视为公网;历史
静态 IPv4 保留唯一默认路由;`private0` 地址只接受 RFC1918 `/32`,另且仅下发一条
当前 VPC CIDR 的 scope-link 路由,关闭 DHCP/RA 且不带 gateway、DNS 或默认路由。
旧的单网卡域可按兼容规则视为公网;历史
`netN`/`ua-net-N` 公网 alias 只有在其他网卡全部明确为私网时才可用于运行态
迁移,其他无角色多网卡域一律停止安全策略操作。面板的
`clean-traffic`/`vc-fw-*` nwfilter 只能绑定 `public0` 产品角色(libvirt alias
@ -161,8 +162,9 @@ PHP 只有在所有参与宿主机的观测 revision/digest 完全匹配时才
如果该服务未安装,API 仍会安全地保留 `queued` 状态,显式
`POST /api/v1/vpcs/{vpc_key}/reconcile` 可用于人工核对,但不能把 queued 当作
已经下发。引擎只证明 Fabric 已就绪、而本阶段没有私网数据面回读时,操作会进入
`waiting_observed`、根同步状态为 `waiting`,专用 worker 不会每 30 秒重复请求;后续
数据面上线后必须由显式 reconcile 重新推进,不能把 waiting 当成 active。
`waiting_observed`、根同步状态为 `waiting`;专用 worker 会周期性执行幂等
reconcile,也可由显式 reconcile 立即推进。重试必须保持 seed、XML 和宿主数据面
幂等,不能把 waiting 当成 active。
VPC 私网数据面现在由引擎的独立 reconciler 执行。每台参与宿主机使用带 owner
标记的独立私网 bridge/tap,VM XML 使用不绑定 nwfilter 的 `ua-private0` alias
@ -173,7 +175,49 @@ VPC 私网数据面现在由引擎的独立 reconciler 执行。每台参与宿
bridge 和 ARP nft 表:ACL 按 priority 处理,bridge 入口校验 MAC/IP/ARP,IPv6、
广播、多播和未知目的地址默认隔离;代理 ARP 仅允许面板登记的精确 `/32`。
云初始化 seed 同步写入 `public0`/`private0` 多网卡配置和 link-scope `/32` 路由。
云初始化 seed 同步写入 `public0`/`private0` 多网卡配置和唯一的 scope-link VPC CIDR
路由。NoCloud 只在 boot 事件重新消费该网络配置:运行中的 VM 首次接入或其 MAC、私网
IP、VPC CIDR
变化时,引擎不会改写 seed、持久 XML 或 live 网卡,而是保持旧配置并返回待重启;VM
完全停机后,下一次 reconcile 才原子写入 seed 和持久 XML,随后一次完整 start 才消费
新网络配置。独立的 guest-network generation 只覆盖 MAC、私网 `/32` 地址和 VPC 前缀;
成员、自定义 route、ACL 等宿主侧变化不改变 generation,避免一次成员变更要求所有运行中
成员重启。只有完整 stop/reconcile/start 后 live domain metadata 带有目标
generation,成员才能通过 active/usable 门禁;
缺失或旧 generation 会保持 `waiting_observed` 并返回
`vpc_private_guest_restart_required`。相同目标的重试不再轮换 NoCloud `instance-id`,
避免重放用户、密码和 sshd 等 per-instance 模块;缺少当前 `boot-v1` 网络更新策略的
旧 seed 以 `vpc_seed_network_policy_migration_required` 失败关闭,必须重装或执行明确的
离线迁移,不能通过轮换 instance-id 静默升级。
live generation 一致仍不等于来宾网络可用。每个运行成员还必须通过 QEMU Guest Agent
回读:`guest-info` 明确启用 `guest-exec`/`guest-exec-status`;链路探测固定使用
`/usr/bin/timeout` 与 `/usr/bin/head` 读取 sysfs,路由探测只允许固定的
`/bin/sh -c 'exec /usr/sbin/ip -j -4 route show table main dev "$1" | /usr/bin/head -c 131073'`
脚本,唯一可变参数是已校验的 guest 接口名。回读必须确认唯一接口的 MAC、精确 IPv4
`/32`、admin/carrier/oper link-up,以及 main table 中恰好一条当前 VPC CIDR 的
scope-link 路由,且没有 gateway、nexthop、`linkdown`、`dead` 或 `unresolved`。
agent/channel/service、命令 allowlist 或固定二进制缺失时分别返回
unsupported/unavailable/uncertain/deadline evidence,成员保持不可用。该 QGA 证据是
租户可控来宾的 readiness 信号,不是 VPC 隔离的安全边界;隔离仍由宿主 bridge、route、
WireGuard 和 nft 回读证明。显式 network suspend 是例外:精确 persistent/live XML 的
`link state='down'` 与宿主配置可以证明隔离已生效,但该成员绝不能因此被标为 usable。
宿主配置提交和成员运行态分开收敛:当前 revision/digest 的 bridge、路由、ACL、seed
与 inactive/live XML 配置证据可完成控制面操作和 tombstone 回收;每个成员的
active/usable 则只由自己的 current-generation QGA 证据推进。一台关机、待重启或
agent 不可用的 VM 不得抹掉同 VPC 其他已验证成员的可用状态,但 VPC 聚合健康仍可显示
degraded,周期 reconcile 会继续推进尚未就绪的成员。
历史 seed 中逐成员 `/32` route 只用于旧 generation;运行中成员保持
`vpc_private_guest_restart_required`,完全停机后的 reconcile 才迁移为单条 VPC CIDR。
4096 成员上限保护记录和回读大小,但普通周期 reconcile 在进入并发 QGA 阶段前仍会按宿主
顺序执行每台 VM 的 seed/libvirt 核对;生命周期目标会优先,整宿主满规模延迟仍需真实压测,
不能把单次 HTTP 预算当作全量收敛时限。
本契约使用 `evidence_version=2`。滚动发布顺序必须是:先应用数据库 schema,再部署能
持久化/校验 v2 的 PHP 与 worker,最后部署 engine;混合版本及历史缺失版本证据一律
fail closed,不能回退接受仅有 XML 的旧 succeeded receipt。
宿主机 apply 在 Fabric 锁和 VPC 锁内执行,持久文件、NetworkManager/ifupdown
配置、路由、策略规则、WireGuard、nft、代理 ARP 和 sysctl 都有实际回读;apply、
实例接入和停机 libvirt 路径中途失败会按 ownership marker 恢复,响应不确定时保持

@ -1,5 +1,32 @@
# 本版验证记录
## 2026-10-08:VPC 来宾网络启动代次门禁
- 私网 MAC、`/32` 地址和唯一的 VPC CIDR scope-link 路由共同生成独立
guest-network generation;成员、自定义 route、ACL/Fabric 等纯宿主变化不改变该
generation。
- 运行中 VM 的新来宾网络不改写 NoCloud seed、inactive XML 或 live device;完全停机
后由 reconcile 写入 seed 与持久 XML,再启动消费。在完成 stop/reconcile/start 且
live metadata 匹配之前,宿主保持
`applying/pending_restart`,PHP 拒绝将缺失 `restart_required=false` 的证据视为
active/usable。
- 相同私网目标重试返回原 seed 字节,不重复轮换 NoCloud `instance-id`;写入
后会在 libvirt define 前重新下载 seed 并核对 MAC/IP/routes;缺少安全 boot 网络策略
的旧 seed 明确要求迁移,不通过新 instance-id 重放 credentials。
- current generation 还要通过 QGA 地址、link 与 route 的精确回读;保留地址/路由但
`NO-CARRIER`/`linkdown` 的接口不能通过。显式 link-down suspension 单独由 XML/config
证明隔离,恢复 link-up 后才重新要求 QGA usable 证据。
- `evidence_version=2` 使历史 XML-only succeeded 证据失效;宿主 config proof、每成员
guest readiness 与 VPC 聚合健康分别收敛,tombstone 不再被无关成员的 QGA 状态阻塞。
- 本地验证:Python compileall 通过;cloud-init、VPC 数据面、libvirt、协调器及删除回执
的定向回归 **394 passed**。完整 engine 回归为 **1071 passed、10 skipped、8
failed**;8 个失败已在本轮修改前的 `6dc0d21` 干净快照复现,属于既有云模板 spec、
重装错误 envelope、Python 3.9 upload timeout/event-loop 与退役宿主盘点式公网池 fixture。
- 隔离 PHP 8.5.10/WASM 下 5 个 VPC 契约/收敛测试全部通过,10 个修改 PHP 文件语法、
`vpc.js` Node 语法和 `git diff --check` 通过。当前宿主没有原生 PHP CLI;真实
libvirt/cloud-init 验收仍需在测试 VM 内核对地址、唯一 VPC CIDR scope-link 路由与跨
VM 连通性。宿主策略路由、代理 ARP 和 WireGuard AllowedIPs 仍必须保持精确 `/32`。
## 2026-10-08:根 VPC 可恢复删除状态机
- 租户 API 新增根 VPC 删除:精确名称确认后,PHP 在事务中把根和子资源固定为

@ -21,20 +21,41 @@
对一个跨两台宿主机的 VPC,至少放置两个成员和一个 ACL,逐台检查:
镜像前置契约:安装并启用 `qemu-guest-agent`,domain 存在 agent channel,
`guest-info` 中 `guest-exec` 与 `guest-exec-status` 均为 enabled;镜像必须提供
`/usr/bin/timeout`、`/usr/bin/head`、`/bin/sh` 和 `/usr/sbin/ip`。不要假定发行版默认允许 `guest-exec`:Alma/RHEL
系列镜像或加固策略可能在 QGA blocklist 中禁用它,必须逐镜像实际核对。任何一项缺失
都应得到 `vpc_private_guest_probe_unsupported`(或 unavailable),成员不得 usable。
0. 运行中 VM 首次加入,或其 MAC、私网 IP、VPC CIDR 变化时,引擎不得改写 NoCloud seed、
inactive XML,也不得对 live domain 执行私网网卡热插/热更新;旧配置必须原样保留。
VM 完全停机后由下一次 reconcile 写入 seed 与 inactive XML,再执行完整 start。
在完成该 stop/reconcile/start 流程前,宿主状态必须是 `applying`、回读为
`pending_restart`、错误码为 `vpc_private_guest_restart_required`,且 VPC
`usable=false`。普通 guest soft reboot 不能替代 libvirt stop/start。
1. 生成的 bridge 名称、owner/state/proxy 文件均为 root-only,bridge/tap 没有
地址、网关或 DNS;`private0` 产品角色对应的 libvirt alias 是
`ua-private0`,其 bridge、稳定 MAC、`/32` 和持久/实时 XML 都一致,私网接口
不绑定公网 `clean-traffic` nwfilter。
2. 每个成员 `/32` 和每条自定义 `/32` route 都带 panel route protocol、正确
2. QGA 回读必须找到唯一的 guest 接口名与 MAC,精确 IPv4 地址只有目标 `/32`;固定
sysfs 探测必须同时确认接口 flags 含 `IFF_UP`、`operstate=up`、`carrier=1`。
接口 down 后即使地址和路由仍保留也必须返回
`vpc_private_guest_link_mismatch`,不能通过 usable 门禁。显式 network suspend
则以 persistent/live XML 都为 down 证明隔离,并保持成员 non-usable。
3. 宿主机上的每个成员 `/32` 和每条自定义 `/32` route 都带 panel route protocol、正确
device/table、`via` 和 metric;不存在 VPC CIDR、默认路由或宽前缀的替代项。
3. `wg show wg-fabric0 allowed-ips` 的每个 peer 与完整期望集合一致:overlay `/32`
来宾 main table 的目标 route 必须只有一条 exact scope-link VPC CIDR,不能含 gateway、
nexthop、`linkdown`、`dead` 或 `unresolved` flags;成员或自定义 route 变化不得改变
兄弟 VM 的 guest generation。历史逐成员 `/32` seed 只能在完全停机后迁移。
4. `wg show wg-fabric0 allowed-ips` 的每个 peer 与完整期望集合一致:overlay `/32`
加精确 tenant/service `/32`,没有 `0.0.0.0/0`、宽前缀或其他 VPC 的未授权地址。
4. nft inet/bridge/arp 三张表的 owner marker、chain priority、ACL comments、
5. nft inet/bridge/arp 三张表的 owner marker、chain priority、ACL comments、
anti-spoof、IPv6/广播/未知目的 drop 和精确 proxy-ARP set 均能被 observe 回读。
bridge 上通用 `proxy_arp=0`、同桥下一跳所需的 `proxy_arp_pvlan=1`、
`rp_filter=2`,代理项集合与 route destination 集合
完全相同;其他地址的 ARP reply 必须被过滤。
5. 从同机和跨机成员分别验证允许的 TCP/UDP/ICMP 流量、ACL deny、伪造源 MAC/IP、
6. 从同机和跨机成员分别验证允许的 TCP/UDP/ICMP 流量、ACL deny、伪造源 MAC/IP、
IPv6、广播/多播和访问宿主机自身地址。验证结果同时记录 guest route、宿主机
route、WireGuard handshake 和 nft counter,不能只看页面的 `active`。
@ -69,3 +90,25 @@ Debian 13:执行 `ifquery <private-bridge>`、`ifup --force <private-bridge>`
跨宿主机最后还要单独验证 underlay 公网防火墙、NAT、MTU、SELinux 和 UDP 监听。
Fabric/VPC 的本地 nft 表不等于 underlay 已放行。
## 发布和规模边界
- 先应用包含 VPC evidence version 字段的数据库 schema,再发布 PHP/worker,最后发布
engine。历史或混合版本 receipt 没有 `evidence_version=2` 时必须重新 reconcile,不能
继承旧 active/usable。
- 用 1、32、256 个成员和至少两台宿主分别压测正常、QGA command disabled、QGA
timeout 三种场景。引擎的协作式探测预算到期后必须停止提交新 QGA/SSH 探测,并取消
未开始任务;它不是对已经开始的 libvirt 打开、XML、seed 或 define 操作的强制 HTTP
截止。PHP HTTP timeout 必须大于该探测预算,operation lease 还要留出明确余量;超时只返回
`vpc_private_guest_probe_deadline_exceeded` 并继续幂等 reconcile,不能 lease steal、
重复派发或遗留不断增长的 guest process/线程/SSH 连接。
- 宿主 libvirt 必须为 upstream 4.2.0+,或由发行版明确回补 CVE-2018-1064 的 QGA 单响应
10 MiB 读取上限;Python 只解析
不超过 192 KiB 的响应,且默认最多并行 4 个来宾探测。`agentSetResponseTimeout` 恢复
默认值失败时必须返回 uncertain,不能在 domain-wide timeout 仍被修改时发布 confirmed。
- 另用单宿主 4096 成员验证周期 reconcile:生命周期目标必须优先;进入 QGA 并发探测前
的逐 VM seed/libvirt 核对仍是已知串行边界,应记录首个目标完成时间和全量收敛时间,
不能仅以请求超时作为容量结论。
- 同一 VPC 放置 A、B 两个成员:A QGA exact、B stopped/unsupported 时,配置 operation
应完成,A 的实例私网可用、B 单独显示未就绪,聚合健康可 degraded;随后只恢复 B 的
agent/启动状态,不做新 mutation,周期 reconcile 必须最终将 B 和聚合健康推进为 active。

@ -18,6 +18,7 @@ from virsh_panel.cloud_init import (
replace_seed_settings,
validate_cloud_network, validate_guest_hostname, validate_private_network,
_parse_network_config,
SeedNetworkPolicyMigrationRequired,
)
@ -251,7 +252,10 @@ def test_seed_iso_can_add_private_role_without_public_route_or_dns():
seed = build_seed_iso(
str(uuid.uuid4()), "multi-nic-vm", "52:54:00:12:34:56", validated,
private_mac="52:54:00:12:34:57",
private_network={"private_ip_cidr": "10.24.3.7/32"},
private_network={
"private_ip_cidr": "10.24.3.7/32",
"routes": ["10.24.0.0/16"],
},
)
network = _network_data(seed)
assert set(network["ethernets"]) == {"public0", "private0"}
@ -261,6 +265,7 @@ def test_seed_iso_can_add_private_role_without_public_route_or_dns():
"dhcp6": False,
"accept-ra": False,
"addresses": ["10.24.3.7/32"],
"routes": [{"to": "10.24.0.0/16", "scope": "link"}],
}
assert read_seed_network(seed) == {
"public_ip_cidr": "8.8.8.18/28",
@ -269,14 +274,14 @@ def test_seed_iso_can_add_private_role_without_public_route_or_dns():
"mac": "52:54:00:12:34:56",
"private_ip_cidr": "10.24.3.7/32",
"private_mac": "52:54:00:12:34:57",
"private_routes": [],
"private_routes": ["10.24.0.0/16"],
"roles": ["public0", "private0"],
}
@pytest.mark.skipif(importlib.util.find_spec("pycdlib") is None,
reason="pycdlib is not installed")
def test_private_seed_routes_are_exact_and_can_be_reconciled_or_removed():
def test_private_seed_migrates_legacy_routes_to_stable_vpc_prefix():
validated = validate_cloud_network(
"8.8.8.18/28", "8.8.8.17", ["1.1.1.1"], PUBLIC_KEY,
)
@ -285,36 +290,154 @@ def test_private_seed_routes_are_exact_and_can_be_reconciled_or_removed():
private_mac="52:54:00:12:34:57",
private_network={
"private_ip_cidr": "10.24.3.7/32",
"routes": ["10.24.3.8/32"],
"routes": ["10.24.0.0/16"],
},
)
assert read_seed_network(seed)["private_routes"] == ["10.24.3.8/32"]
legacy_network = _network_data(seed)
legacy_network["ethernets"]["private0"]["routes"] = [
{"to": "10.24.3.8/32", "scope": "link"},
{"to": "10.24.3.10/32", "scope": "link"},
]
seed = _build_seed_iso((
("/USERDATA.;1", "user-data", "/user-data",
b"#cloud-config\n" + json.dumps(
_user_data(seed), ensure_ascii=True).encode("ascii") + b"\n"),
("/METADATA.;1", "meta-data", "/meta-data",
json.dumps(_meta_data(seed), ensure_ascii=True).encode("ascii") + b"\n"),
("/NETWORK.;1", "network-config", "/network-config",
json.dumps(legacy_network, ensure_ascii=True).encode("ascii") + b"\n"),
))
assert read_seed_network(seed)["private_routes"] == [
"10.24.3.10/32", "10.24.3.8/32",
]
changed = replace_seed_private_network(
seed,
private_mac="52:54:00:12:34:58",
private_ip_cidr="10.24.3.9/32",
private_routes=["10.24.3.10/32", "10.24.3.11/32"],
private_routes=["10.24.0.0/16"],
)
changed_network = read_seed_network(changed)
assert _meta_data(changed)["instance-id"] == _meta_data(seed)["instance-id"]
assert _user_data(changed) == _user_data(seed)
assert changed_network["private_ip_cidr"] == "10.24.3.9/32"
assert changed_network["private_mac"] == "52:54:00:12:34:58"
assert changed_network["private_routes"] == ["10.24.3.10/32", "10.24.3.11/32"]
assert changed_network["private_routes"] == ["10.24.0.0/16"]
unchanged = replace_seed_private_network(
changed,
private_mac="52:54:00:12:34:58",
private_ip_cidr="10.24.3.9/32",
private_routes=["10.24.0.0/16"],
)
assert unchanged == changed
removed = replace_seed_private_network(
changed, private_mac=None, private_ip_cidr=None,
)
removed_network = read_seed_network(removed)
assert _meta_data(removed)["instance-id"] == _meta_data(seed)["instance-id"]
assert _user_data(removed) == _user_data(seed)
assert "private_ip_cidr" not in removed_network
assert "private_routes" not in removed_network
for invalid_routes in (
None,
[],
["10.24.0.0/16", "10.24.3.10/32"],
["10.24.0.0/16", "10.24.0.0/16"],
["10.25.0.0/16"],
["10.24.3.0/25"],
):
with pytest.raises(ValueError):
replace_seed_private_network(
seed,
private_mac="52:54:00:12:34:58",
private_ip_cidr="10.24.3.9/32",
private_routes=invalid_routes,
)
@pytest.mark.skipif(importlib.util.find_spec("pycdlib") is None,
reason="pycdlib is not installed")
def test_private_seed_rejects_public_connected_prefix_overlap():
# The public address itself is global, but this deliberately broad
# connected prefix includes the RFC1918 VPC. private0 must not silently
# steal that connected public route.
validated = validate_cloud_network(
"8.8.8.18/5", "8.8.8.17", ["1.1.1.1"], PUBLIC_KEY,
)
seed = build_seed_iso(
str(uuid.uuid4()), "overlap-vm", "52:54:00:12:34:56", validated,
)
with pytest.raises(ValueError):
replace_seed_private_network(
seed,
private_mac="52:54:00:12:34:58",
private_ip_cidr="10.24.3.9/32",
private_routes=["10.24.3.0/24"],
private_routes=["10.24.0.0/16"],
)
def test_private_route_parser_rejects_mixed_legacy_and_stable_routes():
config = {
"version": 2,
"ethernets": {
"public0": {
"match": {"macaddress": "52:54:00:12:34:56"},
"dhcp4": False, "dhcp6": False,
"addresses": ["8.8.8.18/28"],
"routes": [{"to": "0.0.0.0/0", "via": "8.8.8.17"}],
"nameservers": {"addresses": ["1.1.1.1"]},
},
"private0": {
"match": {"macaddress": "52:54:00:12:34:57"},
"dhcp4": False, "dhcp6": False, "accept-ra": False,
"addresses": ["10.24.3.7/32"],
"routes": [
{"to": "10.24.0.0/16", "scope": "link"},
{"to": "10.24.3.8/32", "scope": "link"},
],
},
},
}
with pytest.raises(ValueError, match="唯一 VPC"):
_parse_network_config(config)
@pytest.mark.skipif(importlib.util.find_spec("pycdlib") is None,
reason="pycdlib is not installed")
def test_private_seed_rejects_legacy_policy_without_rotating_instance_id():
validated = validate_cloud_network(
"8.8.8.18/28", "8.8.8.17", ["1.1.1.1"], PUBLIC_KEY,
)
seed = build_seed_iso(
str(uuid.uuid4()), "legacy-private-vm", "52:54:00:12:34:56",
validated,
)
legacy_user_data = _user_data(seed)
legacy_user_data.pop("updates")
legacy_metadata = _meta_data(seed)
legacy_metadata.pop("virt-cloud-network-policy")
legacy_seed = _build_seed_iso((
("/USERDATA.;1", "user-data", "/user-data",
b"#cloud-config\n" + json.dumps(
legacy_user_data, ensure_ascii=True).encode("ascii") + b"\n"),
("/METADATA.;1", "meta-data", "/meta-data",
json.dumps(legacy_metadata, ensure_ascii=True).encode("ascii")
+ b"\n"),
("/NETWORK.;1", "network-config", "/network-config",
json.dumps(_network_data(seed), ensure_ascii=True).encode("ascii")
+ b"\n"),
))
with pytest.raises(SeedNetworkPolicyMigrationRequired):
replace_seed_private_network(
legacy_seed,
private_mac="52:54:00:12:34:57",
private_ip_cidr="10.24.3.7/32",
)
assert _meta_data(legacy_seed)["instance-id"] == (
legacy_metadata["instance-id"])
@pytest.mark.skipif(importlib.util.find_spec("pycdlib") is None,

@ -412,6 +412,7 @@ def test_vpc_root_delete_endpoint_returns_exact_durable_receipt(boundary):
assert receipt == {
"vpc_id": "vpc-empty-delete",
"operation_id": "vpc-empty-delete-op-0001",
"evidence_version": 2,
"desired_revision": 3,
"desired_digest": target["desired_digest"],
"state": "deleted",

File diff suppressed because it is too large Load Diff

@ -2,8 +2,10 @@
from __future__ import annotations
import asyncio
from concurrent.futures import ThreadPoolExecutor
import json
import threading
import time
from contextlib import asynccontextmanager
from pathlib import Path
from types import SimpleNamespace
@ -13,6 +15,7 @@ import pytest
from virsh_panel.fabric import FabricError, FabricNode, FabricPlan
from virsh_panel.vpc import (
VPC_DELETE_RECEIPT_KIND,
VPC_GUEST_EXEC_KIND,
VPC_SCHEMA_VERSION,
VpcCoordinator,
VpcError,
@ -304,6 +307,77 @@ def test_coordinator_persists_desired_children_and_blocks_without_fabric():
assert set(store.list("vpc_host")) == {"vpc-hk-1.host-a"}
def test_guest_exec_ledger_is_immediate_bounded_private_and_restart_safe():
store = MemoryStore()
plan = parse_vpc_plan(payload())
member = plan.members[0]
coordinator = VpcCoordinator(store, FakeFabric())
state, persist, clear = coordinator._guest_exec_context(plan, member)
assert state == {}
core = {
"version": 1,
"phase": "pending",
"domain_uuid": member.instance_uuid,
"domain_runtime_id": 7,
"qemu_runtime_token": "a" * 32,
"guest_network_generation": "generation-1",
"probe_kind": "link",
"pid": 41,
}
persist(core)
key = f"{plan.vpc_id}.{member.instance_uuid}"
encoded = store.get(VPC_GUEST_EXEC_KIND, key)
assert encoded is not None
assert len(encoded.encode("utf-8")) <= 1024
assert json.loads(encoded)["pid"] == 41
restarted = VpcCoordinator(store, FakeFabric())
loaded, _persist_again, clear_again = restarted._guest_exec_context(
plan, member)
assert loaded == core
asyncio.run(coordinator.apply(payload()))
public = json.dumps(coordinator.get_status(plan.vpc_id), sort_keys=True)
assert '"pid"' not in public
assert "qemu_runtime_token" not in public
assert "a" * 32 not in public
clear_again()
assert store.get(VPC_GUEST_EXEC_KIND, key) is None
persist(core)
coordinator._purge_deleted_vpc(plan.vpc_id)
assert store.get(VPC_GUEST_EXEC_KIND, key) is None
def test_guest_exec_ledger_rejects_oversize_or_unknown_fields():
store = MemoryStore()
plan = parse_vpc_plan(payload())
member = plan.members[0]
key = f"{plan.vpc_id}.{member.instance_uuid}"
store.set(VPC_GUEST_EXEC_KIND, key, "x" * 1025)
with pytest.raises(VpcError, match="账本损坏"):
VpcCoordinator(store, FakeFabric())._guest_exec_context(plan, member)
bad = {
"vpc_id": plan.vpc_id,
"host_id": member.host_id,
"member_id": member.member_id,
"instance_uuid": member.instance_uuid,
"version": 1,
"phase": "launch_intent",
"domain_uuid": member.instance_uuid,
"domain_runtime_id": 7,
"qemu_runtime_token": "a" * 32,
"guest_network_generation": "generation-1",
"probe_kind": "link",
"unexpected": True,
}
store.set(VPC_GUEST_EXEC_KIND, key, json.dumps(bad))
with pytest.raises(VpcError, match="账本损坏"):
VpcCoordinator(store, FakeFabric())._guest_exec_context(plan, member)
def test_observed_projection_keeps_fabric_gate_for_data_plane_deferral():
store = MemoryStore()
result = asyncio.run(VpcCoordinator(store, FakeFabric()).apply(payload()))
@ -942,11 +1016,32 @@ def test_instance_reconcile_returns_host_verified_private_state_and_preserves_li
state="active", observed_revision=plan.desired_revision,
observed_digest=plan.desired_digest, transport_verified=True,
private_dataplane_verified=True,
config_verified=True,
config_observed_revision=plan.desired_revision,
config_observed_digest=plan.desired_digest,
member_nics=[{
"member_id": "member-a", "instance_uuid": INSTANCE_A,
"interface": "private0", "private_ip_cidr": "10.20.0.10/32",
"private_mac": derive_private_mac("vpc-hk-1", "member-a"),
"xml_verified": True,
"config_verified": True,
"seed_verified": True,
"restart_required": False,
"domain_running": True,
"guest_network_verified": requested != "down",
"guest_network_readback": (
"confirmed" if requested != "down" else "mismatch"),
"guest_reason_code": (
None if requested != "down"
else "vpc_private_guest_link_mismatch"),
"guest_private_mac": (derive_private_mac(
"vpc-hk-1", "member-a")
if requested != "down" else None),
"guest_private_ip_cidr": (
"10.20.0.10/32" if requested != "down" else None),
"guest_link_state": (
"up" if requested != "down" else None),
"guest_routes": [],
"link_state": "down" if requested == "down" else "up",
}],
)
@ -970,6 +1065,59 @@ def test_instance_reconcile_returns_host_verified_private_state_and_preserves_li
assert state["link_state"] == "up"
def test_instance_reconcile_invalidates_stale_proof_when_fabric_is_inactive():
store = MemoryStore()
coordinator = VpcCoordinator(
store, FakeFabric(hosts=("host-a",)))
asyncio.run(coordinator.apply(payload()))
async def confirm(plan, record, gate, previous_plan=None,
private_link_states=None):
record["hosts"][0].update(
state="active", observed_revision=plan.desired_revision,
observed_digest=plan.desired_digest, transport_verified=True,
private_dataplane_verified=True, config_verified=True,
config_observed_revision=plan.desired_revision,
config_observed_digest=plan.desired_digest,
member_nics=[{
"member_id": "member-a", "instance_uuid": INSTANCE_A,
"interface": "private0",
"private_ip_cidr": "10.20.0.10/32",
"private_mac": derive_private_mac(
"vpc-hk-1", "member-a"),
"xml_verified": True, "config_verified": True,
"seed_verified": True, "restart_required": False,
"domain_running": True, "guest_network_verified": True,
"guest_network_readback": "confirmed",
"guest_reason_code": None,
"guest_private_mac": derive_private_mac(
"vpc-hk-1", "member-a"),
"guest_private_ip_cidr": "10.20.0.10/32",
"guest_link_state": "up",
"guest_routes": [], "link_state": "up",
}],
)
coordinator._reconcile_dataplane = confirm
active = asyncio.run(coordinator.reconcile_instance(
INSTANCE_A, host_id="host-a",
expected={"vpc_id": "vpc-hk-1", "member_id": "member-a"},
))
assert active["readback"] == "confirmed"
coordinator.fabric = FakeFabric(
state="degraded", gate=False, hosts=("host-a",))
coordinator._reconcile_dataplane = (
VpcCoordinator._reconcile_dataplane.__get__(
coordinator, VpcCoordinator))
stale = asyncio.run(coordinator.reconcile_instance(
INSTANCE_A, host_id="host-a",
expected={"vpc_id": "vpc-hk-1", "member_id": "member-a"},
))
assert stale["state"] == "uncertain"
assert stale["readback"] == "missing"
def test_removed_member_reconcile_uses_durable_retiring_tombstone():
store = MemoryStore()
coordinator = VpcCoordinator(store, FakeFabric(hosts=("host-a",)))
@ -983,13 +1131,7 @@ def test_removed_member_reconcile_uses_durable_retiring_tombstone():
async def fake_reconcile(plan, record, gate, previous_plan=None,
private_link_states=None):
host = record["hosts"][0]
host.update(
state="active", observed_revision=plan.desired_revision,
observed_digest=plan.desired_digest, transport_verified=True,
private_dataplane_verified=True, member_nics=[],
retiring_members=[],
)
confirm_delete_hosts(plan, record)
coordinator._reconcile_dataplane = fake_reconcile
state = asyncio.run(coordinator.reconcile_instance(
@ -1036,10 +1178,23 @@ def test_instance_reconcile_rejects_host_mac_drift():
state="active", observed_revision=plan.desired_revision,
observed_digest=plan.desired_digest, transport_verified=True,
private_dataplane_verified=True,
config_verified=True,
config_observed_revision=plan.desired_revision,
config_observed_digest=plan.desired_digest,
member_nics=[{
"member_id": "member-a", "instance_uuid": INSTANCE_A,
"interface": "private0", "private_ip_cidr": "10.20.0.10/32",
"private_mac": "02:00:00:00:00:01", "xml_verified": True,
"config_verified": True, "seed_verified": True,
"restart_required": False,
"domain_running": True,
"guest_network_verified": True,
"guest_network_readback": "confirmed",
"guest_reason_code": None,
"guest_private_mac": "02:00:00:00:00:01",
"guest_private_ip_cidr": "10.20.0.10/32",
"guest_link_state": "up",
"guest_routes": [],
"link_state": "up",
}],
)
@ -1086,13 +1241,7 @@ def test_removed_member_reconcile_locks_retiring_vm_identity():
async def fake_unlocked(plan, record, gate, previous_plan=None,
private_link_states=None):
host = record["hosts"][0]
host.update(
state="active", observed_revision=plan.desired_revision,
observed_digest=plan.desired_digest, transport_verified=True,
private_dataplane_verified=True, member_nics=[],
retiring_members=[],
)
confirm_delete_hosts(plan, record)
coordinator._reconcile_dataplane_unlocked = fake_unlocked
state = asyncio.run(coordinator.reconcile_instance(
@ -1221,6 +1370,7 @@ def test_root_delete_accepts_revision_jump_and_superseded_unknown_host():
assert result == {
"vpc_id": "vpc-hk-1",
"operation_id": "vpc-delete-revision-jump-0001",
"evidence_version": 2,
"desired_revision": 4,
"desired_digest": target["desired_digest"],
"state": "deleted",
@ -1500,6 +1650,643 @@ def test_delete_pending_projection_does_not_probe_confirmed_offline_host():
record["hosts"][0], plan.desired_revision, plan.desired_digest)
def test_member_tombstone_closes_from_config_proof_when_sibling_not_ready():
target = payload(
desired_revision=2, operation_id="vpc-op-config-proof-0002",
hosts=[{"host_id": "host-a", "members": [{
"member_id": "member-b", "instance_uuid": INSTANCE_B,
"private_ip_cidr": "10.20.0.11/32", "subnet_id": "subnet-a",
}]}],
)
plan = parse_vpc_plan(target)
fabric_plan = FabricPlan(
"fabric-hk", "hk", "100.64.0.0/24", 7, 51820,
(FabricNode(
"host-a", "100.64.0.1", "192.0.2.1:51820", 51820),),
"f" * 64,
)
class RuntimeFabric(FakeFabric):
@staticmethod
def _plan_from_record(_record):
return fabric_plan
async def _components(self, _plan):
return {
"host-a": (
SimpleNamespace(manager=object()),
{"host_id": "host-a"},
),
}
coordinator = VpcCoordinator(
MemoryStore(), RuntimeFabric(hosts=("host-a",)),
registry=object(),
)
coordinator._mark_lifecycle_conflicts = lambda *args, **kwargs: False
coordinator._validate_dataplane_resources = lambda _plan: None
coordinator._fabric_record = lambda _plan: {
"public_keys": {"host-a": "a" * 44},
}
coordinator._fabric_peers_for_plan = lambda _plan, _record: []
coordinator._run_host_dataplane = lambda *args, **kwargs: {
"state": "applying", "transport_verified": True,
"private_dataplane_verified": False,
"config_verified": True,
"config_observed_revision": plan.desired_revision,
"config_observed_digest": plan.desired_digest,
"observed_revision": 0, "observed_digest": None,
"member_nics": [{
"member_id": "member-b", "instance_uuid": INSTANCE_B,
"config_verified": True, "xml_verified": True,
"domain_running": False, "guest_network_verified": False,
"restart_required": True,
}],
"reason_code": "vpc_private_guest_restart_required",
}
record = {
"vpc_id": plan.vpc_id,
"desired_revision": plan.desired_revision,
"desired_digest": plan.desired_digest,
"hosts": [{
"host_id": "host-a", "member_ids": ["member-b"],
"retiring_members": [retiring_member_a()],
"desired_revision": plan.desired_revision,
"desired_digest": plan.desired_digest,
}],
}
asyncio.run(coordinator._reconcile_dataplane_unlocked(
plan, record, {"active": True, "available": True},
previous_plan=plan,
))
assert record["hosts"][0]["state"] == "applying"
assert record["hosts"][0]["retiring_members"] == []
def test_reconcile_skips_exact_host_so_later_host_gets_next_budget():
target = payload(
hosts=[
{"host_id": "host-a", "members": [{
"member_id": "member-a", "instance_uuid": INSTANCE_A,
"private_ip_cidr": "10.20.0.10/32",
"subnet_id": "subnet-a",
}]},
{"host_id": "host-b", "members": [{
"member_id": "member-b", "instance_uuid": INSTANCE_B,
"private_ip_cidr": "10.20.0.11/32",
"subnet_id": "subnet-a",
}]},
],
)
plan = parse_vpc_plan(target)
gate = {"active": True, "available": True}
coordinator = VpcCoordinator(
MemoryStore(), FakeFabric(hosts=("host-a", "host-b")),
registry=object(),
)
record = coordinator._desired_record(
plan, gate, state="applying")
calls = []
def exact(host):
member = next(
item for item in plan.members
if item.host_id == host["host_id"])
private_mac = derive_private_mac(plan.vpc_id, member.member_id)
host.update({
"state": "active",
"transport_verified": True,
"private_dataplane_verified": True,
"config_verified": True,
"observed_revision": plan.desired_revision,
"observed_digest": plan.desired_digest,
"config_observed_revision": plan.desired_revision,
"config_observed_digest": plan.desired_digest,
"retiring_members": [],
"member_nics": [{
"member_id": member.member_id,
"instance_uuid": member.instance_uuid,
"private_ip_cidr": f"{member.private_ip}/32",
"private_mac": private_mac,
"config_verified": True,
"xml_verified": True,
"seed_verified": True,
"domain_running": True,
"guest_network_verified": True,
"restart_required": False,
}],
})
async def consume_one_host(_plan, execution, _gate, _previous, **kwargs):
host_ids = [host["host_id"] for host in execution["hosts"]]
calls.append(host_ids)
if len(calls) == 1:
exact(execution["hosts"][0])
execution["hosts"][1].update({
"state": "uncertain",
"transport_verified": False,
"private_dataplane_verified": False,
"last_error_code":
"vpc_private_guest_probe_deadline_exceeded",
})
else:
assert host_ids == ["host-b"]
exact(execution["hosts"][0])
coordinator._reconcile_dataplane = consume_one_host
asyncio.run(coordinator._reconcile_pending_hosts(
plan, record, gate, plan))
asyncio.run(coordinator._reconcile_pending_hosts(
plan, record, gate, plan))
assert calls == [["host-a", "host-b"], ["host-b"]]
assert all(host["state"] == "active" for host in record["hosts"])
def test_reconcile_reprobes_oldest_active_host_and_revokes_stale_guest_proof():
target = payload(hosts=[
{"host_id": "host-a", "members": [{
"member_id": "member-a", "instance_uuid": INSTANCE_A,
"private_ip_cidr": "10.20.0.10/32",
"subnet_id": "subnet-a",
}]},
{"host_id": "host-b", "members": [{
"member_id": "member-b", "instance_uuid": INSTANCE_B,
"private_ip_cidr": "10.20.0.11/32",
"subnet_id": "subnet-a",
}]},
])
plan = parse_vpc_plan(target)
gate = {"active": True, "available": True}
coordinator = VpcCoordinator(
MemoryStore(), FakeFabric(hosts=("host-a", "host-b")),
registry=object(),
)
record = coordinator._desired_record(plan, gate, state="applying")
attempted_at = {"host-a": 200, "host-b": 100}
for host in record["hosts"]:
member = next(
item for item in plan.members
if item.host_id == host["host_id"])
private_mac = derive_private_mac(plan.vpc_id, member.member_id)
host.update({
"state": "active",
"transport_verified": True,
"private_dataplane_verified": True,
"config_verified": True,
"observed_revision": plan.desired_revision,
"observed_digest": plan.desired_digest,
"config_observed_revision": plan.desired_revision,
"config_observed_digest": plan.desired_digest,
"retiring_members": [],
"probe_attempted_at_ns": attempted_at[host["host_id"]],
"member_nics": [{
"member_id": member.member_id,
"instance_uuid": member.instance_uuid,
"private_ip_cidr": f"{member.private_ip}/32",
"private_mac": private_mac,
"config_verified": True,
"xml_verified": True,
"seed_verified": True,
"domain_running": True,
"guest_network_verified": True,
"restart_required": False,
}],
})
calls = []
async def lose_qga_proof(_plan, execution, _gate, _previous, **kwargs):
calls.append([host["host_id"] for host in execution["hosts"]])
host = execution["hosts"][0]
host.update({
"state": "applying",
"private_dataplane_verified": False,
"observed_revision": 0,
"observed_digest": None,
"last_error_code": "vpc_private_guest_agent_unavailable",
})
host["member_nics"][0].update({
"guest_network_verified": False,
"guest_network_readback": "unavailable",
"guest_reason_code": "vpc_private_guest_agent_unavailable",
})
coordinator._reconcile_dataplane = lose_qga_proof
asyncio.run(coordinator._reconcile_pending_hosts(
plan, record, gate, plan))
assert calls == [["host-b"]]
hosts = {host["host_id"]: host for host in record["hosts"]}
assert hosts["host-a"]["state"] == "active"
assert hosts["host-b"]["state"] == "applying"
assert hosts["host-b"]["private_dataplane_verified"] is False
assert hosts["host-b"]["member_nics"][0][
"guest_network_verified"] is False
assert record["state"] == "applying"
assert record["next_stage_allowed"] is False
assert record["private_dataplane_verified"] is False
def test_reconcile_rotates_pending_host_after_it_consumes_budget():
target = payload(hosts=[
{"host_id": "host-a", "members": [{
"member_id": "member-a", "instance_uuid": INSTANCE_A,
"private_ip_cidr": "10.20.0.10/32",
"subnet_id": "subnet-a",
}]},
{"host_id": "host-b", "members": [{
"member_id": "member-b", "instance_uuid": INSTANCE_B,
"private_ip_cidr": "10.20.0.11/32",
"subnet_id": "subnet-a",
}]},
])
plan = parse_vpc_plan(target)
gate = {"active": True, "available": True}
coordinator = VpcCoordinator(
MemoryStore(), FakeFabric(hosts=("host-a", "host-b")),
registry=object(),
)
record = coordinator._desired_record(
plan, gate, state="applying")
calls = []
async def consume_budget(_plan, execution, _gate, _previous, **kwargs):
calls.append([host["host_id"] for host in execution["hosts"]])
consumer = execution["hosts"][0]
consumer["probe_attempted_at_ns"] = 100 + len(calls)
consumer.update({
"state": "uncertain",
"transport_verified": False,
"private_dataplane_verified": False,
"last_error_code":
"vpc_private_guest_probe_deadline_exceeded",
})
coordinator._reconcile_dataplane = consume_budget
asyncio.run(coordinator._reconcile_pending_hosts(
plan, record, gate, plan))
asyncio.run(coordinator._reconcile_pending_hosts(
plan, record, gate, plan))
assert calls == [["host-a", "host-b"], ["host-b", "host-a"]]
def test_instance_lifecycle_target_precedes_unrelated_pending_host():
target = payload(hosts=[
{"host_id": "host-a", "members": [{
"member_id": "member-a", "instance_uuid": INSTANCE_A,
"private_ip_cidr": "10.20.0.10/32",
"subnet_id": "subnet-a",
}]},
{"host_id": "host-b", "members": [{
"member_id": "member-b", "instance_uuid": INSTANCE_B,
"private_ip_cidr": "10.20.0.11/32",
"subnet_id": "subnet-a",
}]},
])
plan = parse_vpc_plan(target)
gate = {"active": True, "available": True}
coordinator = VpcCoordinator(
MemoryStore(), FakeFabric(hosts=("host-a", "host-b")),
registry=object(),
)
record = coordinator._desired_record(plan, gate, state="applying")
record["hosts"][1]["probe_attempted_at_ns"] = 999
calls = []
async def capture_order(_plan, execution, _gate, _previous, **kwargs):
calls.append([host["host_id"] for host in execution["hosts"]])
coordinator._reconcile_dataplane = capture_order
asyncio.run(coordinator._reconcile_pending_hosts(
plan, record, gate, plan,
private_link_states={INSTANCE_B: "down"},
))
assert calls == [["host-b", "host-a"]]
def test_guest_probe_pool_is_rolling_bounded_and_deadline_fail_closed(
monkeypatch):
import sys
import types
try:
import libvirt # noqa: F401
except ModuleNotFoundError:
stub = types.ModuleType("libvirt")
stub.libvirtError = type("libvirtError", (Exception,), {})
monkeypatch.setitem(sys.modules, "libvirt", stub)
from virsh_panel.libvirt_service import VMService
from virsh_panel.vpc_dataplane import VpcPrivateHostAdapter
members = [{
"member_id": f"member-{index}",
"instance_uuid":
f"00000000-0000-4000-8000-{index + 1:012x}",
"private_ip_cidr": f"10.20.0.{index + 10}/32",
"subnet_id": "subnet-a",
} for index in range(10)]
plan = parse_vpc_plan(payload(hosts=[{
"host_id": "host-a", "members": members,
}]))
monkeypatch.setattr("virsh_panel.vpc.VPC_GUEST_PROBE_WORKERS", 2)
monkeypatch.setattr(VMService, "__init__", lambda self, *args, **kwargs: None)
def configure(self, instance_uuid, **kwargs):
return {
"xml": f"inactive:{instance_uuid}",
"live": f"live:{instance_uuid}",
"restart_required": False,
}
monkeypatch.setattr(VMService, "ensure_private_interface", configure)
lock = threading.Lock()
active = 0
maximum = 0
started = []
def probe(self, instance_uuid, **kwargs):
nonlocal active, maximum
assert isinstance(kwargs.get("guest_probe_deadline"), float)
with lock:
active += 1
maximum = max(maximum, active)
started.append(instance_uuid)
time.sleep(0.1)
with lock:
active -= 1
return {
"inactive": f"inactive:{instance_uuid}",
"live": f"live:{instance_uuid}",
"guest_network": {
"state": "confirmed", "verified": True,
"source": "qemu_guest_agent",
},
}
monkeypatch.setattr(VMService, "read_private_domain_pair", probe)
monkeypatch.setattr(
VpcPrivateHostAdapter, "__init__",
lambda self, manager, profile, **kwargs: None,
)
monkeypatch.setattr(
VpcPrivateHostAdapter, "apply", lambda self, dataplane: {})
captured = {}
def observe(self, dataplane, *, xml_by_instance=None):
captured.update(xml_by_instance or {})
return {
"state": "applying",
"config_verified": True,
"config_observed_revision": plan.desired_revision,
"config_observed_digest": plan.desired_digest,
"member_nics": [],
"reason_code": "vpc_private_guest_probe_deadline_exceeded",
}
monkeypatch.setattr(VpcPrivateHostAdapter, "observe", observe)
coordinator = VpcCoordinator(MemoryStore(), FakeFabric())
result = coordinator._run_host_dataplane(
plan, "host-a", SimpleNamespace(manager=object()),
{"libvirt_uri": "qemu:///system", "storage_pool": "default",
"iso_pool": "iso", "default_network": "default"},
[], None, fabric_public_keys={},
request_deadline=time.monotonic() + 1.05,
)
assert maximum == 2
assert len(started) == 2
assert active == 0
assert len(captured) == 10
deadline_rows = [
pair["guest_network"] for pair in captured.values()
if pair["guest_network"].get("reason_code")
== "vpc_private_guest_probe_deadline_exceeded"
]
assert len(deadline_rows) == 8
assert result["config_verified"] is True
assert result["private_dataplane_verified"] is False
def test_guest_probe_default_pool_bounds_libvirt_reply_amplification():
from virsh_panel.vpc import VPC_GUEST_PROBE_WORKERS
# virDomainQemuAgentCommand materializes libvirt's bounded raw reply before
# Python can enforce its 192 KiB parse limit. Keep the default narrow even
# though individual tests may monkeypatch it lower for deterministic timing.
assert VPC_GUEST_PROBE_WORKERS == 4
def test_guest_probe_limit_is_process_wide_across_vpc_requests(monkeypatch):
import sys
import types
try:
import libvirt # noqa: F401
except ModuleNotFoundError:
stub = types.ModuleType("libvirt")
stub.libvirtError = type("libvirtError", (Exception,), {})
monkeypatch.setitem(sys.modules, "libvirt", stub)
from virsh_panel.libvirt_service import VMService
from virsh_panel.vpc_dataplane import VpcPrivateHostAdapter
def make_plan(label, second_octet, first_uuid):
members = [{
"member_id": f"member-{label}-{index}",
"instance_uuid":
f"00000000-0000-4000-8000-{first_uuid + index:012x}",
"private_ip_cidr":
f"10.{second_octet}.0.{index + 10}/32",
"subnet_id": f"subnet-{label}",
} for index in range(4)]
return parse_vpc_plan(payload(
vpc_id=f"vpc-hk-{label}",
cidr=f"10.{second_octet}.0.0/24",
operation_id=f"vpc-op-{label}-00000001",
subnets=[{
"subnet_id": f"subnet-{label}",
"cidr": f"10.{second_octet}.0.0/25",
}],
hosts=[{"host_id": "host-a", "members": members}],
))
plans = (make_plan("a", 20, 1), make_plan("b", 21, 101))
monkeypatch.setattr(VMService, "__init__", lambda self, *args, **kwargs: None)
monkeypatch.setattr(
VMService, "ensure_private_interface",
lambda self, instance_uuid, **kwargs: {
"xml": f"inactive:{instance_uuid}",
"live": f"live:{instance_uuid}",
"restart_required": False,
},
)
lock = threading.Lock()
release_full_wave = threading.Event()
active = 0
maximum = 0
started = 0
def probe(self, instance_uuid, **kwargs):
nonlocal active, maximum, started
with lock:
active += 1
started += 1
maximum = max(maximum, active)
if active >= 8:
release_full_wave.set()
# Without the process gate, both four-worker request pools enter
# together. With it, a bounded wave of four leaves before the next.
release_full_wave.wait(0.2)
with lock:
active -= 1
return {
"inactive": f"inactive:{instance_uuid}",
"live": f"live:{instance_uuid}",
"guest_network": {
"state": "confirmed", "verified": True,
"source": "qemu_guest_agent",
},
}
monkeypatch.setattr(VMService, "read_private_domain_pair", probe)
monkeypatch.setattr(
VpcPrivateHostAdapter, "__init__",
lambda self, manager, profile, **kwargs: None,
)
monkeypatch.setattr(
VpcPrivateHostAdapter, "apply", lambda self, dataplane: {})
def observe(self, dataplane, *, xml_by_instance=None):
return {
"state": "active",
"config_verified": True,
"config_observed_revision": dataplane.vpc.desired_revision,
"config_observed_digest": dataplane.vpc.desired_digest,
"member_nics": [],
}
monkeypatch.setattr(VpcPrivateHostAdapter, "observe", observe)
def run(plan):
return VpcCoordinator(
MemoryStore(), FakeFabric(),
)._run_host_dataplane(
plan, "host-a", SimpleNamespace(manager=object()),
{"libvirt_uri": "qemu:///system", "storage_pool": "default",
"iso_pool": "iso", "default_network": "default"},
[], None, fabric_public_keys={},
request_deadline=time.monotonic() + 3,
)
with ThreadPoolExecutor(max_workers=2) as requests:
results = [future.result() for future in (
requests.submit(run, plans[0]),
requests.submit(run, plans[1]),
)]
assert started == 8
assert maximum == 4
assert all(result["config_verified"] is True for result in results)
def test_request_probe_deadline_stops_next_host_and_keeps_config_proof(
monkeypatch):
target = payload(hosts=[
{"host_id": "host-a", "members": [{
"member_id": "member-a", "instance_uuid": INSTANCE_A,
"private_ip_cidr": "10.20.0.10/32",
"subnet_id": "subnet-a",
}]},
{"host_id": "host-b", "members": [{
"member_id": "member-b", "instance_uuid": INSTANCE_B,
"private_ip_cidr": "10.20.0.11/32",
"subnet_id": "subnet-a",
}]},
])
plan = parse_vpc_plan(target)
fabric_plan = FabricPlan(
"fabric-hk", "hk", "100.64.0.0/24", 7, 51820,
(
FabricNode(
"host-a", "100.64.0.1", "192.0.2.1:51820", 51820),
FabricNode(
"host-b", "100.64.0.2", "192.0.2.2:51820", 51820),
),
"f" * 64,
)
class DeadlineFabric(FakeFabric):
@staticmethod
def _plan_from_record(_record):
return fabric_plan
async def _components(self, _plan):
return {
host_id: (
SimpleNamespace(manager=object()),
{"host_id": host_id},
)
for host_id in ("host-a", "host-b")
}
coordinator = VpcCoordinator(
MemoryStore(), DeadlineFabric(hosts=("host-a", "host-b")),
registry=object(),
)
coordinator._mark_lifecycle_conflicts = lambda *args, **kwargs: False
coordinator._validate_dataplane_resources = lambda _plan: None
coordinator._fabric_record = lambda _plan: {
"public_keys": {"host-a": "a" * 44, "host-b": "b" * 44},
}
coordinator._fabric_peers_for_plan = lambda _plan, _record: []
monkeypatch.setattr("virsh_panel.vpc.VPC_REQUEST_DEADLINE_SECONDS", 0.05)
calls = []
def slow_host(*args, **kwargs):
host_id = args[1]
calls.append(host_id)
time.sleep(0.08)
return {
"state": "active",
"transport_verified": True,
"private_dataplane_verified": True,
"config_verified": True,
"config_observed_revision": plan.desired_revision,
"config_observed_digest": plan.desired_digest,
"member_nics": [{
"member_id": "member-a",
"guest_network_verified": True,
}],
"reason_code": None,
}
coordinator._run_host_dataplane = slow_host
gate = {"active": True, "available": True}
record = coordinator._desired_record(
plan, gate, state="applying")
asyncio.run(coordinator._reconcile_dataplane_unlocked(
plan, record, gate, previous_plan=plan))
assert calls == ["host-a"]
first, second = record["hosts"]
assert first["transport_verified"] is True
assert first["probe_attempted_at_ns"] > 0
assert first["config_verified"] is True
assert first["config_observed_revision"] == plan.desired_revision
assert first["private_dataplane_verified"] is False
assert first["member_nics"][0]["guest_network_verified"] is False
assert first["last_error_code"] \
== "vpc_private_guest_probe_deadline_exceeded"
assert second["transport_verified"] is False
assert second["last_error_code"] \
== "vpc_private_guest_probe_deadline_exceeded"
def test_root_delete_surfaces_only_allowlisted_deterministic_host_errors():
deterministic_codes = (
"vpc_private_remove_requires_stopped",
@ -1712,7 +2499,7 @@ def test_delete_manifest_removes_unknown_member_nic_before_host_resources(
monkeypatch.setattr(VMService, "remove_private_interface", remove_nic)
monkeypatch.setattr(
VpcPrivateHostAdapter, "__init__",
lambda self, manager, profile: None,
lambda self, manager, profile, **kwargs: None,
)
def remove_host(self, dataplane):

@ -10,16 +10,25 @@ import os
import re
import subprocess
import tempfile
import xml.etree.ElementTree as ET
from dataclasses import replace
from pathlib import Path
import pytest
from virsh_panel.shell_contracts import IP_RULE_CONVERGENCE_SH
from virsh_panel.vpc import VPC_SCHEMA_VERSION
from virsh_panel.vpc_dataplane import (
VpcDataplaneError,
VpcPrivateHostAdapter,
_allowed,
_member_nics,
_markers,
_routes,
parse_vpc_dataplane_plan,
patch_private_interface_xml,
private_guest_network_generation,
private_guest_routes,
render_apply_script,
render_observe_script,
render_remove_script,
@ -188,6 +197,20 @@ def _plan():
}, "host-a")
def _guest_network_evidence(member, routes):
return {
"state": "confirmed",
"verified": True,
"source": "qemu_guest_agent",
"private_mac": member.private_mac,
"private_ip_cidr": f"{member.private_ip}/32",
"link_state": "up",
"routes": sorted(routes),
"reason_code": None,
"observed_at": 1_700_000_000,
}
def test_rendered_data_plane_has_exact_routes_proxy_arp_and_atomic_nft_batch():
plan = _plan()
apply = render_apply_script(plan)
@ -212,6 +235,11 @@ def test_rendered_data_plane_has_exact_routes_proxy_arp_and_atomic_nft_batch():
assert "anti-spoof-bridge-to-foreign" in nft
assert "anti-spoof-fabric-to-foreign" in nft
assert 'oifname "wg-fabric0" ip daddr @vpc_dest_ips drop' in nft
assert (
"arp operation request arp daddr ip != @local_dest_ips drop"
in nft
)
assert "bridge-unknown-arp-target-drop" in nft
def test_rendered_scripts_are_posix_shell():
@ -613,6 +641,8 @@ def test_private_xml_patch_migrates_pre_fix_aliases_and_leaves_private_filter_fr
</devices></domain>"""
patched = patch_private_interface_xml(
xml, member=member, resources=plan.resources,
guest_network_generation=private_guest_network_generation(
member, private_guest_routes(plan.vpc, member)),
vpc_id=plan.vpc.vpc_id, desired_revision=plan.desired_revision,
desired_digest=plan.desired_digest,
)
@ -636,6 +666,8 @@ def test_private_xml_patch_promotes_legacy_public_aliases_idempotently():
</devices></domain>"""
patched = patch_private_interface_xml(
legacy, member=member, resources=plan.resources,
guest_network_generation=private_guest_network_generation(
member, private_guest_routes(plan.vpc, member)),
vpc_id=plan.vpc.vpc_id, desired_revision=plan.desired_revision,
desired_digest=plan.desired_digest,
)
@ -644,6 +676,8 @@ def test_private_xml_patch_promotes_legacy_public_aliases_idempotently():
# A second reconciliation must still identify the original NIC as public.
reparsed = patch_private_interface_xml(
patched, member=member, resources=plan.resources,
guest_network_generation=private_guest_network_generation(
member, private_guest_routes(plan.vpc, member)),
vpc_id=plan.vpc.vpc_id, desired_revision=plan.desired_revision,
desired_digest=plan.desired_digest,
)
@ -665,6 +699,9 @@ def test_private_xml_patch_rejects_domain_wide_alias_collision():
try:
patch_private_interface_xml(
xml, member=plan.members[0], resources=plan.resources,
guest_network_generation=private_guest_network_generation(
plan.members[0],
private_guest_routes(plan.vpc, plan.members[0])),
vpc_id=plan.vpc.vpc_id, desired_revision=plan.desired_revision,
desired_digest=plan.desired_digest,
)
@ -673,3 +710,382 @@ def test_private_xml_patch_rejects_domain_wide_alias_collision():
assert exc.error_code == "vpc_private_xml_role_invalid"
else:
raise AssertionError("domain-wide duplicate libvirt alias was accepted")
def test_guest_network_generation_is_stable_across_member_and_route_churn():
plan = _plan()
member = plan.local_members[0]
routes = private_guest_routes(plan.vpc, member)
assert routes == ("10.42.0.0/24",)
expected = private_guest_network_generation(member, routes)
assert private_guest_routes(
replace(plan.vpc, routes=()), member) == routes
assert private_guest_routes(
replace(plan.vpc, members=(member,)), member) == routes
wider = private_guest_routes(
replace(plan.vpc, cidr="10.42.0.0/16"), member)
assert private_guest_network_generation(member, wider) != expected
for invalid in ((), ("10.42.0.11/32",),
("10.42.0.0/24", "10.42.0.11/32")):
with pytest.raises(VpcDataplaneError):
private_guest_network_generation(member, invalid)
def test_running_member_requires_booted_guest_network_generation():
plan = _plan()
member = plan.local_members[0]
routes = private_guest_routes(plan.vpc, member)
generation = private_guest_network_generation(member, routes)
original = """<domain><metadata/><devices>
<interface type='bridge'><mac address='52:54:00:00:00:01'/>
<alias name='ua-public0'/><source bridge='br0'/></interface>
</devices></domain>"""
persistent = patch_private_interface_xml(
original, member=member, resources=plan.resources,
guest_network_generation=generation,
vpc_id=plan.vpc.vpc_id, desired_revision=plan.desired_revision,
desired_digest=plan.desired_digest,
)
stale_live_root = ET.fromstring(persistent)
stale_managed = stale_live_root.find(
"./metadata/{urn:virsh-panel:1}managed")
stale_managed.attrib.pop("private-guest-network-generation")
stale_live = ET.tostring(stale_live_root, encoding="unicode")
pending = _member_nics(plan, {
member.instance_uuid: {
"inactive": persistent,
"live": stale_live,
},
})[0]
# Device identity is present in both definitions, but the stale live
# generation means this running guest still needs a full restart before
# a missing guest-agent proof can be treated as potentially curable.
# Legacy v1 observations did not carry physical seed proof. Even with
# matching XML device identity they must remain non-usable.
assert pending["xml_verified"] is False
assert pending["config_verified"] is True
assert pending["guest_network_verified"] is False
assert pending["restart_required"] is True
booted = _member_nics(plan, {
member.instance_uuid: {
"inactive": persistent,
"live": persistent,
"seed_verified": True,
"guest_network": _guest_network_evidence(member, routes),
},
})[0]
assert booted["xml_verified"] is True
assert booted["config_verified"] is True
assert booted["domain_running"] is True
assert booted["guest_network_verified"] is True
assert booted["guest_private_ip_cidr"] == f"{member.private_ip}/32"
assert booted["guest_routes"] == sorted(routes)
assert booted["restart_required"] is False
def test_booted_generation_fails_closed_when_guest_evidence_is_unavailable():
plan = _plan()
member = plan.local_members[0]
routes = private_guest_routes(plan.vpc, member)
persistent = patch_private_interface_xml(
"""<domain><metadata/><devices>
<interface type='bridge'><mac address='52:54:00:00:00:01'/>
<alias name='ua-public0'/><source bridge='br0'/></interface>
</devices></domain>""",
member=member, resources=plan.resources,
guest_network_generation=private_guest_network_generation(
member, routes),
vpc_id=plan.vpc.vpc_id, desired_revision=plan.desired_revision,
desired_digest=plan.desired_digest,
)
observed = _member_nics(plan, {
member.instance_uuid: {
"inactive": persistent,
"live": persistent,
"seed_verified": True,
"guest_network": {
"state": "unavailable", "verified": False,
"reason_code": "vpc_private_guest_agent_unavailable",
},
},
})[0]
assert observed["config_verified"] is True
assert observed["xml_verified"] is True
assert observed["domain_running"] is True
assert observed["guest_network_verified"] is False
assert observed["guest_network_readback"] == "unavailable"
assert observed["guest_reason_code"] == (
"vpc_private_guest_agent_unavailable")
# The generation is current, so a restart would not cure an unavailable
# agent. The host remains non-active for the guest-specific reason.
assert observed["restart_required"] is False
def test_booted_generation_rejects_guest_address_or_route_mismatch():
plan = _plan()
member = plan.local_members[0]
routes = private_guest_routes(plan.vpc, member)
persistent = patch_private_interface_xml(
"""<domain><metadata/><devices>
<interface type='bridge'><mac address='52:54:00:00:00:01'/>
<alias name='ua-public0'/><source bridge='br0'/></interface>
</devices></domain>""",
member=member, resources=plan.resources,
guest_network_generation=private_guest_network_generation(
member, routes),
vpc_id=plan.vpc.vpc_id, desired_revision=plan.desired_revision,
desired_digest=plan.desired_digest,
)
mismatch = _guest_network_evidence(member, routes)
mismatch.update({
"state": "mismatch", "verified": False,
"private_ip_cidr": "10.42.0.99/32",
"routes": [],
"reason_code": "vpc_private_guest_address_mismatch",
})
observed = _member_nics(plan, {
member.instance_uuid: {
"inactive": persistent,
"live": persistent,
"seed_verified": True,
"guest_network": mismatch,
},
})[0]
assert observed["config_verified"] is True
assert observed["xml_verified"] is True
assert observed["guest_network_verified"] is False
assert observed["guest_private_ip_cidr"] == "10.42.0.99/32"
assert observed["guest_routes"] == []
assert observed["guest_reason_code"] == (
"vpc_private_guest_address_mismatch")
assert observed["restart_required"] is False
def test_stopped_member_preserves_config_proof_but_is_not_runtime_verified():
plan = _plan()
member = plan.local_members[0]
routes = private_guest_routes(plan.vpc, member)
persistent = patch_private_interface_xml(
"""<domain><metadata/><devices>
<interface type='bridge'><mac address='52:54:00:00:00:01'/>
<alias name='ua-public0'/><source bridge='br0'/></interface>
</devices></domain>""",
member=member, resources=plan.resources,
guest_network_generation=private_guest_network_generation(
member, routes),
vpc_id=plan.vpc.vpc_id, desired_revision=plan.desired_revision,
desired_digest=plan.desired_digest,
)
observed = _member_nics(plan, {
member.instance_uuid: {
"inactive": persistent,
"live": None,
"seed_verified": True,
"guest_network": {
"state": "stopped", "verified": False,
"reason_code": "vpc_private_guest_stopped",
},
},
})[0]
assert observed["config_verified"] is True
assert observed["domain_running"] is False
assert observed["xml_verified"] is True
assert observed["guest_network_verified"] is False
assert observed["restart_required"] is False
assert observed["guest_reason_code"] == (
"vpc_private_guest_stopped")
def _active_observation_markers(plan):
markers = [
f"__VC_VPC_BRIDGE__={plan.resources.bridge}",
"__VC_VPC_BRIDGE_OWNER__=virt-cloud-vpc-dataplane-v1",
"__VC_VPC_PERSISTENCE__=yes",
"__VC_VPC_BRIDGE_BOUNDARY__=yes",
"__VC_VPC_PROXY_ARP__=0",
"__VC_VPC_PROXY_ARP_PVLAN__=1",
"__VC_VPC_PROXY_STATE__=owned",
"__VC_VPC_RP_FILTER__=2",
f"__VC_VPC_NFT_TABLE__={plan.resources.nft_table}",
"__VC_VPC_STATE_OWNER__=owned",
f"__VC_VPC_STATE_REVISION__={plan.desired_revision}",
f"__VC_VPC_STATE_DIGEST__={plan.desired_digest}",
f"__VC_VPC_ACL_REVISION__={plan.desired_revision}",
f"__VC_VPC_ACL_DIGEST__={plan.desired_digest}",
"__VC_VPC_ANTI_SPOOF__=yes",
"__VC_VPC_BRIDGE_NFT__=yes",
"__VC_VPC_ROUTE_EXTRA__=no",
"__VC_VPC_RULE_EXTRA__=no",
"__VC_VPC_ARP_NFT__=yes",
]
for route in _routes(plan):
markers.append(
"__VC_VPC_ROUTE__="
f"{route['destination_cidr']}|{route['device']}|"
f"{route['table']}|{route.get('via') or '-'}|"
f"{route.get('metric', 100)}|owned")
markers.append(
"__VC_VPC_PROXY_IP__="
f"{route['destination_cidr'].removesuffix('/32')}")
markers.extend((
"__VC_VPC_RULE__="
f"{plan.resources.rule_priority}|-|{plan.vpc.cidr}|"
f"{plan.resources.route_table}|owned",
"__VC_VPC_RULE__="
f"{plan.resources.rule_priority + 1}|wg-fabric0|{plan.vpc.cidr}|"
f"{plan.resources.route_table}|owned",
))
for peer in plan.peers:
markers.extend((
f"__VC_VPC_WG_ALLOWED__={peer.host_id}|{peer.public_key}|"
f"{_allowed(plan, peer)}",
f"__VC_VPC_WG_STATE__={peer.host_id}|owned",
))
return markers
def test_observer_does_not_activate_unavailable_or_mismatched_guest():
plan = _plan()
member = plan.local_members[0]
routes = private_guest_routes(plan.vpc, member)
persistent = patch_private_interface_xml(
"""<domain><metadata/><devices>
<interface type='bridge'><mac address='52:54:00:00:00:01'/>
<alias name='ua-public0'/><source bridge='br0'/></interface>
</devices></domain>""",
member=member, resources=plan.resources,
guest_network_generation=private_guest_network_generation(
member, routes),
vpc_id=plan.vpc.vpc_id, desired_revision=plan.desired_revision,
desired_digest=plan.desired_digest,
)
adapter = VpcPrivateHostAdapter(object(), {})
adapter._deployment_contract = lambda: ("networkmanager", "", "", "")
adapter._run = lambda _script: "\n".join(
_active_observation_markers(plan)) + "\n"
for state, reason in (
("unavailable", "vpc_private_guest_agent_unavailable"),
("mismatch", "vpc_private_guest_address_mismatch"),
):
observed = adapter.observe(plan, xml_by_instance={
member.instance_uuid: {
"inactive": persistent,
"live": persistent,
"seed_verified": True,
"guest_network": {
"state": state, "verified": False,
"reason_code": reason,
},
},
})
assert observed["config_verified"] is True
assert observed["state"] == "applying"
assert observed["observed_revision"] == 0
assert observed["observed_digest"] is None
assert observed["readback"] == "pending_guest"
assert observed["reason_code"] == reason
assert observed["member_nics"][0]["xml_verified"] is True
assert observed["member_nics"][0]["guest_network_verified"] is False
confirmed = adapter.observe(plan, xml_by_instance={
member.instance_uuid: {
"inactive": persistent,
"live": persistent,
"seed_verified": True,
"guest_network": _guest_network_evidence(member, routes),
},
})
assert confirmed["state"] == "active"
assert confirmed["observed_revision"] == plan.desired_revision
assert confirmed["observed_digest"] == plan.desired_digest
assert confirmed["readback"] == "confirmed"
def test_observer_reports_pending_restart_instead_of_active():
plan = _plan()
member = plan.local_members[0]
generation = private_guest_network_generation(
member, private_guest_routes(plan.vpc, member))
original = """<domain><metadata/><devices>
<interface type='bridge'><mac address='52:54:00:00:00:01'/>
<alias name='ua-public0'/><source bridge='br0'/></interface>
</devices></domain>"""
persistent = patch_private_interface_xml(
original, member=member, resources=plan.resources,
guest_network_generation=generation,
vpc_id=plan.vpc.vpc_id, desired_revision=plan.desired_revision,
desired_digest=plan.desired_digest,
)
stale_live_root = ET.fromstring(persistent)
stale_live_root.find(
"./metadata/{urn:virsh-panel:1}managed"
).attrib.pop("private-guest-network-generation")
stale_live = ET.tostring(stale_live_root, encoding="unicode")
markers = [
f"__VC_VPC_BRIDGE__={plan.resources.bridge}",
"__VC_VPC_BRIDGE_OWNER__=virt-cloud-vpc-dataplane-v1",
"__VC_VPC_PERSISTENCE__=yes",
"__VC_VPC_BRIDGE_BOUNDARY__=yes",
"__VC_VPC_PROXY_ARP__=0",
"__VC_VPC_PROXY_ARP_PVLAN__=1",
"__VC_VPC_PROXY_STATE__=owned",
"__VC_VPC_RP_FILTER__=2",
f"__VC_VPC_NFT_TABLE__={plan.resources.nft_table}",
"__VC_VPC_STATE_OWNER__=owned",
f"__VC_VPC_STATE_REVISION__={plan.desired_revision}",
f"__VC_VPC_STATE_DIGEST__={plan.desired_digest}",
f"__VC_VPC_ACL_REVISION__={plan.desired_revision}",
f"__VC_VPC_ACL_DIGEST__={plan.desired_digest}",
"__VC_VPC_ANTI_SPOOF__=yes",
"__VC_VPC_BRIDGE_NFT__=yes",
"__VC_VPC_ROUTE_EXTRA__=no",
"__VC_VPC_RULE_EXTRA__=no",
"__VC_VPC_ARP_NFT__=yes",
]
for route in _routes(plan):
markers.append(
"__VC_VPC_ROUTE__="
f"{route['destination_cidr']}|{route['device']}|"
f"{route['table']}|{route.get('via') or '-'}|"
f"{route.get('metric', 100)}|owned")
markers.append(
"__VC_VPC_PROXY_IP__="
f"{route['destination_cidr'].removesuffix('/32')}")
markers.extend((
"__VC_VPC_RULE__="
f"{plan.resources.rule_priority}|-|{plan.vpc.cidr}|"
f"{plan.resources.route_table}|owned",
"__VC_VPC_RULE__="
f"{plan.resources.rule_priority + 1}|wg-fabric0|{plan.vpc.cidr}|"
f"{plan.resources.route_table}|owned",
))
for peer in plan.peers:
markers.extend((
f"__VC_VPC_WG_ALLOWED__={peer.host_id}|{peer.public_key}|"
f"{_allowed(plan, peer)}",
f"__VC_VPC_WG_STATE__={peer.host_id}|owned",
))
adapter = VpcPrivateHostAdapter(object(), {})
adapter._deployment_contract = lambda: ("networkmanager", "", "", "")
adapter._run = lambda _script: "\n".join(markers) + "\n"
observed = adapter.observe(plan, xml_by_instance={
member.instance_uuid: {
"inactive": persistent,
"live": stale_live,
},
})
assert observed["state"] == "applying"
assert observed["readback"] == "pending_restart"
assert observed["reason_code"] == "vpc_private_guest_restart_required"
assert observed["member_nics"][0]["restart_required"] is True

@ -53,6 +53,15 @@ ROOT_PASSWORD_ALPHABET = (
ROOT_PASSWORD_UPPERCASE + ROOT_PASSWORD_LOWERCASE
+ ROOT_PASSWORD_DIGITS + ROOT_PASSWORD_SYMBOLS
)
_RFC1918_NETWORKS = (
ipaddress.IPv4Network("10.0.0.0/8"),
ipaddress.IPv4Network("172.16.0.0/12"),
ipaddress.IPv4Network("192.168.0.0/16"),
)
class SeedNetworkPolicyMigrationRequired(ValueError):
"""A legacy seed cannot be made reboot-safe without replaying user-data."""
def validate_guest_hostname(value: Any) -> str:
@ -288,10 +297,9 @@ def validate_cloud_network(
def validate_private_network(private_ip_cidr: Any) -> dict[str, str]:
"""Validate one L3 VPC guest address.
VPC members are routed as host/guest ``/32`` endpoints. They do not get
a default route or a provider gateway from this function; those routes are
owned by the host dataplane and are added only after the VPC ACL/readback
barrier succeeds.
Each guest address and the host route to that address use ``/32``. Guests
do not get a default route or provider gateway here; the separate seed
renderer adds exactly one on-link VPC CIDR route.
"""
if (not isinstance(private_ip_cidr, str)
or not private_ip_cidr
@ -309,30 +317,33 @@ def validate_private_network(private_ip_cidr: Any) -> dict[str, str]:
# ``is_private`` also includes loopback/link-local ranges, which are not
# valid tenant addresses. Explicitly keep the product contract to RFC1918
# space so a host-local or metadata address cannot be routed as a member.
rfc1918 = (
ipaddress.IPv4Network("10.0.0.0/8"),
ipaddress.IPv4Network("172.16.0.0/12"),
ipaddress.IPv4Network("192.168.0.0/16"),
)
if (not any(address in network for network in rfc1918)
if (not any(address in network for network in _RFC1918_NETWORKS)
or address.is_loopback or address.is_link_local
or address.is_reserved or address.is_multicast):
raise ValueError("私网 IP 必须是 RFC1918 IPv4 地址")
return {"private_ip_cidr": str(interface)}
def _validate_private_routes(value: Any) -> list[str]:
"""Validate the guest's exact on-link VPC routes.
The host owns inter-host forwarding, while the guest still needs a route
for each remote /32. Keep this list deliberately narrow: no gateways,
default routes, or broad prefixes are accepted on ``private0``.
def _validate_private_routes(
value: Any,
*,
allow_legacy: bool = True,
private_ip_cidr: str | None = None,
public_ip_cidr: str | None = None,
) -> list[str]:
"""Validate one stable VPC prefix or one legacy exact-/32 route set.
New seeds use exactly one RFC1918 /16../24 link-scope prefix. This keeps
the guest contract stable when members or service /32s change; exact host
routes, proxy neighbours and nftables destination sets remain the routing
authority. The parser still accepts an all-/32 legacy set so a stopped VM
can migrate, but mixed or ambiguous representations are rejected.
"""
if value is None:
return []
value = []
if not isinstance(value, (list, tuple)) or len(value) > 4096:
raise ValueError("私网路由列表无效")
result: set[str] = set()
result: list[ipaddress.IPv4Network] = []
for item in value:
if isinstance(item, str):
destination = item
@ -340,26 +351,50 @@ def _validate_private_routes(value: Any) -> list[str]:
if item.get("to") in {"default", "0.0.0.0/0", "::/0"}:
# Keep the legacy diagnostic used by seed validation callers:
# private0 is never allowed to acquire a default route.
raise ValueError("私网网卡必须是无默认路由的静态 /32")
raise ValueError("私网地址必须是静态 /32 且无默认路由")
if set(item) != {"to", "scope"} or item.get("scope") != "link":
raise ValueError("私网路由必须是 link scope 的 /32")
raise ValueError("私网路由必须使用 link scope")
destination = item.get("to")
else:
raise ValueError("私网路由格式无效")
if destination in {"default", "0.0.0.0/0", "::/0"}:
raise ValueError("私网网卡必须是无默认路由的静态 /32")
raise ValueError("私网地址必须是静态 /32 且无默认路由")
try:
network = ipaddress.ip_network(destination, strict=True)
except (TypeError, ValueError) as exc:
raise ValueError("私网路由必须是 IPv4 /32") from exc
raise ValueError("私网路由必须是 IPv4 网络") from exc
if (not isinstance(network, ipaddress.IPv4Network)
or network.prefixlen != 32
or not any(network.subnet_of(ipaddress.IPv4Network(block))
for block in ("10.0.0.0/8", "172.16.0.0/12",
"192.168.0.0/16"))):
raise ValueError("私网路由必须是 RFC1918 IPv4 /32")
result.add(str(network))
return sorted(result)
or not any(network.subnet_of(block)
for block in _RFC1918_NETWORKS)):
raise ValueError("私网路由必须位于 RFC1918 IPv4 空间")
result.append(network)
if len(set(result)) != len(result):
raise ValueError("私网路由不能重复")
legacy = all(network.prefixlen == 32 for network in result)
stable = bool(
len(result) == 1 and 16 <= result[0].prefixlen <= 24
)
if not stable and not (allow_legacy and legacy):
raise ValueError(
"私网路由必须是唯一 VPC /16 到 /24 前缀,"
"或完全由历史 /32 路由组成"
)
if private_ip_cidr is not None:
private_ip = ipaddress.ip_interface(
validate_private_network(private_ip_cidr)["private_ip_cidr"]
).ip
if stable and private_ip not in result[0]:
raise ValueError("私网实例地址必须位于 VPC 路由前缀内")
if public_ip_cidr is not None:
try:
public_network = ipaddress.ip_interface(public_ip_cidr).network
except (TypeError, ValueError) as exc:
raise ValueError("公网直连前缀无效") from exc
if any(network.overlaps(public_network) for network in result):
raise ValueError("VPC 前缀不能与公网网卡直连前缀重叠")
return sorted(str(network) for network in result)
def _validate_network_mac(value: Any, label: str) -> str:
@ -446,7 +481,7 @@ def _parse_network_config(config: Any) -> dict[str, Any]:
or len(private["addresses"]) != 1
or not isinstance(private_match, dict)
or set(private_match) != {"macaddress"}):
raise ValueError("原 cloud-init 私网网卡必须是无默认路由的静态 /32")
raise ValueError("原 cloud-init 私网地址必须是静态 /32 且无默认路由")
private_mac = _validate_network_mac(private_mac, "私网")
private_network = validate_private_network(private["addresses"][0])
if private_mac == public_mac:
@ -454,7 +489,11 @@ def _parse_network_config(config: Any) -> dict[str, Any]:
result.update({
**private_network,
"private_mac": private_mac,
"private_routes": _validate_private_routes(private.get("routes")),
"private_routes": _validate_private_routes(
private.get("routes"),
private_ip_cidr=private_network["private_ip_cidr"],
public_ip_cidr=public_network["public_ip_cidr"],
),
"roles": [PUBLIC_INTERFACE_ROLE, PRIVATE_INTERFACE_ROLE],
})
return result
@ -595,7 +634,11 @@ def build_seed_iso(
)
private_routes = _validate_private_routes(
private_network.get("routes", [])
if isinstance(private_network, dict) else [])
if isinstance(private_network, dict) else [],
allow_legacy=False,
private_ip_cidr=private["private_ip_cidr"],
public_ip_cidr=network["public_ip_cidr"],
)
if normalized_private_mac == normalized_mac:
raise ValueError("公网和私网网卡不能使用同一个 MAC")
network_config["ethernets"][PRIVATE_INTERFACE_ROLE] = {
@ -853,31 +896,66 @@ def replace_seed_private_network(
_parse_network_config(config)
user_data = _enable_network_updates(files["user-data"])
metadata = _decode_seed_metadata(files["meta-data"])
if user_data != files["user-data"]:
# Changing instance-id would make NoCloud replay credential and
# sshd per-instance modules. Keeping the id may leave legacy
# cloud-init versions on their cached network policy. Refuse this
# ambiguous migration; a reinstall or explicit offline migration
# must establish boot-v1 first.
raise SeedNetworkPolicyMigrationRequired(
"原 cloud-init 配置盘缺少安全的启动网络更新策略")
ethernets = config.get("ethernets")
if not isinstance(ethernets, dict):
raise ValueError("原 cloud-init 网卡配置无效")
if private_mac is None:
ethernets.pop(PRIVATE_INTERFACE_ROLE, None)
network_unchanged = PRIVATE_INTERFACE_ROLE not in parsed["roles"]
if not network_unchanged:
ethernets.pop(PRIVATE_INTERFACE_ROLE, None)
else:
normalized_mac = _validate_network_mac(private_mac, "私网")
private = validate_private_network(private_ip_cidr)
routes = _validate_private_routes(private_routes)
routes = _validate_private_routes(
private_routes,
allow_legacy=False,
private_ip_cidr=private["private_ip_cidr"],
public_ip_cidr=parsed["public_ip_cidr"],
)
if normalized_mac == parsed["mac"]:
raise ValueError("公网和私网网卡不能使用同一个 MAC")
ethernets[PRIVATE_INTERFACE_ROLE] = {
"match": {"macaddress": normalized_mac},
"dhcp4": False,
"dhcp6": False,
"accept-ra": False,
"addresses": [private["private_ip_cidr"]],
**({"routes": [
{"to": destination, "scope": "link"}
for destination in routes
]} if routes else {}),
}
# Rotate the datasource identity so both add and remove operations are
# consumed on the next boot. Credentials remain opaque bytes.
metadata["instance-id"] = f"virt-cloud-private-{uuid.uuid4().hex}"
network_unchanged = bool(
PRIVATE_INTERFACE_ROLE in parsed["roles"]
and parsed.get("private_mac") == normalized_mac
and parsed.get("private_ip_cidr")
== private["private_ip_cidr"]
and parsed.get("private_routes", []) == routes
)
if not network_unchanged:
ethernets[PRIVATE_INTERFACE_ROLE] = {
"match": {"macaddress": normalized_mac},
"dhcp4": False,
"dhcp6": False,
"accept-ra": False,
"addresses": [private["private_ip_cidr"]],
**({"routes": [
{"to": destination, "scope": "link"}
for destination in routes
]} if routes else {}),
}
policy_unchanged = bool(
user_data == files["user-data"]
and metadata.get(_NETWORK_POLICY_KEY) == _NETWORK_POLICY_VALUE
)
# A waiting-observed VPC is retried periodically. Returning the exact
# original ISO for an identical target prevents every retry from
# inventing a new NoCloud instance and re-running per-instance modules
# at the next boot.
if network_unchanged and policy_unchanged:
return seed
# Never rotate the NoCloud instance identity for a network-only
# change. A new instance-id would make cloud-init run every
# per-instance module again, including the credentials and sshd
# policy stored in this seed. The explicit boot event above is the
# only mechanism used to consume the refreshed network-config.
metadata[_NETWORK_POLICY_KEY] = _NETWORK_POLICY_VALUE
network_data = json.dumps(config, ensure_ascii=True).encode("ascii") + b"\n"
return _build_seed_iso((
@ -886,6 +964,8 @@ def replace_seed_private_network(
_encode_seed_metadata(metadata)),
("/NETWORK.;1", "network-config", "/network-config", network_data),
))
except SeedNetworkPolicyMigrationRequired:
raise
except (KeyError, TypeError, ValueError, json.JSONDecodeError) as exc:
raise ValueError("原 cloud-init 网络配置格式无效") from exc

@ -818,7 +818,8 @@ class EngineApplication:
raise EngineAuthError("私网核对链路状态无效", 422)
expected = {
key: value[key] for key in (
"required", "vpc_id", "member_id", "private_ip_cidr",
"lifecycle_kind", "required", "vpc_id", "member_id",
"private_ip_cidr",
"private_mac", "desired_revision", "desired_digest",
"desired_state") if key in value
}

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

@ -13,6 +13,7 @@ import hashlib
import ipaddress
import re
import shlex
import time
import xml.etree.ElementTree as ET
from dataclasses import dataclass
from typing import Any, Mapping, Sequence
@ -135,6 +136,9 @@ class VpcDataplaneObservation:
proxy_arp: int | None
proxy_arp_pvlan: int | None
proxy_ips: tuple[str, ...]
config_verified: bool
config_observed_revision: int
config_observed_digest: str | None
readback: str
rollback: str
reason_code: str | None = None
@ -157,6 +161,9 @@ class VpcDataplaneObservation:
"proxy_arp": self.proxy_arp,
"proxy_arp_pvlan": self.proxy_arp_pvlan,
"proxy_ips": list(self.proxy_ips),
"config_verified": self.config_verified,
"config_observed_revision": self.config_observed_revision,
"config_observed_digest": self.config_observed_digest,
"readback": self.readback,
"rollback": self.rollback, "reason_code": self.reason_code,
"removed": self.removed,
@ -203,6 +210,77 @@ def derive_private_mac(vpc_id: str, member_id: str) -> str:
return ":".join(f"{x:02x}" for x in bytes([first]) + digest[1:6])
def private_guest_routes(plan: VpcPlan, member: VpcMember) -> tuple[str, ...]:
"""Return the stable on-link VPC prefix installed inside one guest.
Host ACL, nftables and Fabric changes deliberately do not participate in
this projection. The host keeps exact member/service /32 routes, proxy
neighbours and destination sets; the guest needs only one link-scope VPC
prefix. Member and service-route churn therefore cannot force every
running sibling to consume a new boot-only NoCloud generation.
"""
try:
network = ipaddress.ip_network(plan.cidr, strict=True)
private_ip = ipaddress.ip_address(member.private_ip)
except (TypeError, ValueError) as exc:
raise VpcDataplaneError("VPC 来宾私网路由无效") from exc
if (not isinstance(network, ipaddress.IPv4Network)
or not 16 <= network.prefixlen <= 24
or not isinstance(private_ip, ipaddress.IPv4Address)
or private_ip not in network
or not any(network.subnet_of(block) for block in (
ipaddress.IPv4Network("10.0.0.0/8"),
ipaddress.IPv4Network("172.16.0.0/12"),
ipaddress.IPv4Network("192.168.0.0/16"),
))):
raise VpcDataplaneError("VPC 来宾私网路由无效")
return (str(network),)
def private_guest_network_generation(
member: VpcMember,
private_routes: Sequence[str],
) -> str:
"""Hash only the private network state that a guest consumes at boot."""
private_mac = _mac(member.private_mac)
try:
private_ip = ipaddress.ip_interface(f"{member.private_ip}/32")
except (TypeError, ValueError) as exc:
raise VpcDataplaneError("VPC 成员私网 IP 无效") from exc
if not isinstance(private_ip, ipaddress.IPv4Interface):
raise VpcDataplaneError("VPC 成员私网 IP 无效")
routes: list[ipaddress.IPv4Network] = []
if (not isinstance(private_routes, Sequence)
or isinstance(private_routes, (str, bytes))
or len(private_routes) != 1):
raise VpcDataplaneError("VPC 来宾私网路由无效")
for value in private_routes:
if not isinstance(value, str):
raise VpcDataplaneError("VPC 来宾私网路由无效")
try:
route = ipaddress.ip_network(value, strict=True)
except ValueError as exc:
raise VpcDataplaneError("VPC 来宾私网路由无效") from exc
if (not isinstance(route, ipaddress.IPv4Network)
or not 16 <= route.prefixlen <= 24
or private_ip.ip not in route
or not any(route.subnet_of(block) for block in (
ipaddress.IPv4Network("10.0.0.0/8"),
ipaddress.IPv4Network("172.16.0.0/12"),
ipaddress.IPv4Network("192.168.0.0/16"),
))):
raise VpcDataplaneError(
"VPC 来宾私网路由必须是包含成员地址的 IPv4 /16 到 /24")
routes.append(route)
payload = "\n".join((
"virt-cloud-private-guest-network-v2",
private_mac,
str(private_ip),
str(routes[0]),
)).encode("ascii")
return hashlib.sha256(payload).hexdigest()
def resource_names(vpc_id: str) -> VpcDataplaneResources:
vpc_id = _id(vpc_id, "VPC")
digest = hashlib.sha256(vpc_id.encode()).hexdigest()
@ -551,7 +629,8 @@ def render_vpc_nft_rules(plan: VpcDataplanePlan) -> str:
# VPC destination set; restricting it to local routes would drop every
# same-VPC cross-host flow before the inet routing policy can run.
local_destination_cidrs = set(destination_cidrs)
# Use explicit proxy-neighbour entries for the guest's routed /32 model.
# Use explicit proxy-neighbour entries for the host's exact /32 routes to
# guest addresses.
# The generic kernel proxy responder stays disabled, so a nftables reload
# or a short ordering window cannot make the host answer for arbitrary
# addresses. proxy_arp_pvlan is enabled only for the same-bridge
@ -672,6 +751,11 @@ def render_vpc_nft_rules(plan: VpcDataplanePlan) -> str:
f' meta ibrname "{r.bridge}" ether type arp arp saddr ether . arp saddr ip != @local_member_sources drop comment "{marker}:bridge-anti-spoof-arp"',
f' meta ibrname "{r.bridge}" ether type ip ip saddr != @local_member_ips drop comment "{marker}:bridge-anti-spoof-ip"',
f' meta ibrname "{r.bridge}" ether type arp arp saddr ip != @local_member_ips drop comment "{marker}:bridge-anti-spoof-arp-ip"',
# A stable VPC-prefix route lets a guest ask for every address in
# that prefix. Only exact member/service destinations may reach
# another tap or the host proxy responder; unknown ARP targets are
# dropped before they become a bridge-wide scan/flood primitive.
f' meta ibrname "{r.bridge}" ether type arp arp operation request arp daddr ip != @local_dest_ips drop comment "{marker}:bridge-unknown-arp-target-drop"',
# This is an L3 /32 fabric. Do not let broadcast or multicast
# discovery turn the private bridge back into a shared L2 LAN.
f' meta ibrname "{r.bridge}" ip daddr 224.0.0.0/4 drop comment "{marker}:bridge-multicast-drop"',
@ -2601,6 +2685,7 @@ def _markers(output: str) -> dict[str, Any]:
def patch_private_interface_xml(xml: str | bytes, *, member: VpcMember,
resources: VpcDataplaneResources,
guest_network_generation: str,
link_state: str = "up",
vpc_id: str | None = None,
desired_revision: int | None = None,
@ -2608,6 +2693,9 @@ def patch_private_interface_xml(xml: str | bytes, *, member: VpcMember,
"""Add/validate a private0 NIC on an explicit panel-owned bridge."""
if link_state not in {"up", "down"}:
raise VpcDataplaneError("私网网卡 link 状态无效")
if (not isinstance(guest_network_generation, str)
or _DIGEST.fullmatch(guest_network_generation) is None):
raise VpcDataplaneError("来宾私网 generation 无效")
private_mac = _mac(member.private_mac)
try:
root = ET.fromstring(xml)
@ -2680,6 +2768,8 @@ def patch_private_interface_xml(xml: str | bytes, *, member: VpcMember,
managed.set("private-network-bridge", resources.bridge)
managed.set("private-network-verified", "true")
managed.set("private-dataplane-owner", DATAPLANE_OWNER_MARKER)
managed.set("private-guest-network-generation",
guest_network_generation)
if vpc_id is not None:
managed.set("private-vpc-id", vpc_id)
if desired_revision is not None:
@ -2694,7 +2784,11 @@ def _member_nics(
plan: VpcDataplanePlan,
xml_by_instance: Mapping[str, Any] | None,
) -> tuple[dict[str, Any], ...]:
def xml_details(xml: str, member: VpcMember, *, metadata: bool) -> tuple[bool, str | None]:
def xml_details(
xml: str,
member: VpcMember,
expected_generation: str,
) -> tuple[bool, bool, bool, str | None]:
try:
root = ET.fromstring(xml)
private = select_private_interface(
@ -2715,57 +2809,151 @@ def _member_nics(
link = private.find("link") if private is not None else None
link_state = link.get("state", "up").lower() if link is not None else "up"
if link_state not in {"up", "down"}:
return False, None
if not metadata:
return ok, link_state
return False, False, False, None
managed = root.find(f"./metadata/{{{PANEL_NS}}}managed")
return bool(
ok and managed is not None
guest_generation_ok = bool(
managed is not None
and managed.get("private-vpc-id") == plan.vpc.vpc_id
and managed.get("private-member-id") == member.member_id
and managed.get("private-ip-cidr") == f"{member.private_ip}/32"
and managed.get("private-network-bridge")
== plan.resources.bridge
and managed.get("private-dataplane-owner")
== DATAPLANE_OWNER_MARKER
and managed.get("private-guest-network-generation")
== expected_generation
)
desired_metadata_ok = bool(
guest_generation_ok
and managed.get("private-network-revision")
== str(plan.desired_revision)
and managed.get("private-network-digest")
== plan.desired_digest
), link_state
)
return ok, guest_generation_ok, desired_metadata_ok, link_state
except (ET.ParseError, ValueError):
return False, None
return False, False, False, None
result = []
for member in plan.local_members:
verified = False
config_verified = False
seed_verified = False
restart_required = False
guest_verified = False
guest_readback = "missing"
guest_reason_code = "vpc_private_guest_agent_unavailable"
guest_private_mac: str | None = None
guest_private_ip_cidr: str | None = None
guest_link_state: str | None = None
guest_routes: list[str] = []
domain_running = False
link_state: str | None = None
expected_routes = private_guest_routes(plan.vpc, member)
expected_generation = private_guest_network_generation(
member, expected_routes)
if xml_by_instance and member.instance_uuid in xml_by_instance:
value = xml_by_instance[member.instance_uuid]
if isinstance(value, Mapping):
seed_verified = value.get("seed_verified") is True
inactive = value.get("inactive")
live = value.get("live")
inactive_ok = False
live_ok = True
live_device_ok = False
live_generation_ok = False
inactive_link: str | None = None
live_link: str | None = None
if isinstance(inactive, str):
inactive_ok, inactive_link = xml_details(
inactive, member, metadata=True)
if live is not None:
(inactive_device_ok, inactive_generation_ok,
inactive_metadata_ok, inactive_link) = xml_details(
inactive, member, expected_generation)
inactive_ok = bool(
inactive_device_ok and inactive_generation_ok
and inactive_metadata_ok)
config_verified = inactive_ok
domain_running = live is not None
if domain_running:
if isinstance(live, str):
live_ok, live_link = xml_details(
live, member, metadata=False)
else:
live_ok = False
# A running domain is only healthy when the persistent and
# live role agree. This makes a stale config-only link state
# visible to the lifecycle gate instead of silently choosing
# whichever XML happened to be read first.
verified = inactive_ok and live_ok and (
live is None or inactive_link == live_link)
link_state = live_link if live is not None else inactive_link
(live_device_ok, live_generation_ok,
_live_metadata_ok, live_link) = xml_details(
live, member, expected_generation)
guest = value.get("guest_network")
if isinstance(guest, Mapping):
guest_readback = str(guest.get("state", "missing"))[:32]
reason = guest.get("reason_code")
if isinstance(reason, str) and re.fullmatch(
r"[a-z0-9_]{1,64}", reason):
guest_reason_code = reason
observed_routes = guest.get("routes")
if (isinstance(observed_routes, list)
and len(observed_routes) <= 4096
and all(isinstance(route, str)
for route in observed_routes)):
guest_routes = sorted(set(observed_routes))
observed_mac = guest.get("private_mac")
observed_ip = guest.get("private_ip_cidr")
guest_private_mac = (
observed_mac.lower()
if isinstance(observed_mac, str) else None)
guest_private_ip_cidr = (
observed_ip if isinstance(observed_ip, str) else None)
guest_link_state = (
guest.get("link_state")
if guest.get("link_state") in {"up", "down"}
else None)
guest_verified = bool(
guest.get("verified") is True
and guest_readback == "confirmed"
and guest.get("source") == "qemu_guest_agent"
and guest_link_state == "up"
and guest_private_mac == member.private_mac
and guest_private_ip_cidr == f"{member.private_ip}/32"
and guest_routes == list(expected_routes)
)
if guest_verified:
guest_reason_code = None
# Persistent and live libvirt device identity remains a
# prerequisite, but only a bounded guest-agent observation can
# prove that the address and routes exist inside the guest.
stopped_exact = bool(
inactive_ok and seed_verified and not domain_running)
verified = bool(
stopped_exact
or (inactive_ok and seed_verified and domain_running
and live_device_ok
and inactive_link == live_link))
restart_required = bool(
not stopped_exact and (
not domain_running
or not inactive_ok
or not seed_verified
or not live_generation_ok))
if stopped_exact:
guest_readback = "stopped"
guest_reason_code = "vpc_private_guest_stopped"
if restart_required:
guest_reason_code = "vpc_private_guest_restart_required"
link_state = live_link if domain_running else inactive_link
elif isinstance(value, str):
verified, link_state = xml_details(value, member, metadata=True)
(device_ok, generation_ok, metadata_ok,
link_state) = xml_details(
value, member, expected_generation)
verified = device_ok and generation_ok and metadata_ok
config_verified = verified
result.append({"member_id": member.member_id, "instance_uuid": member.instance_uuid,
"interface": PRIVATE_INTERFACE, "private_ip_cidr": f"{member.private_ip}/32",
"private_mac": member.private_mac, "xml_verified": verified,
"config_verified": config_verified,
"seed_verified": seed_verified,
"restart_required": restart_required,
"domain_running": domain_running,
"guest_network_verified": guest_verified,
"guest_network_readback": guest_readback,
"guest_reason_code": guest_reason_code,
"guest_private_mac": guest_private_mac,
"guest_private_ip_cidr": guest_private_ip_cidr,
"guest_link_state": guest_link_state,
"guest_routes": guest_routes,
"link_state": link_state})
return tuple(result)
@ -2773,9 +2961,16 @@ def _member_nics(
class VpcPrivateHostAdapter:
"""Execute generated scripts through a pinned remote host manager."""
def __init__(self, manager: Any, profile: Mapping[str, Any] | None = None):
def __init__(
self,
manager: Any,
profile: Mapping[str, Any] | None = None,
*,
request_deadline: float | None = None,
):
self.manager = manager
self.profile = dict(profile) if isinstance(profile, Mapping) else None
self.request_deadline = request_deadline
def _run(self, script: str) -> str:
current = self.profile if self.profile is not None else self.manager.current()
@ -2783,9 +2978,18 @@ class VpcPrivateHostAdapter:
raise VpcDataplaneError("宿主机尚未配置", 409, "vpc_host_not_configured")
if current.get("key_locked"):
raise VpcDataplaneError("宿主机 SSH 私钥仍锁定", 409, "vpc_host_key_locked")
timeout = 35
if self.request_deadline is not None:
remaining = self.request_deadline - time.monotonic()
if remaining < 1:
raise VpcDataplaneError(
"VPC 来宾网络核验超过请求预算", 502,
"vpc_private_guest_probe_deadline_exceeded")
timeout = min(timeout, int(remaining))
argv = [getattr(self.manager, "ssh_binary", "ssh"), "-F", current["ssh_config_file"], "-T", "-o", "ConnectTimeout=10", current["ssh_target"], "/bin/sh -s"]
try:
result = self.manager._exec(argv, timeout=35, input_text=script)
result = self.manager._exec(
argv, timeout=timeout, input_text=script)
except Exception as exc:
raise VpcDataplaneError("宿主机 VPC SSH 请求失败", 502, "vpc_host_connection_failed") from exc
if result.returncode != 0:
@ -2866,6 +3070,9 @@ class VpcPrivateHostAdapter:
"observed_revision": plan.desired_revision,
"desired_digest": plan.desired_digest,
"observed_digest": plan.desired_digest,
"config_verified": True,
"config_observed_revision": plan.desired_revision,
"config_observed_digest": plan.desired_digest,
"state": "active",
"removed": True,
"bridge": None,
@ -2960,7 +3167,7 @@ class VpcPrivateHostAdapter:
}
expected_peer_states = {p.host_id: "owned" for p in plan.peers}
expected_bridge_nft = "yes" if plan.local_members else "none"
active = (parsed.get("bridge") == plan.resources.bridge
host_config_verified = (parsed.get("bridge") == plan.resources.bridge
and parsed.get("bridge_owner") == DATAPLANE_OWNER_MARKER
and parsed.get("persistence") == "yes"
and parsed.get("bridge_boundary") == "yes"
@ -2987,7 +3194,22 @@ class VpcPrivateHostAdapter:
or parsed.get("wg_peer_set") == "exact")
and expected_peer_states == parsed.get("wg_states", {})
and parsed.get("arp_nft") == ("yes" if plan.local_members else "none")
and all(x["xml_verified"] for x in members))
and all(x.get("config_verified") is True for x in members))
active = bool(
host_config_verified
and all(
x["xml_verified"]
and x.get("guest_network_verified") is True
for x in members
)
)
restart_required = any(
x.get("restart_required") is True for x in members)
guest_reason_code = next((
x.get("guest_reason_code") for x in members
if x.get("guest_network_verified") is not True
and isinstance(x.get("guest_reason_code"), str)
), None)
projected_wg = tuple(
dict(item, tenant_allowed_ips=sorted(
expected_wg[item.get("peer_host_id")][1]
@ -3005,15 +3227,26 @@ class VpcPrivateHostAdapter:
plan.vpc.vpc_id, plan.host_id, plan.desired_revision,
plan.desired_revision if active else 0, plan.desired_digest,
plan.desired_digest if active else None,
"active" if active else "degraded", plan.resources.bridge,
"active" if active else (
"applying" if restart_required or guest_reason_code else "degraded"),
plan.resources.bridge,
parsed.get("bridge_owner"), members, tuple(parsed.get("routes", [])),
projected_wg, parsed.get("nft_table"),
plan.desired_revision if active else None,
parsed.get("anti_spoof") == "yes", rp_filter, proxy_arp,
proxy_arp_pvlan,
tuple(sorted(actual_proxy_ips)), "confirmed" if active else "mismatch",
"none", None if active else "vpc_private_dataplane_readback_mismatch",
tuple(sorted(actual_proxy_ips)),
host_config_verified,
plan.desired_revision if host_config_verified else 0,
plan.desired_digest if host_config_verified else None,
"confirmed" if active else (
"pending_restart" if restart_required
else "pending_guest" if guest_reason_code else "mismatch"),
"none", None if active else (
"vpc_private_guest_restart_required" if restart_required
else guest_reason_code
or "vpc_private_dataplane_readback_mismatch"),
).as_dict()
__all__ = ["DATAPLANE_OWNER_MARKER", "FABRIC_INTERFACE", "PRIVATE_INTERFACE", "VpcDataplaneError", "VpcFabricPeer", "VpcDataplaneResources", "VpcDataplanePlan", "VpcDataplaneObservation", "derive_private_mac", "resource_names", "parse_vpc_dataplane_plan", "render_vpc_nft_rules", "render_apply_script", "render_remove_script", "render_observe_script", "patch_private_interface_xml", "VpcPrivateHostAdapter"]
__all__ = ["DATAPLANE_OWNER_MARKER", "FABRIC_INTERFACE", "PRIVATE_INTERFACE", "VpcDataplaneError", "VpcFabricPeer", "VpcDataplaneResources", "VpcDataplanePlan", "VpcDataplaneObservation", "derive_private_mac", "private_guest_routes", "private_guest_network_generation", "resource_names", "parse_vpc_dataplane_plan", "render_vpc_nft_rules", "render_apply_script", "render_remove_script", "render_observe_script", "patch_private_interface_xml", "VpcPrivateHostAdapter"]

@ -13,6 +13,7 @@ final class CloudVpcOperation extends Model
'vpc_id' => 'int',
'user_id' => 'int',
'desired_revision' => 'int',
'evidence_version' => 'int',
'created_at' => 'int',
'updated_at' => 'int',
'finished_at' => 'int',

@ -124,6 +124,21 @@ final class CloudOperationErrorCatalog
'vpc_private_observation_missing' => '实例私网状态尚未完成回读,请稍后刷新,切勿重复提交',
'vpc_private_observation_mismatch' => '实例私网状态与当前 VPC 目标不一致,正在核对',
'vpc_private_dataplane_unverified' => 'VPC 私网数据面尚未完成核验,请稍后刷新',
'vpc_private_guest_restart_required' => '私网配置尚未写入当前实例;请完全关机,等待系统核对写入后再开机',
'vpc_private_guest_agent_unavailable' => 'QEMU Guest Agent 暂不可用,无法确认来宾私网地址和路由',
'vpc_private_guest_probe_unsupported' => '来宾缺少受支持的网络探测能力,无法确认私网地址和路由',
'vpc_private_guest_probe_uncertain' => '来宾私网探测结果不完整,正在重新核验',
'vpc_private_guest_probe_deadline_exceeded' => '来宾私网探测超时,将继续后台核验',
'vpc_private_guest_probe_reap_required' => '来宾私网探测清理状态未确认;请完全关机再开机后继续核验',
'vpc_private_guest_address_mismatch' => '来宾私网地址与当前 VPC 目标不一致,正在核对',
'vpc_private_guest_route_mismatch' => '来宾私网路由与当前 VPC 目标不一致,正在核对',
'vpc_private_guest_link_mismatch' => '来宾私网网卡未处于可用状态,正在核对',
'vpc_private_network_suspended' => '实例网络已按策略暂停,私网当前不可用',
'vpc_private_guest_stopped' => '实例已关机;私网配置已核对,但运行态网络不可用',
'vpc_private_guest_shutdown_pending' => '实例正在完成正常关机;私网运行态暂不可用',
'vpc_private_guest_readback_unavailable' => 'QEMU Guest Agent 暂不可用,无法确认来宾私网地址和路由',
'vpc_private_guest_network_mismatch' => '来宾私网地址或路由与当前 VPC 目标不一致,正在核对',
'vpc_observation_contract_outdated' => 'VPC 观测证据版本过旧,正在重新核验来宾网络',
'vpc_private_teardown_uncertain' => '实例私网清理状态尚未确认,请勿重复提交',
'vpc_private_expectation_invalid' => '实例私网目标记录无效,请联系管理员核对',
'vpc_private_expectation_unavailable' => '实例私网目标暂时无法读取,请稍后核对',
@ -328,6 +343,18 @@ final class CloudOperationErrorCatalog
'vpc_private_observation_missing', 'vpc_private_observation_mismatch',
'vpc_private_teardown_uncertain', 'vpc_private_expectation_invalid',
'vpc_private_expectation_unavailable', 'vpc_private_dataplane_unverified',
'vpc_private_guest_restart_required',
'vpc_private_guest_agent_unavailable',
'vpc_private_guest_probe_unsupported', 'vpc_private_guest_probe_uncertain',
'vpc_private_guest_probe_deadline_exceeded',
'vpc_private_guest_probe_reap_required',
'vpc_private_guest_address_mismatch', 'vpc_private_guest_route_mismatch',
'vpc_private_guest_link_mismatch',
'vpc_private_network_suspended',
'vpc_private_guest_stopped',
'vpc_private_guest_shutdown_pending',
'vpc_private_guest_readback_unavailable',
'vpc_private_guest_network_mismatch', 'vpc_observation_contract_outdated',
], true)) {
return $engineCode;
}
@ -588,7 +615,7 @@ final class CloudOperationErrorCatalog
$result[$key] = is_string($item) && $item !== ''
? self::normalize($item) : null;
}
foreach (['state' => ['disabled', 'active', 'suspended', 'removed', 'degraded', 'uncertain'],
foreach (['state' => ['disabled', 'active', 'suspended', 'stopping', 'stopped', 'removed', 'degraded', 'uncertain'],
'desired_state' => ['disabled', 'active', 'removed'],
'readback' => ['confirmed', 'missing', 'mismatch'],
'link_state' => ['up', 'down', 'absent']] as $key => $allowed) {
@ -605,6 +632,20 @@ final class CloudOperationErrorCatalog
$result['desired_revision'] = max(0, (int) ($value['desired_revision'] ?? 0));
$result['observed_revision'] = max(0, (int) ($value['observed_revision'] ?? 0));
$result['observed_at'] = max(0, (int) ($value['observed_at'] ?? 0));
foreach (['xml_verified', 'config_verified', 'seed_verified', 'domain_running',
'guest_network_verified'] as $key) {
$result[$key] = ($value[$key] ?? false) === true;
}
$result['restart_required'] = is_bool($value['restart_required'] ?? null)
? (bool) $value['restart_required'] : null;
$result['guest_network_readback'] = is_string(
$value['guest_network_readback'] ?? null
) && in_array($value['guest_network_readback'], [
'confirmed', 'missing', 'unsupported', 'unavailable',
], true) ? $value['guest_network_readback'] : 'missing';
$result['guest_link_state'] = is_string($value['guest_link_state'] ?? null)
&& in_array($value['guest_link_state'], ['up', 'down'], true)
? $value['guest_link_state'] : null;
foreach (['desired_digest', 'observed_digest'] as $key) {
if ($result[$key] !== null
&& preg_match('/^[a-f0-9]{64}$/Di', $result[$key]) !== 1) {
@ -643,6 +684,18 @@ final class CloudOperationErrorCatalog
'private_source_unverified', 'vpc_private_observation_missing',
'vpc_private_observation_mismatch', 'vpc_private_teardown_uncertain',
'vpc_private_dataplane_unverified',
'vpc_private_guest_restart_required',
'vpc_private_guest_agent_unavailable',
'vpc_private_guest_probe_unsupported', 'vpc_private_guest_probe_uncertain',
'vpc_private_guest_probe_deadline_exceeded',
'vpc_private_guest_probe_reap_required',
'vpc_private_guest_address_mismatch', 'vpc_private_guest_route_mismatch',
'vpc_private_guest_link_mismatch',
'vpc_private_network_suspended',
'vpc_private_guest_stopped',
'vpc_private_guest_shutdown_pending',
'vpc_private_guest_readback_unavailable',
'vpc_private_guest_network_mismatch', 'vpc_observation_contract_outdated',
'vpc_private_expectation_invalid', 'vpc_private_expectation_unavailable',
], true) ? $reason : null;
return $result;

File diff suppressed because it is too large Load Diff

@ -38,13 +38,78 @@ final class CloudVpcWorkerService
})->orWhere(function ($running) use ($now): void {
$running->where('status', 'running')
->where('lease_until', '<=', $now);
})->orWhere(function ($legacy) use ($now): void {
// A rolling upgrade must not keep a previously
// `succeeded` host/libvirt-only receipt usable.
// Re-run only the root's current last operation;
// historical audit rows are never replayed.
$legacy->where('status', 'succeeded')
->where('kind', '!=', 'vpc_delete')
->where('evidence_version', '<',
CloudVpcService::OBSERVATION_EVIDENCE_VERSION)
->where('lease_until', '<=', $now)
->where('updated_at', '<=', $now - 2)
->whereExists(static function ($current): void {
$current->selectRaw('1')
->from('cloud_vpcs as current_vpc')
->whereColumn('current_vpc.id',
'cloud_vpc_operations.vpc_id')
->whereColumn('current_vpc.last_operation_id',
'cloud_vpc_operations.operation_id')
->whereColumn('current_vpc.desired_revision',
'cloud_vpc_operations.desired_revision')
->where('current_vpc.state', '!=', 'deleted');
});
})->orWhere(function ($runtime) use ($now): void {
// Configuration may already be committed while one
// or more live guests are not yet observable. Keep
// the mutation terminal, but idempotently re-apply
// only the current root target until runtime proof
// is exact.
$runtime->where('status', 'succeeded')
->where('kind', '!=', 'vpc_delete')
->where('evidence_version', '>=',
CloudVpcService::OBSERVATION_EVIDENCE_VERSION)
->where('lease_until', '<=', $now)
->where('updated_at', '<=', $now - 30)
->whereExists(static function ($current): void {
$current->selectRaw('1')
->from('cloud_vpcs as current_vpc')
->whereColumn('current_vpc.id',
'cloud_vpc_operations.vpc_id')
->whereColumn('current_vpc.last_operation_id',
'cloud_vpc_operations.operation_id')
->whereColumn('current_vpc.desired_revision',
'cloud_vpc_operations.desired_revision')
->whereColumn('current_vpc.observed_revision',
'current_vpc.desired_revision')
->whereColumn('current_vpc.observed_digest',
'current_vpc.desired_digest')
->where('current_vpc.state', 'degraded')
->where(static function ($runtimeState): void {
$runtimeState->whereIn(
'current_vpc.sync_state',
['waiting', 'uncertain']
)->orWhere(static function ($stopped): void {
$stopped->where(
'current_vpc.sync_state', 'applied'
)->whereIn(
'current_vpc.last_error_code', [
'vpc_private_guest_shutdown_pending',
'vpc_private_guest_stopped',
]
);
});
});
});
});
})
->orderBy('id')->lockForUpdate()->first();
if ($operation === null) {
return null;
}
$allowDeferredRetry = (string) $operation->status === 'waiting_observed';
$allowDeferredRetry = (string) $operation->status === 'waiting_observed'
|| (string) $operation->status === 'succeeded';
$vpc = CloudVpc::where('id', (int) $operation->vpc_id)->first();
if ($vpc === null) {
$operation->status = 'failed';

@ -1190,6 +1190,21 @@ final class CloudWorkerService
$status = is_string($operation['status'] ?? null)
? strtolower((string) $operation['status']) : 'uncertain';
$kind = (string) $job->kind;
$lifecycleKind = $kind;
$payloadBody = is_array($payload['body'] ?? null)
? $payload['body'] : [];
$powerAction = null;
if ($kind === 'action') {
// The generic job kind is not enough to decide whether a stopped
// private-state proof is an expected postcondition. Bind that
// decision to the validated encrypted request payload; never infer
// it from an engine result or from the VM's current state.
$action = is_string($payloadBody['action'] ?? null)
? (string) $payloadBody['action'] : '';
if (CloudOperationCatalog::isPowerAction($action)) {
$powerAction = $action;
}
}
$postCommitReinstall = $kind === 'reinstall'
&& $status === 'failed' && self::reinstallNeedsReview($operation);
$terminalSuccess = in_array($status, ['succeeded', 'completed', 'done'], true);
@ -1208,6 +1223,33 @@ final class CloudWorkerService
}
$instance = $job->instance_id === null ? null : CloudInstance::find($job->instance_id);
// Power operations preserve the independently managed network-policy
// target. Bind private readback to the trusted database state instead
// of treating every successful start/reboot as permission to raise a
// quota- or expiry-blocked link. Transitional/failed policy states are
// deliberately unprovable until their own policy job converges.
$networkPolicy = match ((string) ($instance?->network_status ?? '')) {
'active' => 'active',
'blocked_traffic', 'blocked_expired' => 'blocked',
default => 'unready',
};
if ($powerAction !== null) {
$lifecycleKind = 'action_' . $powerAction
. '_policy_' . $networkPolicy;
} elseif ($kind === 'service_suspend') {
$lifecycleKind = 'service_suspend_policy_' . $networkPolicy;
} elseif ($kind === 'service_resume') {
// The resume endpoint defaults false, but the private-state gate
// accepts a stopped postcondition only when that target was
// explicitly captured in the encrypted job payload. A missing or
// non-boolean target remains fail-closed.
$resumeTarget = array_key_exists('start_if_suspended', $payloadBody)
&& is_bool($payloadBody['start_if_suspended'])
? ($payloadBody['start_if_suspended'] ? 'start' : 'stopped')
: 'unbound';
$lifecycleKind = 'service_resume_' . $resumeTarget
. '_policy_' . $networkPolicy;
}
$currentExpectation = CloudVpcService::instancePrivateExpectation(
$instance === null ? null : (int) $instance->id
);
@ -1342,7 +1384,7 @@ final class CloudWorkerService
'observed_at' => time(),
], 'error_code' => 'vpc_private_observation_mismatch']
: CloudVpcService::validatePrivateObservation(
$expectation, $observation, $kind
$expectation, $observation, $lifecycleKind
);
// A normal private-state GET is intentionally an observation only. A
// lifecycle mutation can have replaced the domain XML (or removed a
@ -1356,15 +1398,18 @@ final class CloudWorkerService
// Only the PHP policy short-circuit carries this private marker.
// An engine result may also expose `skipped` after it has already
// proved a target state; that must retain network_suspend/down or
// network_resume/up semantics rather than weakening the gate.
// network_resume/up semantics rather than weakening the gate. A
// skipped target uses a dedicated preserve validator that accepts
// stopped only with exact persistent configuration proof.
$privateReconcileSkipped = ($operation['_private_policy_skipped'] ?? false) === true;
$reconciled = $this->reconcilePrivateState(
$job, $uuid, $kind, $status, $expectation, $actor,
$job, $uuid, $lifecycleKind, $status, $expectation, $actor,
$privateReconcileSkipped
);
$validation = CloudVpcService::validatePrivateObservation(
$expectation, $reconciled,
$privateReconcileSkipped ? 'action' : $kind, $uuid
$privateReconcileSkipped ? 'policy_skipped' : $lifecycleKind,
$uuid
);
} elseif (($expectation['required'] ?? false) === true
&& $uuid === null && !$expectationStale) {
@ -1372,7 +1417,7 @@ final class CloudWorkerService
// an embedded operation payload. Keep it uncertain until the
// worker can address the host-side reconcile endpoint explicitly.
$validation = CloudVpcService::validatePrivateObservation(
$expectation, null, $kind
$expectation, null, $lifecycleKind
);
}
if ($instance !== null && ($expectation['required'] ?? false) === true

@ -10,7 +10,8 @@ const state = {
const STATUS_LABELS = Object.freeze({
applying: '应用中', waiting: '等待宿主机回读', uncertain: '结果待核对',
degraded: '部分失败', active: '已生效', deleting: '清理中', unknown: '状态未知',
degraded: '部分失败', active: '已生效', suspended: '网络已暂停',
deleting: '清理中', unknown: '状态未知',
});
function element(tag, className = '', textContent = '') {
@ -120,8 +121,8 @@ function rootStatus(root) {
function childStatus(value) {
const raw = String(value || '').toLowerCase();
return ['active', 'applying', 'waiting', 'uncertain', 'degraded', 'deleting', 'deleted']
.includes(raw) ? raw : 'unknown';
return ['active', 'suspended', 'applying', 'waiting', 'uncertain', 'degraded',
'deleting', 'deleted'].includes(raw) ? raw : 'unknown';
}
function statusPill(status, extra = '') {

@ -517,6 +517,7 @@ CREATE TABLE IF NOT EXISTS cloud_vpc_operations (
status VARCHAR(16) CHARACTER SET ascii COLLATE ascii_bin NOT NULL DEFAULT 'queued',
payload_json LONGTEXT NOT NULL,
observed_json LONGTEXT DEFAULT NULL,
evidence_version SMALLINT UNSIGNED NOT NULL DEFAULT 0,
error_code VARCHAR(64) CHARACTER SET ascii COLLATE ascii_bin DEFAULT NULL,
created_at BIGINT UNSIGNED NOT NULL,
updated_at BIGINT UNSIGNED NOT NULL,
@ -538,6 +539,8 @@ ALTER TABLE cloud_vpc_operations ADD COLUMN IF NOT EXISTS
lease_until BIGINT UNSIGNED NOT NULL DEFAULT 0 AFTER lease_token;
ALTER TABLE cloud_vpc_operations ADD COLUMN IF NOT EXISTS
attempts INT UNSIGNED NOT NULL DEFAULT 0 AFTER lease_until;
ALTER TABLE cloud_vpc_operations ADD COLUMN IF NOT EXISTS
evidence_version SMALLINT UNSIGNED NOT NULL DEFAULT 0 AFTER observed_json;
ALTER TABLE cloud_vpc_operations ADD UNIQUE INDEX IF NOT EXISTS
cloud_vpc_operation_idempotency (vpc_id, idempotency_key);

@ -25,6 +25,11 @@ $jobService = $read('app/Services/Cloud/CloudJobService.php');
$orderService = $read('app/Services/Cloud/CloudOrderService.php');
$workerService = $read('app/Services/Cloud/CloudWorkerService.php');
$vpcWorkerService = $read('app/Services/Cloud/CloudVpcWorkerService.php');
$runtimeService = $read('app/Services/Cloud/CloudInstanceRuntimeService.php');
$operationErrorCatalog = $read('app/Services/Cloud/CloudOperationErrorCatalog.php');
$engineVpcService = (string) file_get_contents(
dirname($root) . '/engine/virsh_panel/vpc.py'
);
foreach ([
'cloud_region_fabrics',
@ -43,7 +48,8 @@ $assert(str_contains($schema, 'UNIQUE KEY cloud_vpc_member_instance_global (inst
&& str_contains($schema, 'source_int BIGINT UNSIGNED NOT NULL')
&& str_contains($schema, 'destination_broadcast_int BIGINT UNSIGNED NOT NULL')
&& str_contains($schema, 'desired_digest CHAR(64)')
&& str_contains($schema, 'observed_digest CHAR(64)'),
&& str_contains($schema, 'observed_digest CHAR(64)')
&& str_contains($schema, 'evidence_version SMALLINT UNSIGNED NOT NULL DEFAULT 0'),
'VPC schema lost the single-NIC, idempotency or revision digest guards.');
$assert(str_contains($service, 'lockForUpdate()')
@ -53,10 +59,29 @@ $assert(str_contains($service, 'lockForUpdate()')
&& str_contains($service, 'CloudVpcPrivateIp')
&& str_contains($service, 'lockedPrivateIpForMember')
&& str_contains($service, 'CloudVpcHostDeployment')
&& str_contains($service, 'CloudEngineClient())->vpc')
&& str_contains($service, 'new CloudEngineClient(')
&& str_contains($service, '))->vpc(')
&& str_contains($service, 'canonicalJson')
&& str_contains($service, 'next_stage_allowed'),
'VPC service lost independent locking, desired/observed revisions or Fabric gate.');
$assert(str_contains($service, 'public const ENGINE_TIMEOUT_SECONDS = 60.0;')
&& str_contains($service, 'public const OPERATION_LEASE_SECONDS = 120;')
&& str_contains($engineVpcService, 'VPC_REQUEST_DEADLINE_SECONDS = 40.0')
&& str_contains($service, 'self::ENGINE_TIMEOUT_SECONDS')
&& str_contains($service, 'self::OPERATION_LEASE_SECONDS'),
'VPC PHP timeout/lease no longer contain the engine request-wide probe budget.');
preg_match('/VPC_REQUEST_DEADLINE_SECONDS\s*=\s*([0-9.]+)/',
$engineVpcService, $engineBudgetMatch);
$engineBudget = isset($engineBudgetMatch[1]) ? (float) $engineBudgetMatch[1] : 0.0;
preg_match('/ENGINE_TIMEOUT_SECONDS\s*=\s*([0-9.]+)/',
$service, $phpTimeoutMatch);
preg_match('/OPERATION_LEASE_SECONDS\s*=\s*([0-9]+)/',
$service, $phpLeaseMatch);
$phpTimeout = isset($phpTimeoutMatch[1]) ? (float) $phpTimeoutMatch[1] : 0.0;
$phpLease = isset($phpLeaseMatch[1]) ? (int) $phpLeaseMatch[1] : 0;
$assert($engineBudget > 0.0 && $phpTimeout > $engineBudget
&& $phpLease >= $phpTimeout + 30.0,
'VPC PHP timeout/lease numeric ordering no longer bounds the engine probe.');
$assert(str_contains($service, 'CloudRegionFabric')
&& str_contains($service, 'fabric_region')
&& str_contains($adminResource, 'saveRegionFabric')
@ -185,6 +210,23 @@ $assert(str_contains($service, 'privateMacForMember')
&& str_contains($service, "'private_mac' => \$privateMac")
&& str_contains($service, 'validateHostDataplaneEvidence')
&& str_contains($service, 'member_nics')
&& str_contains($service, 'restart_required')
&& str_contains($service, 'guest_network_verified')
&& str_contains($service, '($nic[\'link_state\'] ?? null) !== \'up\'')
&& str_contains($service, 'vpc_private_guest_link_mismatch')
&& str_contains($service, 'vpc_private_guest_probe_deadline_exceeded')
&& str_contains($service, 'vpc_private_guest_probe_reap_required')
&& str_contains($service, 'guest_private_ip_cidr')
&& str_contains($service, 'guest_routes')
&& str_contains($service, 'config_observed_digest')
&& str_contains($service, 'OBSERVATION_EVIDENCE_VERSION')
&& str_contains($service, "?->network_status ?? '') !== 'active'")
&& str_contains($service, 'CloudInstanceRuntimeService::forInstances')
&& str_contains($service, "['state_fresh'] ?? false")
&& str_contains($service, "['power_state'] ?? '') !== 'running'")
&& str_contains($service, "'state' => \$intentionalStopped ? 'stopped'")
&& str_contains($service, "'vpc_private_guest_stopped'")
&& str_contains($service, "'vpc_private_guest_shutdown_pending'")
&& str_contains($service, 'wg_allowed_ips')
&& str_contains($service, 'privateRouteTableForVpc')
&& str_contains($service, 'expectedRouteSet')
@ -194,10 +236,28 @@ $assert(str_contains($service, 'privateMacForMember')
&& str_contains($service, 'FABRIC_INTERFACE')
&& str_contains($service, 'PRIVATE_DATAPLANE_OWNER'),
'VPC desired DTO or host dataplane readback contract lost stable NIC identity/ACL gates.');
$assert(str_contains($runtimeService, "'shutdown' => 'shutdown'")
&& str_contains($runtimeService, "'poweroff' => 'stopped'")
&& str_contains($runtimeService, "'start', 'reboot', 'reset' => 'running'"),
'Runtime cache lost bounded power-action transition semantics.');
$assert(str_contains(
$operationErrorCatalog,
'私网配置尚未写入当前实例;请完全关机,等待系统核对写入后再开机'
) && str_contains(
$operationErrorCatalog,
'来宾私网探测清理状态未确认;请完全关机再开机后继续核验'
) && !str_contains($operationErrorCatalog, '私网地址或路由已暂存'),
'VPC restart guidance must not claim the pending seed/XML was already staged.');
$assert(str_contains($vpcWorkerService, "'waiting_observed'")
&& str_contains($vpcWorkerService, 'allowDeferredRetry')
&& str_contains($vpcWorkerService, "where('status', 'succeeded')")
&& str_contains($vpcWorkerService, 'current_vpc.last_operation_id')
&& str_contains($vpcWorkerService, "where('kind', '!=', 'vpc_delete')")
&& str_contains($vpcWorkerService, 'current_vpc.sync_state')
&& str_contains($vpcWorkerService, 'vpc_private_guest_shutdown_pending')
&& str_contains($vpcWorkerService, 'OBSERVATION_EVIDENCE_VERSION')
&& str_contains($vpcWorkerService, 'CloudVpcService::reconcile($vpcKey, $actor, $allowDeferredRetry)'),
'VPC worker must retry waiting_observed through its independent apply lane.');
'VPC worker must retry waiting or legacy current evidence through its independent apply lane.');
$assert(str_contains($routes, "'/vpcs'")
&& str_contains($routes, "'/vpcs/{vpc_key:[0-9a-f]{32}}/reconcile'")
@ -257,10 +317,20 @@ $assert(str_contains($workerService, "'_private_policy_skipped' => true")
&& str_contains($workerService, 'Only the PHP policy short-circuit carries this private marker')
&& str_contains($workerService, "'link_state' => self::privateReconcileLinkState"),
'A superseded PHP policy must preserve the private link without weakening real engine target proof.');
$assert(str_contains($workerService, "'blocked_traffic', 'blocked_expired' => 'blocked'")
&& str_contains($workerService, "'_policy_' . \$networkPolicy")
&& str_contains($workerService, "'start_if_suspended'")
&& str_contains($workerService, "? 'start' : 'stopped'")
&& str_contains($service, "service_resume_stopped_policy_")
&& str_contains($service, "service_resume_start_policy_")
&& str_contains($service, "action_reboot_policy_")
&& str_contains($service, '$policyUnready || $unboundResume'),
'Power/service private readback is no longer bound to network policy and payload target.');
$assert(str_contains($service, '?string $expectedVmUuid = null')
&& str_contains($service, '$observedVmUuid')
&& str_contains($service, 'hash_equals(strtolower($expectedVmUuid)')
&& str_contains($workerService, "? 'action' : \$kind, \$uuid"),
&& str_contains($workerService, "? 'policy_skipped' : \$lifecycleKind")
&& str_contains($service, "\$preserveTarget = \$kind === 'policy_skipped'"),
'Private observation validation must enforce the lifecycle VM UUID binding.');
$assert(str_contains($workerService, "'private_mac', 'desired_revision'")
&& str_contains($service, "'private_mac', 'desired_revision'"),

@ -80,6 +80,20 @@ $schema->create('cloud_instances', static function (Blueprint $table): void {
$table->unsignedBigInteger('created_at');
$table->unsignedBigInteger('updated_at');
});
$schema->create('cloud_instance_runtime', static function (Blueprint $table): void {
$table->unsignedInteger('instance_id')->primary();
$table->unsignedInteger('host_id');
$table->string('uuid', 36);
$table->string('power_state', 16)->default('unknown');
$table->string('sync_state', 16)->default('never');
$table->unsignedBigInteger('last_success_at')->default(0);
$table->unsignedBigInteger('last_attempt_at')->default(0);
$table->unsignedBigInteger('revision')->default(0);
$table->string('last_error_code', 64)->default('');
$table->unsignedBigInteger('created_at');
$table->unsignedBigInteger('updated_at');
$table->unique(['host_id', 'uuid']);
});
$schema->create('cloud_vpcs', static function (Blueprint $table): void {
$table->increments('id');
$table->string('vpc_key', 32)->unique();
@ -234,6 +248,7 @@ $schema->create('cloud_vpc_operations', static function (Blueprint $table): void
$table->string('status', 24);
$table->text('payload_json');
$table->text('observed_json')->nullable();
$table->unsignedSmallInteger('evidence_version')->default(0);
$table->string('error_code', 64)->nullable();
$table->unsignedBigInteger('created_at');
$table->unsignedBigInteger('updated_at');
@ -296,7 +311,7 @@ $routeTableMethod = new ReflectionMethod(CloudVpcService::class, 'privateRouteTa
$privateMacMethod = new ReflectionMethod(CloudVpcService::class, 'privateMacForMember');
$activeHostEvidence = static function (
string $vpcKey, int $hostId, int $revision, string $digest,
array $members = [], array $desiredRoutes = []
string $vpcCidr, array $members = [], array $desiredRoutes = []
) use ($bridgeMethod, $nftMethod, $routeTableMethod): array {
$bridge = $bridgeMethod->invoke(null, $vpcKey);
$routeTable = $routeTableMethod->invoke(null, $vpcKey);
@ -316,7 +331,16 @@ $activeHostEvidence = static function (
'interface' => 'private0',
'private_ip_cidr' => (string) $member['private_ip_cidr'],
'private_mac' => (string) $member['private_mac'],
'xml_verified' => true,
'xml_verified' => true, 'config_verified' => true,
'seed_verified' => true,
'restart_required' => false,
'domain_running' => true, 'guest_network_verified' => true,
'guest_network_readback' => 'confirmed',
'guest_reason_code' => null,
'guest_private_mac' => (string) $member['private_mac'],
'guest_private_ip_cidr' => (string) $member['private_ip_cidr'],
'link_state' => 'up', 'guest_link_state' => 'up',
'guest_routes' => [$vpcCidr],
];
} else {
$remoteIpsByHost[(string) $member['host_id']][] =
@ -367,6 +391,8 @@ $activeHostEvidence = static function (
'observed_revision' => $revision, 'observed_digest' => $digest,
'state' => 'active', 'transport_verified' => true,
'private_dataplane_verified' => true, 'removed' => false,
'config_verified' => true, 'config_observed_revision' => $revision,
'config_observed_digest' => $digest,
'bridge' => $bridge,
'bridge_owner' => 'virt-cloud-vpc-dataplane-v1',
'nft_table' => $nftMethod->invoke(null, $vpcKey),
@ -391,6 +417,13 @@ $seedInstance = static function (int $id, int $hostId, int $userId = 9) use (
'service_status' => 'exempt', 'network_status' => 'active',
'expires_at' => null, 'created_at' => $now, 'updated_at' => $now,
]);
Capsule::table('cloud_instance_runtime')->insert([
'instance_id' => $id, 'host_id' => $hostId, 'uuid' => $uuidFor($id),
'power_state' => 'running', 'sync_state' => 'ok',
'last_success_at' => $now, 'last_attempt_at' => $now,
'revision' => 1, 'last_error_code' => '',
'created_at' => $now, 'updated_at' => $now,
]);
};
$seedVpc = static function (
int $id, string $cidr, string $subnetCidr, int $hostId,
@ -437,10 +470,12 @@ $seedVpc = static function (
$payload['desired_digest'] = $digest;
$payload['operation_id'] = $operationId;
$hostEvidence = $activeHostEvidence(
$vpcKey, $hostId, 1, $digest, $memberRows, $desiredRoutes
$vpcKey, $hostId, 1, $digest, $networkIp . '/24',
$memberRows, $desiredRoutes
);
if ($proofMode === 'deleted') {
$observed = [
'evidence_version' => CloudVpcService::OBSERVATION_EVIDENCE_VERSION,
'revision' => 0, 'digest' => null, 'state' => 'degraded',
'next_stage_allowed' => false, 'private_dataplane_verified' => false,
'fabric_gate' => ['active' => false, 'error_code' => 'vpc_no_members'],
@ -455,6 +490,7 @@ $seedVpc = static function (
unset($hostEvidence['nft_table']);
}
$observed = [
'evidence_version' => CloudVpcService::OBSERVATION_EVIDENCE_VERSION,
'revision' => 1, 'digest' => $digest, 'state' => 'active',
'next_stage_allowed' => true,
'private_dataplane_verified' => $proofMode === 'active',
@ -508,6 +544,7 @@ $seedVpc = static function (
'observed_json' => json_encode(
$observed, JSON_THROW_ON_ERROR | JSON_UNESCAPED_SLASHES
),
'evidence_version' => CloudVpcService::OBSERVATION_EVIDENCE_VERSION,
'error_code' => null, 'created_at' => $now, 'updated_at' => $now,
'finished_at' => $now, 'lease_token' => null, 'lease_until' => 0,
'attempts' => 1,
@ -635,10 +672,11 @@ $installCurrentProof = static function (
foreach ($observedHostIds as $hostId) {
$observedHosts[(string) $hostId] = $activeHostEvidence(
(string) $vpc['vpc_key'], (int) $hostId,
$revision, $digest, $memberRows, []
$revision, $digest, (string) $root->cidr, $memberRows, []
);
}
$observed = [
'evidence_version' => CloudVpcService::OBSERVATION_EVIDENCE_VERSION,
'revision' => $revision, 'digest' => $digest, 'state' => 'active',
'next_stage_allowed' => true, 'private_dataplane_verified' => true,
'fabric_gate' => ['active' => true], 'hosts' => $observedHosts,
@ -656,6 +694,7 @@ $installCurrentProof = static function (
'observed_json' => json_encode(
$observed, JSON_THROW_ON_ERROR | JSON_UNESCAPED_SLASHES
),
'evidence_version' => CloudVpcService::OBSERVATION_EVIDENCE_VERSION,
'error_code' => null, 'created_at' => $currentTime,
'updated_at' => $currentTime, 'finished_at' => $currentTime,
'lease_token' => null, 'lease_until' => 0, 'attempts' => 1,
@ -733,6 +772,7 @@ $installRemovalProof = static function (
'proxy_ips' => [],
];
$observed = [
'evidence_version' => CloudVpcService::OBSERVATION_EVIDENCE_VERSION,
'revision' => $revision, 'digest' => $digest, 'state' => 'degraded',
'next_stage_allowed' => true, 'private_dataplane_verified' => true,
'fabric_gate' => ['active' => true],
@ -751,6 +791,7 @@ $installRemovalProof = static function (
'observed_json' => json_encode(
$observed, JSON_THROW_ON_ERROR | JSON_UNESCAPED_SLASHES
),
'evidence_version' => CloudVpcService::OBSERVATION_EVIDENCE_VERSION,
'error_code' => null, 'created_at' => $proofTime,
'updated_at' => $proofTime, 'finished_at' => $proofTime,
'lease_token' => null, 'lease_until' => 0, 'attempts' => 1,
@ -816,14 +857,44 @@ $boundedHosts = [];
for ($index = 1; $index <= 257; $index++) {
$boundedHosts[(string) $index] = [
'host_id' => (string) $index,
'config_verified' => true, 'config_observed_revision' => 9,
'config_observed_digest' => str_repeat('a', 64),
'member_nics' => $index === 1 ? [[
'member_id' => str_repeat('b', 32),
'instance_uuid' => 'bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb',
'interface' => 'private0', 'private_ip_cidr' => '10.200.0.1/32',
'private_mac' => '02:00:00:00:00:01',
'xml_verified' => true, 'config_verified' => true,
'seed_verified' => true,
'restart_required' => false, 'domain_running' => true,
'guest_network_verified' => true,
'guest_network_readback' => 'confirmed',
'guest_reason_code' => null,
'guest_private_mac' => '02:00:00:00:00:01',
'guest_private_ip_cidr' => '10.200.0.1/32',
'link_state' => 'up', 'guest_link_state' => 'up',
'guest_routes' => ['10.200.0.0/24'],
]] : [],
'routes' => $index === 1 ? $boundedRoutes : [],
'wg_allowed_ips' => $index === 1 ? $boundedPeers : [],
];
}
$boundedObserved = $boundedObservedMethod->invoke(null, ['hosts' => $boundedHosts]);
$boundedObserved = $boundedObservedMethod->invoke(null, [
'evidence_version' => CloudVpcService::OBSERVATION_EVIDENCE_VERSION,
'hosts' => $boundedHosts,
]);
$assert(count($boundedObserved['hosts'] ?? []) === 257
&& count($boundedObserved['hosts']['1']['routes'] ?? []) === 257
&& count($boundedObserved['hosts']['1']['wg_allowed_ips'] ?? []) === 257,
&& count($boundedObserved['hosts']['1']['wg_allowed_ips'] ?? []) === 257
&& ($boundedObserved['hosts']['1']['config_verified'] ?? false) === true
&& ($boundedObserved['hosts']['1']['member_nics'][0]['guest_network_verified']
?? false) === true
&& ($boundedObserved['hosts']['1']['member_nics'][0]['link_state'] ?? null)
=== 'up'
&& ($boundedObserved['hosts']['1']['member_nics'][0]['guest_routes'] ?? [])
=== ['10.200.0.0/24']
&& ($boundedObserved['evidence_version'] ?? 0)
=== CloudVpcService::OBSERVATION_EVIDENCE_VERSION,
'Observed proof persistence still truncates hosts, routes or peers at 256.');
// A fully proven historical pair must release both uniqueness keys before a

@ -157,6 +157,7 @@ $schema->create('cloud_vpc_operations', static function (Blueprint $table): void
$table->string('status', 16);
$table->text('payload_json');
$table->text('observed_json')->nullable();
$table->unsignedSmallInteger('evidence_version')->default(0);
$table->string('error_code', 64)->nullable();
$table->string('lease_token', 32)->nullable();
$table->unsignedBigInteger('lease_until')->default(0);

File diff suppressed because it is too large Load Diff

@ -126,7 +126,8 @@ $assert(str_contains($script, 'X-CSRF-Token')
&& str_contains($script, 'rule_key: row.dataset.aclKey'),
'VPC mutations must use same-origin CSRF protection and separate idempotency/entity keys.');
foreach (['applying', 'degraded', 'uncertain', 'waiting', 'active', 'deleting'] as $status) {
foreach (['applying', 'degraded', 'uncertain', 'waiting', 'active', 'suspended',
'deleting'] as $status) {
$assert(str_contains($script, $status),
'VPC UI lost state projection for ' . $status . '.');
}

Loading…
Cancel
Save