fix(vpc): recover failed cross-host attachments

venix
admin 1 month ago
parent 9fb2624668
commit 7fe16eef4d

@ -1,5 +1,53 @@
# 本版验证记录
## 2026-10-14:新 VPC 加入回滚与原 operation 显式恢复
- 现场固定 operation `708fa7c660383d7ee945d1543de1e0a8`、VPC
`5518137d9479fc40bf153c26a3f4c350` 的 revision 3 在 AlmaLinux 宿主完成桥、
nft、路由和策略规则后,于 `fabric_acl/vpc_state_load` 退出并整体回滚;目标实例
的 live/persistent XML 因而仍只有公网网卡。Debian 当时没有 Alma 上相同的历史
state 组合,所以结果不同;两台宿主使用的是同一 Engine/helper 方法,差异来自
持久状态历史,不是发行版分支脚本。
- 根因是旧 state 把完整 Fabric peer 聚合复制到每个 VPC。旧、新 VPC 同时保存
同一公钥和 `/32` 时,loader 将合法重复误判成冲突。新 state 只保存本 VPC 中
有本 state `wg-fabric0` route 支撑的 peer `/32`;loader 可读取旧聚合候选,
但最终只保留 route-backed 唯一归属。相同地址映射到不同公钥、单文件重复、
route/peer 不成对和越界 route 仍拒绝。
- apply、observe 和 retiring-host remove 均从完整 Fabric 计划取得权威
`public key -> overlay /32` 映射,再与宿主全部有效 state 的 route-backed tenant
union 合并。每个 peer 的实时 AllowedIPs、完整公钥集合和 overlay 映射均 exact
回读;撤销最后一条 route 后遗留的 tenant `/32` 会使回读失败并在应用中被收敛。
helper 契约升级为 `vpc-state-route-scope-contract-v4`,旧 v3 helper 在任何 VPC
mutation 前 fail closed,须先通过原 Fabric 的显式核对发布并完整回读 v4。
- VPC HTTP/周期 root 现在先绑定完整 operation、VPC、revision 和 digest。
仅签名 actor 中的真布尔 `vpc_reconcile_explicit` 可触发一次性接管;Engine 只在
原 uncertain root 为唯一、无资源、无任何同 execution child、目标与持久 VPC
或删除回执完全一致时,以单次 CAS 删除旧 root、绑定当前 request 并写有界回执。
marker 在绑定时被消费,不进入 child。worker、periodic、普通重试、旧目标、多个
候选、残留 child 和混合 CAS 状态均不接管;claim/release 回包丢失与取消按 exact
snapshot 分类并保留 fence,禁止把未知写结果冒充成功。
- 宿主 apply journal 仅写白名单阶段、状态和退出码,不记录命令、密钥或配置内容。
Mac 本机使用 `engine/.venv` 的 Python 3.12.14、pytest 9.1.1 和项目
`requirements.txt`;18 个 VPC/Fabric/协调相关模块统一回归 **1,234 passed**,
保留 39 条既有 XML truthiness 弃用警告。两位独立审查分别复核 operation CAS、
完整 peer/overlay 闭集和退出宿主路径;Python 3.9/3.12 compile、apply/observe/remove
生成脚本 `/bin/sh -n` 及 `git diff --check` 通过。
- 完整 Engine 结果为 **3,975 passed、24 skipped、118 subtests passed、6 failed**;
六个失败均已逐项在未修改的 `9fb2624` 隔离副本中同样复现,且三个公网池用例在
独立 pytest 进程中仍失败,不是本轮状态泄漏或测试顺序污染。其中三项属于旧测试
漂移:cloud template 创建规格新增 `platform_network_policy_snapshot` 后旧字段集合
断言未更新;reinstall cleanup 的 409 测试仍读取已删除的 `detail` 字段;Fabric
全局权限测试的 readiness fixture 缺当前必需能力证据而得到 `vpc_ready=false`。
另三项揭示既有公网池缺陷:分配器可能重复选择 libvirt 可见 VM 已占用的
`8.8.8.4`,且 `/28` 池耗尽后两个参数组合仍返回 201 而不是 409。公网池缺陷会
影响新实例公网地址分配,应单独修复;它们不进入本次已有实例的 VPC 私网加入路径,
六项均不是现场 rollback 的触发点。因此本轮不宣称全项目测试全绿,真实
helper/Fabric 回读仍是发布门禁。
- 本节当前记录源码与本地回归,尚未提交、部署或重放网络写请求。部署后先显式核对
原 Fabric 并确认两台宿主 v4/helper、peer/overlay、route 和 nft exact;随后只对
原 VPC operation 执行一次显式“重新核对”。只有宿主 state、WireGuard、nft、
libvirt XML/seed、QGA `/32` 和真实跨宿主流量全部通过,才能记录现场恢复。
## 2026-10-14:Fabric 旧未决占用阻断显式核对
- 现场已部署 `d50e6d5`,Engine 于 `08:47:49 UTC` 重启。`08:48:06 UTC`

@ -65,12 +65,17 @@
每份受管 state 的 `/32` route 必须在完整解析该文件后,仅按该文件唯一的 CIDR
校验;不能把累计的全宿主路由重新按最后一个 VPC 的 CIDR 校验,也不能按“属于
任一 VPC 网段”放行。验证完成的 route、tenant IP 和 peer IP 仍需完整聚合,
跨 state 的重复 peer IP 继续拒绝。至少以两个不重叠网段验证正反文件顺序、CIDR
后置,以及跨到另一 VPC 网段的恶意 route 仍被拒绝。
任一 VPC 网段”放行。每条 `wg-fabric0` route 还必须在同一 state 中存在同公钥的
peer 归属,另一份旧 state 的全局 peer 快照不能替它补齐。新 state 只保存本 VPC
有 route 支撑的 peer `/32`;共享 loader 可读取旧版同公钥聚合快照并去重,但最终
只保留仍有唯一 `wg-fabric0` route 的地址。相同 IP 对应不同公钥、单文件重复 peer、
重复 route 或越界 route 均继续拒绝。删除一个 state 后,即使其他旧快照仍抄有该
地址,也必须立即从 WireGuard 与共享 nft 授权中撤销。至少以两个不重叠网段验证
正反文件顺序、CIDR 后置、旧新 state 混用、删除撤权和跨文件缺失 peer 掩盖。
宿主 `/usr/local/libexec/virt-cloud-wg-fabric` 还须包含第七行契约
`# vpc-state-route-scope-contract-v3`。仅更新 Engine 不会替换已安装 helper;
`# vpc-state-route-scope-contract-v4`。v3 仍使用旧的全局 peer 聚合 loader;仅更新
Engine 不会替换已安装 helper,且 VPC mutation 必须在 v4 发布前 fail closed。
同计划旧 helper 会被 Fabric 观察判为需要升级,普通 GET 和周期观察不执行升级。
已有邻居契约、状态和在途操作均完整的环境,可通过原 Fabric 的显式“重新核对”
受控发布当前 helper;有旧邻居状态或未完成事务时,仍按上节先核实迁移证据。
@ -83,14 +88,18 @@ helper 升级并通过完整回读后,正常关闭首次接入尚未完成的
## Fabric tenant 集合的统一口径
`VPC_TENANT_FILE`、Fabric nft `vpc_tenant_ips` 和观察期望必须统一为全部受管
state 的已验证 route 与 peer `/32` 的去重并集。peer 行可携带认证计划中的全局
Fabric 地址,即使该地址不在当前 state 的 CIDR 中;这不授权当前 VPC 创建该
地址的 route。仅派生 tenant 集合可去重,原始重复 route 或 peer 仍必须拒绝。
至少验证另一个 VPC 的 peer-only 地址、route/peer 自然重叠、多个 state 与文件
顺序、CIDR 后置、nft refresh 后完整回读;故意漏掉任一 peer 地址时观察不得
active。集合对齐不等于每份旧全局 peer 快照已完成归属/删除收敛;涉及历史快照
清理时保留原操作与严格迁移边界,不通过删除旧 VPC 或改写成功状态绕过。
state 的已验证 route `/32` 去重并集,包含本地 bridge route 与远端
`wg-fabric0` route;没有 route 的 peer-only 聚合地址不得进入 nft 集合。
`VPC_PEER_FILE` 先兼容读取旧聚合候选,再按 route-backed 归属筛选。共享 WireGuard
AllowedIPs 的完整期望由宿主全部有效 state 的筛选结果加每个 peer 的 overlay `/32`
组成,不能用某次控制面请求携带的全 Fabric 聚合代替宿主持久归属。
观察必须对完整宿主 state union 与实时 WG 做 exact 比较,同时证明当前 VPC 的
peer 公钥、overlay、成员和 service route 是其子集;生产计划还须提供完整、唯一的
`公钥 → overlay /32` 权威映射,不能从实时 AllowedIPs 反推 overlay。至少验证
A+B 与 B+C 的交错拓扑、同 peer 多 VPC union、peer-only
地址不授权、删除后旧地址拒绝、nft refresh 完整回读和 stale/extra AllowedIPs。
旧 state 迁移保留原 operation 与严格边界,不通过删库或改写成功状态绕过。
## NetworkManager 带运行网卡的桥恢复
@ -154,8 +163,10 @@ UUID、ifindex 和完整端口集合。恢复必须先证明 ACL、sysctl、prox
来宾 main table 的目标 route 必须只有一条 exact scope-link VPC CIDR,不能含 gateway、
nexthop、`linkdown`、`dead` 或 `unresolved` flags;成员或自定义 route 变化不得改变
兄弟 VM 的 guest generation。历史逐成员 `/32` seed 只能在完全停机后迁移。
4. `wg show wg-fabric0 allowed-ips` 的每个 peer 与完整期望集合一致:overlay `/32`
加精确 tenant/service `/32`,没有 `0.0.0.0/0`、宽前缀或其他 VPC 的未授权地址。
4. `wg show wg-fabric0 allowed-ips` 的每个 peer 与宿主 route-backed state union
完整一致:overlay `/32` 加全部仍有归属的 tenant/service `/32`。同宿主的其他
VPC 地址可以存在,但必须由对应 state 的 route 支撑;控制面聚合中尚未落盘或
已删除 VPC 的地址不得残留。没有 `0.0.0.0/0`、宽前缀或未知公钥。
5. nft inet/bridge/arp 三张表的 owner marker、chain priority、ACL comments、
anti-spoof、IPv6/广播/未知目的 drop 和精确 proxy-ARP set 均能被 observe 回读。
bridge 上通用 `proxy_arp=0`、同桥下一跳所需的 `proxy_arp_pvlan=1`、

@ -10,6 +10,43 @@ tags:
# 当前项目交接:Virt Cloud
## 2026-10-14 新 VPC 加入在第二台宿主反复回滚
- **现场根因**:原 operation `708fa7c660383d7ee945d1543de1e0a8` 在 AlmaLinux
宿主的 VPC apply 已完成桥、nft、路由和策略规则,随后于 `fabric_acl` 的
`vpc_state_load` 退出并整体回滚。旧 loader 把每份 state 中的全 Fabric peer
聚合当作全局唯一来源;同一公钥、同一 `/32` 被旧 VPC 与新 VPC state 同时记录时
被误判冲突。Debian 当时尚无同样的历史 state,所以两台系统结果不同;根因是
持久状态历史,不是发行版使用了不同核对脚本。
- **数据面修复**:新 state 仅保存本 VPC 的 route-backed peer 贡献;loader 兼容读取
旧同公钥聚合候选,但只授权仍有唯一 `wg-fabric0` route 的地址,并要求 route 与
peer 在同一 state 内成对。删除 state 会撤销旧聚合地址,相同 IP 不同公钥仍
fail closed。观察按宿主全部有效 state union 精确核对实时 WireGuard,同时要求
当前 VPC 的公钥、overlay、成员和 service route 子集成立;apply、observe 和
retiring-host remove 都从完整 Fabric 计划携带唯一的 `公钥 → overlay /32` 映射,
不从宿主实时 AllowedIPs 自举权威 overlay。
- **升级边界**:Fabric helper 路由归属契约升为
`vpc-state-route-scope-contract-v4`;v3 helper 使用旧 loader。部署 Engine 后须先用
原 Fabric 显式核对发布并回读 v4,VPC mutation 在此之前拒绝执行,不能让旧 timer
随后把新 union 写回旧语义。
- **operation 安全**:HTTP 与周期 VPC root 绑定 operation/revision/digest 的不可变
身份;claim/release 回包丢失、取消和残留 child 均按 exact execution 回读。
任何未结算 child 或 execution unknown 都把 root/child 保留为 uncertain 并返回
503,不再出现“页面 200、下一轮却被孤儿占用阻断”。只有用户显式“重新核对”携带
的签名一次性 marker 可在旧 root 唯一、childless、完整目标与持久记录一致时,以
单次 CAS 交接原 root;worker/periodic、旧目标和残留 child 均不接管。宿主 apply
只向本地 journal 记录白名单阶段与退出码,不输出命令或配置内容。
- **本机环境与边界**:Mac 已建立 `engine/.venv`(Python 3.12.14)并按项目依赖运行
回归;18 个相关模块 1,234 passed,完整 Engine 为 3,975 passed、24 skipped、
118 subtests passed、6 failed。六个失败均在未修改 `9fb2624` 复现:三项是旧测试
漂移,另三项是既有公网池占用扫描/耗尽判断缺陷;后者会影响新实例公网地址分配,
但不进入本次已有实例的 VPC 私网加入路径,详见 `docs/VALIDATION.md`。源码验证不
代替 helper v4、两宿主 state/WG/nft、libvirt XML、seed、QGA 和来宾私网实流量
验收。本节当前仍待提交、部署并以原 operation 完成现场核对。
- 详细迁移与验收见 `docs/VALIDATION.md`、`docs/VPC_DATAPLANE_ACCEPTANCE.md`;
关系沿用 [[Concepts/Virt Cloud WireGuard Fabric与VPC L3私网]] 与
[[Concepts/Virt Cloud VPC永久邻居与重复协调]]。
## 2026-10-14 Fabric 更新后 502 与原 operation 占用
- **现场**:`d50e6d5` 已部署,Engine 重启后原 `791e…` 的终态时间仍为

@ -2178,6 +2178,17 @@ def test_hmac_binds_actor_payload_target_and_rejects_replay():
hashlib.sha256(body).hexdigest(), store, now=1000,
)
invalid_vpc_reconcile = {**USER, "vpc_reconcile_explicit": "true"}
invalid_headers = {key.lower(): value for key, value in signed_headers(
SECRET, "POST", target, body, invalid_vpc_reconcile,
timestamp=1000, nonce="v" * 24,
).items()}
with pytest.raises(EngineAuthError):
authenticate(
invalid_headers, SECRET, "POST", target,
hashlib.sha256(body).hexdigest(), store, now=1000,
)
def test_private_route_allowlist_and_no_browser_admin_cookie(boundary):
client, *_ = boundary

@ -341,7 +341,7 @@ def test_observe_artifact_state_accepts_only_kind_specific_header(
f"# fabric-plan-digest={current.digest}\n"
"# libvirt-startup-independent-v1\n"
"# vpc-local-neighbor-contract-v1\n"
"# vpc-state-route-scope-contract-v3\n",
"# vpc-state-route-scope-contract-v4\n",
encoding="utf-8",
)
artifact.chmod(int(mode, 8))

@ -247,3 +247,133 @@ def test_local_catalog_completion_preserves_outer_remote_claim():
fabric.close()
asyncio.run(exercise())
def _assert_exact_child_identity(row, owner, parent, resources, metadata):
assert row['state'] != 'active'
assert row['owner_id'] == owner.owner_id
assert row['execution_id'] == parent.execution_id
assert row['operation_id'] == parent.operation_id
assert row['kind'] == 'resource'
assert row['resources'] == sorted(resources)
assert row['metadata'] == metadata
def test_catalog_claim_lost_commit_response_recovers_exact_child_as_uncertain():
class LostClaimReply(DurableCoordinator):
lose_reply = True
def claim(self, **options):
result = super().claim(**options)
if options.get('kind') == 'resource' and self.lose_reply:
self.lose_reply = False
raise RuntimeError('catalog claim commit response lost')
return result
async def exercise():
store = MemoryCAS()
owner = LostClaimReply(store)
metadata = {'job_kind': 'vpc_operation', 'vpc_id': 'vpc-hk-1'}
parent = owner.claim(operation_id='operation-lock-recovery',
kind='request', metadata=metadata)
resources = ('catalog:network',)
catalog = _DurableAsyncLock(owner, resources)
with execution_context(owner, parent):
with pytest.raises(RuntimeError, match='commit response lost'):
async with catalog:
pytest.fail('lost admission response entered the mutation body')
rows = owner.snapshot()
children = [row for token, row in rows.items() if token != parent.token]
assert len(children) == 1
_assert_exact_child_identity(
children[0], owner, parent, resources, metadata)
assert not catalog.locked()
asyncio.run(exercise())
def test_catalog_claim_response_cancellation_drains_and_releases_known_child():
entered, resume = threading.Event(), threading.Event()
class BlockingClaimReply(DurableCoordinator):
blocked = False
def claim(self, **options):
result = super().claim(**options)
if options.get('kind') == 'resource' and not self.blocked:
self.blocked = True
entered.set()
assert resume.wait(3)
return result
async def exercise():
store = MemoryCAS()
owner = BlockingClaimReply(store)
metadata = {'job_kind': 'vpc_operation', 'vpc_id': 'vpc-hk-1'}
parent = owner.claim(operation_id='operation-lock-cancel',
kind='request', metadata=metadata)
resources = ('catalog:network',)
catalog = _DurableAsyncLock(owner, resources)
body_entered = []
async def admitted():
async with catalog:
body_entered.append(True)
with execution_context(owner, parent):
task = asyncio.create_task(admitted())
try:
assert await asyncio.to_thread(entered.wait, 3)
task.cancel()
await asyncio.sleep(0)
assert not task.done()
finally:
resume.set()
with pytest.raises(asyncio.CancelledError):
await task
assert body_entered == []
rows = owner.snapshot()
children = [row for token, row in rows.items() if token != parent.token]
assert not any(row['state'] == 'active' for row in children)
assert children == []
assert not catalog.locked()
asyncio.run(exercise())
def test_catalog_release_precommit_failure_retains_exact_child_as_uncertain():
class FailedRelease(DurableCoordinator):
fail_release = True
def release(self, claim):
if claim.kind == 'resource' and self.fail_release:
self.fail_release = False
raise CoordinationError('coordination_unavailable', 503)
return super().release(claim)
async def exercise():
store = MemoryCAS()
owner = FailedRelease(store)
metadata = {'job_kind': 'vpc_operation', 'vpc_id': 'vpc-hk-1'}
parent = owner.claim(operation_id='operation-lock-release',
kind='request', metadata=metadata)
resources = ('catalog:network',)
catalog = _DurableAsyncLock(owner, resources)
with execution_context(owner, parent):
with pytest.raises(CoordinationError) as raised:
async with catalog:
pass
assert raised.value.error_code == 'coordination_unavailable'
rows = owner.snapshot()
children = [row for token, row in rows.items() if token != parent.token]
assert len(children) == 1
_assert_exact_child_identity(
children[0], owner, parent, resources, metadata)
assert not catalog.locked()
asyncio.run(exercise())

@ -149,7 +149,7 @@ def startup_files(root, current, *, legacy=False, gate=True,
if legacy_neighbors and name == "helper":
content = content.replace("# vpc-local-neighbor-contract-v1\n", "")
if legacy_routes and name == "helper":
content = content.replace("# vpc-state-route-scope-contract-v3\n", "")
content = content.replace("# vpc-state-route-scope-contract-v4\n", "")
target = root / name
target.write_text(content)
target.chmod(mode)
@ -231,7 +231,7 @@ def test_explicit_reconcile_upgrades_same_plan_then_skips_current_hosts(tmp_path
rendered = _apply_script(actual_plan, host, keys)
assert FABRIC_STARTUP_ISOLATION_MARKER in rendered
assert "# vpc-local-neighbor-contract-v1" in rendered
assert "# vpc-state-route-scope-contract-v3" in rendered
assert "# vpc-state-route-scope-contract-v4" in rendered
assert 'record_desired_absent gate_generator' in rendered
startup_files(self.manager.startup_root, actual_plan, gate=False)
return super().dispatch_apply(actual_plan, host, keys, operation_id)

@ -1331,6 +1331,31 @@ def test_fabric_gate_keeps_revision_mismatch_as_reconcilable_state():
assert result["state"] == "degraded"
def test_same_vpc_revision_and_digest_cannot_borrow_another_operation_receipt(monkeypatch):
async def run():
store = MemoryStore()
coordinator = VpcCoordinator(store, FakeFabric(hosts=("host-a",)))
first = await coordinator.apply(payload(operation_id="a" * 32))
before = store.get("vpc_desired", "vpc-hk-1")
retries = []
async def observe_only(*_args, **_kwargs):
retries.append(True)
monkeypatch.setattr(coordinator, "_reconcile_pending_hosts", observe_only)
with pytest.raises(VpcError) as conflict:
await coordinator.apply(payload(operation_id="b" * 32))
assert conflict.value.status == 409
assert conflict.value.error_code == "vpc_revision_conflict"
assert store.get("vpc_desired", "vpc-hk-1") == before
assert retries == []
replay = await coordinator.apply(payload(operation_id="a" * 32))
assert replay["operation_id"] == first["operation_id"]
assert retries == [True]
asyncio.run(run())
def test_coordinator_revision_conflict_and_tampered_record_fail_closed():
store = MemoryStore()
coordinator = VpcCoordinator(store, FakeFabric(hosts=("host-a",)))
@ -2605,9 +2630,11 @@ def test_delete_pending_projection_does_not_probe_confirmed_offline_host():
}
coordinator._fabric_peers_for_plan = lambda _plan, _record: []
host_digests = []
host_peer_overlays = []
def confirm_host(*args, **kwargs):
host_digests.append(args[8])
host_peer_overlays.append(kwargs["fabric_peer_overlays"])
return {
"state": "active", "removed": True,
"transport_verified": True,
@ -2641,6 +2668,10 @@ def test_delete_pending_projection_does_not_probe_confirmed_offline_host():
assert probed == ["host-b"]
assert host_digests == [fabric_plan.digest]
assert host_peer_overlays == [{
"a" * 44: "100.64.0.1/32",
"b" * 44: "100.64.0.2/32",
}]
assert coordinator._confirmed_removal(
record["hosts"][0], plan.desired_revision, plan.desired_digest)
@ -4104,6 +4135,8 @@ def test_delete_manifest_removes_unknown_member_nic_before_host_resources(
"private_mac": member.private_mac,
} for member in manifest]
events = []
local_key = "AQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQE="
remote_key = "AgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgI="
monkeypatch.setattr(VMService, "__init__", lambda self, *args, **kwargs: None)
@ -4118,6 +4151,10 @@ def test_delete_manifest_removes_unknown_member_nic_before_host_resources(
)
def remove_host(self, dataplane):
assert dataplane.fabric_peer_keys == frozenset({remote_key})
assert dataplane.fabric_peer_overlays == frozenset({
(remote_key, "100.64.0.3/32"),
})
events.append(("host", dataplane.host_id, dataplane.vpc.vpc_id))
return {
"state": "active", "removed": True,
@ -4134,7 +4171,12 @@ def test_delete_manifest_removes_unknown_member_nic_before_host_resources(
plan, "host-a", SimpleNamespace(manager=object()),
{"libvirt_uri": "qemu:///system", "storage_pool": "default",
"iso_pool": "iso", "default_network": "default"},
[], None, retiring_members=rows, fabric_public_keys={},
[], None, retiring_members=rows,
fabric_public_keys={"host-a": local_key, "host-b": remote_key},
fabric_peer_overlays={
local_key: "100.64.0.2/32",
remote_key: "100.64.0.3/32",
},
fabric_plan_digest="ef" * 32,
)
assert events == [

@ -32,7 +32,12 @@ def empty_receipts():
@pytest.fixture
def plan():
p = _plan()
return replace(p, fabric_peer_keys=frozenset(x.public_key for x in p.peers))
return replace(
p,
fabric_peer_keys=frozenset(x.public_key for x in p.peers),
fabric_peer_overlays=frozenset(
(x.public_key, x.overlay_cidr) for x in p.peers),
)
class Adapter(dp.VpcPrivateHostAdapter):
@ -362,8 +367,9 @@ def test_rendered_host_receipt_requires_exact_persistent_bytes(tmp_path, shell,
f'PLAN_DIGEST="{plan.fabric_plan_digest}"\n'
'# libvirt-startup-independent-v1\n'
'# vpc-local-neighbor-contract-v1\n')
if mutation != 'old_route_scope':
header += '# vpc-state-route-scope-contract-v3\n'
header += ('# vpc-state-route-scope-contract-v3\n'
if mutation == 'old_route_scope'
else '# vpc-state-route-scope-contract-v4\n')
helper.write_text(header)
helper.chmod(0o700)
stat = executable(tmp_path / 'stat', 'import sys\n'

@ -0,0 +1,159 @@
"""VPC apply diagnostics must stay useful without exposing remote output."""
from __future__ import annotations
import os
from pathlib import Path
import shlex
import subprocess
import tempfile
import unittest
from virsh_panel.vpc import VPC_SCHEMA_VERSION
from virsh_panel.vpc_dataplane import (
VPC_APPLY_SAFE_LOG_SH,
parse_vpc_dataplane_plan,
render_apply_script,
)
_SECRET = "NEVER_LOG_vpc_private_fixture_or_sensitive_path"
def _plan():
return parse_vpc_dataplane_plan({
"schema_version": VPC_SCHEMA_VERSION,
"vpc_id": "vpc-safe-log",
"region": "hk",
"fabric_id": "fabric-hk",
"fabric_revision": 3,
"cidr": "10.42.0.0/24",
"desired_revision": 1,
"operation_id": "vpc-op-safe-log-0001",
"subnets": [{"subnet_id": "subnet-a", "cidr": "10.42.0.0/25"}],
"hosts": [{
"host_id": "host-a",
"members": [{
"member_id": "member-a",
"instance_uuid": "11111111-1111-4111-8111-111111111111",
"private_ip_cidr": "10.42.0.10/32",
"subnet_id": "subnet-a",
}],
}],
"routes": [],
"acls": [],
"fabric_peers": [],
}, "host-a", fabric_plan_digest="ab" * 32)
class VpcApplySafeLogTest(unittest.TestCase):
def _run_logger(self, body: str):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
output = root / "journal"
logger = root / "logger"
logger.write_text(
"#!/bin/sh\nprintf '%s\\n' \"$*\" >> \"$VC_VPC_TEST_LOG\"\n",
encoding="utf-8",
)
logger.chmod(0o700)
script = "\n".join((
"set -eu",
VPC_APPLY_SAFE_LOG_SH,
f"LOGGER={shlex.quote(str(logger))}",
body,
))
result = subprocess.run(
["/bin/sh", "-c", script],
env={**os.environ, "VC_VPC_TEST_LOG": str(output)},
capture_output=True,
text=True,
check=False,
)
lines = output.read_text(encoding="utf-8").splitlines() \
if output.exists() else []
return result, lines
def test_exit_trap_preserves_status_and_logs_only_fixed_fields(self):
result, lines = self._run_logger(
"VC_VPC_APPLY_PHASE=post_activation_sysctl\nexit 72")
self.assertEqual(result.returncode, 72)
self.assertEqual(result.stdout, "")
self.assertEqual(result.stderr, "")
self.assertEqual(lines, [
"-t virt-cloud-vpc -- apply_state=failed "
"apply_phase=post_activation_sysctl exit_status=72",
])
def test_unknown_phase_and_status_are_sanitized(self):
result, lines = self._run_logger("\n".join((
"VC_VPC_APPLY_PHASE=" + shlex.quote("bad\n" + _SECRET),
f"vc_vpc_apply_failure_log {shlex.quote(_SECRET)}",
)))
self.assertEqual(result.returncode, 0)
self.assertEqual(lines, [
"-t virt-cloud-vpc -- apply_state=failed "
"apply_phase=unknown exit_status=1",
])
self.assertNotIn(_SECRET, "\n".join(lines))
def test_success_does_not_create_failure_record(self):
result, lines = self._run_logger("VC_VPC_APPLY_PHASE=commit\nexit 0")
self.assertEqual(result.returncode, 0)
self.assertEqual(lines, [])
def test_term_signal_uses_nonzero_status_and_safe_phase(self):
result, lines = self._run_logger(
'VC_VPC_APPLY_PHASE=routes\nkill -TERM "$$"\nexit 99')
self.assertEqual(result.returncode, 143)
self.assertEqual(result.stdout, "")
self.assertEqual(result.stderr, "")
self.assertEqual(lines, [
"-t virt-cloud-vpc -- apply_state=failed "
"apply_phase=routes exit_status=143",
])
def test_rendered_phases_bracket_activation_and_post_activation_work(self):
script = render_apply_script(_plan(), deployment_mode="networkmanager_nmcli")
rollback_start = script.index("rollback() {")
rollback_end = script.index("\ntrap rollback EXIT HUP INT TERM", rollback_start)
rollback = script[rollback_start:rollback_end]
disable_traps = rollback.index("trap - EXIT HUP INT TERM")
safe_log = rollback.index('vc_vpc_apply_failure_log "$status"')
rollback_mutations = rollback.index("set +e")
self.assertLess(disable_traps, safe_log)
self.assertLess(safe_log, rollback_mutations)
persistence = script.index("VC_VPC_APPLY_PHASE=persistence")
activation = script.index("VC_VPC_APPLY_PHASE=activation", persistence)
nm_up = script.index(
'"$NMCLI" connection up uuid "$NM_PROFILE_UUID"', activation)
post_activation = script.index(
"VC_VPC_APPLY_PHASE=post_activation_sysctl", nm_up)
rp_filter = script.index("vc_vpc_rp_filter_set || exit 72", post_activation)
runtime = script.index("VC_VPC_APPLY_PHASE=runtime", rp_filter)
proxy = script.index("VC_VPC_APPLY_PHASE=proxy", runtime)
neighbors = script.index("VC_VPC_APPLY_PHASE=neighbors", proxy)
routes = script.index("VC_VPC_APPLY_PHASE=routes", neighbors)
policy = script.index("VC_VPC_APPLY_PHASE=policy_rules", routes)
fabric_acl = script.index("VC_VPC_APPLY_PHASE=fabric_acl", policy)
wireguard = script.index("VC_VPC_APPLY_PHASE=wireguard", fabric_acl)
final = script.index("VC_VPC_APPLY_PHASE=final_readback", wireguard)
commit = script.index("VC_VPC_APPLY_PHASE=commit", final)
self.assertLess(activation, nm_up)
self.assertLess(nm_up, post_activation)
self.assertLess(post_activation, rp_filter)
self.assertEqual(
sorted((runtime, proxy, neighbors, routes, policy, fabric_acl,
wireguard, final, commit)),
[runtime, proxy, neighbors, routes, policy, fabric_acl,
wireguard, final, commit],
)
if __name__ == "__main__":
unittest.main()

@ -1013,11 +1013,38 @@ def test_observe_can_require_the_complete_authoritative_fabric_peer_set():
"host-b": ["10.42.0.11/32", "10.42.0.100/32"],
},
fabric_peer_keys=[WG_KEY, WG_KEY_2],
fabric_peer_overlays={
WG_KEY: "100.64.0.2/32",
WG_KEY_2: "100.64.0.3/32",
},
)
assert strict.fabric_peer_keys == frozenset({WG_KEY, WG_KEY_2})
assert strict.fabric_peer_overlays == frozenset({
(WG_KEY, "100.64.0.2/32"),
(WG_KEY_2, "100.64.0.3/32"),
})
observe = render_observe_script(strict)
assert "__VC_VPC_WG_PEER_SET__" in observe
assert "live_fabric_peer_keys" in observe
with pytest.raises(VpcDataplaneError) as overlay_error:
parse_vpc_dataplane_plan(
plan.vpc.canonical(), "host-a",
fabric_plan_digest=FABRIC_DIGEST,
fabric_peers=[{
"host_id": "host-b", "public_key": WG_KEY,
"overlay_ip": "100.64.0.2/32",
"tenant_allowed_ips": ["10.42.0.11/32", "10.42.0.100/32"],
}],
trusted_tenant_allowed_ips={
"host-b": ["10.42.0.11/32", "10.42.0.100/32"],
},
fabric_peer_keys=[WG_KEY, WG_KEY_2],
fabric_peer_overlays={
WG_KEY: "100.64.0.9/32",
WG_KEY_2: "100.64.0.3/32",
},
)
assert overlay_error.value.error_code == "vpc_fabric_peer_set_invalid"
try:
parse_vpc_dataplane_plan(
plan.vpc.canonical(), "host-a",
@ -1360,6 +1387,99 @@ def _active_observation_markers(plan):
return markers
def _observe_with_route_backed_wg_markers(plan, *, allowed, key=None,
wg_state="owned",
peer_set="exact"):
"""Run adapter readback with an explicitly host-derived WG union."""
member = plan.local_members[0]
routes = private_guest_routes(plan.vpc, member)
persistent = patch_private_interface_xml(
"""<domain><metadata/><devices>
<interface type='bridge'><mac address='52:54:00:00:00:01'/>
<alias name='ua-public0'/><source bridge='br0'/></interface>
</devices></domain>""",
member=member, resources=plan.resources,
guest_network_generation=private_guest_network_generation(
member, routes),
vpc_id=plan.vpc.vpc_id, desired_revision=plan.desired_revision,
desired_digest=plan.desired_digest,
)
peer = plan.peers[0]
markers = []
for item in _active_observation_markers(plan):
if item.startswith("__VC_VPC_WG_ALLOWED__="):
item = (f"__VC_VPC_WG_ALLOWED__={peer.host_id}|"
f"{key or peer.public_key}|{allowed}")
elif item.startswith("__VC_VPC_WG_STATE__="):
item = f"__VC_VPC_WG_STATE__={peer.host_id}|{wg_state}"
markers.append(item)
markers.append(f"__VC_VPC_WG_PEER_SET__={peer_set}")
adapter = VpcPrivateHostAdapter(object(), {})
adapter._deployment_contract = lambda: ("networkmanager", "", "", "")
adapter._run = lambda _script: "\n".join(markers) + "\n"
return adapter.observe(plan, xml_by_instance={
member.instance_uuid: {
"inactive": persistent,
"live": persistent,
"seed_verified": True,
"guest_network": _guest_network_evidence(member, routes),
},
})
def test_adapter_accepts_host_state_union_without_catalog_only_tenant_ip():
base = _plan()
route_backed_allowed = _allowed(base, base.peers[0])
peer = replace(
base.peers[0],
tenant_allowed_ips=tuple(sorted(
set(base.peers[0].tenant_allowed_ips) | {'10.45.0.11/32'})),
)
plan = replace(
base, peers=(peer,),
fabric_peer_keys=frozenset({peer.public_key}),
fabric_peer_overlays=frozenset({
(peer.public_key, peer.overlay_cidr),
}))
observed = _observe_with_route_backed_wg_markers(
plan, allowed=route_backed_allowed)
assert observed["state"] == "active"
assert observed["config_verified"] is True
assert observed["wg_allowed_ips"] == [{
"peer_host_id": peer.host_id,
"public_key": peer.public_key,
"tenant_allowed_ips": ["10.42.0.100/32", "10.42.0.11/32"],
}]
@pytest.mark.parametrize(('mutation', 'kwargs'), [
('wrong-key', {'key': WG_KEY_2}),
('missing-overlay', {
'allowed': '10.42.0.11/32,10.42.0.100/32',
}),
('missing-vpc-route', {
'allowed': '100.64.0.2/32,10.42.0.11/32',
}),
('shell-union-mismatch', {'wg_state': 'missing'}),
('fabric-peer-set-mismatch', {'peer_set': 'extra'}),
])
def test_adapter_rejects_incomplete_or_unproven_route_backed_wg_union(
mutation, kwargs):
plan = replace(
_plan(), fabric_peer_keys=frozenset({WG_KEY}),
fabric_peer_overlays=frozenset({(WG_KEY, "100.64.0.2/32")}))
peer = plan.peers[0]
values = {'allowed': _allowed(plan, peer)}
values.update(kwargs)
observed = _observe_with_route_backed_wg_markers(plan, **values)
assert observed["config_verified"] is False, mutation
assert observed["state"] == "degraded", mutation
@pytest.mark.parametrize("acl_proof", ("exact", "old_revision", "wrong_digest", "missing_nft"))
def test_stopped_host_reports_acl_revision_only_after_exact_configuration_proof(acl_proof):
plan = _plan()
@ -1414,7 +1534,9 @@ def test_stopped_host_reports_acl_revision_only_after_exact_configuration_proof(
def test_stopped_linkdown_host_config_remains_reusable_without_becoming_active():
plan = replace(_plan(), fabric_peer_keys=frozenset({WG_KEY}))
plan = replace(
_plan(), fabric_peer_keys=frozenset({WG_KEY}),
fabric_peer_overlays=frozenset({(WG_KEY, "100.64.0.2/32")}))
member = plan.local_members[0]
routes = private_guest_routes(plan.vpc, member)
persistent = patch_private_interface_xml(
@ -1470,7 +1592,9 @@ def test_stopped_linkdown_host_config_remains_reusable_without_becoming_active()
def test_topology_mismatch_revokes_reusable_host_config_proof():
plan = replace(_plan(), fabric_peer_keys=frozenset({WG_KEY}))
plan = replace(
_plan(), fabric_peer_keys=frozenset({WG_KEY}),
fabric_peer_overlays=frozenset({(WG_KEY, "100.64.0.2/32")}))
member = plan.local_members[0]
routes = private_guest_routes(plan.vpc, member)
persistent = patch_private_interface_xml(

@ -208,8 +208,10 @@ def test_helper_upgrade_gate_runs_before_network_mutation(tmp_path, shell, case)
header = ("#!/bin/sh\n# virt-cloud-wg-fabric-v1\n# fabric-plan-digest=" + digest +
'\nPLAN_DIGEST="' + digest + '"\n# libvirt-startup-independent-v1\n')
if case != "old": header += "# vpc-local-neighbor-contract-v1\n"
if case not in {"old", "old_route_scope"}:
if case == "old_route_scope":
header += "# vpc-state-route-scope-contract-v3\n"
elif case != "old":
header += "# vpc-state-route-scope-contract-v4\n"
if case == "wrong_digest": header = header.replace(digest, "cd" * 32)
if case != "missing":
helper.write_text(header)

@ -306,6 +306,7 @@ NFT=vc_test_nft
guard_functions,
loader_functions,
'VPC_TENANT_FILE="$EXPECTED"',
'VPC_STATE_STATUS=owned',
'FABRIC_NFT_SNAPSHOT="$SNAPSHOT"',
"vpc_state_load() { :; }",
observe_function,

@ -105,7 +105,7 @@ opposite_clean=yes; nft_file_ok=yes; sysctl_file_ok=yes; persistence=no
])
def test_observer_empty_proxy_set_reports_missing_without_aborting(
tmp_path, shell, actual, expected):
body = _fragment("proxy_records=", 'if "$NFT" list table inet "$TABLE"')
body = _fragment("proxy_records=", "# vc-local-neighbor-observe-begin")
proxy_file = tmp_path / "proxy"
proxy_file.write_text("# virt-cloud-vpc-proxy-v1\n10.44.0.10\n10.44.0.11\n")
result = _run(shell, '''

File diff suppressed because it is too large Load Diff

@ -13,6 +13,8 @@ import virsh_panel.vpc_dataplane as dataplane_module
from virsh_panel.shell_contracts import WG_ALLOWED_IPS_SH
OTHER_WG_KEY = "AgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgI="
def _loader_from_script(script, *, observe=False):
start = script.index('VPC_STATE_DIR="/etc/virt-cloud/vpc-dataplane"')
@ -55,6 +57,19 @@ def _state_text(plan, *, cidr_after_routes=False):
return text
def _legacy_aggregate_state_text(plan, *, cidr_after_routes=False):
"""Render the pre-migration state shape with full Fabric peer closure."""
lines = _state_text(
plan, cidr_after_routes=cidr_after_routes).splitlines()
lines = [line for line in lines if not line.startswith('peer|')]
for peer in plan.peers:
if peer.tenant_allowed_ips:
lines.append(
f"peer|{peer.host_id}|{peer.public_key}|"
+ ','.join(peer.tenant_allowed_ips))
return '\n'.join(lines) + '\n'
def _with_global_peer_tenants(plan, *extra_ips):
"""Use the same trusted peer closure that the coordinator supplies."""
payload = plan.vpc.canonical()
@ -114,6 +129,68 @@ printf 'state=%s\n' "$VPC_STATE_STATUS"
capture_output=True, check=False)
def _run_observe_wg_contract(
tmp_path, plan, texts, actual_by_key, *, peer_read_status=0):
"""Run the rendered observer's exact state-union/AllowedIPs block."""
state_dir = tmp_path / 'states'
state_dir.mkdir(mode=0o700)
for index, state_text in enumerate(texts, 1):
state_file = state_dir / f'{index:011x}.state'
state_file.write_text(state_text)
state_file.chmod(0o600)
wg_rows = tmp_path / 'wg-rows'
wg_rows.write_text(''.join(
f'{key}\t{allowed}\n' for key, allowed in actual_by_key.items()))
script = dataplane_module.render_observe_script(plan)
block_start = script.index('vpc_state_loaded=no')
block_end = script.index('fabric_nft_state=no', block_start)
harness = '\n'.join((
'set -eu',
_loader_from_script(script, observe=True),
WG_ALLOWED_IPS_SH,
'VPC_STATE_DIR=' + shlex.quote(str(state_dir)),
'VPC_PEER_FILE=' + shlex.quote(str(tmp_path / 'peers')),
'VPC_ROUTE_FILE=' + shlex.quote(str(tmp_path / 'routes')),
'VPC_TENANT_FILE=' + shlex.quote(str(tmp_path / 'tenants')),
'VPC_WG_ROWS=' + shlex.quote(str(wg_rows)),
'VPC_WG_PEER_READ_STATUS=' + str(peer_read_status),
'FIREWALL_MODE=native_nftables',
r'''
vc_fixture_stat() {
[ "$1" = -c ] || return 9
case "$2" in
'%u:%a')
if [ "$3" = "$VPC_STATE_DIR" ]; then printf '0:700\n'
elif [ "${3%/*}" = "$VPC_STATE_DIR" ] && [ -f "$3" ]; then
printf '0:600\n'
else return 9; fi ;;
'%s') [ "${3%/*}" = "$VPC_STATE_DIR" ] && [ -f "$3" ] || return 9
wc -c < "$3" | tr -d ' ' ;;
*) return 9 ;;
esac
}
vc_fixture_wg() {
[ "$1" = show ] && [ "$2" = wg-fabric0 ] && [ "$#" = 3 ] || return 9
case "$3" in
allowed-ips) cat "$VPC_WG_ROWS" ;;
peers)
[ "$VPC_WG_PEER_READ_STATUS" = 0 ] ||
return "$VPC_WG_PEER_READ_STATUS"
cut -f1 "$VPC_WG_ROWS" ;;
*) return 9 ;;
esac
}
STAT=vc_fixture_stat
WG=vc_fixture_wg
WG_IFACE=wg-fabric0
'''.strip(),
script[block_start:block_end],
))
return subprocess.run(['/bin/sh', '-c', harness], text=True,
capture_output=True, check=False)
@pytest.mark.parametrize('mode', ['apply', 'observe'])
@pytest.mark.parametrize('reverse', [False, True])
@pytest.mark.parametrize('cidr_after_routes', [False, True])
@ -163,30 +240,32 @@ def test_global_peer_tenants_join_routes_once_in_the_fabric_address_universe(
assert result.stdout == 'state=owned\n'
route_ips = {row['destination_cidr'] for plan in plans
for row in dataplane_module._routes(plan)}
expected = sorted(route_ips | {extra})
expected = sorted(route_ips)
actual = (tmp_path / 'tenants').read_text().splitlines()
assert sorted(actual) == expected
assert len(actual) == len(set(actual))
expected_peers = sorted(
peer.public_key + '|' + address
for plan in plans for peer in plan.peers
for address in peer.tenant_allowed_ips)
plan.peers[0].public_key + '|' + row['destination_cidr']
for plan in plans for row in dataplane_module._routes(plan)
if row['device'] == 'wg-fabric0')
assert sorted((tmp_path / 'peers').read_text().splitlines()) == expected_peers
# A global peer authorization contributes no route in this VPC's table.
# It must not weaken the independent per-record route/CIDR contract.
# A trusted aggregate entry contributes no durable route, peer, or nft
# authorization. Runtime convergence still checks it through _allowed().
routes = (tmp_path / 'routes').read_text().splitlines()
assert {row.split('|', 1)[0] for row in routes} == route_ips
assert extra not in route_ips
assert extra not in actual
assert all(not row.endswith('|' + extra)
for row in (tmp_path / 'peers').read_text().splitlines())
@pytest.mark.parametrize('mode', ['apply', 'observe'])
@pytest.mark.parametrize('cidr_after_routes', [False, True])
def test_peer_only_state_keeps_its_global_tenants_without_local_routes(
def test_peer_only_state_does_not_authorize_aggregate_without_owned_routes(
tmp_path, mode, cidr_after_routes):
carrier = _with_global_peer_tenants(_plan(), '10.46.0.21/32')
# The loader supports a retained peer-only record after local members and
# routes disappear. Render that record with the real state producer;
# extract the loader from an ordinary participating-host transaction.
# A retained record may outlive all local routes. Render it with the new
# producer and extract the loader from a participating-host transaction.
peer_only = replace(carrier, members=())
text = _state_text(peer_only, cidr_after_routes=cidr_after_routes)
assert '\nroute|' not in text and '\nlocal_neighbor|' not in text
@ -194,25 +273,264 @@ def test_peer_only_state_keeps_its_global_tenants_without_local_routes(
assert result.returncode == 0, result.stderr
assert result.stdout == 'state=owned\n'
assert (tmp_path / 'routes').read_text() == ''
actual = (tmp_path / 'tenants').read_text().splitlines()
assert sorted(actual) == ['10.44.0.11/32', '10.46.0.21/32']
assert len(actual) == len(set(actual))
assert (tmp_path / 'peers').read_text() == ''
assert (tmp_path / 'tenants').read_text() == ''
@pytest.mark.parametrize('mode', ['apply', 'observe'])
@pytest.mark.parametrize('reverse', [False, True])
@pytest.mark.parametrize('cidr_after_routes', [False, True])
def test_global_tenant_union_still_rejects_duplicate_peer_ownership(
def test_identical_global_peer_closure_is_deduplicated_across_vpc_states(
tmp_path, mode, reverse, cidr_after_routes):
first, second = _two_vpc_plans()
first = _with_global_peer_tenants(first, '10.46.0.21/32')
second = _with_global_peer_tenants(second, '10.46.0.21/32')
texts = [_state_text(plan, cidr_after_routes=cidr_after_routes)
aggregate = (
'10.44.0.11/32', '10.45.0.11/32', '10.46.0.21/32',
)
first = _with_global_peer_tenants(first, *aggregate)
second = _with_global_peer_tenants(second, *aggregate)
assert first.peers[0].tenant_allowed_ips == aggregate
assert second.peers[0].tenant_allowed_ips == aggregate
texts = [_legacy_aggregate_state_text(
plan, cidr_after_routes=cidr_after_routes)
for plan in (first, second)]
result = _run_multi_vpc_loader(
tmp_path, first, texts[::-1] if reverse else texts, mode)
assert result.returncode == (1 if mode == 'observe' else 71), result.stderr
assert 'state=owned' not in result.stdout
assert result.returncode == 0, result.stderr
assert result.stdout == 'state=owned\n'
key = first.peers[0].public_key
peer_rows = (tmp_path / 'peers').read_text().splitlines()
route_backed = aggregate[:2]
assert sorted(peer_rows) == [key + '|' + address
for address in route_backed]
assert len(peer_rows) == len(set(peer_rows))
route_ips = {
row['destination_cidr']
for plan in (first, second)
for row in dataplane_module._routes(plan)
}
tenant_ips = (tmp_path / 'tenants').read_text().splitlines()
assert tenant_ips == sorted(route_ips)
assert len(tenant_ips) == len(set(tenant_ips))
assert all(not row.endswith('|' + aggregate[2]) for row in peer_rows)
assert aggregate[2] not in tenant_ips
def test_new_state_persists_member_and_service_routes_not_fabric_aggregate():
base = _plan()
payload = base.vpc.canonical()
payload['routes'] = [{
'route_id': 'service-b',
'destination_cidr': '10.44.0.100/32',
'next_hop_member_id': 'member-b',
'metric': 20,
}]
tenants = ('10.44.0.100/32', '10.44.0.11/32', '10.46.0.21/32')
peer = base.peers[0]
payload['fabric_peers'] = [{
'host_id': peer.host_id,
'public_key': peer.public_key,
'overlay_ip': peer.overlay_cidr,
'tenant_allowed_ips': list(tenants),
}]
plan = dataplane_module.parse_vpc_dataplane_plan(
payload, base.host_id, fabric_plan_digest=base.fabric_plan_digest,
trusted_tenant_allowed_ips={peer.host_id: tenants})
state = _state_text(plan)
key = plan.peers[0].public_key
assert (f'peer|host-b|{key}|'
'10.44.0.100/32,10.44.0.11/32\n') in state
assert '10.46.0.21/32' not in state
assert set(dataplane_module._allowed(plan, plan.peers[0]).split(',')) == {
plan.peers[0].overlay_cidr,
'10.44.0.100/32',
'10.44.0.11/32',
'10.46.0.21/32',
}
@pytest.mark.parametrize('mode', ['apply', 'observe'])
@pytest.mark.parametrize('reverse', [False, True])
@pytest.mark.parametrize('cidr_after_routes', [False, True])
def test_legacy_and_new_state_mix_keeps_unique_route_backed_peer_rows(
tmp_path, mode, reverse, cidr_after_routes):
first, second = _two_vpc_plans()
aggregate = (
'10.44.0.11/32', '10.45.0.11/32', '10.46.0.21/32',
)
first = _with_global_peer_tenants(first, *aggregate)
second = _with_global_peer_tenants(second, *aggregate)
texts = [
_legacy_aggregate_state_text(
first, cidr_after_routes=cidr_after_routes),
_state_text(second, cidr_after_routes=cidr_after_routes),
]
result = _run_multi_vpc_loader(
tmp_path, first, texts[::-1] if reverse else texts, mode)
assert result.returncode == 0, result.stderr
key = first.peers[0].public_key
assert sorted((tmp_path / 'peers').read_text().splitlines()) == [
key + '|10.44.0.11/32', key + '|10.45.0.11/32',
]
assert (tmp_path / 'tenants').read_text().splitlines() == [
'10.44.0.10/32', '10.44.0.11/32',
'10.45.0.10/32', '10.45.0.11/32',
]
@pytest.mark.parametrize('mode', ['apply', 'observe'])
def test_legacy_aggregate_withdraws_deleted_vpc_route_ownership(
tmp_path, mode):
first, second = _two_vpc_plans()
aggregate = ('10.44.0.11/32', '10.45.0.11/32')
first = _with_global_peer_tenants(first, *aggregate)
second = _with_global_peer_tenants(second, *aggregate)
legacy = [_legacy_aggregate_state_text(plan)
for plan in (first, second)]
before = tmp_path / 'before'
before.mkdir()
loaded = _run_multi_vpc_loader(before, first, legacy, mode)
assert loaded.returncode == 0, loaded.stderr
key = first.peers[0].public_key
assert sorted((before / 'peers').read_text().splitlines()) == [
key + '|10.44.0.11/32', key + '|10.45.0.11/32',
]
assert (before / 'tenants').read_text().splitlines() == [
'10.44.0.10/32', '10.44.0.11/32',
'10.45.0.10/32', '10.45.0.11/32',
]
after = tmp_path / 'after'
after.mkdir()
removed = _run_multi_vpc_loader(after, first, [legacy[0]], mode)
assert removed.returncode == 0, removed.stderr
assert (after / 'peers').read_text() == key + '|10.44.0.11/32\n'
assert (after / 'tenants').read_text().splitlines() == [
'10.44.0.10/32', '10.44.0.11/32',
]
def test_observer_uses_only_local_route_backed_state_for_interleaved_topology(
tmp_path):
# Fabric A/B/C has VPC1 on A+B and VPC2 on B+C. The catalog aggregate for
# peer B includes both B addresses, but host A has only VPC1 state and must
# therefore require only overlay(B)+VPC1(B).
first, _second = _two_vpc_plans()
catalog_view = _with_global_peer_tenants(first, '10.45.0.11/32')
peer = catalog_view.peers[0]
result = _run_observe_wg_contract(
tmp_path, catalog_view, [_state_text(first)], {
peer.public_key: peer.overlay_cidr + ',10.44.0.11/32',
})
assert result.returncode == 0, result.stderr
assert (f'__VC_VPC_WG_STATE__={peer.host_id}|owned\n'
in result.stdout)
allowed = next(
line for line in result.stdout.splitlines()
if line.startswith('__VC_VPC_WG_ALLOWED__='))
assert '10.44.0.11/32' in allowed
assert '10.45.0.11/32' not in allowed
def test_observer_exact_union_withdraws_deleted_same_peer_state(tmp_path):
first, second = _two_vpc_plans()
aggregate = ('10.44.0.11/32', '10.45.0.11/32')
observed_plan = _with_global_peer_tenants(first, *aggregate)
peer = observed_plan.peers[0]
union = peer.overlay_cidr + ',' + ','.join(aggregate)
before = tmp_path / 'before'
before.mkdir()
combined = _run_observe_wg_contract(
before, observed_plan, [_state_text(first), _state_text(second)],
{peer.public_key: union})
assert combined.returncode == 0, combined.stderr
assert f'__VC_VPC_WG_STATE__={peer.host_id}|owned' in combined.stdout
stale_dir = tmp_path / 'stale-after-delete'
stale_dir.mkdir()
stale = _run_observe_wg_contract(
stale_dir, observed_plan, [_state_text(first)],
{peer.public_key: union})
assert stale.returncode == 0, stale.stderr
assert f'__VC_VPC_WG_STATE__={peer.host_id}|missing' in stale.stdout
exact_dir = tmp_path / 'exact-after-delete'
exact_dir.mkdir()
exact = _run_observe_wg_contract(
exact_dir, observed_plan, [_state_text(first)], {
peer.public_key: peer.overlay_cidr + ',10.44.0.11/32',
})
assert exact.returncode == 0, exact.stderr
assert f'__VC_VPC_WG_STATE__={peer.host_id}|owned' in exact.stdout
def test_observer_rejects_stale_tenant_on_authoritative_non_vpc_peer(
tmp_path):
first, second = _two_vpc_plans()
first_key = first.peers[0].public_key
second = replace(second, peers=(replace(
second.peers[0], public_key=OTHER_WG_KEY,
overlay_ip='100.64.0.3'),))
observed = replace(
first,
fabric_peer_keys=frozenset({first_key, OTHER_WG_KEY}),
fabric_peer_overlays=frozenset({
(first_key, '100.64.0.2/32'),
(OTHER_WG_KEY, '100.64.0.3/32'),
}),
)
rows = {
first_key: '100.64.0.2/32,10.44.0.11/32',
OTHER_WG_KEY: '100.64.0.3/32,10.45.0.11/32',
}
exact_dir = tmp_path / 'exact-closure'
exact_dir.mkdir()
exact = _run_observe_wg_contract(
exact_dir, observed, [_state_text(first), _state_text(second)], rows)
assert exact.returncode == 0, exact.stderr
assert '__VC_VPC_WG_PEER_SET__=exact\n' in exact.stdout
wrong_overlay_dir = tmp_path / 'wrong-overlay-closure'
wrong_overlay_dir.mkdir()
wrong_overlay = _run_observe_wg_contract(
wrong_overlay_dir, observed,
[_state_text(first), _state_text(second)], {
**rows,
OTHER_WG_KEY: '100.64.0.4/32,10.45.0.11/32',
})
assert wrong_overlay.returncode == 0, wrong_overlay.stderr
assert '__VC_VPC_WG_PEER_SET__=missing\n' in wrong_overlay.stdout
# VPC2's final route/state was withdrawn while its host remains a valid
# Fabric peer. The stale AllowedIP must invalidate VPC1's host readback too;
# otherwise WireGuard can still route VPC2's deleted address to that key.
stale_dir = tmp_path / 'stale-closure'
stale_dir.mkdir()
stale = _run_observe_wg_contract(
stale_dir, observed, [_state_text(first)], rows)
assert stale.returncode == 0, stale.stderr
assert '__VC_VPC_WG_PEER_SET__=missing\n' in stale.stdout
assert '__VC_VPC_WG_PEER_SET__=exact\n' not in stale.stdout
def test_observer_empty_authoritative_peer_set_requires_successful_readback(
tmp_path):
base = _plan()
plan = replace(
base,
members=tuple(member for member in base.members
if member.host_id == base.host_id),
peers=(),
fabric_peer_keys=frozenset(),
fabric_peer_overlays=frozenset(),
)
result = _run_observe_wg_contract(
tmp_path, plan, [_state_text(plan)], {}, peer_read_status=1)
assert result.returncode == 0, result.stderr
assert '__VC_VPC_WG_PEER_SET__=missing\n' in result.stdout
assert '__VC_VPC_WG_PEER_SET__=exact\n' not in result.stdout
@pytest.mark.parametrize('mode', ['apply', 'observe'])
@ -264,7 +582,7 @@ elif args[:2]==['-c','-f'] or args[:1]==['-f']:
assert len(values)==len(set(values)),values
(root/('checked.batch' if args[0]=='-c' else 'applied.batch')).write_text(batch)
if args[0]!='-c':
if MUTATION=='missing':values.remove('10.46.0.21/32')
if MUTATION=='missing':values.remove('10.44.0.11/32')
elif MUTATION=='extra':values.append('10.47.0.99/32')
(root/'live-set.json').write_text(json.dumps(sorted(values)))
elif args==['list','set','inet','virt_cloud_wg_fabric','vpc_tenant_ips']:
@ -304,7 +622,7 @@ def test_global_tenant_refresh_and_fabric_observer_share_the_exact_universe(
result = _run_multi_vpc_loader(
tmp_path, carrier, [_state_text(carrier)], mode, extra_script=script)
assert result.returncode == 0, result.stderr
wanted = ['10.44.0.10/32', '10.44.0.11/32', '10.46.0.21/32']
wanted = ['10.44.0.10/32', '10.44.0.11/32']
assert (tmp_path / 'checked.batch').read_text() == (
tmp_path / 'applied.batch').read_text()
matches = mutation == 'none'
@ -369,13 +687,14 @@ def test_multi_vpc_state_load_rejects_routes_outside_the_owning_record(
@pytest.mark.parametrize('mode', ['apply', 'observe'])
@pytest.mark.parametrize('reverse', [False, True])
def test_multi_vpc_state_load_rejects_duplicate_peer_ips_across_records(
def test_multi_vpc_state_load_rejects_one_peer_ip_with_different_keys(
tmp_path, mode, reverse):
first, second = _two_vpc_plans()
key = first.peers[0].public_key
duplicate = _state_text(second).replace(
'peer|' + second.peers[0].host_id + '|' + key + '|10.45.0.11/32',
'peer|' + second.peers[0].host_id + '|' + key + '|10.44.0.11/32')
'peer|' + second.peers[0].host_id + '|' + OTHER_WG_KEY
+ '|10.44.0.11/32,10.45.0.11/32')
texts = [_state_text(first), duplicate]
result = _run_multi_vpc_loader(
tmp_path, first, texts[::-1] if reverse else texts, mode)
@ -383,6 +702,46 @@ def test_multi_vpc_state_load_rejects_duplicate_peer_ips_across_records(
assert 'state=owned' not in result.stdout
@pytest.mark.parametrize('mode', ['apply', 'observe'])
def test_vpc_state_load_rejects_duplicate_peer_ip_within_one_state(
tmp_path, mode):
plan = _plan()
text = _state_text(plan)
text += next(line for line in text.splitlines()
if line.startswith('peer|')) + '\n'
result = _run_multi_vpc_loader(tmp_path, plan, [text], mode)
assert result.returncode == (1 if mode == 'observe' else 71), result.stderr
assert 'state=owned' not in result.stdout
@pytest.mark.parametrize('mode', ['apply', 'observe'])
def test_vpc_state_load_requires_one_key_for_every_wireguard_route(
tmp_path, mode):
plan = _plan()
text = '\n'.join(line for line in _state_text(plan).splitlines()
if not line.startswith('peer|')) + '\n'
result = _run_multi_vpc_loader(tmp_path, plan, [text], mode)
assert result.returncode == (1 if mode == 'observe' else 71), result.stderr
assert 'state=owned' not in result.stdout
@pytest.mark.parametrize('mode', ['apply', 'observe'])
@pytest.mark.parametrize('reverse', [False, True])
def test_legacy_aggregate_cannot_mask_missing_peer_in_route_owner(
tmp_path, mode, reverse):
first, second = _two_vpc_plans()
broken_owner = '\n'.join(
line for line in _state_text(first).splitlines()
if not line.startswith('peer|')) + '\n'
legacy_carrier = _legacy_aggregate_state_text(
_with_global_peer_tenants(second, '10.44.0.11/32'))
texts = [broken_owner, legacy_carrier]
result = _run_multi_vpc_loader(
tmp_path, first, texts[::-1] if reverse else texts, mode)
assert result.returncode == (1 if mode == 'observe' else 71), result.stderr
assert 'state=owned' not in result.stdout
@pytest.mark.parametrize('peer_mode', ['none', 'valid', 'duplicate_ip', 'public_ip'])
def test_full_vpc_state_load_handles_peers_under_nounset(tmp_path, peer_mode):
plan = _plan()
@ -432,14 +791,11 @@ printf 'state=%s\n' "$VPC_STATE_STATUS"
))
result = subprocess.run(['/bin/sh', '-c', harness], text=True,
capture_output=True, check=False)
if peer_mode in ('none', 'valid'):
if peer_mode == 'valid':
assert result.returncode == 0, result.stderr
assert result.stdout == 'state=owned\n'
peers = (tmp_path / 'peers').read_text()
if peer_mode == 'none':
assert peers == ''
else:
assert peers == plan.peers[0].public_key + '|10.44.0.11/32\n'
assert peers == plan.peers[0].public_key + '|10.44.0.11/32\n'
else:
assert result.returncode != 0
assert 'state=owned' not in result.stdout
@ -471,17 +827,19 @@ def test_vpc_apply_and_remove_merge_actual_full_fabric_overlay(
plan.peers[0], overlay_ip=expected_overlay.removesuffix('/32')),))
if transaction == 'apply':
script = dataplane_module.render_apply_script(plan)
start = script.index('while IFS=\'|\' read -r peer_host key allowed; do')
start = script.index('expected_fabric_peer_keys=')
end = script.index('# The shared Fabric quarantine table', start)
else:
script = dataplane_module.render_remove_script(replace(
plan, members=tuple(m for m in plan.members
if m.host_id != plan.host_id)))
start = script.index('for key in $("$WG" show "$WG_IFACE" peers')
start = script.index('expected_fabric_peer_keys=')
end = script.index('vc_vpc_fabric_nft_live_ready || exit 74', start)
merge = script[start:end]
peer_file = tmp_path / 'peers'
peer_file.write_text(plan.peers[0].public_key + '|10.44.0.11/32\n')
actual_file = tmp_path / 'actual'
actual_file.write_text(actual)
mutations = tmp_path / 'mutations'
mutations.write_text('')
harness = '\n'.join((
@ -489,7 +847,7 @@ def test_vpc_apply_and_remove_merge_actual_full_fabric_overlay(
_loader_from_script(script),
WG_ALLOWED_IPS_SH,
'VPC_PEER_FILE=' + shlex.quote(str(peer_file)),
'VC_ACTUAL=' + shlex.quote(actual),
'VC_ACTUAL_FILE=' + shlex.quote(str(actual_file)),
'VC_KEY=' + shlex.quote(plan.peers[0].public_key),
'VC_MUTATIONS=' + shlex.quote(str(mutations)),
'WG_IFACE=wg-fabric0',
@ -498,10 +856,11 @@ vc_fixture_wg() {
if [ "$*" = 'show wg-fabric0 peers' ]; then
printf '%s\n' "$VC_KEY"
elif [ "$*" = 'show wg-fabric0 allowed-ips' ]; then
printf '%s\t%s\n' "$VC_KEY" "$VC_ACTUAL"
printf '%s\t%s\n' "$VC_KEY" "$(cat "$VC_ACTUAL_FILE")"
elif [ "$1" = set ] && [ "$2" = wg-fabric0 ] &&
[ "$3" = peer ] && [ "$4" = "$VC_KEY" ] &&
[ "$5" = allowed-ips ]; then
printf '%s' "$6" > "$VC_ACTUAL_FILE"
printf '%s\n' "$6" >> "$VC_MUTATIONS"
else return 9; fi
}
@ -519,6 +878,143 @@ WG=vc_fixture_wg
assert mutations.read_text() == ''
def test_vpc_apply_converges_full_authoritative_fabric_peer_closure(tmp_path):
base = _plan()
first_key = base.peers[0].public_key
plan = replace(
base,
fabric_peer_keys=frozenset({first_key, OTHER_WG_KEY}),
fabric_peer_overlays=frozenset({
(first_key, '100.64.0.2/32'),
(OTHER_WG_KEY, '100.64.0.3/32'),
}),
)
script = dataplane_module.render_apply_script(plan)
block_start = script.index(
'expected_fabric_peer_keys=',
script.index('VC_VPC_APPLY_PHASE=wireguard'))
block_end = script.index(
'# The shared Fabric quarantine table', block_start)
peer_file = tmp_path / 'peers'
peer_file.write_text(first_key + '|10.44.0.11/32\n')
wg_rows = tmp_path / 'wg-rows'
wg_rows.write_text(
first_key + '\t100.64.0.2/32,10.44.0.11/32\n'
+ OTHER_WG_KEY + '\t100.64.0.4/32,10.45.0.11/32\n')
mutations = tmp_path / 'mutations'
mutations.write_text('')
harness = '\n'.join((
'set -eu',
_loader_from_script(script),
WG_ALLOWED_IPS_SH,
'VPC_PEER_FILE=' + shlex.quote(str(peer_file)),
'VC_WG_ROWS=' + shlex.quote(str(wg_rows)),
'VC_WG_MUTATIONS=' + shlex.quote(str(mutations)),
'WG_IFACE=wg-fabric0',
r'''
vc_fixture_wg() {
if [ "$*" = 'show wg-fabric0 peers' ]; then
cut -f1 "$VC_WG_ROWS"
elif [ "$*" = 'show wg-fabric0 allowed-ips' ]; then
cat "$VC_WG_ROWS"
elif [ "$1" = set ] && [ "$2" = wg-fabric0 ] &&
[ "$3" = peer ] && [ "$5" = allowed-ips ] && [ "$#" = 6 ]; then
awk -F '\t' -v OFS='\t' -v wanted="$4" -v allowed="$6" '
$1 == wanted {$2=allowed; found++} {print}
END {exit found == 1 ? 0 : 1}
' "$VC_WG_ROWS" > "$VC_WG_ROWS.next" || return 9
mv -f "$VC_WG_ROWS.next" "$VC_WG_ROWS" || return 9
printf '%s|%s\n' "$4" "$6" >> "$VC_WG_MUTATIONS"
else return 9; fi
}
WG=vc_fixture_wg
'''.strip(),
script[block_start:block_end],
))
result = subprocess.run(['/bin/sh', '-c', harness], text=True,
capture_output=True, check=False)
assert result.returncode == 0, result.stderr
assert wg_rows.read_text().splitlines() == [
first_key + '\t100.64.0.2/32,10.44.0.11/32',
OTHER_WG_KEY + '\t100.64.0.3/32',
]
assert mutations.read_text().splitlines() == [
first_key + '|100.64.0.2/32,10.44.0.11/32',
OTHER_WG_KEY + '|100.64.0.3/32',
]
def test_vpc_remove_converges_authoritative_closure_for_retiring_host(
tmp_path):
base = _plan()
first_key = base.peers[0].public_key
plan = replace(
base,
members=tuple(member for member in base.members
if member.host_id != base.host_id),
fabric_peer_keys=frozenset({first_key, OTHER_WG_KEY}),
fabric_peer_overlays=frozenset({
(first_key, '100.64.0.2/32'),
(OTHER_WG_KEY, '100.64.0.3/32'),
}),
)
script = dataplane_module.render_remove_script(plan)
block_start = script.index('expected_fabric_peer_keys=')
block_end = script.index(
'vc_vpc_fabric_nft_live_ready || exit 74', block_start)
peer_file = tmp_path / 'peers'
peer_file.write_text(OTHER_WG_KEY + '|10.45.0.11/32\n')
wg_rows = tmp_path / 'wg-rows'
wg_rows.write_text(
first_key + '\t100.64.0.8/32,10.44.0.11/32\n'
+ OTHER_WG_KEY + '\t100.64.0.9/32,10.45.0.11/32\n')
mutations = tmp_path / 'mutations'
mutations.write_text('')
harness = '\n'.join((
'set -eu',
_loader_from_script(script),
WG_ALLOWED_IPS_SH,
'VPC_PEER_FILE=' + shlex.quote(str(peer_file)),
'VC_WG_ROWS=' + shlex.quote(str(wg_rows)),
'VC_WG_MUTATIONS=' + shlex.quote(str(mutations)),
'WG_IFACE=wg-fabric0',
r'''
vc_fixture_wg() {
if [ "$*" = 'show wg-fabric0 peers' ]; then
cut -f1 "$VC_WG_ROWS"
elif [ "$*" = 'show wg-fabric0 allowed-ips' ]; then
cat "$VC_WG_ROWS"
elif [ "$1" = set ] && [ "$2" = wg-fabric0 ] &&
[ "$3" = peer ] && [ "$5" = allowed-ips ] && [ "$#" = 6 ]; then
awk -F '\t' -v OFS='\t' -v wanted="$4" -v allowed="$6" '
$1 == wanted {$2=allowed; found++} {print}
END {exit found == 1 ? 0 : 1}
' "$VC_WG_ROWS" > "$VC_WG_ROWS.next" || return 9
mv -f "$VC_WG_ROWS.next" "$VC_WG_ROWS" || return 9
printf '%s|%s\n' "$4" "$6" >> "$VC_WG_MUTATIONS"
else return 9; fi
}
WG=vc_fixture_wg
'''.strip(),
script[block_start:block_end],
))
result = subprocess.run(['/bin/sh', '-c', harness], text=True,
capture_output=True, check=False)
assert result.returncode == 0, result.stderr
assert wg_rows.read_text().splitlines() == [
first_key + '\t100.64.0.2/32',
OTHER_WG_KEY + '\t100.64.0.3/32,10.45.0.11/32',
]
assert mutations.read_text().splitlines() == [
first_key + '|100.64.0.2/32',
OTHER_WG_KEY + '|100.64.0.3/32,10.45.0.11/32',
]
@pytest.mark.parametrize('malformed', [
'100.99.106.2/32\n100.64.0.2/32',
'100.99.106.2/32 10.44.0.99/32',

@ -113,18 +113,20 @@ def test_generated_production_paths_read_complete_unordered_peer_rows(
body = ''
if operation == 'apply':
script = dataplane.render_apply_script(plan)
start = script.index("while IFS='|' read -r peer_host key allowed; do")
start = script.index(
'expected_fabric_peer_keys=',
script.index('VC_VPC_APPLY_PHASE=wireguard'))
end = script.index('# The shared Fabric quarantine table', start)
body = script[start:end]
elif operation == 'remove':
script = dataplane.render_remove_script(replace(
plan, members=tuple(m for m in plan.members if m.host_id != plan.host_id)))
start = script.index('for key in $("$WG" show "$WG_IFACE" peers')
start = script.index('expected_fabric_peer_keys=')
end = script.index('vc_vpc_fabric_nft_live_ready || exit 74', start)
body = script[start:end]
elif operation == 'vpc_observe':
script = dataplane.render_observe_script(plan)
start = script.index("while IFS='|' read -r peer_host key expected_allowed; do")
start = script.index('vpc_state_loaded=no')
end = script.index('expected_fabric_peer_keys=', start)
body = script[start:end]
else:
@ -136,6 +138,10 @@ def test_generated_production_paths_read_complete_unordered_peer_rows(
if operation in ('apply', 'remove'):
body = ('vpc_state_allowed_for_peer() { printf "10.44.0.11/32,10.44.0.99/32"; }\n'
+ body)
elif operation == 'vpc_observe':
body = ('vpc_state_load() { VPC_STATE_STATUS=owned; }\n'
'vpc_state_allowed_for_peer() { printf "10.44.0.11/32,10.44.0.99/32"; }\n'
+ body)
result = run_shell(body, rows, mutation_file=mutations)
assert result.returncode == 0, result.stderr
if operation in ('apply', 'remove'):
@ -173,22 +179,25 @@ def test_generated_paths_never_accept_partial_or_failed_readback(
mutations = tmp_path / 'mutations'
if operation == 'apply':
script = dataplane.render_apply_script(plan)
start = script.index("while IFS='|' read -r peer_host key allowed; do")
start = script.index(
'expected_fabric_peer_keys=',
script.index('VC_VPC_APPLY_PHASE=wireguard'))
end = script.index('# The shared Fabric quarantine table', start)
elif operation == 'remove':
script = dataplane.render_remove_script(replace(
plan, members=tuple(m for m in plan.members if m.host_id != plan.host_id)))
start = script.index('for key in $("$WG" show "$WG_IFACE" peers')
start = script.index('expected_fabric_peer_keys=')
end = script.index('vc_vpc_fabric_nft_live_ready || exit 74', start)
elif operation == 'vpc_observe':
script = dataplane.render_observe_script(plan)
start = script.index("while IFS='|' read -r peer_host key expected_allowed; do")
start = script.index('vpc_state_loaded=no')
end = script.index('expected_fabric_peer_keys=', start)
else:
script = fabric._observe_script((), expected_digest=FABRIC_DIGEST)
start = script.index(' $WG show "$IFACE" peers 2>/dev/null | while')
end = script.index(' $IP -N -o route show', start)
body = ('vpc_state_allowed_for_peer() { printf "10.44.0.11/32"; }\n'
body = ('vpc_state_load() { VPC_STATE_STATUS=owned; }\n'
'vpc_state_allowed_for_peer() { printf "10.44.0.11/32,10.44.0.99/32"; }\n'
+ script[start:end])
result = run_shell(body, rows, mutation_file=mutations,
command_status=1 if scenario == 'command_failed' else 0)

@ -11,6 +11,7 @@ from __future__ import annotations
from contextlib import contextmanager
from contextvars import ContextVar
from dataclasses import dataclass, field
import hashlib
import json
import re
import secrets
@ -24,6 +25,8 @@ RECORD_KEY = "admission-v1"
_IDENTIFIER = re.compile(r"[A-Za-z0-9_.:-]{1,191}\Z")
_TOKEN = re.compile(r"[A-Za-z0-9_-]{16,128}\Z")
_KIND = re.compile(r"[a-z][a-z0-9_-]{0,63}\Z")
_EXPLICIT_VPC_RECONCILE_METADATA = {"explicit_vpc_reconcile": True}
_RESOLVE_AND_BIND_RECEIPT_KIND = "resolve_and_bind_v1"
_UNKNOWN_CODES = frozenset({
"libvirt_execution_uncertain", "libvirt_execution_blocked",
"coordination_uncertain", "operation_uncertain",
@ -117,6 +120,18 @@ def _resources(values) -> tuple[str, ...]:
return result
def _digest(value: Any) -> str:
return hashlib.sha256(_encode(value).encode("ascii")).hexdigest()
def _claim_envelope(record: Mapping[str, Any]) -> dict[str, Any]:
"""Return the immutable admission identity used by handoff receipts."""
return {key: record.get(key) for key in (
"token", "execution_id", "owner_id", "operation_id", "resources",
"maintenance", "kind", "metadata",
)}
class DurableCoordinator:
def __init__(self, store, *, owner_id: str | None = None, cas_attempts: int = 32,
max_claims: int = 1024):
@ -356,6 +371,119 @@ class DurableCoordinator:
def snapshot(self) -> dict[str, dict]:
return self._read()[1]["claims"]
def bind_operation(self, claim: Claim, operation_id: str,
metadata: Mapping[str, Any]) -> Claim:
"""Bind a validated operation envelope before its first resource claim.
Identity, ownership and execution evidence stay unchanged. An existing
binding is immutable, and another execution's same operation remains
fenced even when its resource set is empty.
"""
if not isinstance(operation_id, str) or not _IDENTIFIER.fullmatch(operation_id):
raise ValueError("Invalid coordination operation")
if not isinstance(metadata, Mapping):
raise ValueError("Invalid coordination metadata")
additions = json.loads(_encode(dict(metadata)))
def mutate(document):
current = self._owned(document, claim)
if current is None:
raise CoordinationError("coordination_unavailable", 503)
if current["state"] != "active":
raise CoordinationClaimRejected("coordination_uncertain", 503)
if (current["operation_id"] not in {None, operation_id}
or any(key in current["metadata"] and current["metadata"][key] != value
for key, value in additions.items())):
raise CoordinationClaimRejected("coordination_busy", 409)
for existing in document["claims"].values():
if existing["token"] == current["token"]:
continue
if existing["maintenance"] or (
(existing["state"] == "uncertain"
or existing["execution_id"] != current["execution_id"])
and existing["operation_id"] == operation_id
):
unknown = existing["state"] == "uncertain"
raise CoordinationClaimRejected(
"coordination_uncertain" if unknown else "coordination_busy",
503 if unknown else 409)
combined = {**current["metadata"], **additions}
if len(_encode(combined)) > 8192:
raise ValueError("Coordination metadata is too large")
changed = current["operation_id"] != operation_id or combined != current["metadata"]
current.update(operation_id=operation_id, metadata=combined,
updated_at=int(time.time()))
return changed, Claim(current["token"], current["execution_id"], current["owner_id"],
operation_id, tuple(current["resources"]), current["kind"],
json.loads(_encode(combined)))
return self._change(mutate)
def bind_explicit_operation(self, claim: Claim | Mapping[str, Any],
operation_id: str,
metadata: Mapping[str, Any]) -> Claim:
"""Consume the one-shot VPC reconcile capability while binding.
This stricter sibling of :meth:`bind_operation` is used only by an
explicitly authorized VPC request. The marker is replaced by the
signed target identity in the same CAS, so nested resource claims
cannot inherit a capability that was meant only for the HTTP root.
"""
replacement_identity = _identity(claim)
if not isinstance(operation_id, str) or not _IDENTIFIER.fullmatch(operation_id):
raise ValueError("Invalid coordination operation")
if not isinstance(metadata, Mapping):
raise ValueError("Invalid coordination metadata")
target_metadata = json.loads(_encode(dict(metadata)))
if len(_encode(target_metadata)) > 8192:
raise ValueError("Coordination metadata is too large")
def claim_value(name, default=None):
if isinstance(claim, Mapping):
return claim.get(name, default)
return getattr(claim, name, default)
if (replacement_identity[2] != self.owner_id
or claim_value("operation_id") is not None
or tuple(claim_value("resources", ())) != ()
or claim_value("kind") != "request"
or claim_value("metadata", {}) != _EXPLICIT_VPC_RECONCILE_METADATA):
raise ValueError("Invalid explicit reconciliation root")
def mutate(document):
current = self._owned(document, claim)
if (current is None
or current["state"] != "active"
or current["maintenance"]
or current["operation_id"] is not None
or current["resources"] != []
or current["kind"] != "request"
or current["metadata"] != _EXPLICIT_VPC_RECONCILE_METADATA):
raise CoordinationClaimRejected("coordination_uncertain", 503)
conflicts = [
existing for existing in document["claims"].values()
if existing["token"] != current["token"]
and (existing["execution_id"] == current["execution_id"]
or existing["operation_id"] == operation_id)
]
if conflicts:
unknown = any(existing["state"] == "uncertain"
for existing in conflicts)
raise CoordinationClaimRejected(
"coordination_uncertain" if unknown else "coordination_busy",
503 if unknown else 409)
current.update(
operation_id=operation_id,
metadata=json.loads(_encode(target_metadata)),
updated_at=int(time.time()),
)
return True, Claim(
current["token"], current["execution_id"], current["owner_id"],
operation_id, (), "request",
json.loads(_encode(target_metadata)))
return self._change(mutate)
def operation_claim(self, operation_id: str) -> dict | None:
matches = [claim for claim in self.snapshot().values()
if claim["operation_id"] == operation_id]
@ -448,6 +576,208 @@ class DurableCoordinator:
return self._change(mutate)
def resolve_and_bind(self, claim: Mapping[str, Any], replacement: Claim | Mapping[str, Any],
evidence: str, *, operation_id: str,
metadata: Mapping[str, Any]) -> Claim:
"""Atomically retire an exact uncertain owner into an active request.
The replacement must already be an active request owned by this
coordinator. Its token is preserved, so an HTTP request can take over
an explicitly reconciled operation without a release/admission gap.
"""
if not isinstance(evidence, str) or not evidence.strip() or len(evidence) > 1024:
raise ValueError("Explicit reconciliation evidence is required")
_identity(claim)
replacement_identity = _identity(replacement)
if not isinstance(operation_id, str) or not _IDENTIFIER.fullmatch(operation_id):
raise ValueError("Invalid coordination operation")
if not isinstance(metadata, Mapping):
raise ValueError("Invalid coordination metadata")
target_metadata = json.loads(_encode(dict(metadata)))
if len(_encode(target_metadata)) > 8192:
raise ValueError("Coordination metadata is too large")
def replacement_value(name, default=None):
if isinstance(replacement, Mapping):
return replacement.get(name, default)
return getattr(replacement, name, default)
if (replacement_identity[2] != self.owner_id
or replacement_value("operation_id") is not None
or tuple(replacement_value("resources", ())) != ()
or replacement_value("kind") != "request"
or replacement_value("metadata", {})
!= _EXPLICIT_VPC_RECONCILE_METADATA):
raise ValueError("Invalid explicit reconciliation root")
def mutate(document):
original = self._owned(document, claim)
current = document["claims"].get(replacement_identity[0])
if (original is None or original != dict(claim)
or original["state"] != "uncertain"
or original["operation_id"] != operation_id
or original["resources"] != []
or original["maintenance"]
or original["kind"] not in {"background", "request"}
or original["metadata"] != target_metadata
or current is None
or (current["execution_id"], current["owner_id"])
!= replacement_identity[1:]
or current["owner_id"] != self.owner_id
or current["token"] == original["token"]
or current["state"] != "active"
or current["maintenance"]
or current["kind"] != "request"
or current["operation_id"] is not None
or current["resources"] != []
or current["metadata"]
!= _EXPLICIT_VPC_RECONCILE_METADATA):
raise CoordinationClaimRejected("coordination_uncertain", 503)
conflicts = [
existing for existing in document["claims"].values()
if existing["token"] not in {original["token"], current["token"]}
and (existing["execution_id"] in {
original["execution_id"], current["execution_id"]}
or existing["operation_id"] == operation_id)
]
if conflicts:
unknown = any(existing["state"] == "uncertain"
for existing in conflicts)
raise CoordinationClaimRejected(
"coordination_uncertain" if unknown else "coordination_busy",
503 if unknown else 409)
now = int(time.time())
current.update(
operation_id=operation_id, resources=[],
metadata=json.loads(_encode(target_metadata)), updated_at=now)
receipt = {
"kind": _RESOLVE_AND_BIND_RECEIPT_KIND,
"token": original["token"],
"operation_id": operation_id,
"resolved_by": self.owner_id,
"resolved_at": now,
"evidence": evidence.strip(),
"source_claim_sha256": _digest(original),
"replacement_token": current["token"],
"replacement_identity_sha256": _digest(
_claim_envelope(current)),
}
receipts = document.setdefault("resolutions", [])
receipts.append(receipt)
document["resolutions"] = receipts[-100:]
del document["claims"][original["token"]]
return True, Claim(
current["token"], current["execution_id"], current["owner_id"],
operation_id, (), "request",
json.loads(_encode(target_metadata)))
return self._change(mutate)
def classify_resolve_and_bind(
self, claim: Mapping[str, Any], replacement: Claim | Mapping[str, Any],
evidence: str, *, operation_id: str,
metadata: Mapping[str, Any]) -> tuple[str, dict[str, Any] | None]:
"""Classify a lost handoff response from one exact durable snapshot.
``committed`` requires the source to be gone, the request to carry the
exact bound VPC identity, and one exact receipt written by the same CAS.
``uncommitted`` requires both original rows to remain unchanged and no
matching receipt. Every mixed state is deliberately unclassifiable.
"""
if not isinstance(evidence, str) or not evidence.strip() or len(evidence) > 1024:
raise ValueError("Explicit reconciliation evidence is required")
_identity(claim)
replacement_identity = _identity(replacement)
if not isinstance(operation_id, str) or not _IDENTIFIER.fullmatch(operation_id):
raise ValueError("Invalid coordination operation")
if not isinstance(metadata, Mapping):
raise ValueError("Invalid coordination metadata")
target_metadata = json.loads(_encode(dict(metadata)))
if len(_encode(target_metadata)) > 8192:
raise ValueError("Coordination metadata is too large")
def replacement_value(name, default=None):
if isinstance(replacement, Mapping):
return replacement.get(name, default)
return getattr(replacement, name, default)
if (replacement_identity[2] != self.owner_id
or replacement_value("operation_id") is not None
or tuple(replacement_value("resources", ())) != ()
or replacement_value("kind") != "request"
or replacement_value("metadata", {})
!= _EXPLICIT_VPC_RECONCILE_METADATA):
raise ValueError("Invalid explicit reconciliation root")
_raw, document = self._read()
original = document["claims"].get(claim["token"])
current = document["claims"].get(replacement_identity[0])
other_conflict = any(
existing["token"] not in {claim["token"], replacement_identity[0]}
and (existing["execution_id"] in {
claim["execution_id"], replacement_identity[1]}
or existing["operation_id"] == operation_id)
for existing in document["claims"].values()
)
anonymous = bool(
current is not None
and (current["execution_id"], current["owner_id"])
== replacement_identity[1:]
and current["state"] == "active"
and current["maintenance"] is False
and current["operation_id"] is None
and current["resources"] == []
and current["kind"] == "request"
and current["metadata"] == _EXPLICIT_VPC_RECONCILE_METADATA)
bound = bool(
current is not None
and (current["execution_id"], current["owner_id"])
== replacement_identity[1:]
and current["state"] in {"active", "uncertain"}
and current["maintenance"] is False
and current["operation_id"] == operation_id
and current["resources"] == []
and current["kind"] == "request"
and current["metadata"] == target_metadata)
expected_receipt = {
"kind": _RESOLVE_AND_BIND_RECEIPT_KIND,
"token": claim["token"],
"operation_id": operation_id,
"resolved_by": self.owner_id,
"evidence": evidence.strip(),
"source_claim_sha256": _digest(dict(claim)),
"replacement_token": replacement_identity[0],
"replacement_identity_sha256": (
_digest(_claim_envelope(current)) if bound else None),
}
def receipt_matches(receipt) -> bool:
return bool(
isinstance(receipt, dict)
and set(receipt) == {*expected_receipt, "resolved_at"}
and isinstance(receipt.get("resolved_at"), int)
and not isinstance(receipt.get("resolved_at"), bool)
and receipt["resolved_at"] > 0
and all(receipt.get(key) == value
for key, value in expected_receipt.items()))
related_receipts = [
receipt for receipt in document.get("resolutions", [])
if (isinstance(receipt, dict)
and (receipt.get("token") == claim["token"]
or receipt.get("replacement_token")
== replacement_identity[0]))
]
matching_receipts = [receipt for receipt in related_receipts
if receipt_matches(receipt)]
if (original is None and bound and not other_conflict
and len(matching_receipts) == len(related_receipts) == 1):
return "committed", current
if (original == dict(claim) and anonymous and not other_conflict
and not related_receipts):
return "uncommitted", current
return "mixed", current
def configure_default_coordinator(coordinator: DurableCoordinator | None) -> None:
global _default_coordinator

@ -66,7 +66,7 @@ def parse_actor(encoded: str) -> dict:
raise EngineAuthError()
if value["is_super_admin"] and not value["is_admin"]:
raise EngineAuthError()
for key in ("can_provision", "system_cleanup"):
for key in ("can_provision", "system_cleanup", "vpc_reconcile_explicit"):
if key in value and not isinstance(value[key], bool):
raise EngineAuthError()
if "operation_id" in value and not re.fullmatch(r"[A-Za-z0-9_-]{8,80}", str(value["operation_id"])):

@ -395,7 +395,7 @@ vpc_state_load() {
: > "$VPC_TENANT_FILE" || vpc_state_fail || return 1
chmod 600 "$VPC_TENANT_FILE" || vpc_state_fail || return 1
vpc_state_files=0
vpc_state_seen_ips=
vpc_state_seen_peer_records=
vpc_state_seen_route_ips=
for vpc_state_file in "$VPC_STATE_DIR"/*.state; do
[ -e "$vpc_state_file" ] || [ -L "$vpc_state_file" ] || continue
@ -425,6 +425,7 @@ vpc_state_load() {
vpc_state_firewall_mode=
vpc_state_table=
vpc_state_bridge=
vpc_state_file_seen_ips=
while IFS='|' read -r vpc_kind vpc_a vpc_b vpc_c vpc_d; do
case "$vpc_kind" in
owner)
@ -540,19 +541,27 @@ vpc_state_load() {
vpc_state_allowed "$vpc_ip" || { IFS=$vpc_old_ifs; vpc_state_fail || return 1; }
vpc_state_private "$vpc_ip" || { IFS=$vpc_old_ifs; vpc_state_fail || return 1; }
vpc_state_in_fabric "$vpc_ip" && { IFS=$vpc_old_ifs; vpc_state_fail || return 1; }
case " $vpc_state_seen_ips " in
case " $vpc_state_file_seen_ips " in
*" $vpc_ip "*) IFS=$vpc_old_ifs; vpc_state_fail || return 1 ;;
esac
vpc_state_seen_ips="$vpc_state_seen_ips $vpc_ip"
printf '%s|%s\n' "$vpc_b" "$vpc_ip" >> "$VPC_PEER_FILE" || {
IFS=$vpc_old_ifs; vpc_state_fail || return 1;
}
# A trusted peer snapshot may include another VPC on
# this Fabric. The shared nft set must cover the same
# route/peer universe that the observer verifies.
printf '%s\n' "$vpc_ip" >> "$VPC_TENANT_FILE" || {
IFS=$vpc_old_ifs; vpc_state_fail || return 1;
}
vpc_state_file_seen_ips="$vpc_state_file_seen_ips $vpc_ip"
vpc_peer_record="$vpc_ip|$vpc_b"
case " $vpc_state_seen_peer_records " in
*" $vpc_peer_record "*) : ;;
*" $vpc_ip|"*)
IFS=$vpc_old_ifs
vpc_state_fail || return 1 ;;
*)
vpc_state_seen_peer_records="$vpc_state_seen_peer_records $vpc_peer_record"
printf '%s|%s\n' "$vpc_b" "$vpc_ip" >> "$VPC_PEER_FILE" || {
IFS=$vpc_old_ifs; vpc_state_fail || return 1;
}
# Keep validated legacy aggregate entries as
# candidates only. After every state is parsed,
# live wg-fabric0 routes select the exact peer
# contribution that remains authorized.
: ;;
esac
done
IFS=$vpc_old_ifs ;;
'') : ;;
@ -590,11 +599,48 @@ vpc_state_load() {
[ -z "$vpc_route_extra" ] || vpc_state_fail || return 1
vpc_state_cidr_contains "$vpc_route_prefix" "$vpc_state_cidr_value" ||
vpc_state_fail || return 1
if [ "$vpc_route_dev" = wg-fabric0 ]; then
case " $vpc_state_file_seen_ips " in
*" $vpc_route_prefix "*) : ;;
*) vpc_state_fail || return 1 ;;
esac
fi
done < "$vpc_state_file"
done
# Route and peer records naturally overlap. Deduplicate only the derived
# tenant universe; the independent route/peer conflict checks above must
# still reject ambiguous source records before any nft refresh.
# Older state writers copied the complete Fabric peer aggregate into each
# VPC record. Validate those entries above, but authorize only IPs that
# still have one globally unique wg-fabric0 route after all records have
# passed their own CIDR checks. This preserves mixed-version reads while
# deleting one state immediately withdraws its stale aggregate entries.
vpc_peer_final="$VPC_PEER_FILE.route-backed"
rm -f "$vpc_peer_final" 2>/dev/null || vpc_state_fail || return 1
: > "$vpc_peer_final" || vpc_state_fail || return 1
chmod 600 "$vpc_peer_final" || {
rm -f "$vpc_peer_final"; vpc_state_fail || return 1;
}
if ! awk -F'|' '
stage != "routes" {
if (NF != 2 || $1 == "" || $2 == "" || candidate[$2] != "") bad=1
else candidate[$2]=$1
next
}
{
if (NF != 5 || $1 == "" || $2 == "") {bad=1; next}
if ($2 == "wg-fabric0") {
if (!($1 in candidate)) bad=1
else print candidate[$1] "|" $1
}
}
END {exit bad ? 1 : 0}
' "$VPC_PEER_FILE" stage=routes "$VPC_ROUTE_FILE" > "$vpc_peer_final"; then
rm -f "$vpc_peer_final"
vpc_state_fail || return 1
fi
mv -f "$vpc_peer_final" "$VPC_PEER_FILE" || {
rm -f "$vpc_peer_final"; vpc_state_fail || return 1;
}
# The Fabric nft set covers both ends of routed traffic, including local
# bridge routes. Aggregate-only peer entries never enter this file.
LC_ALL=C sort -u -o "$VPC_TENANT_FILE" "$VPC_TENANT_FILE" || vpc_state_fail || return 1
if [ "$vpc_state_files" -gt 0 ]; then VPC_STATE_STATUS=owned; fi
return 0
@ -2774,7 +2820,7 @@ _BOOT_HELPER = r'''#!/bin/sh
PLAN_DIGEST="__VC_FABRIC_DIGEST__"
# libvirt-startup-independent-v1
# vpc-local-neighbor-contract-v1
# vpc-state-route-scope-contract-v3
# vpc-state-route-scope-contract-v4
EXPECTED_DIGEST="$PLAN_DIGEST"
EXPECTED_NFT_RULES_SHA256="__VC_FABRIC_NFT_RULES_SHA256__"
DEPLOYMENT_MODE="__VC_FABRIC_MODE__"
@ -8430,7 +8476,7 @@ artifact_state() {{
if [ "$artifact_kind" = helper ]; then
grep -Fqx -- '# vpc-local-neighbor-contract-v1' "$artifact_path" ||
{{ printf '%s' foreign; return; }}
grep -Fqx -- '# vpc-state-route-scope-contract-v3' "$artifact_path" ||
grep -Fqx -- '# vpc-state-route-scope-contract-v4' "$artifact_path" ||
{{ printf '%s' foreign; return; }}
fi
if [ "$artifact_kind" = helper ] &&

@ -48,10 +48,10 @@ from .engine_context import (EngineComponents, HostStore, LogStore, MainStore,
TRUSTED_ACTOR, actor_session)
from .engine_security import (EngineAuthError, authenticate, decode_secret,
signed_headers)
from .fabric import FabricCoordinator, FabricError
from .fabric import FabricCoordinator, FabricError, _await_mutation_completion
from .libvirt_rpc import EXECUTION_ERROR_CODES as LIBVIRT_EXECUTION_ERROR_CODES
from .coordination import (
DurableCoordinator, CoordinationError, CoordinationClaimRejected,
Claim, DurableCoordinator, CoordinationError, CoordinationClaimRejected,
current_execution, execution_context,
configure_default_coordinator, update_current_claim, _uncertain_exception,
connection_failure_before_write, execution_failure_before_write,
@ -419,43 +419,181 @@ class _DurableAsyncLock(asyncio.Lock):
self.coordinator = coordinator
self.resources = resources
def _matches_expected_claim(self, record, *, token=None):
return (isinstance(record, dict)
and (token is None or record.get("token") == token)
and record.get("execution_id") == self._claim_execution_id
and record.get("owner_id") == self.coordinator.owner_id
and record.get("operation_id") == self._claim_operation_id
and tuple(record.get("resources", ())) == self._claim_resources
and record.get("kind") == "resource"
and record.get("metadata") == self._claim_metadata
and record.get("maintenance") is False
and record.get("state") in {"active", "uncertain"})
async def _coordination_call(self, function, *args, **kwargs):
return await _await_mutation_completion(
asyncio.to_thread(function, *args, **kwargs))
async def _recover_expected_claim(self, *, token=None):
rows = await self._coordination_call(self.coordinator.snapshot)
matches = [record for record in rows.values()
if self._matches_expected_claim(record, token=token)]
if len(matches) > 1:
raise CoordinationError("coordination_unavailable", 503)
if not matches:
return None, None
record = matches[0]
claim = Claim(record["token"], record["execution_id"],
record["owner_id"], record["operation_id"],
tuple(record["resources"]), record["kind"],
json.loads(json.dumps(record["metadata"])))
return claim, record["state"]
async def _retain_expected_claim(self):
token = self.claim.token if self.claim is not None else None
claim, state = await self._recover_expected_claim(token=token)
if claim is None:
return False
self.claim = claim
if state == "uncertain":
return True
try:
retained = await self._coordination_call(
self.coordinator.retain, claim)
except BaseException:
# Retain is idempotent. A lost CAS response is confirmed from the
# exact immutable child identity before deciding whether to retry.
recovered, recovered_state = await self._recover_expected_claim(
token=claim.token)
if recovered is None:
return False
self.claim = recovered
if recovered_state == "uncertain":
return True
retained = await self._coordination_call(
self.coordinator.retain, recovered)
if retained:
return True
recovered, recovered_state = await self._recover_expected_claim(
token=claim.token)
return recovered is not None and recovered_state == "uncertain"
async def _release_or_retain_expected_claim(self):
try:
released = await self._coordination_call(
self.coordinator.release, self.claim)
except BaseException:
# Never replay an unconfirmed release. If it did not commit, turn
# the exact still-owned child into an uncertainty fence instead.
recovered, _state = await self._recover_expected_claim(
token=self.claim.token)
if recovered is None:
return
self.claim = recovered
if not await self._retain_expected_claim():
raise CoordinationError("coordination_unavailable", 503)
raise
if released:
return
recovered, _state = await self._recover_expected_claim(
token=self.claim.token)
if recovered is None:
return
self.claim = recovered
if not await self._retain_expected_claim():
raise CoordinationError("coordination_unavailable", 503)
raise CoordinationError("coordination_uncertain", 503)
async def _claim_and_publish(self):
try:
self.claim = await self._coordination_call(
self.coordinator.claim, resources=self._claim_resources,
operation_id=self._claim_operation_id,
execution_id=self._claim_execution_id,
kind="resource", metadata=self._claim_metadata)
except CoordinationClaimRejected:
# A confirmed rejection proves this execution installed no child.
raise
except BaseException:
# A failed/lost claim response can follow a committed CAS. Recover
# only one exact child; a broader match could seize another task.
recovered, _state = await self._recover_expected_claim()
if recovered is not None:
self._claim_response_unknown = True
self.claim = recovered
if not await self._retain_expected_claim():
raise CoordinationError("coordination_unavailable", 503)
raise
async def __aenter__(self):
await self.acquire()
context = current_execution()
claim_operation_id = context.claim.operation_id if context else None
claim_execution_id = (context.claim.execution_id if context else
secrets.token_hex(32))
while claim_execution_id == self.coordinator.owner_id:
claim_execution_id = secrets.token_hex(32)
claim_resources = tuple(sorted(set(self.resources)))
claim_metadata = json.loads(json.dumps(
context.claim.metadata if context else {}, ensure_ascii=True,
allow_nan=False, separators=(",", ":"), sort_keys=True))
await self.acquire()
try:
self.claim = await asyncio.to_thread(
self.coordinator.claim, resources=self.resources,
operation_id=context.claim.operation_id if context else None,
execution_id=context.claim.execution_id if context else None,
kind="resource", metadata=context.claim.metadata if context else None)
self.claim = None
self.context = None
self._claim_response_unknown = False
self._claim_operation_id = claim_operation_id
self._claim_execution_id = claim_execution_id
self._claim_resources = claim_resources
self._claim_metadata = claim_metadata
# Publish self.claim inside the shielded child so caller
# cancellation cannot lose a successfully returned admission.
await _await_mutation_completion(self._claim_and_publish())
self.context = execution_context(self.coordinator, self.claim)
self.context.__enter__()
except BaseException:
self.release()
try:
if self.claim is not None:
settlement = (self._retain_expected_claim()
if self._claim_response_unknown else
self._release_or_retain_expected_claim())
settled = await _await_mutation_completion(settlement)
if self._claim_response_unknown and not settled:
raise CoordinationError("coordination_unavailable", 503)
finally:
self.release()
raise
self.context = execution_context(self.coordinator, self.claim)
self.context.__enter__()
return self
async def _settle(self, exc):
not_started = False
if (exc is not None and self.claim.operation_id
and self.claim.metadata.get("job_kind") != "vpc_operation"):
try:
raw = await self._coordination_call(
self.coordinator.store.get, "operation",
self.claim.operation_id)
record = json.loads(raw) if raw else {}
not_started = record.get("mutation_started") is False
except Exception:
pass
context = current_execution()
if context is not None and context.execution_unknown.is_set():
not_started = False
if exc is not None and _uncertain_exception(exc) and not not_started:
if not await self._retain_expected_claim():
raise CoordinationError("coordination_unavailable", 503)
return
await self._release_or_retain_expected_claim()
async def __aexit__(self, exc_type, exc, tb):
try:
not_started = False
if exc is not None and self.claim.operation_id:
try:
raw = await asyncio.to_thread(
self.coordinator.store.get, "operation", self.claim.operation_id)
record = json.loads(raw) if raw else {}
not_started = record.get("mutation_started") is False
except Exception:
pass
if current_execution().execution_unknown.is_set():
not_started = False
if exc is not None and _uncertain_exception(exc) and not not_started:
await asyncio.to_thread(self.coordinator.retain, self.claim)
else:
if not await asyncio.to_thread(self.coordinator.release, self.claim):
raise CoordinationError("coordination_uncertain", 503)
await _await_mutation_completion(self._settle(exc))
finally:
self.context.__exit__(exc_type, exc, tb)
self.release()
try:
self.context.__exit__(exc_type, exc, tb)
finally:
self.release()
class EngineApplication:
@ -490,6 +628,7 @@ class EngineApplication:
store, self.fabric, self.registry,
vm_lock=lambda host_id, vm_id: self._target_lock(host_id, vm_id),
catalog_lock=self._network_catalog_mutation_lock,
coordinator=self.coordinator,
)
self.introspection = introspection
project_dir = Path(config.get("project_dir", Path(__file__).resolve().parents[2]))
@ -874,6 +1013,9 @@ class EngineApplication:
method == "POST" and path == "/internal/v1/hosts/scan")
fabric_admission_request = bool(method == "POST" and re.fullmatch(
r"/internal/v1/fabric/(?:apply|[A-Za-z0-9][A-Za-z0-9_.-]{0,63}/reconcile)", path))
vpc_write_request = bool(
method != "GET" and (path == "/internal/v1/vpcs/apply"
or path.startswith("/internal/v1/vpcs/")))
async def execute():
await asyncio.to_thread(self._fence_legacy_operations)
@ -882,7 +1024,11 @@ class EngineApplication:
claim = await asyncio.to_thread(
self.coordinator.claim,
operation_id=op_id if isinstance(op_id, str) and OP_ID.fullmatch(op_id) else None,
resources=() if cancel_queued_snapshot else resources, kind="request")
resources=() if cancel_queued_snapshot else resources, kind="request",
metadata={"explicit_vpc_reconcile": True}
if (vpc_write_request
and delegated.get("vpc_reconcile_explicit") is True)
else None)
except CoordinationClaimRejected as exc:
if (type(exc) is not CoordinationClaimRejected or not fabric_admission_request
or exc.error_code not in {"coordination_busy", "coordination_uncertain"}):
@ -894,9 +1040,118 @@ class EngineApplication:
"error": str(exc), "error_code": exc.error_code})
return messages
dispatch_started = False
vpc_finalized = False
messages = []
async def capture(message):
messages.append(message)
def same_claim_identity(expected, current):
fields = (
"token", "execution_id", "owner_id", "operation_id",
"resources", "maintenance", "kind", "metadata",
"created_at",
)
return (isinstance(current, dict)
and all(current.get(field) == expected.get(field)
for field in fields))
async def exact_current(expected):
rows = await asyncio.to_thread(self.coordinator.snapshot)
current = rows.get(expected["token"])
if current is not None and not same_claim_identity(
expected, current):
raise CoordinationError("coordination_unavailable", 503)
return current
async def retain_exact(expected, *, allow_absent):
"""Idempotently retain one snapshot owner, with readback/retry."""
current = expected
last_error = None
for _attempt in range(2):
if current["state"] == "uncertain":
return True
try:
if await asyncio.to_thread(
self.coordinator.retain, current):
return True
except BaseException as retain_error:
last_error = retain_error
current = await exact_current(expected)
if current is None:
if allow_absent:
return False
raise CoordinationError(
"coordination_unavailable", 503) from last_error
if current["state"] == "uncertain":
return True
raise CoordinationError(
"coordination_unavailable", 503) from last_error
async def release_exact_root(expected):
"""Release once, then prove absence or retain the exact root."""
try:
if await asyncio.to_thread(
self.coordinator.release, expected):
return True
except BaseException:
pass
current = await exact_current(expected)
if current is None:
# The CAS may have committed, but a false/lost response is
# still an unconfirmed VPC final result and must become 503.
return False
await retain_exact(current, allow_absent=False)
return False
async def finalize_vpc_execution(*, keep_root):
"""Settle one VPC root and all children without a replay gap."""
rows = await asyncio.to_thread(self.coordinator.snapshot)
root = rows.get(claim.token)
if root is not None and root["execution_id"] != claim.execution_id:
raise CoordinationError("coordination_unavailable", 503)
execution_claims = {
token: row for token, row in rows.items()
if row["execution_id"] == claim.execution_id}
children = {
token: row for token, row in execution_claims.items()
if token != claim.token}
execution_unknown = current_execution().execution_unknown.is_set()
residual = bool(children) or any(
row["state"] == "uncertain"
for row in execution_claims.values())
forced_uncertain = execution_unknown or residual
if not (keep_root or forced_uncertain):
if root is None:
return False, True
released = await release_exact_root(root)
return released, not released
# Fence the root first. A child retain failure can then be
# retried without ever making this execution look complete.
root_error = None
if root is None:
root_error = CoordinationError(
"coordination_unavailable", 503)
else:
try:
await retain_exact(root, allow_absent=False)
except BaseException as exc:
root_error = exc
child_error = None
for row in children.values():
if row["state"] != "active":
continue
try:
await retain_exact(row, allow_absent=True)
except BaseException as exc:
child_error = child_error or exc
if root_error is not None:
raise root_error
if child_error is not None:
raise child_error
return False, forced_uncertain
with execution_context(self.coordinator, claim,
connection_failure_safe=libvirt_only,
tracked_mutations=bool(operation_request)):
@ -938,7 +1193,8 @@ class EngineApplication:
if item["type"] == "http.response.start"), 503)
raw = await asyncio.to_thread(self.store.get, "operation", op_id) if op_id else None
record = json.loads(raw) if raw else {}
current = (await asyncio.to_thread(self.coordinator.snapshot)).get(claim.token, {})
claims = await asyncio.to_thread(self.coordinator.snapshot)
current = claims.get(claim.token, {})
response_error_code = None
if status >= 500:
with suppress(TypeError, ValueError, AttributeError):
@ -955,8 +1211,20 @@ class EngineApplication:
and bool(record.get("cleanup_pending")))
if current_execution().execution_unknown.is_set():
safe_failure = False
if (current_execution().execution_unknown.is_set()
or (status >= 500 or record.get("status") == "uncertain") and not safe_failure):
keep_root = (
current_execution().execution_unknown.is_set()
or (status >= 500
or record.get("status") == "uncertain")
and not safe_failure)
if vpc_write_request:
released, forced_uncertain = (
await _await_mutation_completion(
finalize_vpc_execution(keep_root=keep_root)))
vpc_finalized = True
if not released and (status < 500 or forced_uncertain):
raise CoordinationError(
"coordination_uncertain", 503)
elif keep_root:
await asyncio.to_thread(self.coordinator.retain, claim)
else:
await asyncio.to_thread(self.coordinator.release, claim)
@ -999,7 +1267,28 @@ class EngineApplication:
mark_execution_unknown()
if current_execution().execution_unknown.is_set():
not_started = False
if _uncertain_exception(exc) and not not_started and dispatch_started:
keep_root = (_uncertain_exception(exc)
and not not_started and dispatch_started)
if (vpc_write_request and dispatch_started
and not vpc_finalized):
settlement_error = None
released = False
forced_uncertain = False
try:
released, forced_uncertain = (
await _await_mutation_completion(
finalize_vpc_execution(
keep_root=keep_root)))
except BaseException as final_error:
settlement_error = final_error
if isinstance(exc, asyncio.CancelledError):
raise
if settlement_error is not None:
raise settlement_error from exc
if not released and (keep_root or forced_uncertain):
raise CoordinationError(
"coordination_uncertain", 503) from exc
elif keep_root:
await asyncio.to_thread(self.coordinator.retain, claim)
else:
await asyncio.to_thread(self.coordinator.release, claim)

@ -17,12 +17,16 @@ import hashlib
import ipaddress
import json
import re
import secrets
import threading
import time
from dataclasses import dataclass, replace
from typing import Any, Collection, Mapping, Sequence
from .coordination import CoordinationAdmissionRejected, CoordinationError
from .coordination import (
CoordinationAdmissionRejected, CoordinationClaimRejected, CoordinationError,
current_execution, execution_context, _uncertain_exception,
)
from .fabric import (
FabricCoordinator,
FabricError,
@ -1015,10 +1019,12 @@ class VpcCoordinator:
def __init__(self, store: Any, fabric: Any, registry: Any | None = None,
vm_lock: Any | None = None,
catalog_lock: Any | None = None):
catalog_lock: Any | None = None,
coordinator: Any | None = None):
self.store = store
self.fabric = fabric
self._catalog_lock_provider = catalog_lock
self._coordinator = coordinator
# The registry is optional for pure control-plane/unit callers. The
# production EngineApplication supplies it so a VPC revision can only
# become active after host bridge/WireGuard/nft and libvirt readback.
@ -1037,6 +1043,540 @@ class VpcCoordinator:
self._fallback_catalog_lock: Any | None = None
self._reconcile_wakeup: Any | None = None
@staticmethod
def _operation_identity(plan: VpcPlan) -> dict[str, Any]:
return {"job_kind": "vpc_operation", "vpc_id": plan.vpc_id,
"desired_revision": plan.desired_revision,
"desired_digest": plan.desired_digest}
def _retained_operation_target_matches(self, plan: VpcPlan) -> bool:
"""Revalidate a retained root against the durable target under its lock."""
receipt = self._load_delete_receipt(plan.vpc_id)
if receipt is not None:
return bool(
receipt["operation_id"] == plan.operation_id
and receipt["desired_revision"] == plan.desired_revision
and receipt["desired_digest"] == plan.desired_digest)
record = self._load("vpc_desired", plan.vpc_id)
if record is None:
return False
durable = _plan_from_record(record)
return bool(
durable.operation_id == plan.operation_id
and durable.desired_revision == plan.desired_revision
and durable.desired_digest == plan.desired_digest
and durable.canonical() == plan.canonical())
@asynccontextmanager
async def _operation_scope(self, plan: VpcPlan):
"""Carry one validated target through HTTP and periodic resource claims."""
parent = current_execution()
coordinator = parent.coordinator if parent is not None else self._coordinator
if coordinator is None:
yield # Pure control-plane embeddings have no durable owner.
return
metadata = self._operation_identity(plan)
explicit_marker = {"explicit_vpc_reconcile": True}
handoff_evidence = (
f"vpc-explicit-reconcile-v1 operation={plan.operation_id} "
f"vpc={plan.vpc_id} revision={plan.desired_revision} "
f"digest={plan.desired_digest}; exact childless retained root")
claim = None
background_execution_id = None
handoff_original = None
handoff_attempted = False
explicit_bind_attempted = False
def claim_value(value, name: str, default: Any = None) -> Any:
if isinstance(value, Mapping):
return value.get(name, default)
return getattr(value, name, default)
def exact_parent(record: Mapping[str, Any] | None, *, bound: bool) -> bool:
if parent is None or not isinstance(record, Mapping):
return False
expected_metadata = metadata if bound else explicit_marker
expected_operation = plan.operation_id if bound else None
return bool(
record.get("token") == parent.claim.token
and record.get("execution_id") == parent.claim.execution_id
and record.get("owner_id") == parent.claim.owner_id
and record.get("operation_id") == expected_operation
and record.get("resources") == []
and record.get("kind") == "request"
and record.get("metadata") == expected_metadata
and record.get("maintenance") is False
and record.get("state") in {"active", "uncertain"})
async def retained_root_for_explicit_reconcile():
"""Return only one exact, childless VPC root eligible for handoff."""
if parent is None:
return None
marker_present = "explicit_vpc_reconcile" in parent.claim.metadata
explicit_root = bool(
parent.claim.operation_id is None
and tuple(parent.claim.resources) == ()
and parent.claim.kind == "request"
and parent.claim.metadata == explicit_marker)
if not explicit_root:
if marker_present:
raise CoordinationError("coordination_unavailable", 503)
return None
rows = await _await_mutation_completion(
asyncio.to_thread(coordinator.snapshot))
current = rows.get(parent.claim.token)
if not exact_parent(current, bound=False) or current["state"] != "active":
raise CoordinationError("coordination_unavailable", 503)
if not self._retained_operation_target_matches(plan):
return None
eligible = []
for row in rows.values():
if (row.get("token") != parent.claim.token
and row.get("state") == "uncertain"
and row.get("kind") in {"background", "request"}
and row.get("operation_id") == plan.operation_id
and row.get("resources") == []
and row.get("maintenance") is False
and row.get("metadata") == metadata):
eligible.append(row)
if len(eligible) != 1:
return None
original = eligible[0]
if any(
row.get("token") not in {original["token"], parent.claim.token}
and (row.get("execution_id") in {
original["execution_id"], parent.claim.execution_id}
or row.get("operation_id") == plan.operation_id)
for row in rows.values()
):
return None
return original
async def admit():
nonlocal claim, background_execution_id, handoff_attempted
nonlocal handoff_original, explicit_bind_attempted
if parent is not None:
handoff_original = await retained_root_for_explicit_reconcile()
if handoff_original is not None:
handoff_attempted = True
claim = await asyncio.to_thread(
coordinator.resolve_and_bind, handoff_original, parent.claim,
handoff_evidence,
operation_id=plan.operation_id, metadata=metadata)
return
explicit_bind_attempted = parent.claim.metadata == explicit_marker
binder = (coordinator.bind_explicit_operation
if explicit_bind_attempted else coordinator.bind_operation)
claim = await asyncio.to_thread(
binder, parent.claim, plan.operation_id, metadata)
else:
background_execution_id = secrets.token_hex(32)
claim = await asyncio.to_thread(
coordinator.claim, operation_id=plan.operation_id,
execution_id=background_execution_id,
kind="background", metadata=metadata)
def root_value(name: str, default: Any = None) -> Any:
return claim_value(claim, name, default)
def same_root(record: Mapping[str, Any] | None) -> bool:
return bool(
isinstance(record, Mapping)
and record.get("token") == root_value("token")
and record.get("execution_id") == root_value("execution_id")
and record.get("owner_id") == root_value("owner_id")
and record.get("operation_id") == root_value("operation_id")
and tuple(record.get("resources", ()))
== tuple(root_value("resources", ()))
and record.get("kind") == root_value("kind")
and record.get("metadata") == root_value("metadata", {})
and record.get("maintenance")
== root_value("maintenance", False))
async def read_root() -> Mapping[str, Any] | None:
rows = await _await_mutation_completion(
asyncio.to_thread(coordinator.snapshot))
current = rows.get(root_value("token"))
if current is not None and not same_root(current):
raise CoordinationError("coordination_unavailable", 503)
return current
async def retain_root_exact() -> None:
try:
retained = await _await_mutation_completion(
asyncio.to_thread(coordinator.retain, claim))
except BaseException as first_error:
current = await read_root()
if current is None:
raise first_error
if current.get("state") == "uncertain":
if isinstance(first_error, asyncio.CancelledError):
raise first_error
return
if current.get("state") != "active":
raise CoordinationError(
"coordination_unavailable", 503) from first_error
try:
retained = await _await_mutation_completion(
asyncio.to_thread(coordinator.retain, current))
except BaseException as retry_error:
final = await read_root()
if final is not None and final.get("state") == "uncertain":
if isinstance(retry_error, asyncio.CancelledError):
raise retry_error
if isinstance(first_error, asyncio.CancelledError):
raise first_error
return
raise retry_error from first_error
if isinstance(first_error, asyncio.CancelledError):
raise first_error
if not retained:
current = await read_root()
if current is not None and current.get("state") == "uncertain":
return
raise CoordinationError("coordination_uncertain", 503)
async def settle_background(keep: bool) -> None:
if keep:
await retain_root_exact()
return
try:
released = await _await_mutation_completion(
asyncio.to_thread(coordinator.release, claim))
except BaseException as release_error:
current = await read_root()
if current is not None:
try:
await retain_root_exact()
except BaseException as retain_error:
raise retain_error from release_error
raise
if not released:
# A false release result is not a successful VPC settlement.
# The caller must reconcile even when a concurrent read shows
# that the exact root has disappeared.
raise CoordinationError("coordination_uncertain", 503)
async def retain_residual(row: Mapping[str, Any]) -> None:
try:
retained = await _await_mutation_completion(
asyncio.to_thread(coordinator.retain, row))
except BaseException as first_error:
# Fence the root before trying to classify a child settlement
# whose CAS response may have been lost.
try:
await settle_background(True)
recovered_rows = await _await_mutation_completion(
asyncio.to_thread(coordinator.snapshot))
except BaseException as fence_error:
raise fence_error from first_error
current = recovered_rows.get(row["token"])
if current is None:
if isinstance(first_error, asyncio.CancelledError):
raise first_error
return
identity_keys = (
"token", "execution_id", "owner_id", "operation_id",
"resources", "kind", "metadata", "maintenance",
)
if any(current.get(key) != row.get(key)
for key in identity_keys):
raise CoordinationError(
"coordination_unavailable", 503) from first_error
if current.get("state") == "uncertain":
if isinstance(first_error, asyncio.CancelledError):
raise first_error
return
if current.get("state") != "active":
raise CoordinationError(
"coordination_unavailable", 503) from first_error
try:
retained = await _await_mutation_completion(
asyncio.to_thread(coordinator.retain, current))
except BaseException as retry_error:
try:
final_rows = await _await_mutation_completion(
asyncio.to_thread(coordinator.snapshot))
except BaseException as readback_error:
raise readback_error from retry_error
final = final_rows.get(row["token"])
if final is None or final.get("state") == "uncertain":
if isinstance(retry_error, asyncio.CancelledError):
raise retry_error
if isinstance(first_error, asyncio.CancelledError):
raise first_error
return
raise retry_error from first_error
if isinstance(first_error, asyncio.CancelledError):
raise first_error
if not retained:
# A definite false means the exact child disappeared before
# settlement. The root still remains uncertain because the
# final snapshot observed an unsettled execution.
return
async def finalize_background(
force_keep: bool, cancellation_seen: asyncio.Event) -> bool:
try:
rows = await _await_mutation_completion(
asyncio.to_thread(coordinator.snapshot))
except BaseException:
# Final readback/cancellation can race after host work but
# before root settlement. Preserve the root before returning
# the readback failure.
await settle_background(True)
raise
residual = [
row for row in rows.values()
if (row["execution_id"] == claim.execution_id
and row["token"] != claim.token)
]
for row in residual:
if row["state"] == "active":
await retain_residual(row)
unknown = (force_keep
or cancellation_seen.is_set()
or current_execution().execution_unknown.is_set()
or bool(residual)
or any(
row["execution_id"] == claim.execution_id
and row["state"] == "uncertain"
for row in rows.values()))
await settle_background(unknown)
return unknown
async def drain_background_finalizer(force_keep: bool) -> bool:
"""Finish every owner settlement before propagating cancellation."""
cancellation_seen = asyncio.Event()
task = asyncio.create_task(
finalize_background(force_keep, cancellation_seen))
caller_cancelled = False
while True:
try:
result = await asyncio.shield(task)
except asyncio.CancelledError:
if task.cancelled():
raise
caller_cancelled = True
cancellation_seen.set()
if not task.done():
continue
try:
result = task.result()
except BaseException as exc:
raise asyncio.CancelledError() from exc
except BaseException as exc:
if caller_cancelled:
raise asyncio.CancelledError() from exc
raise
if caller_cancelled:
raise asyncio.CancelledError()
return result
async def recover_background_admission() -> None:
"""Fence a background owner whose claim response was interrupted."""
nonlocal claim
if claim is None and background_execution_id is not None:
# A CAS response can be lost after the background claim was
# durably installed. Recover only this exact execution; an
# operation id alone could name another live worker. No host
# work has been admitted yet, so retaining the recovered row
# records the uncertain database outcome without replaying it.
rows = await _await_mutation_completion(
asyncio.to_thread(coordinator.snapshot))
recovered = [row for row in rows.values()
if (row.get("execution_id") == background_execution_id
and row.get("owner_id") == coordinator.owner_id
and row.get("operation_id") == plan.operation_id
and row.get("resources") == []
and row.get("kind") == "background"
and row.get("metadata") == metadata
and row.get("maintenance") is False)]
if len(recovered) > 1:
raise CoordinationError("coordination_unavailable", 503)
if recovered:
claim = recovered[0]
if claim is not None:
await retain_root_exact()
async def recover_parent_handoff() -> None:
"""Fence an atomically rebound request after a lost CAS response."""
nonlocal claim
if parent is None or not handoff_attempted or handoff_original is None:
return
outcome, current = await _await_mutation_completion(
asyncio.to_thread(
coordinator.classify_resolve_and_bind,
handoff_original, parent.claim, handoff_evidence,
operation_id=plan.operation_id, metadata=metadata))
if outcome == "uncommitted":
try:
claim = await _await_mutation_completion(
asyncio.to_thread(
coordinator.resolve_and_bind,
handoff_original, parent.claim, handoff_evidence,
operation_id=plan.operation_id, metadata=metadata))
except BaseException as retry_error:
outcome, current = await _await_mutation_completion(
asyncio.to_thread(
coordinator.classify_resolve_and_bind,
handoff_original, parent.claim, handoff_evidence,
operation_id=plan.operation_id, metadata=metadata))
if outcome != "committed":
if (isinstance(current, Mapping)
and current.get("token") == parent.claim.token
and current.get("execution_id")
== parent.claim.execution_id
and current.get("owner_id")
== parent.claim.owner_id):
claim = current
await retain_root_exact()
raise CoordinationError(
"coordination_unavailable", 503) from retry_error
else:
await retain_root_exact()
return
if outcome == "committed" and exact_parent(current, bound=True):
claim = current
await retain_root_exact()
return
if (isinstance(current, Mapping)
and current.get("token") == parent.claim.token
and current.get("execution_id") == parent.claim.execution_id
and current.get("owner_id") == parent.claim.owner_id):
claim = current
await retain_root_exact()
raise CoordinationError("coordination_unavailable", 503)
async def classify_explicit_bind():
rows = await _await_mutation_completion(
asyncio.to_thread(coordinator.snapshot))
current = rows.get(parent.claim.token)
conflicts = any(
row.get("token") != parent.claim.token
and (row.get("execution_id") == parent.claim.execution_id
or row.get("operation_id") == plan.operation_id)
for row in rows.values())
if not conflicts and exact_parent(current, bound=True):
return "committed", current
if (not conflicts and exact_parent(current, bound=False)
and current.get("state") == "active"):
return "uncommitted", current
return "mixed", current
async def recover_explicit_bind() -> None:
"""Classify/retry a marker-consuming bind without losing its root."""
nonlocal claim
if parent is None or not explicit_bind_attempted:
return
outcome, current = await classify_explicit_bind()
if outcome == "uncommitted":
try:
claim = await _await_mutation_completion(
asyncio.to_thread(
coordinator.bind_explicit_operation,
parent.claim, plan.operation_id, metadata))
except BaseException as retry_error:
outcome, current = await classify_explicit_bind()
if outcome != "committed":
if (isinstance(current, Mapping)
and current.get("token") == parent.claim.token
and current.get("execution_id")
== parent.claim.execution_id
and current.get("owner_id")
== parent.claim.owner_id):
claim = current
await retain_root_exact()
raise CoordinationError(
"coordination_unavailable", 503) from retry_error
else:
await retain_root_exact()
return
if outcome == "committed" and exact_parent(current, bound=True):
claim = current
await retain_root_exact()
return
if (isinstance(current, Mapping)
and current.get("token") == parent.claim.token
and current.get("execution_id") == parent.claim.execution_id
and current.get("owner_id") == parent.claim.owner_id):
claim = current
await retain_root_exact()
raise CoordinationError("coordination_unavailable", 503)
try:
await _await_mutation_completion(admit())
except CoordinationClaimRejected as exc:
if handoff_attempted:
outcome, current = await _await_mutation_completion(
asyncio.to_thread(
coordinator.classify_resolve_and_bind,
handoff_original, parent.claim, handoff_evidence,
operation_id=plan.operation_id, metadata=metadata))
if outcome == "committed" and exact_parent(current, bound=True):
claim = current
await _await_mutation_completion(retain_root_exact())
raise CoordinationError("coordination_uncertain", 503) from exc
elif explicit_bind_attempted:
outcome, current = await classify_explicit_bind()
if outcome == "committed":
claim = current
await _await_mutation_completion(retain_root_exact())
raise CoordinationError("coordination_uncertain", 503) from exc
# Only a proven binding/admission rejection is side-effect free.
# Unavailable storage and existing execution uncertainty retain
# the normal conservative handling.
raise CoordinationAdmissionRejected(exc.error_code, exc.status) from exc
except BaseException as exc:
# Cancellation can arrive after CAS committed but before admit
# returns. Preserve the known background owner before losing its
# handle; HTTP ownership is settled by the outer request guard.
if parent is None:
try:
await _await_mutation_completion(
recover_background_admission())
except BaseException as settle_exc:
raise settle_exc from exc
elif handoff_attempted:
try:
await _await_mutation_completion(recover_parent_handoff())
except BaseException as settle_exc:
raise settle_exc from exc
elif explicit_bind_attempted:
try:
await _await_mutation_completion(recover_explicit_bind())
except BaseException as settle_exc:
raise settle_exc from exc
raise
with execution_context(coordinator, claim):
try:
yield
except BaseException as exc:
if parent is None:
try:
unknown = await drain_background_finalizer(
_uncertain_exception(exc))
except BaseException as settle_exc:
raise settle_exc from exc
if unknown and not isinstance(exc, asyncio.CancelledError):
raise CoordinationError(
"coordination_uncertain", 503) from exc
raise
else:
if parent is None:
unknown = await drain_background_finalizer(False)
if unknown:
raise CoordinationError("coordination_uncertain", 503)
def _check_operation_identity(self, plan: VpcPlan) -> None:
context = current_execution()
if (context is not None and context.claim.metadata.get("job_kind") == "vpc_operation"
and (context.claim.operation_id != plan.operation_id
or any(context.claim.metadata.get(key) != value
for key, value in self._operation_identity(plan).items()))):
raise VpcError("VPC 目标已由其他操作替代", 409, "vpc_revision_conflict")
@staticmethod
def _key(vpc_id: str, child_id: str) -> str:
return f"{vpc_id}.{child_id}"
@ -1889,6 +2429,7 @@ class VpcCoordinator:
fabric_public_keys: Mapping[str, str] | None = None,
fabric_plan_digest: str | None = None,
private_link_states: Mapping[str, str] | None = None,
fabric_peer_overlays: Mapping[str, str] | None = None,
request_deadline: float | None = None,
previous_host: Mapping[str, Any] | None = None,
) -> dict[str, Any]:
@ -1917,10 +2458,32 @@ class VpcCoordinator:
key for peer_host, key in fabric_public_keys.items()
if peer_host != host_id
]
authoritative_peer_overlays = None
if fabric_peer_overlays is not None:
if not isinstance(fabric_peer_overlays, Mapping):
raise VpcError("Fabric peer overlay 闭集无效", 500,
"vpc_fabric_record_corrupt")
local_public_key = (
fabric_public_keys.get(host_id)
if fabric_public_keys is not None else None)
if fabric_peer_overlays and not isinstance(local_public_key, str):
raise VpcError("Fabric 本机公钥记录不完整", 500,
"vpc_fabric_record_corrupt")
authoritative_peer_overlays = {
key: overlay
for key, overlay in fabric_peer_overlays.items()
if key != local_public_key
}
if (authoritative_peer_keys is not None
and set(authoritative_peer_overlays)
!= set(authoritative_peer_keys)):
raise VpcError("Fabric 公钥与 overlay 闭集不一致", 500,
"vpc_fabric_record_corrupt")
dataplane = parse_vpc_dataplane_plan(
payload, host_id,
trusted_tenant_allowed_ips=trusted_peer_tenant_ips,
fabric_peer_keys=authoritative_peer_keys,
fabric_peer_overlays=authoritative_peer_overlays,
fabric_plan_digest=fabric_plan_digest)
# A root-delete plan extends the canonical digest with its signed
# retiring-member manifest. The host projection intentionally
@ -1932,10 +2495,33 @@ class VpcCoordinator:
# plan. Construct a typed empty-local plan directly so the
# removal transaction can validate the deterministic VPC
# resources without reintroducing the old host into Fabric.
authoritative_peer_keys = None
authoritative_peer_overlays = None
if fabric_public_keys is not None:
authoritative_peer_keys = frozenset(
key for peer_host, key in fabric_public_keys.items()
if peer_host != host_id)
if fabric_peer_overlays is not None:
local_public_key = (
fabric_public_keys.get(host_id)
if fabric_public_keys is not None else None)
if fabric_peer_overlays and not isinstance(local_public_key, str):
raise VpcError("Fabric 本机公钥记录不完整", 500,
"vpc_fabric_record_corrupt")
authoritative_peer_overlays = frozenset(
(key, overlay)
for key, overlay in fabric_peer_overlays.items()
if key != local_public_key)
if (authoritative_peer_keys is not None
and {key for key, _overlay in authoritative_peer_overlays}
!= authoritative_peer_keys):
raise VpcError("Fabric 公钥与 overlay 闭集不一致", 500,
"vpc_fabric_record_corrupt")
dataplane = VpcDataplanePlan(
plan.vpc, host_id, tuple(plan.members), tuple(),
resource_names(plan.vpc_id),
fabric_plan_digest)
fabric_plan_digest, authoritative_peer_keys,
authoritative_peer_overlays)
if request_deadline is None:
request_deadline = (
time.monotonic() + VPC_REQUEST_DEADLINE_SECONDS)
@ -2609,6 +3195,13 @@ class VpcCoordinator:
if any(not isinstance(key, str) for key in fabric_public_keys.values()):
raise VpcError("Fabric 公钥记录不完整", 500,
"vpc_fabric_record_corrupt")
fabric_peer_overlays = {
str(fabric_public_keys[node.host_id]): node.overlay_ip + "/32"
for node in full_fabric_plan.nodes
}
if len(fabric_peer_overlays) != len(full_fabric_plan.nodes):
raise VpcError("Fabric 公钥与 overlay 闭集重复", 500,
"vpc_fabric_record_corrupt")
fabric_plan = full_fabric_plan
# ``record`` is the execution closure for this pass. Root-delete
# retries deliberately project it down to hosts that do not yet
@ -2673,6 +3266,7 @@ class VpcCoordinator:
profile, all_peers, previous_plan,
host.get("retiring_members", []), fabric_public_keys,
fabric_plan_digest, private_link_states,
fabric_peer_overlays=fabric_peer_overlays,
request_deadline=request_deadline,
previous_host=host)
if time.monotonic() >= request_deadline:
@ -3722,13 +4316,13 @@ class VpcCoordinator:
# the global order per-VPC -> catalog, and do not release either lock
# while a cancelled coroutine still has a blocking host mutation or
# durable receipt commit in flight.
async with self._lock(vpc_id), _pre_dataplane_lock(
async with self._lock(vpc_id), self._operation_scope(plan), _pre_dataplane_lock(
self._catalog_mutation_lock()):
return await _await_mutation_completion(delete_locked())
async def apply(self, value: Mapping[str, Any]) -> dict[str, Any]:
plan = parse_vpc_plan(value)
async with self._lock(plan.vpc_id), _pre_dataplane_lock(
async with self._lock(plan.vpc_id), self._operation_scope(plan), _pre_dataplane_lock(
self._catalog_mutation_lock()):
return await _await_mutation_completion(
self._apply_locked(plan))
@ -3763,6 +4357,9 @@ class VpcCoordinator:
if old_digest != plan.desired_digest:
raise VpcError("相同 desired_revision 的 VPC 计划已存在", 409,
"vpc_revision_conflict")
if plan.operation_id != previous_plan.operation_id:
raise VpcError("相同 VPC 目标已绑定其他操作编号", 409,
"vpc_revision_conflict")
# Preserve exact host absence proofs monotonically. A
# partial pass may have removed host A while host B was
# unavailable; replay must retry only B instead of making
@ -4551,6 +5148,7 @@ class VpcCoordinator:
if current.get("desired_digest") != plan.desired_digest:
raise VpcError("VPC 计划摘要与持久化记录不一致", 500,
"vpc_record_corrupt")
self._check_operation_identity(plan)
gate = self._fabric_gate(plan)
# Reconcile is the durable retry path after a host response was
# lost or the engine restarted. Exact active hosts return without
@ -4562,10 +5160,18 @@ class VpcCoordinator:
self._save_children(current)
return self._public(current), None
async with self._lock(vpc_id), _pre_dataplane_lock(
self._catalog_mutation_lock()):
result, delete_payload = await _await_mutation_completion(
reconcile_locked())
async with self._lock(vpc_id):
identity = self._load("vpc_desired", vpc_id)
if identity is None:
raise VpcError("VPC 不存在", 404, "vpc_not_found")
if self._delete_marker(identity) is not None:
result, delete_payload = None, self._delete_payload_from_record(identity)
else:
plan = _plan_from_record(identity)
async with self._operation_scope(plan), _pre_dataplane_lock(
self._catalog_mutation_lock()):
result, delete_payload = await _await_mutation_completion(
reconcile_locked())
# The locked recheck may observe a delete target that was installed
# after the optimistic read. Release the non-reentrant lock before
# entering ``delete``, which acquires the same per-VPC and catalog

@ -65,6 +65,49 @@ _APPLY_RECEIPT_LIMIT = 1024
_APPLY_RECEIPTS: OrderedDict[tuple[str, ...], tuple[str, str]] = OrderedDict()
_APPLY_RECEIPT_LOCK = Lock()
# The synchronous VPC SSH boundary intentionally discards arbitrary remote
# stderr because host tools may print paths or configuration. Keep one
# closed, host-local journal record instead so an apply failure can be placed
# before or after network activation without exposing command output.
VPC_APPLY_SAFE_LOG_SH = r'''
LOGGER=
for vc_vpc_logger_path in /usr/bin/logger /bin/logger /usr/sbin/logger /sbin/logger; do
if [ -x "$vc_vpc_logger_path" ]; then LOGGER=$vc_vpc_logger_path; break; fi
done
VC_VPC_APPLY_PHASE=entry
vc_vpc_apply_failure_log() {
vc_vpc_log_status=$1
case "$vc_vpc_log_status" in ''|*[!0-9]*) vc_vpc_log_status=1 ;; esac
[ "$vc_vpc_log_status" -le 255 ] 2>/dev/null || vc_vpc_log_status=1
[ "$vc_vpc_log_status" != 0 ] || return 0
case "${VC_VPC_APPLY_PHASE:-}" in
entry|tools|preflight|transaction_snapshot|quarantine|firewall|bridge|persistence|activation|post_activation_sysctl|runtime|proxy|neighbors|routes|policy_rules|fabric_acl|wireguard|final_readback|commit|rollback)
vc_vpc_log_phase=$VC_VPC_APPLY_PHASE ;;
*) vc_vpc_log_phase=unknown ;;
esac
[ -n "${LOGGER:-}" ] || return 0
"$LOGGER" -t virt-cloud-vpc -- \
"apply_state=failed apply_phase=$vc_vpc_log_phase exit_status=$vc_vpc_log_status" \
>/dev/null 2>&1 || true
return 0
}
vc_vpc_apply_log_exit() {
vc_vpc_log_exit_status=$?
trap - EXIT
vc_vpc_apply_failure_log "$vc_vpc_log_exit_status" || true
exit "$vc_vpc_log_exit_status"
}
vc_vpc_apply_signal_exit() {
vc_vpc_signal_status=$1
trap - HUP INT TERM
exit "$vc_vpc_signal_status"
}
trap vc_vpc_apply_log_exit EXIT
trap 'vc_vpc_apply_signal_exit 129' HUP
trap 'vc_vpc_apply_signal_exit 130' INT
trap 'vc_vpc_apply_signal_exit 143' TERM
'''.strip()
# iproute2 prints host routes without /32 and suppresses dev when its query
# already selected that device. Infer only the explicitly supplied filter;
# unfiltered inventories must still contain a unique output device. A private
@ -243,7 +286,7 @@ vc_vpc_neighbor_helper_ready() {
[ "$(sed -n '3p' "$vc_ln_helper")" = "# fabric-plan-digest=$EXPECTED_DIGEST" ] &&
[ "$(sed -n '4p' "$vc_ln_helper")" = "PLAN_DIGEST=\"$EXPECTED_DIGEST\"" ] &&
[ "$(sed -n '6p' "$vc_ln_helper")" = '# vpc-local-neighbor-contract-v1' ] &&
[ "$(sed -n '7p' "$vc_ln_helper")" = '# vpc-state-route-scope-contract-v3' ]
[ "$(sed -n '7p' "$vc_ln_helper")" = '# vpc-state-route-scope-contract-v4' ]
}
vc_vpc_neighbors_snapshot() {
vc_ln_snapshot=$(vc_local_neighbor_records "$BRIDGE") || return 1
@ -933,6 +976,11 @@ class VpcDataplanePlan:
# separate from ``peers`` because one VPC can use only a subset of the
# peers present on a shared Fabric.
fabric_peer_keys: frozenset[str] | None = None
# Authoritative public-key -> overlay /32 closure from the complete
# engine-validated Fabric plan. The key-only field remains for legacy
# callers, but only this mapping can prove and converge cryptokey routing
# for peers outside this VPC's participant set.
fabric_peer_overlays: frozenset[tuple[str, str]] | None = None
def __post_init__(self) -> None:
_fabric_plan_digest(self.fabric_plan_digest)
@ -1033,6 +1081,23 @@ def _key(value: Any) -> str:
return value
def _fabric_overlay_cidr(value: Any) -> str:
try:
interface = ipaddress.ip_interface(value)
except (TypeError, ValueError) as exc:
raise VpcDataplaneError("Fabric peer overlay 地址无效", 409,
"vpc_fabric_peer_set_invalid") from exc
fabric_overlay = ipaddress.IPv4Network("100.64.0.0/10")
if (not isinstance(interface, ipaddress.IPv4Interface)
or interface.network.prefixlen != 32
or interface.ip not in fabric_overlay
or interface.ip.is_loopback
or interface.ip.is_link_local):
raise VpcDataplaneError("Fabric peer overlay 地址无效", 409,
"vpc_fabric_peer_set_invalid")
return f"{interface.ip}/32"
def _mac(value: Any, label: str = "私网 MAC") -> str:
if not isinstance(value, str) or _MAC.fullmatch(value) is None:
raise VpcDataplaneError(f"{label}无效")
@ -1252,6 +1317,7 @@ def parse_vpc_dataplane_plan(
fabric_peers: Sequence[Mapping[str, Any]] | None = None,
trusted_tenant_allowed_ips: Mapping[str, Sequence[str]] | None = None,
fabric_peer_keys: Sequence[str] | None = None,
fabric_peer_overlays: Mapping[str, str] | None = None,
fabric_plan_digest: str | None = None,
) -> VpcDataplanePlan:
"""Parse existing VPC DTO plus an optional engine-only Fabric snapshot."""
@ -1337,9 +1403,65 @@ def parse_vpc_dataplane_plan(
raise VpcDataplaneError("VPC peer 不属于 Fabric 公钥闭集", 409,
"vpc_fabric_peer_set_invalid")
authoritative_keys = frozenset(canonical_keys)
authoritative_overlays: frozenset[tuple[str, str]] | None = None
if fabric_peer_overlays is not None:
if (not isinstance(fabric_peer_overlays, Mapping)
or len(fabric_peer_overlays) > 64):
raise VpcDataplaneError("Fabric peer overlay 闭集无效", 409,
"vpc_fabric_peer_set_invalid")
canonical_overlays: dict[str, str] = {}
seen_overlay_cidrs: set[str] = set()
for raw_key, raw_overlay in fabric_peer_overlays.items():
key = _key(raw_key)
overlay_cidr = _fabric_overlay_cidr(raw_overlay)
if key in canonical_overlays or overlay_cidr in seen_overlay_cidrs:
raise VpcDataplaneError("Fabric peer overlay 闭集重复", 409,
"vpc_fabric_peer_set_invalid")
canonical_overlays[key] = overlay_cidr
seen_overlay_cidrs.add(overlay_cidr)
overlay_keys = frozenset(canonical_overlays)
if authoritative_keys is not None and overlay_keys != authoritative_keys:
raise VpcDataplaneError("Fabric peer 公钥与 overlay 闭集不一致", 409,
"vpc_fabric_peer_set_invalid")
for peer in peers:
if canonical_overlays.get(peer.public_key) != peer.overlay_cidr:
raise VpcDataplaneError("VPC peer overlay 不属于 Fabric 闭集", 409,
"vpc_fabric_peer_set_invalid")
authoritative_keys = overlay_keys
authoritative_overlays = frozenset(canonical_overlays.items())
return VpcDataplanePlan(
vpc, host_id, tuple(members), peers, resource_names(vpc.vpc_id),
fabric_plan_digest, authoritative_keys)
fabric_plan_digest, authoritative_keys, authoritative_overlays)
def _authoritative_fabric_peer_overlays(
plan: VpcDataplanePlan,
) -> dict[str, str] | None:
"""Validate and materialize the engine-only full Fabric peer closure."""
if plan.fabric_peer_overlays is None:
return None
mapping: dict[str, str] = {}
seen_overlays: set[str] = set()
for row in plan.fabric_peer_overlays:
if not isinstance(row, tuple) or len(row) != 2:
raise VpcDataplaneError("Fabric peer overlay 闭集无效", 409,
"vpc_fabric_peer_set_invalid")
key = _key(row[0])
overlay = _fabric_overlay_cidr(row[1])
if key in mapping or overlay in seen_overlays:
raise VpcDataplaneError("Fabric peer overlay 闭集重复", 409,
"vpc_fabric_peer_set_invalid")
mapping[key] = overlay
seen_overlays.add(overlay)
if (plan.fabric_peer_keys is None
or frozenset(mapping) != plan.fabric_peer_keys):
raise VpcDataplaneError("Fabric peer 公钥与 overlay 闭集不一致", 409,
"vpc_fabric_peer_set_invalid")
for peer in plan.peers:
if mapping.get(peer.public_key) != peer.overlay_cidr:
raise VpcDataplaneError("VPC peer overlay 不属于 Fabric 闭集", 409,
"vpc_fabric_peer_set_invalid")
return dict(sorted(mapping.items()))
def _routes(plan: VpcDataplanePlan) -> tuple[dict[str, Any], ...]:
@ -1674,11 +1796,18 @@ def _state(
metric=route.get("metric", 100)))
for member in sorted(plan.local_members, key=lambda item: item.private_ip):
lines.append(f"local_neighbor|{member.private_ip}|{member.private_mac}")
wg_route_ips = {
str(route["destination_cidr"])
for route in routes if route["device"] == FABRIC_INTERFACE
}
owned_peer_ips = _expected_peer_tenant_ips(plan.vpc)
for peer in plan.peers:
ips = sorted(set(
[m.private_ip + "/32" for m in plan.remote_members
if m.host_id == peer.host_id]
+ list(peer.tenant_allowed_ips)))
# Persist only this VPC's route-backed contribution. The authenticated
# aggregate remains on ``peer.tenant_allowed_ips`` for live WireGuard
# convergence/readback, while old aggregate state is filtered by the
# loader during a mixed-version rollout.
ips = sorted(set(owned_peer_ips.get(peer.host_id, frozenset()))
& wg_route_ips)
if ips:
lines.append(f"peer|{peer.host_id}|{peer.public_key}|{','.join(ips)}")
return "\n".join(lines) + "\n"
@ -1860,9 +1989,36 @@ def _render_apply_script_v2(
proxy_lines = "\n".join(
str(ipaddress.ip_network(item["destination_cidr"], strict=True).network_address)
for item in routes)
peers_by_key = {peer.public_key: peer for peer in plan.peers}
authoritative_peer_overlays = _authoritative_fabric_peer_overlays(plan)
if (authoritative_peer_overlays is None
and plan.fabric_peer_keys is not None
and not plan.fabric_peer_keys.issubset(peers_by_key)):
raise VpcDataplaneError(
"完整 Fabric peer 收敛缺少权威 overlay 闭集", 409,
"vpc_fabric_peer_set_invalid")
# A VPC revision can stop using one Fabric peer while the peer itself
# remains part of the host's WireGuard Fabric. Converge that full
# authoritative closure after rewriting durable state, otherwise the
# removed peer retains the withdrawn tenant /32 indefinitely. Direct
# parser callers without the engine-only closure retain the historical
# current-VPC scope; production plans always provide the full key-to-overlay
# closure.
apply_fabric_peer_keys = tuple(sorted(
authoritative_peer_overlays
if authoritative_peer_overlays is not None
else (plan.fabric_peer_keys
if plan.fabric_peer_keys is not None else peers_by_key)
))
peer_lines = "\n".join(
f"{peer.host_id}|{peer.public_key}|{_allowed(plan, peer)}"
for peer in plan.peers)
f"{peers_by_key[key].host_id if key in peers_by_key else '-'}|"
f"{key}|{authoritative_peer_overlays[key]}"
if authoritative_peer_overlays is not None
else f"{peers_by_key[key].host_id}|{key}|{peers_by_key[key].overlay_cidr}"
for key in apply_fabric_peer_keys)
apply_fabric_peer_key_words = " ".join(apply_fabric_peer_keys)
apply_fabric_peer_keys_provided = (
"yes" if authoritative_peer_overlays is not None else "no")
state = _state(plan, routes, firewall_mode)
# Local service /32s use a guest address as the next hop on an otherwise
# unnumbered private bridge. Linux therefore needs the explicit `onlink`
@ -1965,6 +2121,7 @@ set -eu
umask 077
PATH=/usr/sbin:/usr/bin:/sbin:/bin
export PATH
{VPC_APPLY_SAFE_LOG_SH}
IP=$(for p in /usr/sbin/ip /usr/bin/ip /sbin/ip /bin/ip; do [ -x "$p" ] && {{ printf '%s' "$p"; break; }}; done)
WG=$(for p in /usr/bin/wg /usr/sbin/wg /bin/wg /sbin/wg; do [ -x "$p" ] && {{ printf '%s' "$p"; break; }}; done)
NFT=$(for p in /usr/sbin/nft /usr/bin/nft /sbin/nft /bin/nft; do [ -x "$p" ] && {{ printf '%s' "$p"; break; }}; done)
@ -2009,12 +2166,14 @@ nft_plan_owned() {{
[ "$(printf '%s\\n' "$nft_plan_text" |
grep -Foc 'comment "virt-cloud-wg-fabric-v1:'"$nft_plan_digest"'"' || true)" = 8 ]
}}
VC_VPC_APPLY_PHASE=tools
[ -n "$IP" ] && [ -n "$WG" ] && [ -n "$NFT" ] && [ -n "$FLOCK" ] && [ -n "$STAT" ] && [ -n "$CMP" ] && [ -n "$SYSCTL" ] && [ -n "$BRIDGE_CMD" ] && [ -n "$SYSTEMCTL" ] || exit 70
if [ "$MODE" = networkmanager_nmcli ]; then
NMCLI=$(for p in /usr/bin/nmcli /usr/sbin/nmcli /bin/nmcli /sbin/nmcli; do [ -x "$p" ] && {{ printf '%s' "$p"; break; }}; done)
[ -n "$NMCLI" ] || exit 70
fi
[ "$FIREWALL_MODE" = native_nftables ] || exit 71
VC_VPC_APPLY_PHASE=preflight
if [ -L /run/virt-cloud ] || {{ [ -e /run/virt-cloud ] && [ ! -d /run/virt-cloud ]; }}; then exit 71; fi
if [ -L /etc/virt-cloud ] || {{ [ -e /etc/virt-cloud ] && [ ! -d /etc/virt-cloud ]; }}; then exit 71; fi
if [ -L {state_dir} ] || {{ [ -e {state_dir} ] && [ ! -d {state_dir} ]; }}; then exit 71; fi
@ -2280,13 +2439,17 @@ fi
vc_ip_rule_scope_check {_q(plan.vpc.cidr)} "$ROUTE_TABLE" "$RULE_PRIORITY" "$WG_IFACE" managed 73 73 || exit $?
# Read-only preflight leaves the original ACL and link intact. The journal
# must contain the runtime baseline before quarantine can flush kernel state.
VC_VPC_APPLY_PHASE=transaction_snapshot
mkdir -m 700 "$TXN"
vc_vpc_write_native_transaction_mode || exit 79
printf '%s\\n' '{DATAPLANE_OWNER_MARKER}:{plan.vpc.vpc_id}:{plan.host_id}:{plan.desired_revision}:{plan.desired_digest}' > "$TXN/inflight"
chmod 600 "$TXN/inflight"
printf '%s\\n' preparing > "$TXN/state"; chmod 600 "$TXN/state"
rollback() {{
status=$?; [ "$COMMITTED" = yes ] && exit "$status"; trap - EXIT HUP INT TERM
status=$?
trap - EXIT HUP INT TERM
vc_vpc_apply_failure_log "$status" || true
[ "$COMMITTED" = yes ] && exit "$status"
set +e
if [ "$MUTATION_STARTED" = no ]; then
# Snapshot/preflight failure has not touched any dataplane resource.
@ -2679,10 +2842,17 @@ rollback() {{
fi
"$IP" link set dev "$BRIDGE" down >/dev/null 2>&1 || true
fi
if [ "$ROLLBACK_FAILED" = yes ]; then printf '%s\\n' rollback_failed > "$TXN/state"; chmod 600 "$TXN/state"; exit 79; fi
if [ "$ROLLBACK_FAILED" = yes ]; then
VC_VPC_APPLY_PHASE=rollback
vc_vpc_apply_failure_log 79 || true
printf '%s\\n' rollback_failed > "$TXN/state"; chmod 600 "$TXN/state"; exit 79
fi
rm -rf "$TXN"; exit "$status"
}}
trap rollback EXIT HUP INT TERM
trap 'vc_vpc_apply_signal_exit 129' HUP
trap 'vc_vpc_apply_signal_exit 130' INT
trap 'vc_vpc_apply_signal_exit 143' TERM
if [ -f "$OWNER_FILE" ]; then
[ ! -L "$OWNER_FILE" ] && grep -Fqx "$OWNER_MARKER" "$OWNER_FILE" || exit 71
cp -p "$OWNER_FILE" "$TXN/owner.before"
@ -2734,6 +2904,7 @@ if [ "$BRIDGE_TOPOLOGY_CAPTURED" = yes ]; then
# Kernel DOWN and NM unmanaged can flush proxy neighbours and routes. The
# original ACL, owner, manager, topology and runtime are now journaled.
vc_vpc_bridge_topology_matches || exit 72
VC_VPC_APPLY_PHASE=quarantine
MUTATION_STARTED=yes
RUNTIME_QUARANTINED=yes
if [ "$MODE" = networkmanager_nmcli ]; then
@ -2795,8 +2966,10 @@ vc_vpc_nft_matches_snapshot "$TXN/nft.desired" || exit 72
}}
# Install and exactly read back the raw tenant guard before restoring link
# administration in either persistence mode.
VC_VPC_APPLY_PHASE=firewall
MUTATION_STARTED=yes
vc_vpc_install_rules
VC_VPC_APPLY_PHASE=bridge
if "$IP" link show dev "$BRIDGE" >/dev/null 2>&1; then
"$IP" -d link show dev "$BRIDGE" | grep -Eq '(^|[[:space:]])bridge[[:space:]]+(forward_delay|stp_state|hello_time|ageing_time)([[:space:]]|$)' || exit 71
[ -f "$OWNER_FILE" ] || exit 71
@ -2884,6 +3057,7 @@ for vc_runtime_key in proxy_arp proxy_arp_pvlan; do
cat "$vc_runtime_path" > "$TXN/$vc_runtime_key.before" || exit 72
fi
done
VC_VPC_APPLY_PHASE=persistence
if [ "$MODE" = networkmanager_nmcli ]; then
[ -n "$NMCLI" ] || exit 70
NM_PROFILE_UUID=
@ -2910,11 +3084,13 @@ if [ "$MODE" = networkmanager_nmcli ]; then
vc_nm_device_release || exit 72
NM_DEVICE_QUARANTINED=no
fi
VC_VPC_APPLY_PHASE=activation
if [ "$NM_PRESERVE_ACTIVE_BRIDGE" = yes ]; then
vc_nm_preserve_bridge_ready || exit 72
else
"$NMCLI" connection up uuid "$NM_PROFILE_UUID" >/dev/null
fi
VC_VPC_APPLY_PHASE=post_activation_sysctl
vc_vpc_rp_filter_set || exit 72
else
[ -x "$IFUP" ] && [ -x "$IFQUERY" ] && [ -x "$IFDOWN" ] || exit 70
@ -2949,9 +3125,12 @@ else
# have acquired a libvirt tap before a later step fails. The rollback path
# below inspects ports first, then deletes only an empty newly-created
# bridge or restores an empty existing bridge to its sampled DOWN state.
VC_VPC_APPLY_PHASE=activation
VC_FABRIC_APPLYING=1 "$IFUP" --force "$BRIDGE" >/dev/null 2>&1 || exit 72
VC_VPC_APPLY_PHASE=post_activation_sysctl
vc_vpc_rp_filter_set || exit 72
fi
VC_VPC_APPLY_PHASE=runtime
if [ "$BRIDGE_TOPOLOGY_CAPTURED" = yes ]; then
vc_vpc_bridge_topology_matches || exit 72
fi
@ -2961,6 +3140,7 @@ printf '%s\\n' 0 > "/proc/sys/net/ipv4/conf/$BRIDGE/proxy_arp" || exit 72
PROXY_ARP_PVLAN_CHANGED=yes
printf '%s\\n' 1 > "/proc/sys/net/ipv4/conf/$BRIDGE/proxy_arp_pvlan" || exit 72
[ "$(cat "/proc/sys/net/ipv4/conf/$BRIDGE/proxy_arp_pvlan" 2>/dev/null)" = 1 ] || exit 72
VC_VPC_APPLY_PHASE=proxy
vc_vpc_proxy_snapshot "$TXN/proxy.activation.current" || exit 72
vc_vpc_runtime_subset "$TXN/proxy.activation.current" "$TXN/proxy.before" || exit 71
PROXY_ENTRIES_CHANGED=yes
@ -2989,7 +3169,9 @@ cat <<'{proxy_marker}' | LC_ALL=C sort -u > "$TXN/proxy.desired"
{proxy_marker}
"$CMP" -s "$TXN/proxy.after" "$TXN/proxy.desired" || exit 72
# Restore exact member bindings after the manager may have cleared them.
VC_VPC_APPLY_PHASE=neighbors
vc_vpc_neighbors_reconcile || exit 72
VC_VPC_APPLY_PHASE=routes
ROUTES_CHANGED=yes
cat > "$TXN/routes.desired" <<'{route_marker}'
{route_lines}
@ -3039,6 +3221,7 @@ done <<'{route_marker}'
# Recheck the complete RPDB scope after route convergence, immediately before
# adding missing rules. An out-of-band rule created during the transaction
# must force rollback rather than make a successful apply read back degraded.
VC_VPC_APPLY_PHASE=policy_rules
# vc-policy-rule-preflight-begin
: > "$TXN/rules.before"
: > "$TXN/rules.attempted"
@ -3052,6 +3235,7 @@ vc_ip_rule_preflight "$WG_RULE_PRIORITY" "$WG_IFACE" {_q(plan.vpc.cidr)} "$ROUTE
vc_ip_rule_ensure "$RULE_PRIORITY" - {_q(plan.vpc.cidr)} "$ROUTE_TABLE" 73 73 "$TXN/rules.attempted" "$TXN/rules.created" || exit $?
vc_ip_rule_ensure "$WG_RULE_PRIORITY" "$WG_IFACE" {_q(plan.vpc.cidr)} "$ROUTE_TABLE" 73 73 "$TXN/rules.attempted" "$TXN/rules.created" || exit $?
# vc-policy-rule-ensure-end
VC_VPC_APPLY_PHASE=fabric_acl
cat > "$STATE_FILE" <<'{state_marker}'
{state.rstrip()}
{state_marker}
@ -3068,23 +3252,54 @@ vc_nft_table_snapshot inet virt_cloud_wg_fabric \
[ ! -e "$TXN/fabric.nft.desired.absent" ] || exit 74
vc_nft_table_matches_snapshot inet virt_cloud_wg_fabric \
"$TXN/fabric.nft.desired" || exit 74
while IFS='|' read -r peer_host key allowed; do
VC_VPC_APPLY_PHASE=wireguard
expected_fabric_peer_keys={_q(apply_fabric_peer_key_words)}
if [ "{apply_fabric_peer_keys_provided}" = yes ]; then
live_fabric_peer_keys=$("$WG" show "$WG_IFACE" peers 2>/dev/null) || exit 74
for live_peer_key in $live_fabric_peer_keys; do
case " $expected_fabric_peer_keys " in
*" $live_peer_key "*) : ;;
*) exit 74 ;;
esac
done
for expected_peer_key in $expected_fabric_peer_keys; do
printf '%s\n' "$live_fabric_peer_keys" |
grep -Fqx "$expected_peer_key" || exit 74
done
fi
while IFS='|' read -r peer_host key expected_overlay; do
[ -n "$peer_host" ] || continue
"$WG" show "$WG_IFACE" peers | grep -Fqx "$key" || exit 74
case ",$allowed," in *',0.0.0.0/0,'*|*',0.0.0.0/1,'*|*',128.0.0.0/1,'* ) exit 74 ;; esac
case ",$expected_overlay," in *',0.0.0.0/0,'*|*',0.0.0.0/1,'*|*',128.0.0.0/1,'* ) exit 74 ;; esac
# Fabric has one AllowedIPs list per public key. Merge the complete,
# validated tenant set from every panel-owned VPC state file while keeping
# the peer's single overlay /32. Never replace the list with this VPC's
# partial view: doing so would black-hole another concurrently active VPC.
merge_actual=$(vc_wg_allowed_for_peer "$WG_IFACE" "$key") || exit 74
merge_overlay=$(vpc_state_overlay_from_allowed "$merge_actual") || exit 74
vpc_state_overlay_from_allowed "$merge_actual" >/dev/null || exit 74
merge_extra=$(vpc_state_allowed_for_peer "$key") || exit 74
merge_allowed="$merge_overlay"
merge_allowed="$expected_overlay"
[ -z "$merge_extra" ] || merge_allowed="$merge_allowed,$merge_extra"
"$WG" set "$WG_IFACE" peer "$key" allowed-ips "$merge_allowed" || exit 74
done <<'{peer_marker}'
{peer_lines}
{peer_marker}
# `wg set` success is not an exact-state receipt. Re-read every authoritative
# peer and prove its Fabric overlay and route-backed tenant union exactly.
while IFS='|' read -r final_peer_host final_peer_key final_overlay; do
[ -n "$final_peer_host" ] || continue
final_actual=$(vc_wg_allowed_for_peer "$WG_IFACE" "$final_peer_key") || exit 74
vpc_state_overlay_from_allowed "$final_actual" >/dev/null || exit 74
final_extra=$(vpc_state_allowed_for_peer "$final_peer_key") || exit 74
final_expected="$final_overlay"
[ -z "$final_extra" ] || final_expected="$final_expected,$final_extra"
final_actual_set=$(printf '%s\n' "$final_actual" | tr ',' '\n' | awk 'NF' | LC_ALL=C sort -u)
final_expected_set=$(printf '%s\n' "$final_expected" | tr ',' '\n' | awk 'NF' | LC_ALL=C sort -u)
[ -n "$final_actual_set" ] &&
[ "$final_actual_set" = "$final_expected_set" ] || exit 74
done <<'{peer_marker}'
{peer_lines}
{peer_marker}
# The shared Fabric quarantine table is the last enforcement boundary before
# a VPC bridge is admitted. Reparse the complete stateless grammar,
# bind every owner comment to the engine-authenticated full Fabric digest,
@ -3092,6 +3307,7 @@ done <<'{peer_marker}'
# prove the whole table is still the post-refresh snapshot. A retained
# comment with altered drop semantics, a range element, or a late mutation
# therefore fails closed.
VC_VPC_APPLY_PHASE=final_readback
vc_vpc_fabric_nft_live_ready || exit 74
vpc_state_nft_set_matches || exit 74
vc_nft_table_matches_snapshot inet virt_cloud_wg_fabric \
@ -3105,6 +3321,7 @@ vc_ln_final_expected=$(cat "$TXN/neighbors.desired") || exit 74
vc_ln_final_actual=$(vc_local_neighbor_records "$BRIDGE") || exit 74
vc_ln_final_wanted=$(printf '%s\\n' "$vc_ln_final_expected" | awk 'NF {{print $0 "|PERMANENT"}}') || exit 74
[ "$vc_ln_final_actual" = "$vc_ln_final_wanted" ] || exit 74
VC_VPC_APPLY_PHASE=commit
rm -f "$VPC_PEER_FILE" "$VPC_ROUTE_FILE" "$VPC_TENANT_FILE" "$VPC_TENANT_FILE.nft"
printf '%s\\n' ready > "$TXN/state"; chmod 600 "$TXN/state"
printf '%s\\n' '__VC_VPC_APPLIED__=yes'
@ -3160,6 +3377,15 @@ def render_remove_script(
fragment = f"/etc/network/interfaces.d/virt-cloud-vpc-{r.token}"
sysctl_file = f"/etc/sysctl.d/99-virt-cloud-vpc-{r.token}.conf"
proxy_file = f"/etc/virt-cloud/vpc-dataplane/{r.token}.proxy"
authoritative_peer_overlays = _authoritative_fabric_peer_overlays(plan)
remove_peer_lines = "\n".join(
f"{key}|{overlay}"
for key, overlay in (authoritative_peer_overlays or {}).items())
remove_peer_key_words = " ".join(authoritative_peer_overlays or ())
remove_peer_marker = "__VC_VPC_REMOVE_FABRIC_PEERS_" + hashlib.sha256(
remove_peer_lines.encode()).hexdigest()[:12]
remove_peer_closure_provided = (
"yes" if authoritative_peer_overlays is not None else "no")
state_loader = _VPC_STATE_LOADER.replace(
"__VC_FABRIC_HOST_ID__", plan.host_id).replace(
"__VC_FABRIC_ID__", plan.vpc.fabric_id).replace(
@ -3877,13 +4103,57 @@ vc_nft_table_snapshot inet virt_cloud_wg_fabric \
[ ! -e "$TXN/fabric.nft.desired.absent" ] || exit 74
vc_nft_table_matches_snapshot inet virt_cloud_wg_fabric \
"$TXN/fabric.nft.desired" || exit 74
for key in $("$WG" show "$WG_IFACE" peers 2>/dev/null); do
actual=$(vc_wg_allowed_for_peer "$WG_IFACE" "$key") || exit 74
overlay=$(vpc_state_overlay_from_allowed "$actual") || exit 74
extra=$(vpc_state_allowed_for_peer "$key") || exit 74
merged="$overlay"; [ -z "$extra" ] || merged="$merged,$extra"
"$WG" set "$WG_IFACE" peer "$key" allowed-ips "$merged" || exit 74
done
expected_fabric_peer_keys={_q(remove_peer_key_words)}
if [ "{remove_peer_closure_provided}" = yes ]; then
live_fabric_peer_keys=$("$WG" show "$WG_IFACE" peers 2>/dev/null) || exit 74
for live_peer_key in $live_fabric_peer_keys; do
case " $expected_fabric_peer_keys " in
*" $live_peer_key "*) : ;;
*) exit 74 ;;
esac
done
for expected_peer_key in $expected_fabric_peer_keys; do
printf '%s\n' "$live_fabric_peer_keys" |
grep -Fqx "$expected_peer_key" || exit 74
done
while IFS='|' read -r key expected_overlay; do
[ -n "$key" ] || continue
actual=$(vc_wg_allowed_for_peer "$WG_IFACE" "$key") || exit 74
vpc_state_overlay_from_allowed "$actual" >/dev/null || exit 74
extra=$(vpc_state_allowed_for_peer "$key") || exit 74
merged="$expected_overlay"; [ -z "$extra" ] || merged="$merged,$extra"
"$WG" set "$WG_IFACE" peer "$key" allowed-ips "$merged" || exit 74
done <<'{remove_peer_marker}'
{remove_peer_lines}
{remove_peer_marker}
else
# Legacy direct callers lack a full authenticated overlay mapping. Keep
# their historical route-withdrawal behavior without treating it as an
# authoritative Fabric closure receipt.
for key in $("$WG" show "$WG_IFACE" peers 2>/dev/null); do
actual=$(vc_wg_allowed_for_peer "$WG_IFACE" "$key") || exit 74
overlay=$(vpc_state_overlay_from_allowed "$actual") || exit 74
extra=$(vpc_state_allowed_for_peer "$key") || exit 74
merged="$overlay"; [ -z "$extra" ] || merged="$merged,$extra"
"$WG" set "$WG_IFACE" peer "$key" allowed-ips "$merged" || exit 74
done
fi
if [ "{remove_peer_closure_provided}" = yes ]; then
while IFS='|' read -r final_peer_key final_overlay; do
[ -n "$final_peer_key" ] || continue
final_actual=$(vc_wg_allowed_for_peer "$WG_IFACE" "$final_peer_key") || exit 74
vpc_state_overlay_from_allowed "$final_actual" >/dev/null || exit 74
final_extra=$(vpc_state_allowed_for_peer "$final_peer_key") || exit 74
final_expected="$final_overlay"
[ -z "$final_extra" ] || final_expected="$final_expected,$final_extra"
final_actual_set=$(printf '%s\n' "$final_actual" | tr ',' '\n' | awk 'NF' | LC_ALL=C sort -u)
final_expected_set=$(printf '%s\n' "$final_expected" | tr ',' '\n' | awk 'NF' | LC_ALL=C sort -u)
[ -n "$final_actual_set" ] &&
[ "$final_actual_set" = "$final_expected_set" ] || exit 74
done <<'{remove_peer_marker}'
{remove_peer_lines}
{remove_peer_marker}
fi
vc_vpc_fabric_nft_live_ready || exit 74
vpc_state_nft_set_matches || exit 74
vc_nft_table_matches_snapshot inet virt_cloud_wg_fabric \
@ -3953,15 +4223,25 @@ def render_observe_script(
f"{x['destination_cidr']}|{x['device']}|{x.get('via') or '-'}|{x.get('metric', 100)}"
for x in routes)
peer_lines = "\n".join(
f"{p.host_id}|{p.public_key}|{_allowed(plan, p)}"
# The host's validated state files, rather than the control-plane
# catalog aggregate, own the tenant half of AllowedIPs. The current
# plan still supplies the authenticated peer identity and overlay /32.
f"{p.host_id}|{p.public_key}|{p.overlay_cidr}"
for p in plan.peers)
fabric_peer_key_words = " ".join(sorted(plan.fabric_peer_keys or ()))
authoritative_peer_overlays = _authoritative_fabric_peer_overlays(plan)
fabric_peer_overlay_lines = "\n".join(
f"{key}|{overlay}"
for key, overlay in (authoritative_peer_overlays or {}).items())
fabric_peer_key_words = " ".join(authoritative_peer_overlays or ())
fabric_peer_key_arg = _q(fabric_peer_key_words)
fabric_peer_keys_provided = "yes" if plan.fabric_peer_keys is not None else "no"
fabric_peer_keys_provided = (
"yes" if authoritative_peer_overlays is not None else "no")
route_marker = "__VC_VPC_OBS_ROUTES_" + hashlib.sha256(
route_lines.encode()).hexdigest()[:12]
peer_marker = "__VC_VPC_OBS_PEERS_" + hashlib.sha256(
peer_lines.encode()).hexdigest()[:12]
fabric_peer_marker = "__VC_VPC_OBS_FABRIC_PEERS_" + hashlib.sha256(
fabric_peer_overlay_lines.encode()).hexdigest()[:12]
state_loader = _VPC_STATE_LOADER.replace(
"__VC_FABRIC_HOST_ID__", plan.host_id).replace(
"__VC_FABRIC_ID__", plan.vpc.fabric_id).replace(
@ -4114,7 +4394,7 @@ vc_vpc_observe_cleanup() {{
trap 'vc_vpc_observe_cleanup' EXIT
trap 'exit 79' HUP INT TERM
vc_vpc_fabric_observe_ready() {{
vpc_state_load || return 1
[ "$VPC_STATE_STATUS" = owned ] || return 1
vc_nft_table_snapshot inet virt_cloud_wg_fabric \
"$FABRIC_NFT_SNAPSHOT" || return 1
[ -f "$FABRIC_NFT_SNAPSHOT" ] &&
@ -4383,18 +4663,30 @@ route_extra=$(printf '%s\\n' "$route_records" | awk -F'|' '{{
}} END {{ if (count != {len(routes)}) extra=1; print extra ? "yes" : "no" }}')
printf '%s\\n' "__VC_VPC_ROUTE_EXTRA__=$route_extra"
{route_checks}
while IFS='|' read -r peer_host key expected_allowed; do
vpc_state_loaded=no
if vpc_state_load && [ "$VPC_STATE_STATUS" = owned ]; then
vpc_state_loaded=yes
fi
while IFS='|' read -r peer_host key expected_overlay; do
[ -n "$peer_host" ] || continue
actual=$(vc_wg_allowed_for_peer "$WG_IFACE" "$key") || actual=invalid
# ``wg show`` does not promise AllowedIPs order. Canonicalize both
# normalized lists before comparing, and retain the complete set in
# the marker. The Python adapter projects this to this VPC's tenant /32s
# after proving that no extra or missing entry exists.
expected_allowed=$expected_overlay
state=missing
if [ "$vpc_state_loaded" = yes ]; then
state_allowed=$(vpc_state_allowed_for_peer "$key") || state_allowed=invalid
if [ "$state_allowed" != invalid ] && [ -n "$state_allowed" ]; then
expected_allowed="$expected_allowed,$state_allowed"
fi
fi
# ``wg show`` does not promise AllowedIPs order. Canonicalize the live set
# and the exact overlay + route-backed union derived by the shared state
# loader. The Python adapter separately proves this VPC's required routes
# are a subset before projecting its own tenant /32s.
expected_set=$(printf '%s\\n' "$expected_allowed" | tr ',' '\\n' | awk 'NF' | LC_ALL=C sort -u)
actual_set=$(printf '%s\\n' "$actual" | tr ',' '\\n' | awk 'NF' | LC_ALL=C sort -u)
allowed=$(printf '%s\\n' "$actual_set" | awk 'BEGIN {{ ORS=""; sep="" }} NF {{ printf "%s%s", sep, $0; sep="," }}')
state=missing
if [ -n "$actual_set" ] && [ "$actual_set" = "$expected_set" ]; then
if [ "$vpc_state_loaded" = yes ] && [ "$state_allowed" != invalid ] &&
[ -n "$actual_set" ] && [ "$actual_set" = "$expected_set" ]; then
state=owned
else
allowed=missing
@ -4407,8 +4699,13 @@ done <<'{peer_marker}'
expected_fabric_peer_keys={fabric_peer_key_arg}
wg_peer_set=unverified
if [ "{fabric_peer_keys_provided}" = yes ]; then
live_fabric_peer_keys=$("$WG" show "$WG_IFACE" peers 2>/dev/null || true)
wg_peer_set=exact
if live_fabric_peer_keys=$("$WG" show "$WG_IFACE" peers 2>/dev/null); then
:
else
live_fabric_peer_keys=
wg_peer_set=missing
fi
for live_peer_key in $live_fabric_peer_keys; do
case " $expected_fabric_peer_keys " in
*" $live_peer_key "*) : ;;
@ -4418,6 +4715,41 @@ if [ "{fabric_peer_keys_provided}" = yes ]; then
for expected_peer_key in $expected_fabric_peer_keys; do
printf '%s\n' "$live_fabric_peer_keys" | grep -Fqx "$expected_peer_key" || wg_peer_set=missing
done
# Key presence alone cannot prove the shared cryptokey-routing boundary.
# A VPC revision may have withdrawn the final route for a peer that remains
# in the Fabric. Require every authoritative key to carry exactly one live
# overlay /32 plus only the route-backed tenant union from durable host
# state, including peers outside this VPC's current participant set.
if [ "$wg_peer_set" = exact ] && [ "$vpc_state_loaded" = yes ]; then
while IFS='|' read -r expected_peer_key expected_overlay; do
[ -n "$expected_peer_key" ] || continue
closure_actual=$(vc_wg_allowed_for_peer "$WG_IFACE" "$expected_peer_key") || {{
wg_peer_set=missing
break
}}
vpc_state_overlay_from_allowed "$closure_actual" >/dev/null || {{
wg_peer_set=missing
break
}}
closure_extra=$(vpc_state_allowed_for_peer "$expected_peer_key") || {{
wg_peer_set=missing
break
}}
closure_expected="$expected_overlay"
[ -z "$closure_extra" ] || closure_expected="$closure_expected,$closure_extra"
closure_actual_set=$(printf '%s\n' "$closure_actual" | tr ',' '\n' | awk 'NF' | LC_ALL=C sort -u)
closure_expected_set=$(printf '%s\n' "$closure_expected" | tr ',' '\n' | awk 'NF' | LC_ALL=C sort -u)
if [ -z "$closure_actual_set" ] ||
[ "$closure_actual_set" != "$closure_expected_set" ]; then
wg_peer_set=missing
break
fi
done <<'{fabric_peer_marker}'
{fabric_peer_overlay_lines}
{fabric_peer_marker}
elif [ "$wg_peer_set" = exact ]; then
wg_peer_set=missing
fi
fi
printf '%s\n' "__VC_VPC_WG_PEER_SET__=$wg_peer_set"
fabric_nft_state=no
@ -4998,41 +5330,39 @@ class VpcPrivateHostAdapter:
x.get("via"), str(x.get("metric", 100)))
for x in parsed.get("routes", [])
}
expected_wg_full = {
# The live `wg show` value includes the peer's overlay /32 as the
# first AllowedIP. Compare the complete cryptokey-routing set;
# project tenant entries only after that exact check.
p.host_id: (p.public_key, set(_allowed(plan, p).split(",")))
for p in plan.peers
}
wg_allowed_rows = parsed.get("wg_allowed_ips", [])
actual_wg_full = {
x.get("peer_host_id"): (x.get("public_key"),
set(x.get("tenant_allowed_ips", [])))
for x in parsed.get("wg_allowed_ips", [])
for x in wg_allowed_rows
}
# The live peer may carry /32s belonging to other VPCs sharing this
# Fabric. Verify the complete aggregate here, then expose only this
# VPC's tenant set to the PHP contract.
# Durable host state owns the complete live tenant union. The shell
# proves that union is exact; here retain the plan-level obligation:
# this VPC's peer key, overlay, members and service routes must all be
# present before the result can be accepted or projected to PHP.
required_tenant = _expected_peer_tenant_ips(plan.vpc)
expected_tenant: dict[str, set[str]] = {
p.host_id: {m.private_ip + "/32" for m in plan.remote_members
if m.host_id == p.host_id}
p.host_id: set(required_tenant.get(p.host_id, frozenset()))
for p in plan.peers
}
members_by_id = {m.member_id: m for m in plan.members}
for route in plan.vpc.routes:
target = (members_by_id.get(route.next_hop_member_id)
if route.next_hop_member_id is not None else None)
target_host = target.host_id if target is not None else route.next_hop_host_id
if target_host in expected_tenant:
expected_tenant[target_host].add(route.destination_cidr)
expected_wg = {
p.host_id: (p.public_key, expected_tenant.get(p.host_id, set()))
for p in plan.peers
}
actual_wg = {
host: (key, ips & expected_wg.get(host, (None, set()))[1])
for host, (key, ips) in actual_wg_full.items()
required_wg = {
p.host_id: (p.public_key,
{p.overlay_cidr} | expected_tenant.get(p.host_id, set()))
for p in plan.peers
}
wg_required_verified = (
len(wg_allowed_rows) == len(actual_wg_full) == len(required_wg)
and set(actual_wg_full) == set(required_wg)
and all(
actual_wg_full[host][0] == key
and required_ips.issubset(actual_wg_full[host][1])
for host, (key, required_ips) in required_wg.items()
)
)
members = _member_nics(plan, xml_by_instance)
actual_route_state = {
(x.get("destination_cidr"), x.get("device"), str(x.get("table")),
@ -5059,6 +5389,14 @@ class VpcPrivateHostAdapter:
for x in parsed.get("rules", []) if x.get("state") == "owned"
}
expected_peer_states = {p.host_id: "owned" for p in plan.peers}
wg_state_marker_count = sum(
line.startswith("__VC_VPC_WG_STATE__=")
for line in output.splitlines()
)
wg_peer_set_marker_count = sum(
line.startswith("__VC_VPC_WG_PEER_SET__=")
for line in output.splitlines()
)
expected_bridge_nft = "yes" if plan.local_members else "none"
host_dataplane_verified = (parsed.get("bridge") == plan.resources.bridge
and parsed.get("bridge_owner") == DATAPLANE_OWNER_MARKER
@ -5094,9 +5432,13 @@ class VpcPrivateHostAdapter:
and actual_routes == actual_route_state
and route_flags_valid
and expected_rules == actual_rules
and expected_wg_full == actual_wg_full
and (plan.fabric_peer_keys is None
or parsed.get("wg_peer_set") == "exact")
and wg_required_verified
and ((plan.fabric_peer_keys is None
and plan.fabric_peer_overlays is None)
or (plan.fabric_peer_overlays is not None
and wg_peer_set_marker_count == 1
and parsed.get("wg_peer_set") == "exact"))
and wg_state_marker_count == len(expected_peer_states)
and expected_peer_states == parsed.get("wg_states", {})
and parsed.get("arp_nft") == ("yes" if plan.local_members else "none"))
host_config_verified = host_dataplane_verified and all(
@ -5108,7 +5450,7 @@ class VpcPrivateHostAdapter:
and parsed.get("firewall_mode") == firewall_mode
and parsed.get("firewall_state") == "owned"
and parsed.get("state_firewall_mode") == firewall_mode
and plan.fabric_peer_keys is not None
and plan.fabric_peer_overlays is not None
and parsed.get("wg_peer_set") == "exact"
and parsed.get("helper_contract") == "owned"
and parsed.get("bridge_admin") == "up"

@ -1098,6 +1098,12 @@ final class CloudVpcService
}
[$operation, $payload, $runtimeProofRefresh, $previousOutcomePending] = $claim;
$leaseToken = (string) $operation->lease_token;
// Only the explicit user reconcile route may transfer an exact,
// childless retained Engine root for this immutable target. Keep the
// capability inside the signed actor and off worker/power requests.
$engineActor = $allowFailedRetry
? array_replace($actor, ['vpc_reconcile_explicit' => true])
: $actor;
try {
// A previous apply may have finished while its PHP receipt was
// lost. Recover that exact proof before requiring a stopped guest
@ -1118,7 +1124,7 @@ final class CloudVpcService
null, self::ENGINE_TIMEOUT_SECONDS
))->vpc(
'POST', '/internal/v1/vpcs/' . $vpcKey . '/delete',
$actor, $payload, (string) $operation->operation_id
$engineActor, $payload, (string) $operation->operation_id
);
$expectedRevision = (int) ($payload['desired_revision'] ?? 0);
$expectedDigest = (string) ($payload['desired_digest'] ?? '');
@ -1146,7 +1152,7 @@ final class CloudVpcService
$result = (new CloudEngineClient(
null, self::ENGINE_TIMEOUT_SECONDS
))->vpc('POST', '/internal/v1/vpcs/apply',
$actor, $payload, (string) $operation->operation_id);
$engineActor, $payload, (string) $operation->operation_id);
$observed = self::applyObservation($operation, $payload, $result);
self::recordObserved($vpcKey, (int) ($payload['desired_revision'] ?? 0),
(string) ($payload['desired_digest'] ?? ''), $observed,

@ -14,7 +14,7 @@ namespace App\Services\Cloud {
public function vpc(string $method, string $path, array $actor,
?array $body, ?string $operationId): array
{
self::$requests[] = compact('method', 'path', 'body', 'operationId');
self::$requests[] = compact('method', 'path', 'actor', 'body', 'operationId');
return (self::$respond)($body);
}
}
@ -63,6 +63,7 @@ foreach (['vpc_create', 'member_attach', 'member_detach'] as $index => $kind) {
$after = Capsule::table('cloud_vpc_operations')
->where('operation_id', $vpc['operation_id'])->first();
$assert($request['operationId'] === $vpc['operation_id']
&& ($request['actor']['vpc_reconcile_explicit'] ?? null) === true
&& $request['body'] === json_decode($before->payload_json, true)
&& $after->payload_json === $before->payload_json
&& $after->status === 'uncertain'
@ -297,6 +298,7 @@ $expectCloudError(static fn () => (new \App\Controllers\CloudVpcController())->r
), 502);
$assert(count(CloudEngineClient::$requests) === $requestCount + 1
&& end(CloudEngineClient::$requests)['operationId'] === $replacementId
&& (end(CloudEngineClient::$requests)['actor']['vpc_reconcile_explicit'] ?? null) === true
&& end(CloudEngineClient::$requests)['body'] === $replacementPayload,
'The explicit user reconcile did not retry the exact failed target.');

Loading…
Cancel
Save