diff --git a/docs/VALIDATION.md b/docs/VALIDATION.md index ae3c95a0..8e4f687b 100644 --- a/docs/VALIDATION.md +++ b/docs/VALIDATION.md @@ -1,5 +1,53 @@ # 本版验证记录 +## 2026-10-14:新 VPC 加入回滚与原 operation 显式恢复 + +- 现场固定 operation `708fa7c660383d7ee945d1543de1e0a8`、VPC + `5518137d9479fc40bf153c26a3f4c350` 的 revision 3 在 AlmaLinux 宿主完成桥、 + nft、路由和策略规则后,于 `fabric_acl/vpc_state_load` 退出并整体回滚;目标实例 + 的 live/persistent XML 因而仍只有公网网卡。Debian 当时没有 Alma 上相同的历史 + state 组合,所以结果不同;两台宿主使用的是同一 Engine/helper 方法,差异来自 + 持久状态历史,不是发行版分支脚本。 +- 根因是旧 state 把完整 Fabric peer 聚合复制到每个 VPC。旧、新 VPC 同时保存 + 同一公钥和 `/32` 时,loader 将合法重复误判成冲突。新 state 只保存本 VPC 中 + 有本 state `wg-fabric0` route 支撑的 peer `/32`;loader 可读取旧聚合候选, + 但最终只保留 route-backed 唯一归属。相同地址映射到不同公钥、单文件重复、 + route/peer 不成对和越界 route 仍拒绝。 +- apply、observe 和 retiring-host remove 均从完整 Fabric 计划取得权威 + `public key -> overlay /32` 映射,再与宿主全部有效 state 的 route-backed tenant + union 合并。每个 peer 的实时 AllowedIPs、完整公钥集合和 overlay 映射均 exact + 回读;撤销最后一条 route 后遗留的 tenant `/32` 会使回读失败并在应用中被收敛。 + helper 契约升级为 `vpc-state-route-scope-contract-v4`,旧 v3 helper 在任何 VPC + mutation 前 fail closed,须先通过原 Fabric 的显式核对发布并完整回读 v4。 +- VPC HTTP/周期 root 现在先绑定完整 operation、VPC、revision 和 digest。 + 仅签名 actor 中的真布尔 `vpc_reconcile_explicit` 可触发一次性接管;Engine 只在 + 原 uncertain root 为唯一、无资源、无任何同 execution child、目标与持久 VPC + 或删除回执完全一致时,以单次 CAS 删除旧 root、绑定当前 request 并写有界回执。 + marker 在绑定时被消费,不进入 child。worker、periodic、普通重试、旧目标、多个 + 候选、残留 child 和混合 CAS 状态均不接管;claim/release 回包丢失与取消按 exact + snapshot 分类并保留 fence,禁止把未知写结果冒充成功。 +- 宿主 apply journal 仅写白名单阶段、状态和退出码,不记录命令、密钥或配置内容。 + Mac 本机使用 `engine/.venv` 的 Python 3.12.14、pytest 9.1.1 和项目 + `requirements.txt`;18 个 VPC/Fabric/协调相关模块统一回归 **1,234 passed**, + 保留 39 条既有 XML truthiness 弃用警告。两位独立审查分别复核 operation CAS、 + 完整 peer/overlay 闭集和退出宿主路径;Python 3.9/3.12 compile、apply/observe/remove + 生成脚本 `/bin/sh -n` 及 `git diff --check` 通过。 +- 完整 Engine 结果为 **3,975 passed、24 skipped、118 subtests passed、6 failed**; + 六个失败均已逐项在未修改的 `9fb2624` 隔离副本中同样复现,且三个公网池用例在 + 独立 pytest 进程中仍失败,不是本轮状态泄漏或测试顺序污染。其中三项属于旧测试 + 漂移:cloud template 创建规格新增 `platform_network_policy_snapshot` 后旧字段集合 + 断言未更新;reinstall cleanup 的 409 测试仍读取已删除的 `detail` 字段;Fabric + 全局权限测试的 readiness fixture 缺当前必需能力证据而得到 `vpc_ready=false`。 + 另三项揭示既有公网池缺陷:分配器可能重复选择 libvirt 可见 VM 已占用的 + `8.8.8.4`,且 `/28` 池耗尽后两个参数组合仍返回 201 而不是 409。公网池缺陷会 + 影响新实例公网地址分配,应单独修复;它们不进入本次已有实例的 VPC 私网加入路径, + 六项均不是现场 rollback 的触发点。因此本轮不宣称全项目测试全绿,真实 + helper/Fabric 回读仍是发布门禁。 +- 本节当前记录源码与本地回归,尚未提交、部署或重放网络写请求。部署后先显式核对 + 原 Fabric 并确认两台宿主 v4/helper、peer/overlay、route 和 nft exact;随后只对 + 原 VPC operation 执行一次显式“重新核对”。只有宿主 state、WireGuard、nft、 + libvirt XML/seed、QGA `/32` 和真实跨宿主流量全部通过,才能记录现场恢复。 + ## 2026-10-14:Fabric 旧未决占用阻断显式核对 - 现场已部署 `d50e6d5`,Engine 于 `08:47:49 UTC` 重启。`08:48:06 UTC` diff --git a/docs/VPC_DATAPLANE_ACCEPTANCE.md b/docs/VPC_DATAPLANE_ACCEPTANCE.md index ab4cfa09..b77ca747 100644 --- a/docs/VPC_DATAPLANE_ACCEPTANCE.md +++ b/docs/VPC_DATAPLANE_ACCEPTANCE.md @@ -65,12 +65,17 @@ 每份受管 state 的 `/32` route 必须在完整解析该文件后,仅按该文件唯一的 CIDR 校验;不能把累计的全宿主路由重新按最后一个 VPC 的 CIDR 校验,也不能按“属于 -任一 VPC 网段”放行。验证完成的 route、tenant IP 和 peer IP 仍需完整聚合, -跨 state 的重复 peer IP 继续拒绝。至少以两个不重叠网段验证正反文件顺序、CIDR -后置,以及跨到另一 VPC 网段的恶意 route 仍被拒绝。 +任一 VPC 网段”放行。每条 `wg-fabric0` route 还必须在同一 state 中存在同公钥的 +peer 归属,另一份旧 state 的全局 peer 快照不能替它补齐。新 state 只保存本 VPC +有 route 支撑的 peer `/32`;共享 loader 可读取旧版同公钥聚合快照并去重,但最终 +只保留仍有唯一 `wg-fabric0` route 的地址。相同 IP 对应不同公钥、单文件重复 peer、 +重复 route 或越界 route 均继续拒绝。删除一个 state 后,即使其他旧快照仍抄有该 +地址,也必须立即从 WireGuard 与共享 nft 授权中撤销。至少以两个不重叠网段验证 +正反文件顺序、CIDR 后置、旧新 state 混用、删除撤权和跨文件缺失 peer 掩盖。 宿主 `/usr/local/libexec/virt-cloud-wg-fabric` 还须包含第七行契约 -`# vpc-state-route-scope-contract-v3`。仅更新 Engine 不会替换已安装 helper; +`# vpc-state-route-scope-contract-v4`。v3 仍使用旧的全局 peer 聚合 loader;仅更新 +Engine 不会替换已安装 helper,且 VPC mutation 必须在 v4 发布前 fail closed。 同计划旧 helper 会被 Fabric 观察判为需要升级,普通 GET 和周期观察不执行升级。 已有邻居契约、状态和在途操作均完整的环境,可通过原 Fabric 的显式“重新核对” 受控发布当前 helper;有旧邻居状态或未完成事务时,仍按上节先核实迁移证据。 @@ -83,14 +88,18 @@ helper 升级并通过完整回读后,正常关闭首次接入尚未完成的 ## Fabric tenant 集合的统一口径 `VPC_TENANT_FILE`、Fabric nft `vpc_tenant_ips` 和观察期望必须统一为全部受管 -state 的已验证 route 与 peer `/32` 的去重并集。peer 行可携带认证计划中的全局 -Fabric 地址,即使该地址不在当前 state 的 CIDR 中;这不授权当前 VPC 创建该 -地址的 route。仅派生 tenant 集合可去重,原始重复 route 或 peer 仍必须拒绝。 - -至少验证另一个 VPC 的 peer-only 地址、route/peer 自然重叠、多个 state 与文件 -顺序、CIDR 后置、nft refresh 后完整回读;故意漏掉任一 peer 地址时观察不得 -active。集合对齐不等于每份旧全局 peer 快照已完成归属/删除收敛;涉及历史快照 -清理时保留原操作与严格迁移边界,不通过删除旧 VPC 或改写成功状态绕过。 +state 的已验证 route `/32` 去重并集,包含本地 bridge route 与远端 +`wg-fabric0` route;没有 route 的 peer-only 聚合地址不得进入 nft 集合。 +`VPC_PEER_FILE` 先兼容读取旧聚合候选,再按 route-backed 归属筛选。共享 WireGuard +AllowedIPs 的完整期望由宿主全部有效 state 的筛选结果加每个 peer 的 overlay `/32` +组成,不能用某次控制面请求携带的全 Fabric 聚合代替宿主持久归属。 + +观察必须对完整宿主 state union 与实时 WG 做 exact 比较,同时证明当前 VPC 的 +peer 公钥、overlay、成员和 service route 是其子集;生产计划还须提供完整、唯一的 +`公钥 → overlay /32` 权威映射,不能从实时 AllowedIPs 反推 overlay。至少验证 +A+B 与 B+C 的交错拓扑、同 peer 多 VPC union、peer-only +地址不授权、删除后旧地址拒绝、nft refresh 完整回读和 stale/extra AllowedIPs。 +旧 state 迁移保留原 operation 与严格边界,不通过删库或改写成功状态绕过。 ## NetworkManager 带运行网卡的桥恢复 @@ -154,8 +163,10 @@ UUID、ifindex 和完整端口集合。恢复必须先证明 ACL、sysctl、prox 来宾 main table 的目标 route 必须只有一条 exact scope-link VPC CIDR,不能含 gateway、 nexthop、`linkdown`、`dead` 或 `unresolved` flags;成员或自定义 route 变化不得改变 兄弟 VM 的 guest generation。历史逐成员 `/32` seed 只能在完全停机后迁移。 -4. `wg show wg-fabric0 allowed-ips` 的每个 peer 与完整期望集合一致:overlay `/32` - 加精确 tenant/service `/32`,没有 `0.0.0.0/0`、宽前缀或其他 VPC 的未授权地址。 +4. `wg show wg-fabric0 allowed-ips` 的每个 peer 与宿主 route-backed state union + 完整一致:overlay `/32` 加全部仍有归属的 tenant/service `/32`。同宿主的其他 + VPC 地址可以存在,但必须由对应 state 的 route 支撑;控制面聚合中尚未落盘或 + 已删除 VPC 的地址不得残留。没有 `0.0.0.0/0`、宽前缀或未知公钥。 5. nft inet/bridge/arp 三张表的 owner marker、chain priority、ACL comments、 anti-spoof、IPv6/广播/未知目的 drop 和精确 proxy-ARP set 均能被 observe 回读。 bridge 上通用 `proxy_arp=0`、同桥下一跳所需的 `proxy_arp_pvlan=1`、 diff --git a/docs/知识库/Memory/CURRENT.md b/docs/知识库/Memory/CURRENT.md index fd5309c2..bd2ac13e 100644 --- a/docs/知识库/Memory/CURRENT.md +++ b/docs/知识库/Memory/CURRENT.md @@ -10,6 +10,43 @@ tags: # 当前项目交接:Virt Cloud +## 2026-10-14 新 VPC 加入在第二台宿主反复回滚 + +- **现场根因**:原 operation `708fa7c660383d7ee945d1543de1e0a8` 在 AlmaLinux + 宿主的 VPC apply 已完成桥、nft、路由和策略规则,随后于 `fabric_acl` 的 + `vpc_state_load` 退出并整体回滚。旧 loader 把每份 state 中的全 Fabric peer + 聚合当作全局唯一来源;同一公钥、同一 `/32` 被旧 VPC 与新 VPC state 同时记录时 + 被误判冲突。Debian 当时尚无同样的历史 state,所以两台系统结果不同;根因是 + 持久状态历史,不是发行版使用了不同核对脚本。 +- **数据面修复**:新 state 仅保存本 VPC 的 route-backed peer 贡献;loader 兼容读取 + 旧同公钥聚合候选,但只授权仍有唯一 `wg-fabric0` route 的地址,并要求 route 与 + peer 在同一 state 内成对。删除 state 会撤销旧聚合地址,相同 IP 不同公钥仍 + fail closed。观察按宿主全部有效 state union 精确核对实时 WireGuard,同时要求 + 当前 VPC 的公钥、overlay、成员和 service route 子集成立;apply、observe 和 + retiring-host remove 都从完整 Fabric 计划携带唯一的 `公钥 → overlay /32` 映射, + 不从宿主实时 AllowedIPs 自举权威 overlay。 +- **升级边界**:Fabric helper 路由归属契约升为 + `vpc-state-route-scope-contract-v4`;v3 helper 使用旧 loader。部署 Engine 后须先用 + 原 Fabric 显式核对发布并回读 v4,VPC mutation 在此之前拒绝执行,不能让旧 timer + 随后把新 union 写回旧语义。 +- **operation 安全**:HTTP 与周期 VPC root 绑定 operation/revision/digest 的不可变 + 身份;claim/release 回包丢失、取消和残留 child 均按 exact execution 回读。 + 任何未结算 child 或 execution unknown 都把 root/child 保留为 uncertain 并返回 + 503,不再出现“页面 200、下一轮却被孤儿占用阻断”。只有用户显式“重新核对”携带 + 的签名一次性 marker 可在旧 root 唯一、childless、完整目标与持久记录一致时,以 + 单次 CAS 交接原 root;worker/periodic、旧目标和残留 child 均不接管。宿主 apply + 只向本地 journal 记录白名单阶段与退出码,不输出命令或配置内容。 +- **本机环境与边界**:Mac 已建立 `engine/.venv`(Python 3.12.14)并按项目依赖运行 + 回归;18 个相关模块 1,234 passed,完整 Engine 为 3,975 passed、24 skipped、 + 118 subtests passed、6 failed。六个失败均在未修改 `9fb2624` 复现:三项是旧测试 + 漂移,另三项是既有公网池占用扫描/耗尽判断缺陷;后者会影响新实例公网地址分配, + 但不进入本次已有实例的 VPC 私网加入路径,详见 `docs/VALIDATION.md`。源码验证不 + 代替 helper v4、两宿主 state/WG/nft、libvirt XML、seed、QGA 和来宾私网实流量 + 验收。本节当前仍待提交、部署并以原 operation 完成现场核对。 +- 详细迁移与验收见 `docs/VALIDATION.md`、`docs/VPC_DATAPLANE_ACCEPTANCE.md`; + 关系沿用 [[Concepts/Virt Cloud WireGuard Fabric与VPC L3私网]] 与 + [[Concepts/Virt Cloud VPC永久邻居与重复协调]]。 + ## 2026-10-14 Fabric 更新后 502 与原 operation 占用 - **现场**:`d50e6d5` 已部署,Engine 重启后原 `791e…` 的终态时间仍为 diff --git a/engine/tests/test_engine_boundary.py b/engine/tests/test_engine_boundary.py index f098befb..90eb872b 100644 --- a/engine/tests/test_engine_boundary.py +++ b/engine/tests/test_engine_boundary.py @@ -2178,6 +2178,17 @@ def test_hmac_binds_actor_payload_target_and_rejects_replay(): hashlib.sha256(body).hexdigest(), store, now=1000, ) + invalid_vpc_reconcile = {**USER, "vpc_reconcile_explicit": "true"} + invalid_headers = {key.lower(): value for key, value in signed_headers( + SECRET, "POST", target, body, invalid_vpc_reconcile, + timestamp=1000, nonce="v" * 24, + ).items()} + with pytest.raises(EngineAuthError): + authenticate( + invalid_headers, SECRET, "POST", target, + hashlib.sha256(body).hexdigest(), store, now=1000, + ) + def test_private_route_allowlist_and_no_browser_admin_cookie(boundary): client, *_ = boundary diff --git a/engine/tests/test_fabric.py b/engine/tests/test_fabric.py index af914a0f..6d65bd2a 100644 --- a/engine/tests/test_fabric.py +++ b/engine/tests/test_fabric.py @@ -341,7 +341,7 @@ def test_observe_artifact_state_accepts_only_kind_specific_header( f"# fabric-plan-digest={current.digest}\n" "# libvirt-startup-independent-v1\n" "# vpc-local-neighbor-contract-v1\n" - "# vpc-state-route-scope-contract-v3\n", + "# vpc-state-route-scope-contract-v4\n", encoding="utf-8", ) artifact.chmod(int(mode, 8)) diff --git a/engine/tests/test_fabric_catalog_coordination.py b/engine/tests/test_fabric_catalog_coordination.py index 3257fb75..01551f5d 100644 --- a/engine/tests/test_fabric_catalog_coordination.py +++ b/engine/tests/test_fabric_catalog_coordination.py @@ -247,3 +247,133 @@ def test_local_catalog_completion_preserves_outer_remote_claim(): fabric.close() asyncio.run(exercise()) + + +def _assert_exact_child_identity(row, owner, parent, resources, metadata): + assert row['state'] != 'active' + assert row['owner_id'] == owner.owner_id + assert row['execution_id'] == parent.execution_id + assert row['operation_id'] == parent.operation_id + assert row['kind'] == 'resource' + assert row['resources'] == sorted(resources) + assert row['metadata'] == metadata + + +def test_catalog_claim_lost_commit_response_recovers_exact_child_as_uncertain(): + class LostClaimReply(DurableCoordinator): + lose_reply = True + + def claim(self, **options): + result = super().claim(**options) + if options.get('kind') == 'resource' and self.lose_reply: + self.lose_reply = False + raise RuntimeError('catalog claim commit response lost') + return result + + async def exercise(): + store = MemoryCAS() + owner = LostClaimReply(store) + metadata = {'job_kind': 'vpc_operation', 'vpc_id': 'vpc-hk-1'} + parent = owner.claim(operation_id='operation-lock-recovery', + kind='request', metadata=metadata) + resources = ('catalog:network',) + catalog = _DurableAsyncLock(owner, resources) + + with execution_context(owner, parent): + with pytest.raises(RuntimeError, match='commit response lost'): + async with catalog: + pytest.fail('lost admission response entered the mutation body') + + rows = owner.snapshot() + children = [row for token, row in rows.items() if token != parent.token] + assert len(children) == 1 + _assert_exact_child_identity( + children[0], owner, parent, resources, metadata) + assert not catalog.locked() + + asyncio.run(exercise()) + + +def test_catalog_claim_response_cancellation_drains_and_releases_known_child(): + entered, resume = threading.Event(), threading.Event() + + class BlockingClaimReply(DurableCoordinator): + blocked = False + + def claim(self, **options): + result = super().claim(**options) + if options.get('kind') == 'resource' and not self.blocked: + self.blocked = True + entered.set() + assert resume.wait(3) + return result + + async def exercise(): + store = MemoryCAS() + owner = BlockingClaimReply(store) + metadata = {'job_kind': 'vpc_operation', 'vpc_id': 'vpc-hk-1'} + parent = owner.claim(operation_id='operation-lock-cancel', + kind='request', metadata=metadata) + resources = ('catalog:network',) + catalog = _DurableAsyncLock(owner, resources) + body_entered = [] + + async def admitted(): + async with catalog: + body_entered.append(True) + + with execution_context(owner, parent): + task = asyncio.create_task(admitted()) + try: + assert await asyncio.to_thread(entered.wait, 3) + task.cancel() + await asyncio.sleep(0) + assert not task.done() + finally: + resume.set() + with pytest.raises(asyncio.CancelledError): + await task + + assert body_entered == [] + rows = owner.snapshot() + children = [row for token, row in rows.items() if token != parent.token] + assert not any(row['state'] == 'active' for row in children) + assert children == [] + assert not catalog.locked() + + asyncio.run(exercise()) + + +def test_catalog_release_precommit_failure_retains_exact_child_as_uncertain(): + class FailedRelease(DurableCoordinator): + fail_release = True + + def release(self, claim): + if claim.kind == 'resource' and self.fail_release: + self.fail_release = False + raise CoordinationError('coordination_unavailable', 503) + return super().release(claim) + + async def exercise(): + store = MemoryCAS() + owner = FailedRelease(store) + metadata = {'job_kind': 'vpc_operation', 'vpc_id': 'vpc-hk-1'} + parent = owner.claim(operation_id='operation-lock-release', + kind='request', metadata=metadata) + resources = ('catalog:network',) + catalog = _DurableAsyncLock(owner, resources) + + with execution_context(owner, parent): + with pytest.raises(CoordinationError) as raised: + async with catalog: + pass + assert raised.value.error_code == 'coordination_unavailable' + + rows = owner.snapshot() + children = [row for token, row in rows.items() if token != parent.token] + assert len(children) == 1 + _assert_exact_child_identity( + children[0], owner, parent, resources, metadata) + assert not catalog.locked() + + asyncio.run(exercise()) diff --git a/engine/tests/test_fabric_libvirt_independence.py b/engine/tests/test_fabric_libvirt_independence.py index bed488f9..df70edef 100644 --- a/engine/tests/test_fabric_libvirt_independence.py +++ b/engine/tests/test_fabric_libvirt_independence.py @@ -149,7 +149,7 @@ def startup_files(root, current, *, legacy=False, gate=True, if legacy_neighbors and name == "helper": content = content.replace("# vpc-local-neighbor-contract-v1\n", "") if legacy_routes and name == "helper": - content = content.replace("# vpc-state-route-scope-contract-v3\n", "") + content = content.replace("# vpc-state-route-scope-contract-v4\n", "") target = root / name target.write_text(content) target.chmod(mode) @@ -231,7 +231,7 @@ def test_explicit_reconcile_upgrades_same_plan_then_skips_current_hosts(tmp_path rendered = _apply_script(actual_plan, host, keys) assert FABRIC_STARTUP_ISOLATION_MARKER in rendered assert "# vpc-local-neighbor-contract-v1" in rendered - assert "# vpc-state-route-scope-contract-v3" in rendered + assert "# vpc-state-route-scope-contract-v4" in rendered assert 'record_desired_absent gate_generator' in rendered startup_files(self.manager.startup_root, actual_plan, gate=False) return super().dispatch_apply(actual_plan, host, keys, operation_id) diff --git a/engine/tests/test_vpc.py b/engine/tests/test_vpc.py index bca93eb2..07a3044f 100644 --- a/engine/tests/test_vpc.py +++ b/engine/tests/test_vpc.py @@ -1331,6 +1331,31 @@ def test_fabric_gate_keeps_revision_mismatch_as_reconcilable_state(): assert result["state"] == "degraded" +def test_same_vpc_revision_and_digest_cannot_borrow_another_operation_receipt(monkeypatch): + async def run(): + store = MemoryStore() + coordinator = VpcCoordinator(store, FakeFabric(hosts=("host-a",))) + first = await coordinator.apply(payload(operation_id="a" * 32)) + before = store.get("vpc_desired", "vpc-hk-1") + retries = [] + + async def observe_only(*_args, **_kwargs): + retries.append(True) + + monkeypatch.setattr(coordinator, "_reconcile_pending_hosts", observe_only) + with pytest.raises(VpcError) as conflict: + await coordinator.apply(payload(operation_id="b" * 32)) + assert conflict.value.status == 409 + assert conflict.value.error_code == "vpc_revision_conflict" + assert store.get("vpc_desired", "vpc-hk-1") == before + assert retries == [] + replay = await coordinator.apply(payload(operation_id="a" * 32)) + assert replay["operation_id"] == first["operation_id"] + assert retries == [True] + + asyncio.run(run()) + + def test_coordinator_revision_conflict_and_tampered_record_fail_closed(): store = MemoryStore() coordinator = VpcCoordinator(store, FakeFabric(hosts=("host-a",))) @@ -2605,9 +2630,11 @@ def test_delete_pending_projection_does_not_probe_confirmed_offline_host(): } coordinator._fabric_peers_for_plan = lambda _plan, _record: [] host_digests = [] + host_peer_overlays = [] def confirm_host(*args, **kwargs): host_digests.append(args[8]) + host_peer_overlays.append(kwargs["fabric_peer_overlays"]) return { "state": "active", "removed": True, "transport_verified": True, @@ -2641,6 +2668,10 @@ def test_delete_pending_projection_does_not_probe_confirmed_offline_host(): assert probed == ["host-b"] assert host_digests == [fabric_plan.digest] + assert host_peer_overlays == [{ + "a" * 44: "100.64.0.1/32", + "b" * 44: "100.64.0.2/32", + }] assert coordinator._confirmed_removal( record["hosts"][0], plan.desired_revision, plan.desired_digest) @@ -4104,6 +4135,8 @@ def test_delete_manifest_removes_unknown_member_nic_before_host_resources( "private_mac": member.private_mac, } for member in manifest] events = [] + local_key = "AQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQE=" + remote_key = "AgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgI=" monkeypatch.setattr(VMService, "__init__", lambda self, *args, **kwargs: None) @@ -4118,6 +4151,10 @@ def test_delete_manifest_removes_unknown_member_nic_before_host_resources( ) def remove_host(self, dataplane): + assert dataplane.fabric_peer_keys == frozenset({remote_key}) + assert dataplane.fabric_peer_overlays == frozenset({ + (remote_key, "100.64.0.3/32"), + }) events.append(("host", dataplane.host_id, dataplane.vpc.vpc_id)) return { "state": "active", "removed": True, @@ -4134,7 +4171,12 @@ def test_delete_manifest_removes_unknown_member_nic_before_host_resources( plan, "host-a", SimpleNamespace(manager=object()), {"libvirt_uri": "qemu:///system", "storage_pool": "default", "iso_pool": "iso", "default_network": "default"}, - [], None, retiring_members=rows, fabric_public_keys={}, + [], None, retiring_members=rows, + fabric_public_keys={"host-a": local_key, "host-b": remote_key}, + fabric_peer_overlays={ + local_key: "100.64.0.2/32", + remote_key: "100.64.0.3/32", + }, fabric_plan_digest="ef" * 32, ) assert events == [ diff --git a/engine/tests/test_vpc_apply_receipts.py b/engine/tests/test_vpc_apply_receipts.py index 9d7fcd15..d87a342f 100644 --- a/engine/tests/test_vpc_apply_receipts.py +++ b/engine/tests/test_vpc_apply_receipts.py @@ -32,7 +32,12 @@ def empty_receipts(): @pytest.fixture def plan(): p = _plan() - return replace(p, fabric_peer_keys=frozenset(x.public_key for x in p.peers)) + return replace( + p, + fabric_peer_keys=frozenset(x.public_key for x in p.peers), + fabric_peer_overlays=frozenset( + (x.public_key, x.overlay_cidr) for x in p.peers), + ) class Adapter(dp.VpcPrivateHostAdapter): @@ -362,8 +367,9 @@ def test_rendered_host_receipt_requires_exact_persistent_bytes(tmp_path, shell, f'PLAN_DIGEST="{plan.fabric_plan_digest}"\n' '# libvirt-startup-independent-v1\n' '# vpc-local-neighbor-contract-v1\n') - if mutation != 'old_route_scope': - header += '# vpc-state-route-scope-contract-v3\n' + header += ('# vpc-state-route-scope-contract-v3\n' + if mutation == 'old_route_scope' + else '# vpc-state-route-scope-contract-v4\n') helper.write_text(header) helper.chmod(0o700) stat = executable(tmp_path / 'stat', 'import sys\n' diff --git a/engine/tests/test_vpc_apply_safe_log.py b/engine/tests/test_vpc_apply_safe_log.py new file mode 100644 index 00000000..42219c4c --- /dev/null +++ b/engine/tests/test_vpc_apply_safe_log.py @@ -0,0 +1,159 @@ +"""VPC apply diagnostics must stay useful without exposing remote output.""" +from __future__ import annotations + +import os +from pathlib import Path +import shlex +import subprocess +import tempfile +import unittest + +from virsh_panel.vpc import VPC_SCHEMA_VERSION +from virsh_panel.vpc_dataplane import ( + VPC_APPLY_SAFE_LOG_SH, + parse_vpc_dataplane_plan, + render_apply_script, +) + + +_SECRET = "NEVER_LOG_vpc_private_fixture_or_sensitive_path" + + +def _plan(): + return parse_vpc_dataplane_plan({ + "schema_version": VPC_SCHEMA_VERSION, + "vpc_id": "vpc-safe-log", + "region": "hk", + "fabric_id": "fabric-hk", + "fabric_revision": 3, + "cidr": "10.42.0.0/24", + "desired_revision": 1, + "operation_id": "vpc-op-safe-log-0001", + "subnets": [{"subnet_id": "subnet-a", "cidr": "10.42.0.0/25"}], + "hosts": [{ + "host_id": "host-a", + "members": [{ + "member_id": "member-a", + "instance_uuid": "11111111-1111-4111-8111-111111111111", + "private_ip_cidr": "10.42.0.10/32", + "subnet_id": "subnet-a", + }], + }], + "routes": [], + "acls": [], + "fabric_peers": [], + }, "host-a", fabric_plan_digest="ab" * 32) + + +class VpcApplySafeLogTest(unittest.TestCase): + def _run_logger(self, body: str): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + output = root / "journal" + logger = root / "logger" + logger.write_text( + "#!/bin/sh\nprintf '%s\\n' \"$*\" >> \"$VC_VPC_TEST_LOG\"\n", + encoding="utf-8", + ) + logger.chmod(0o700) + script = "\n".join(( + "set -eu", + VPC_APPLY_SAFE_LOG_SH, + f"LOGGER={shlex.quote(str(logger))}", + body, + )) + result = subprocess.run( + ["/bin/sh", "-c", script], + env={**os.environ, "VC_VPC_TEST_LOG": str(output)}, + capture_output=True, + text=True, + check=False, + ) + lines = output.read_text(encoding="utf-8").splitlines() \ + if output.exists() else [] + return result, lines + + def test_exit_trap_preserves_status_and_logs_only_fixed_fields(self): + result, lines = self._run_logger( + "VC_VPC_APPLY_PHASE=post_activation_sysctl\nexit 72") + + self.assertEqual(result.returncode, 72) + self.assertEqual(result.stdout, "") + self.assertEqual(result.stderr, "") + self.assertEqual(lines, [ + "-t virt-cloud-vpc -- apply_state=failed " + "apply_phase=post_activation_sysctl exit_status=72", + ]) + + def test_unknown_phase_and_status_are_sanitized(self): + result, lines = self._run_logger("\n".join(( + "VC_VPC_APPLY_PHASE=" + shlex.quote("bad\n" + _SECRET), + f"vc_vpc_apply_failure_log {shlex.quote(_SECRET)}", + ))) + + self.assertEqual(result.returncode, 0) + self.assertEqual(lines, [ + "-t virt-cloud-vpc -- apply_state=failed " + "apply_phase=unknown exit_status=1", + ]) + self.assertNotIn(_SECRET, "\n".join(lines)) + + def test_success_does_not_create_failure_record(self): + result, lines = self._run_logger("VC_VPC_APPLY_PHASE=commit\nexit 0") + + self.assertEqual(result.returncode, 0) + self.assertEqual(lines, []) + + def test_term_signal_uses_nonzero_status_and_safe_phase(self): + result, lines = self._run_logger( + 'VC_VPC_APPLY_PHASE=routes\nkill -TERM "$$"\nexit 99') + + self.assertEqual(result.returncode, 143) + self.assertEqual(result.stdout, "") + self.assertEqual(result.stderr, "") + self.assertEqual(lines, [ + "-t virt-cloud-vpc -- apply_state=failed " + "apply_phase=routes exit_status=143", + ]) + + def test_rendered_phases_bracket_activation_and_post_activation_work(self): + script = render_apply_script(_plan(), deployment_mode="networkmanager_nmcli") + rollback_start = script.index("rollback() {") + rollback_end = script.index("\ntrap rollback EXIT HUP INT TERM", rollback_start) + rollback = script[rollback_start:rollback_end] + disable_traps = rollback.index("trap - EXIT HUP INT TERM") + safe_log = rollback.index('vc_vpc_apply_failure_log "$status"') + rollback_mutations = rollback.index("set +e") + self.assertLess(disable_traps, safe_log) + self.assertLess(safe_log, rollback_mutations) + + persistence = script.index("VC_VPC_APPLY_PHASE=persistence") + activation = script.index("VC_VPC_APPLY_PHASE=activation", persistence) + nm_up = script.index( + '"$NMCLI" connection up uuid "$NM_PROFILE_UUID"', activation) + post_activation = script.index( + "VC_VPC_APPLY_PHASE=post_activation_sysctl", nm_up) + rp_filter = script.index("vc_vpc_rp_filter_set || exit 72", post_activation) + runtime = script.index("VC_VPC_APPLY_PHASE=runtime", rp_filter) + proxy = script.index("VC_VPC_APPLY_PHASE=proxy", runtime) + neighbors = script.index("VC_VPC_APPLY_PHASE=neighbors", proxy) + routes = script.index("VC_VPC_APPLY_PHASE=routes", neighbors) + policy = script.index("VC_VPC_APPLY_PHASE=policy_rules", routes) + fabric_acl = script.index("VC_VPC_APPLY_PHASE=fabric_acl", policy) + wireguard = script.index("VC_VPC_APPLY_PHASE=wireguard", fabric_acl) + final = script.index("VC_VPC_APPLY_PHASE=final_readback", wireguard) + commit = script.index("VC_VPC_APPLY_PHASE=commit", final) + + self.assertLess(activation, nm_up) + self.assertLess(nm_up, post_activation) + self.assertLess(post_activation, rp_filter) + self.assertEqual( + sorted((runtime, proxy, neighbors, routes, policy, fabric_acl, + wireguard, final, commit)), + [runtime, proxy, neighbors, routes, policy, fabric_acl, + wireguard, final, commit], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/engine/tests/test_vpc_dataplane.py b/engine/tests/test_vpc_dataplane.py index fb169067..346e1cd4 100644 --- a/engine/tests/test_vpc_dataplane.py +++ b/engine/tests/test_vpc_dataplane.py @@ -1013,11 +1013,38 @@ def test_observe_can_require_the_complete_authoritative_fabric_peer_set(): "host-b": ["10.42.0.11/32", "10.42.0.100/32"], }, fabric_peer_keys=[WG_KEY, WG_KEY_2], + fabric_peer_overlays={ + WG_KEY: "100.64.0.2/32", + WG_KEY_2: "100.64.0.3/32", + }, ) assert strict.fabric_peer_keys == frozenset({WG_KEY, WG_KEY_2}) + assert strict.fabric_peer_overlays == frozenset({ + (WG_KEY, "100.64.0.2/32"), + (WG_KEY_2, "100.64.0.3/32"), + }) observe = render_observe_script(strict) assert "__VC_VPC_WG_PEER_SET__" in observe assert "live_fabric_peer_keys" in observe + with pytest.raises(VpcDataplaneError) as overlay_error: + parse_vpc_dataplane_plan( + plan.vpc.canonical(), "host-a", + fabric_plan_digest=FABRIC_DIGEST, + fabric_peers=[{ + "host_id": "host-b", "public_key": WG_KEY, + "overlay_ip": "100.64.0.2/32", + "tenant_allowed_ips": ["10.42.0.11/32", "10.42.0.100/32"], + }], + trusted_tenant_allowed_ips={ + "host-b": ["10.42.0.11/32", "10.42.0.100/32"], + }, + fabric_peer_keys=[WG_KEY, WG_KEY_2], + fabric_peer_overlays={ + WG_KEY: "100.64.0.9/32", + WG_KEY_2: "100.64.0.3/32", + }, + ) + assert overlay_error.value.error_code == "vpc_fabric_peer_set_invalid" try: parse_vpc_dataplane_plan( plan.vpc.canonical(), "host-a", @@ -1360,6 +1387,99 @@ def _active_observation_markers(plan): return markers +def _observe_with_route_backed_wg_markers(plan, *, allowed, key=None, + wg_state="owned", + peer_set="exact"): + """Run adapter readback with an explicitly host-derived WG union.""" + member = plan.local_members[0] + routes = private_guest_routes(plan.vpc, member) + persistent = patch_private_interface_xml( + """ + + + """, + member=member, resources=plan.resources, + guest_network_generation=private_guest_network_generation( + member, routes), + vpc_id=plan.vpc.vpc_id, desired_revision=plan.desired_revision, + desired_digest=plan.desired_digest, + ) + peer = plan.peers[0] + markers = [] + for item in _active_observation_markers(plan): + if item.startswith("__VC_VPC_WG_ALLOWED__="): + item = (f"__VC_VPC_WG_ALLOWED__={peer.host_id}|" + f"{key or peer.public_key}|{allowed}") + elif item.startswith("__VC_VPC_WG_STATE__="): + item = f"__VC_VPC_WG_STATE__={peer.host_id}|{wg_state}" + markers.append(item) + markers.append(f"__VC_VPC_WG_PEER_SET__={peer_set}") + adapter = VpcPrivateHostAdapter(object(), {}) + adapter._deployment_contract = lambda: ("networkmanager", "", "", "") + adapter._run = lambda _script: "\n".join(markers) + "\n" + return adapter.observe(plan, xml_by_instance={ + member.instance_uuid: { + "inactive": persistent, + "live": persistent, + "seed_verified": True, + "guest_network": _guest_network_evidence(member, routes), + }, + }) + + +def test_adapter_accepts_host_state_union_without_catalog_only_tenant_ip(): + base = _plan() + route_backed_allowed = _allowed(base, base.peers[0]) + peer = replace( + base.peers[0], + tenant_allowed_ips=tuple(sorted( + set(base.peers[0].tenant_allowed_ips) | {'10.45.0.11/32'})), + ) + plan = replace( + base, peers=(peer,), + fabric_peer_keys=frozenset({peer.public_key}), + fabric_peer_overlays=frozenset({ + (peer.public_key, peer.overlay_cidr), + })) + + observed = _observe_with_route_backed_wg_markers( + plan, allowed=route_backed_allowed) + + assert observed["state"] == "active" + assert observed["config_verified"] is True + assert observed["wg_allowed_ips"] == [{ + "peer_host_id": peer.host_id, + "public_key": peer.public_key, + "tenant_allowed_ips": ["10.42.0.100/32", "10.42.0.11/32"], + }] + + +@pytest.mark.parametrize(('mutation', 'kwargs'), [ + ('wrong-key', {'key': WG_KEY_2}), + ('missing-overlay', { + 'allowed': '10.42.0.11/32,10.42.0.100/32', + }), + ('missing-vpc-route', { + 'allowed': '100.64.0.2/32,10.42.0.11/32', + }), + ('shell-union-mismatch', {'wg_state': 'missing'}), + ('fabric-peer-set-mismatch', {'peer_set': 'extra'}), +]) +def test_adapter_rejects_incomplete_or_unproven_route_backed_wg_union( + mutation, kwargs): + plan = replace( + _plan(), fabric_peer_keys=frozenset({WG_KEY}), + fabric_peer_overlays=frozenset({(WG_KEY, "100.64.0.2/32")})) + peer = plan.peers[0] + values = {'allowed': _allowed(plan, peer)} + values.update(kwargs) + + observed = _observe_with_route_backed_wg_markers(plan, **values) + + assert observed["config_verified"] is False, mutation + assert observed["state"] == "degraded", mutation + + @pytest.mark.parametrize("acl_proof", ("exact", "old_revision", "wrong_digest", "missing_nft")) def test_stopped_host_reports_acl_revision_only_after_exact_configuration_proof(acl_proof): plan = _plan() @@ -1414,7 +1534,9 @@ def test_stopped_host_reports_acl_revision_only_after_exact_configuration_proof( def test_stopped_linkdown_host_config_remains_reusable_without_becoming_active(): - plan = replace(_plan(), fabric_peer_keys=frozenset({WG_KEY})) + plan = replace( + _plan(), fabric_peer_keys=frozenset({WG_KEY}), + fabric_peer_overlays=frozenset({(WG_KEY, "100.64.0.2/32")})) member = plan.local_members[0] routes = private_guest_routes(plan.vpc, member) persistent = patch_private_interface_xml( @@ -1470,7 +1592,9 @@ def test_stopped_linkdown_host_config_remains_reusable_without_becoming_active() def test_topology_mismatch_revokes_reusable_host_config_proof(): - plan = replace(_plan(), fabric_peer_keys=frozenset({WG_KEY})) + plan = replace( + _plan(), fabric_peer_keys=frozenset({WG_KEY}), + fabric_peer_overlays=frozenset({(WG_KEY, "100.64.0.2/32")})) member = plan.local_members[0] routes = private_guest_routes(plan.vpc, member) persistent = patch_private_interface_xml( diff --git a/engine/tests/test_vpc_local_neighbor_transaction.py b/engine/tests/test_vpc_local_neighbor_transaction.py index 7a63dc0f..08495d83 100644 --- a/engine/tests/test_vpc_local_neighbor_transaction.py +++ b/engine/tests/test_vpc_local_neighbor_transaction.py @@ -208,8 +208,10 @@ def test_helper_upgrade_gate_runs_before_network_mutation(tmp_path, shell, case) header = ("#!/bin/sh\n# virt-cloud-wg-fabric-v1\n# fabric-plan-digest=" + digest + '\nPLAN_DIGEST="' + digest + '"\n# libvirt-startup-independent-v1\n') if case != "old": header += "# vpc-local-neighbor-contract-v1\n" - if case not in {"old", "old_route_scope"}: + if case == "old_route_scope": header += "# vpc-state-route-scope-contract-v3\n" + elif case != "old": + header += "# vpc-state-route-scope-contract-v4\n" if case == "wrong_digest": header = header.replace(digest, "cd" * 32) if case != "missing": helper.write_text(header) diff --git a/engine/tests/test_vpc_nft_dataplane.py b/engine/tests/test_vpc_nft_dataplane.py index 1656da90..2e23ae29 100644 --- a/engine/tests/test_vpc_nft_dataplane.py +++ b/engine/tests/test_vpc_nft_dataplane.py @@ -306,6 +306,7 @@ NFT=vc_test_nft guard_functions, loader_functions, 'VPC_TENANT_FILE="$EXPECTED"', + 'VPC_STATE_STATUS=owned', 'FABRIC_NFT_SNAPSHOT="$SNAPSHOT"', "vpc_state_load() { :; }", observe_function, diff --git a/engine/tests/test_vpc_observer_optional_results.py b/engine/tests/test_vpc_observer_optional_results.py index e89603b8..7921fe72 100644 --- a/engine/tests/test_vpc_observer_optional_results.py +++ b/engine/tests/test_vpc_observer_optional_results.py @@ -105,7 +105,7 @@ opposite_clean=yes; nft_file_ok=yes; sysctl_file_ok=yes; persistence=no ]) def test_observer_empty_proxy_set_reports_missing_without_aborting( tmp_path, shell, actual, expected): - body = _fragment("proxy_records=", 'if "$NFT" list table inet "$TABLE"') + body = _fragment("proxy_records=", "# vc-local-neighbor-observe-begin") proxy_file = tmp_path / "proxy" proxy_file.write_text("# virt-cloud-vpc-proxy-v1\n10.44.0.10\n10.44.0.11\n") result = _run(shell, ''' diff --git a/engine/tests/test_vpc_operation_identity.py b/engine/tests/test_vpc_operation_identity.py new file mode 100644 index 00000000..c238b177 --- /dev/null +++ b/engine/tests/test_vpc_operation_identity.py @@ -0,0 +1,1722 @@ +"""VPC claims bind exact immutable target identity before host admission.""" + +import asyncio +import json +import threading + +import pytest + +from virsh_panel.coordination import ( + CoordinationAdmissionRejected, CoordinationClaimRejected, CoordinationError, + DurableCoordinator, current_execution, execution_context, + mark_execution_unknown, +) +from virsh_panel.vpc import ( + VpcCoordinator, VpcError, parse_vpc_delete_plan, parse_vpc_plan, +) +from virsh_panel.vpc_dataplane import derive_private_mac +from test_durable_fabric_jobs import FabricStore +from test_engine_boundary import SUPER, cloud_engine_request +from test_engine_coordination import local_backends, request, running_engine +from test_vpc import ( + FakeFabric, confirm_delete_hosts, delete_payload, payload, +) +from test_vpc_admission_coordination import vpc_payload + + +OPERATION_ID = "a" * 32 + + +def identity(plan): + return {"job_kind": "vpc_operation", "vpc_id": plan.vpc_id, + "desired_revision": plan.desired_revision, + "desired_digest": plan.desired_digest} + + +async def explicit_request(engine, method, target, value=None): + actor = {**SUPER, "vpc_reconcile_explicit": True} + body = b"" if value is None else json.dumps(value).encode() + return await cloud_engine_request( + engine, method, target, body, actor, + headers={"Content-Type": "application/json"}, + ) + + +def retained_root(coordinator, plan, *, kind="background"): + claim = coordinator.claim( + operation_id=plan.operation_id, kind=kind, metadata=identity(plan)) + assert coordinator.retain(claim) + return coordinator.snapshot()[claim.token] + + +def coordination_document(store): + return json.loads(store.get("coordination", "admission-v1")) + + +def retiring_members(value): + plan = parse_vpc_plan(value) + return tuple({ + "member_id": member.member_id, + "instance_uuid": member.instance_uuid, + "host_id": member.host_id, + "private_ip_cidr": f"{member.private_ip}/32", + "private_mac": (member.private_mac + or derive_private_mac(plan.vpc_id, member.member_id)), + } for member in plan.members) + + +async def seed(engine, store, *, empty=False): + target = payload(hosts=[], members=[], operation_id=OPERATION_ID) if empty else { + **vpc_payload(), "operation_id": OPERATION_ID} + vpc = VpcCoordinator(store, FakeFabric(hosts=("host-a",)), + catalog_lock=engine._network_catalog_mutation_lock, + vm_lock=engine._target_lock, coordinator=engine.coordinator) + await vpc.apply(target) + assert engine.coordinator.snapshot() == {} + engine.vpc = vpc + return vpc, target + + +@pytest.mark.parametrize("old_kind", ["background", "request"]) +def test_explicit_http_apply_atomically_takes_over_exact_retained_root( + tmp_path, monkeypatch, old_kind): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + old = retained_root(engine.coordinator, plan, kind=old_kind) + seen = [] + + async def host(*_args, **_kwargs): + rows = list(engine.coordinator.snapshot().values()) + seen.extend(rows) + assert rows + assert all(row["operation_id"] == plan.operation_id + and row["metadata"] == identity(plan) + and "explicit_vpc_reconcile" not in row["metadata"] + for row in rows) + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + status, result = await explicit_request( + engine, "POST", "/internal/v1/vpcs/apply", target) + assert status == 200, result + assert seen + assert engine.coordinator.snapshot() == {} + document = coordination_document(store) + receipts = [item for item in document["resolutions"] + if item.get("kind") == "resolve_and_bind_v1"] + assert len(receipts) == 1 + receipt = receipts[0] + assert receipt["token"] == old["token"] + assert receipt["operation_id"] == plan.operation_id + assert receipt["replacement_token"] != old["token"] + assert set(receipt) == { + "kind", "token", "operation_id", "resolved_by", + "resolved_at", "evidence", "source_claim_sha256", + "replacement_token", "replacement_identity_sha256", + } + + asyncio.run(run()) + + +def test_explicit_http_apply_without_retained_root_consumes_marker( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + seen = [] + + async def host(*_args, **_kwargs): + seen.extend(engine.coordinator.snapshot().values()) + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + status, result = await explicit_request( + engine, "POST", "/internal/v1/vpcs/apply", target) + assert status == 200, result + assert seen + assert all(row["metadata"] == identity(plan) + and "explicit_vpc_reconcile" not in row["metadata"] + for row in seen) + assert engine.coordinator.snapshot() == {} + assert coordination_document(store)["resolutions"] == [] + + asyncio.run(run()) + + +def test_http_apply_without_explicit_marker_keeps_exact_retained_root( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + old = retained_root( + engine.coordinator, parse_vpc_plan(target)) + before = engine.coordinator.snapshot() + + async def forbidden(*_args, **_kwargs): + pytest.fail("unmarked retry reached host dispatch") + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", forbidden) + status, result = await request( + engine, "POST", "/internal/v1/vpcs/apply", target) + assert status == 503, result + assert result["error_code"] == "coordination_uncertain" + assert engine.coordinator.snapshot() == before + assert old["token"] in before + assert coordination_document(store)["resolutions"] == [] + + asyncio.run(run()) + + +def test_explicit_http_delete_takes_over_only_durable_delete_target( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, applied = await seed(engine, store) + target = delete_payload( + operation_id="b" * 32, + retiring_members=retiring_members(applied), + ) + plan = parse_vpc_delete_plan(target)[0] + release = engine.coordinator.release + failed_root_release = False + + async def confirmed(plan_value, record, *_args, **_kwargs): + confirm_delete_hosts(plan_value, record) + + def lose_background_release(claim): + nonlocal failed_root_release + kind = (claim.get("kind") if isinstance(claim, dict) + else claim.kind) + if kind == "background" and not failed_root_release: + failed_root_release = True + raise RuntimeError("lost background release response") + return release(claim) + + monkeypatch.setattr(vpc, "_reconcile_dataplane", confirmed) + monkeypatch.setattr( + engine.coordinator, "release", lose_background_release) + with pytest.raises(RuntimeError): + await vpc.delete("vpc-hk-1", target) + assert failed_root_release + rows = engine.coordinator.snapshot() + assert len(rows) == 1 + old = next(iter(rows.values())) + assert old["state"] == "uncertain" + assert old["operation_id"] == plan.operation_id + assert old["metadata"] == identity(plan) + assert store.get("vpc_desired", "vpc-hk-1") is None + + monkeypatch.setattr(engine.coordinator, "release", release) + monkeypatch.setattr( + vpc, "_reconcile_dataplane", + lambda *_args, **_kwargs: pytest.fail( + "confirmed delete receipt retried host dispatch")) + status, result = await explicit_request( + engine, "POST", "/internal/v1/vpcs/vpc-hk-1/delete", target) + assert status == 200, result + assert result["deleted"] is True + assert engine.coordinator.snapshot() == {} + receipts = coordination_document(store)["resolutions"] + assert any(item.get("kind") == "resolve_and_bind_v1" + and item.get("token") == old["token"] + for item in receipts) + + asyncio.run(run()) + + +def test_http_delete_without_explicit_marker_keeps_retained_delete_root( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, applied = await seed(engine, store) + target = delete_payload( + operation_id="c" * 32, + retiring_members=retiring_members(applied), + ) + plan = parse_vpc_delete_plan(target)[0] + release = engine.coordinator.release + failed_root_release = False + + async def confirmed(plan_value, record, *_args, **_kwargs): + confirm_delete_hosts(plan_value, record) + + def lose_background_release(claim): + nonlocal failed_root_release + kind = (claim.get("kind") if isinstance(claim, dict) + else claim.kind) + if kind == "background" and not failed_root_release: + failed_root_release = True + raise RuntimeError("lost background release response") + return release(claim) + + monkeypatch.setattr(vpc, "_reconcile_dataplane", confirmed) + monkeypatch.setattr( + engine.coordinator, "release", lose_background_release) + with pytest.raises(RuntimeError): + await vpc.delete("vpc-hk-1", target) + assert failed_root_release + monkeypatch.setattr(engine.coordinator, "release", release) + before = engine.coordinator.snapshot() + + async def forbidden(*_args, **_kwargs): + pytest.fail("unmarked delete retry reached host dispatch") + + monkeypatch.setattr(vpc, "_reconcile_dataplane", forbidden) + status, result = await request( + engine, "POST", "/internal/v1/vpcs/vpc-hk-1/delete", target) + assert status == 503, result + assert result["error_code"] == "coordination_uncertain" + assert engine.coordinator.snapshot() == before + assert coordination_document(store)["resolutions"] == [] + + asyncio.run(run()) + + +def test_explicit_delete_does_not_take_over_root_with_residual_child( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, applied = await seed(engine, store) + target = delete_payload( + operation_id="e" * 32, + retiring_members=retiring_members(applied), + ) + + async def interrupted(*_args, **_kwargs): + raise CoordinationError("coordination_unavailable", 503) + + monkeypatch.setattr(vpc, "_reconcile_dataplane", interrupted) + with pytest.raises(CoordinationError): + await vpc.delete("vpc-hk-1", target) + before = engine.coordinator.snapshot() + assert len(before) == 2 + execution_ids = {row["execution_id"] for row in before.values()} + assert len(execution_ids) == 1 + assert any(row["kind"] == "background" for row in before.values()) + assert any(row["kind"] == "resource" for row in before.values()) + + async def forbidden(*_args, **_kwargs): + pytest.fail("residual-child delete reached host dispatch") + + monkeypatch.setattr(vpc, "_reconcile_dataplane", forbidden) + status, result = await explicit_request( + engine, "POST", "/internal/v1/vpcs/vpc-hk-1/delete", target) + assert status == 503, result + assert result["error_code"] == "coordination_uncertain" + assert engine.coordinator.snapshot() == before + assert coordination_document(store)["resolutions"] == [] + + asyncio.run(run()) + + +def test_explicit_apply_revalidates_durable_target_before_handoff( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, stale = await seed(engine, store) + stale_plan = parse_vpc_plan(stale) + old = retained_root(engine.coordinator, stale_plan) + newer = { + **stale, + "desired_revision": stale_plan.desired_revision + 1, + "operation_id": "f" * 32, + } + await vpc.apply(newer) + before = engine.coordinator.snapshot() + assert set(before) == {old["token"]} + + async def forbidden(*_args, **_kwargs): + pytest.fail("stale retained target reached host dispatch") + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", forbidden) + status, result = await explicit_request( + engine, "POST", "/internal/v1/vpcs/apply", stale) + assert status == 503, result + assert result["error_code"] == "coordination_uncertain" + assert engine.coordinator.snapshot() == before + assert coordination_document(store)["resolutions"] == [] + + asyncio.run(run()) + + +@pytest.mark.parametrize("fault", ["before", "after", "false_after"]) +def test_explicit_handoff_cas_fault_is_read_back_fenced_and_retryable( + tmp_path, monkeypatch, fault): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + old = retained_root(engine.coordinator, plan) + compare = engine.coordinator.store.compare_and_set + triggered = False + host_calls = 0 + + def fail_handoff(kind, key, expected, value): + nonlocal triggered + desired = json.loads(value) + handoff = any( + item.get("kind") == "resolve_and_bind_v1" + and item.get("token") == old["token"] + for item in desired.get("resolutions", [])) + if handoff and not triggered: + triggered = True + if fault == "before": + raise RuntimeError("handoff did not commit") + assert compare(kind, key, expected, value) + if fault == "after": + raise RuntimeError("handoff response lost") + return False + return compare(kind, key, expected, value) + + async def host(*_args, **_kwargs): + nonlocal host_calls + host_calls += 1 + + monkeypatch.setattr( + engine.coordinator.store, "compare_and_set", fail_handoff) + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + status, result = await explicit_request( + engine, "POST", "/internal/v1/vpcs/apply", target) + assert status == 503, result + assert result["error_code"] in { + "coordination_uncertain", "coordination_unavailable"} + assert triggered and host_calls == 0 + rows = engine.coordinator.snapshot() + assert old["token"] not in rows + assert len(rows) == 1 + root = next(iter(rows.values())) + assert root["state"] == "uncertain" + assert root["kind"] == "request" + assert root["resources"] == [] + assert root["operation_id"] == plan.operation_id + assert root["metadata"] == identity(plan) + receipts = [item for item in coordination_document(store)["resolutions"] + if item.get("kind") == "resolve_and_bind_v1"] + assert len(receipts) == 1 + + monkeypatch.setattr( + engine.coordinator.store, "compare_and_set", compare) + status, result = await explicit_request( + engine, "POST", "/internal/v1/vpcs/apply", target) + assert status == 200, result + assert host_calls == 1 + assert engine.coordinator.snapshot() == {} + + asyncio.run(run()) + + +@pytest.mark.parametrize("fault", ["before", "after", "false_after"]) +def test_marker_only_bind_cas_fault_strips_capability_and_is_retryable( + tmp_path, monkeypatch, fault): + async def run(): + async with running_engine(tmp_path) as (engine, _store, _services): + vpc, target = await seed(engine, _store) + plan = parse_vpc_plan(target) + compare = engine.coordinator.store.compare_and_set + triggered = False + host_calls = 0 + + def fail_bind(kind, key, expected, value): + nonlocal triggered + before = json.loads(expected).get("claims", {}) + after = json.loads(value).get("claims", {}) + bound = any( + token in before + and before[token].get("operation_id") is None + and before[token].get("metadata") + == {"explicit_vpc_reconcile": True} + and row.get("operation_id") == plan.operation_id + and row.get("metadata") == identity(plan) + for token, row in after.items()) + if bound and not triggered: + triggered = True + if fault == "before": + raise RuntimeError("bind did not commit") + assert compare(kind, key, expected, value) + if fault == "after": + raise RuntimeError("bind response lost") + return False + return compare(kind, key, expected, value) + + async def host(*_args, **_kwargs): + nonlocal host_calls + host_calls += 1 + + monkeypatch.setattr( + engine.coordinator.store, "compare_and_set", fail_bind) + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + status, result = await explicit_request( + engine, "POST", "/internal/v1/vpcs/apply", target) + assert status == 503, result + assert triggered and host_calls == 0 + rows = engine.coordinator.snapshot() + assert len(rows) == 1 + root = next(iter(rows.values())) + assert root["state"] == "uncertain" + assert root["operation_id"] == plan.operation_id + assert root["metadata"] == identity(plan) + assert "explicit_vpc_reconcile" not in root["metadata"] + + monkeypatch.setattr( + engine.coordinator.store, "compare_and_set", compare) + status, result = await explicit_request( + engine, "POST", "/internal/v1/vpcs/apply", target) + assert status == 200, result + assert host_calls == 1 + assert engine.coordinator.snapshot() == {} + + asyncio.run(run()) + + +@pytest.mark.parametrize("phase", ["handoff", "bind"]) +def test_explicit_admission_cancellation_fences_exact_bound_root( + tmp_path, monkeypatch, phase): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + old = (retained_root(engine.coordinator, plan) + if phase == "handoff" else None) + method_name = ("resolve_and_bind" if phase == "handoff" + else "bind_explicit_operation") + original = getattr(engine.coordinator, method_name) + entered = threading.Event() + resume = threading.Event() + + def commit_then_block(*args, **kwargs): + result = original(*args, **kwargs) + entered.set() + assert resume.wait(5), "test did not release admission CAS" + return result + + monkeypatch.setattr(engine.coordinator, method_name, commit_then_block) + parent = engine.coordinator.claim( + kind="request", metadata={"explicit_vpc_reconcile": True}) + + async def apply_in_parent(): + with execution_context(engine.coordinator, parent): + return await vpc.apply(target) + + task = asyncio.create_task(apply_in_parent()) + try: + assert await asyncio.to_thread(entered.wait, 5) + task.cancel() + finally: + resume.set() + with pytest.raises(asyncio.CancelledError): + await task + + rows = engine.coordinator.snapshot() + if old is not None: + assert old["token"] not in rows + assert len(rows) == 1 + root = next(iter(rows.values())) + assert root["token"] == parent.token + assert root["state"] == "uncertain" + assert root["operation_id"] == plan.operation_id + assert root["metadata"] == identity(plan) + + asyncio.run(run()) + + +@pytest.mark.parametrize("child_execution", ["old", "replacement"]) +def test_atomic_handoff_rejects_any_child_of_either_execution( + tmp_path, child_execution): + store = FabricStore() + coordinator = DurableCoordinator(store, owner_id="d" * 64) + target = payload(operation_id=OPERATION_ID) + plan = parse_vpc_plan(target) + old_claim = coordinator.claim( + operation_id=plan.operation_id, kind="background", + metadata=identity(plan)) + parent = coordinator.claim( + kind="request", metadata={"explicit_vpc_reconcile": True}) + execution_id = (old_claim.execution_id if child_execution == "old" + else parent.execution_id) + child = coordinator.claim( + operation_id=(plan.operation_id if child_execution == "old" else None), + resources=(f"test:{child_execution}-execution-child",), + execution_id=execution_id, kind="resource", + metadata=(identity(plan) if child_execution == "old" + else {"explicit_vpc_reconcile": True})) + assert coordinator.retain(old_claim) + before = coordinator.snapshot() + old = before[old_claim.token] + + with pytest.raises(CoordinationClaimRejected): + coordinator.resolve_and_bind( + old, parent, "exact retained root evidence", + operation_id=plan.operation_id, metadata=identity(plan)) + + assert coordinator.snapshot() == before + assert child.token in before + assert coordination_document(store)["resolutions"] == [] + + +@pytest.mark.parametrize("invalid", ["kind", "resources", "metadata"]) +def test_atomic_handoff_rejects_noncanonical_retained_root(invalid): + store = FabricStore() + coordinator = DurableCoordinator(store, owner_id="d" * 64) + plan = parse_vpc_plan(payload(operation_id=OPERATION_ID)) + kwargs = { + "operation_id": plan.operation_id, + "kind": "background", + "metadata": identity(plan), + } + if invalid == "kind": + kwargs["kind"] = "resource" + elif invalid == "resources": + kwargs["resources"] = ("test:noncanonical-root",) + else: + kwargs["metadata"] = {**identity(plan), "desired_revision": 999} + old_claim = coordinator.claim(**kwargs) + assert coordinator.retain(old_claim) + parent = coordinator.claim( + kind="request", metadata={"explicit_vpc_reconcile": True}) + before = coordinator.snapshot() + + with pytest.raises(CoordinationClaimRejected): + coordinator.resolve_and_bind( + before[old_claim.token], parent, + "exact retained root evidence", + operation_id=plan.operation_id, metadata=identity(plan)) + + assert coordinator.snapshot() == before + assert coordination_document(store)["resolutions"] == [] + + +def test_handoff_readback_rejects_mismatched_related_receipt(): + store = FabricStore() + coordinator = DurableCoordinator(store, owner_id="d" * 64) + plan = parse_vpc_plan(payload(operation_id=OPERATION_ID)) + old_claim = coordinator.claim( + operation_id=plan.operation_id, kind="background", + metadata=identity(plan)) + assert coordinator.retain(old_claim) + old = coordinator.snapshot()[old_claim.token] + parent = coordinator.claim( + kind="request", metadata={"explicit_vpc_reconcile": True}) + evidence = "exact retained root evidence" + coordinator.resolve_and_bind( + old, parent, evidence, + operation_id=plan.operation_id, metadata=identity(plan)) + + document = coordination_document(store) + document["resolutions"][-1]["evidence"] = "different evidence" + store.set("coordination", "admission-v1", json.dumps(document)) + outcome, current = coordinator.classify_resolve_and_bind( + old, parent, evidence, + operation_id=plan.operation_id, metadata=identity(plan)) + assert outcome == "mixed" + assert current["token"] == parent.token + + +@pytest.mark.parametrize("entry", ["apply", "reconcile"]) +@pytest.mark.parametrize("unknown", [False, True]) +def test_http_vpc_claims_use_signed_target_and_never_generic_vm_receipt( + tmp_path, monkeypatch, entry, unknown): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + seen = [] + # An unrelated generic VM receipt must not settle VPC work just + # because it happens to share the opaque operation identifier. + store.set("operation", OPERATION_ID, json.dumps({"mutation_started": False})) + + async def host(*_args, **_kwargs): + rows = list(engine.coordinator.snapshot().values()) + seen.append(rows) + assert len(rows) == 4 # request, catalog, both member VM locks + assert {row["operation_id"] for row in rows} == {OPERATION_ID} + assert all(row["metadata"] == identity(plan) for row in rows) + assert current_execution().claim.operation_id == OPERATION_ID + if unknown: + raise CoordinationError("coordination_uncertain", 503) + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + status, result = await request(engine, "POST", + "/internal/v1/vpcs/apply" if entry == "apply" else "/internal/v1/vpcs/vpc-hk-1/reconcile", + target if entry == "apply" else {}, operation_id="actor-id-is-not-the-vpc-target") + assert seen + assert status == (503 if unknown else 200), result + rows = list(engine.coordinator.snapshot().values()) + if unknown: + assert len(rows) == 4 + assert all(row["state"] == "uncertain" and row["operation_id"] == OPERATION_ID + and row["metadata"] == identity(plan) for row in rows) + else: + assert rows == [] + asyncio.run(run()) + + +@pytest.mark.parametrize("unknown", [False, True]) +def test_periodic_vpc_claims_are_named_and_unknown_retry_keeps_original_fence( + tmp_path, monkeypatch, unknown): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + seen = [] + + async def host(*_args, **_kwargs): + rows = list(engine.coordinator.snapshot().values()) + seen.append(rows) + assert len(rows) == 4 + assert any(row["kind"] == "background" for row in rows) + assert all(row["operation_id"] == OPERATION_ID + and row["metadata"] == identity(plan) for row in rows) + if unknown: + raise CoordinationError("coordination_uncertain", 503) + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + if unknown: + with pytest.raises(CoordinationError): + await vpc.reconcile("vpc-hk-1") + before = engine.coordinator.snapshot() + assert len(before) == 4 + assert all(row["state"] == "uncertain" for row in before.values()) + with pytest.raises(CoordinationAdmissionRejected): + await vpc.reconcile("vpc-hk-1") + await vpc.reconcile_all() + assert engine.coordinator.snapshot() == before + assert len(seen) == 1 + else: + await vpc.reconcile("vpc-hk-1") + assert seen and engine.coordinator.snapshot() == {} + asyncio.run(run()) + + +@pytest.mark.parametrize("state", ["active", "uncertain"]) +@pytest.mark.parametrize("entry", ["http", "periodic"]) +def test_same_operation_admission_rejection_does_not_create_anonymous_uncertainty( + tmp_path, monkeypatch, state, entry): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + owner = DurableCoordinator(store) + blocker = owner.claim(operation_id=OPERATION_ID, kind="resource") + if state == "uncertain": owner.retain(blocker) + before = owner.snapshot() + + async def forbidden(*_args, **_kwargs): + pytest.fail("rejected operation reached host dispatch") + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", forbidden) + if entry == "http": + status, result = await request(engine, "POST", "/internal/v1/vpcs/apply", target) + assert status == (503 if state == "uncertain" else 409), result + else: + with pytest.raises(CoordinationAdmissionRejected): + await vpc.reconcile("vpc-hk-1") + assert owner.snapshot() == before + asyncio.run(run()) + + +def test_http_delete_claim_binds_the_delete_target_before_receipt_write(tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, _ = await seed(engine, store, empty=True) + target = delete_payload(hosts=(), operation_id="b" * 32) + observed = [] + original = vpc._save + + def save(kind, key, value): + if kind == "vpc_deleted": + rows = list(engine.coordinator.snapshot().values()) + observed.extend(rows) + assert len(rows) == 2 + assert all(row["operation_id"] == target["operation_id"] for row in rows) + assert all(row["metadata"]["vpc_id"] == "vpc-hk-1" + and row["metadata"]["desired_revision"] == 2 for row in rows) + return original(kind, key, value) + + monkeypatch.setattr(vpc, "_save", save) + status, result = await request(engine, "POST", "/internal/v1/vpcs/vpc-hk-1/delete", target) + assert status == 200 and result["deleted"] is True, result + assert observed and engine.coordinator.snapshot() == {} + asyncio.run(run()) + + +@pytest.mark.parametrize("entry", ["http", "periodic"]) +def test_legacy_anonymous_catalog_claim_is_preserved_without_new_uncertainty( + tmp_path, monkeypatch, entry): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + owner = DurableCoordinator(store) + old = owner.claim(resources=("catalog:network",), kind="resource") + owner.retain(old) + before = owner.snapshot() + + async def forbidden(*_args, **_kwargs): + pytest.fail("legacy unknown owner was bypassed") + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", forbidden) + if entry == "http": + status, result = await request(engine, "POST", "/internal/v1/vpcs/apply", target) + assert status == 503 and result["error_code"] == "coordination_uncertain", result + else: + with pytest.raises(CoordinationAdmissionRejected): + await vpc.reconcile("vpc-hk-1") + assert owner.snapshot() == before + asyncio.run(run()) + + +@pytest.mark.parametrize("fault", ["missing_id", "invalid_id", "digest", "revision"]) +def test_invalid_signed_vpc_envelope_never_binds_or_dispatches(tmp_path, monkeypatch, fault): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + target = dict(target) + if fault == "missing_id": target.pop("operation_id") + elif fault == "invalid_id": target["operation_id"] = "../unsafe" + elif fault == "digest": target["desired_digest"] = "0" * 64 + else: target["desired_revision"] = 0 + bound = [] + bind = engine.coordinator.bind_operation + + def track(*args, **kwargs): + bound.append(True) + return bind(*args, **kwargs) + + monkeypatch.setattr(engine.coordinator, "bind_operation", track) + status, _ = await request(engine, "POST", "/internal/v1/vpcs/apply", target) + assert status in {400, 409, 422} + assert not bound and engine.coordinator.snapshot() == {} + asyncio.run(run()) + + +def test_reconcile_identity_change_before_catalog_admission_is_rejected(tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + bind = engine.coordinator.bind_operation + + def supersede(*args, **kwargs): + result = bind(*args, **kwargs) + record = json.loads(store.get("vpc_desired", "vpc-hk-1")) + record["operation_id"] = "b" * 32 + store.set("vpc_desired", "vpc-hk-1", json.dumps(record)) + return result + + async def forbidden(*_args, **_kwargs): + pytest.fail("stale bound identity reached host dispatch") + + monkeypatch.setattr(engine.coordinator, "bind_operation", supersede) + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", forbidden) + status, result = await request(engine, "POST", "/internal/v1/vpcs/vpc-hk-1/reconcile", {}) + assert status == 409 and result["error_code"] == "vpc_revision_conflict", result + assert engine.coordinator.snapshot() == {} + asyncio.run(run()) + + +def test_bind_operation_is_immutable_and_keeps_ownership_and_extra_metadata(): + owner = DurableCoordinator(FabricStore()) + claim = owner.claim(kind="request", metadata={"audit": "preserved"}) + metadata = identity(parse_vpc_plan(payload(operation_id=OPERATION_ID))) + bound = owner.bind_operation(claim, OPERATION_ID, metadata) + assert (bound.token, bound.execution_id, bound.owner_id) == (claim.token, claim.execution_id, claim.owner_id) + assert bound.metadata == {"audit": "preserved", **metadata} + assert owner.bind_operation(bound, OPERATION_ID, metadata) == bound + before = owner.snapshot() + for operation, changed in [("b" * 32, metadata), (OPERATION_ID, {**metadata, "vpc_id": "another-vpc"})]: + with pytest.raises(CoordinationClaimRejected): + owner.bind_operation(bound, operation, changed) + assert owner.snapshot() == before + assert owner.release(claim) and owner.snapshot() == {} + + +def test_lost_bind_response_preserves_committed_operation_identity(): + class LostReply(FabricStore): + fail = False + def compare_and_set(self, kind, key, expected, value): + result = super().compare_and_set(kind, key, expected, value) + if result and self.fail: + self.fail = False + raise RuntimeError("committed bind reply lost") + return result + store = LostReply() + owner = DurableCoordinator(store) + claim = owner.claim(kind="request") + store.fail = True + with execution_context(owner, claim): + with pytest.raises(CoordinationError): + owner.bind_operation(claim, OPERATION_ID, {"job_kind": "vpc_operation"}) + assert current_execution().execution_unknown.is_set() + assert owner.snapshot()[claim.token]["operation_id"] == OPERATION_ID + + +@pytest.mark.parametrize("entry", ["http", "periodic"]) +def test_legacy_anonymous_empty_claim_is_not_borrowed_or_a_vpc_blocker( + tmp_path, entry): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, _target = await seed(engine, store) + legacy_owner = DurableCoordinator(store) + legacy = legacy_owner.claim(kind="request") + assert legacy_owner.retain(legacy) + before = legacy_owner.snapshot() + row = before[legacy.token] + assert row["state"] == "uncertain" + assert row["operation_id"] is None + assert row["resources"] == [] + assert row["metadata"] == {} + + if entry == "http": + status, result = await request( + engine, "POST", "/internal/v1/vpcs/vpc-hk-1/reconcile", {}) + assert status == 200, result + else: + await vpc.reconcile("vpc-hk-1") + + # The legacy row has no target identity or resource fence, so it + # cannot safely be borrowed by this operation. It also does not + # intersect the exact VPC operation/catalog claims admitted above. + assert legacy_owner.snapshot() == before + # It still blocks process-wide maintenance until an operator can + # supply explicit evidence to DurableCoordinator.resolve(). + with pytest.raises(CoordinationClaimRejected) as blocked: + legacy_owner.claim(maintenance=True, kind="update") + assert blocked.value.error_code == "coordination_uncertain" + + asyncio.run(run()) + + +def test_periodic_cancel_after_claim_commit_retains_exact_owner(tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + committed = threading.Event() + reply = threading.Event() + claim = engine.coordinator.claim + + def delayed_reply(*args, **kwargs): + result = claim(*args, **kwargs) + if kwargs.get("kind") == "background": + committed.set() + assert reply.wait(5), "test did not release committed claim response" + return result + + async def forbidden(*_args, **_kwargs): + pytest.fail("cancelled admission dispatched host work") + + monkeypatch.setattr(engine.coordinator, "claim", delayed_reply) + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", forbidden) + task = asyncio.create_task(vpc.reconcile("vpc-hk-1")) + try: + assert await asyncio.to_thread(committed.wait, 5) + task.cancel() + finally: + reply.set() + with pytest.raises(asyncio.CancelledError): + await task + before = engine.coordinator.snapshot() + assert len(before) == 1 + row = next(iter(before.values())) + assert row["state"] == "uncertain" and row["kind"] == "background" + assert row["operation_id"] == OPERATION_ID + assert row["metadata"] == identity(parse_vpc_plan(target)) + with pytest.raises(CoordinationAdmissionRejected): + await vpc.reconcile("vpc-hk-1") + assert engine.coordinator.snapshot() == before + + asyncio.run(run()) + + +def test_periodic_cancel_during_lost_claim_recovery_fences_exact_owner( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + claim = engine.coordinator.claim + snapshot = engine.coordinator.snapshot + entered = threading.Event() + resume = threading.Event() + blocked = False + + def commit_then_raise(*args, **kwargs): + result = claim(*args, **kwargs) + if kwargs.get("kind") == "background": + raise RuntimeError("committed background claim reply lost") + return result + + def block_recovery_snapshot(): + nonlocal blocked + rows = snapshot() + if (not blocked and len(rows) == 1 + and next(iter(rows.values())).get("kind") == "background"): + blocked = True + entered.set() + assert resume.wait(5), "test did not release recovery snapshot" + return rows + + async def forbidden(*_args, **_kwargs): + pytest.fail("unknown background admission dispatched host work") + + monkeypatch.setattr(engine.coordinator, "claim", commit_then_raise) + monkeypatch.setattr(engine.coordinator, "snapshot", block_recovery_snapshot) + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", forbidden) + task = asyncio.create_task(vpc.reconcile("vpc-hk-1")) + try: + assert await asyncio.to_thread(entered.wait, 5) + task.cancel() + finally: + resume.set() + with pytest.raises(asyncio.CancelledError): + await task + + rows = snapshot() + assert len(rows) == 1 + row = next(iter(rows.values())) + assert row["state"] == "uncertain" + assert row["operation_id"] == OPERATION_ID + assert row["metadata"] == identity(plan) + + asyncio.run(run()) + + +def test_periodic_lost_claim_response_recovers_exact_owner_as_uncertain( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + compare = engine.coordinator.store.compare_and_set + lose_background_reply = True + + def commit_then_disconnect(kind, key, expected, value): + nonlocal lose_background_reply + result = compare(kind, key, expected, value) + document = json.loads(value) + if (result and lose_background_reply + and any(row.get("kind") == "background" + for row in document.get("claims", {}).values())): + lose_background_reply = False + raise RuntimeError("committed background claim reply lost") + return result + + async def forbidden(*_args, **_kwargs): + pytest.fail("unknown background admission dispatched host work") + + monkeypatch.setattr( + engine.coordinator.store, "compare_and_set", commit_then_disconnect) + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", forbidden) + with pytest.raises(CoordinationError) as raised: + await vpc.reconcile("vpc-hk-1") + assert raised.value.error_code == "coordination_unavailable" + + before = engine.coordinator.snapshot() + assert len(before) == 1 + row = next(iter(before.values())) + assert row["state"] == "uncertain" + assert row["kind"] == "background" + assert row["operation_id"] == OPERATION_ID + assert row["metadata"] == identity(plan) + + with pytest.raises(CoordinationAdmissionRejected) as blocked: + await vpc.reconcile("vpc-hk-1") + assert blocked.value.error_code == "coordination_uncertain" + assert engine.coordinator.snapshot() == before + + asyncio.run(run()) + + +def test_periodic_runtime_after_claim_commit_recovers_exact_owner_as_uncertain( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + claim = engine.coordinator.claim + + def commit_then_raise(*args, **kwargs): + result = claim(*args, **kwargs) + if kwargs.get("kind") == "background": + raise RuntimeError("committed background claim reply lost") + return result + + async def forbidden(*_args, **_kwargs): + pytest.fail("unknown background admission dispatched host work") + + monkeypatch.setattr(engine.coordinator, "claim", commit_then_raise) + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", forbidden) + with pytest.raises(RuntimeError, match="claim reply lost"): + await vpc.reconcile("vpc-hk-1") + + rows = engine.coordinator.snapshot() + assert len(rows) == 1 + row = next(iter(rows.values())) + assert row["state"] == "uncertain" + assert row["kind"] == "background" + assert row["operation_id"] == OPERATION_ID + assert row["resources"] == [] + assert row["metadata"] == identity(plan) + assert row["maintenance"] is False + + asyncio.run(run()) + + +def test_periodic_recovered_claim_retain_retries_precommit_failure( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + claim = engine.coordinator.claim + retain = engine.coordinator.retain + retain_calls = 0 + + def commit_then_raise(*args, **kwargs): + result = claim(*args, **kwargs) + if kwargs.get("kind") == "background": + raise RuntimeError("committed background claim reply lost") + return result + + def fail_first_retain(value, *args, **kwargs): + nonlocal retain_calls + kind = (value.get("kind") if isinstance(value, dict) + else value.kind) + if kind == "background": + retain_calls += 1 + if retain_calls == 1: + raise CoordinationError("coordination_unavailable", 503) + return retain(value, *args, **kwargs) + + async def forbidden(*_args, **_kwargs): + pytest.fail("unknown background admission dispatched host work") + + monkeypatch.setattr(engine.coordinator, "claim", commit_then_raise) + monkeypatch.setattr(engine.coordinator, "retain", fail_first_retain) + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", forbidden) + with pytest.raises(RuntimeError, match="claim reply lost"): + await vpc.reconcile("vpc-hk-1") + + assert retain_calls == 2 + rows = engine.coordinator.snapshot() + assert len(rows) == 1 + row = next(iter(rows.values())) + assert row["state"] == "uncertain" + assert row["operation_id"] == OPERATION_ID + assert row["metadata"] == identity(plan) + + asyncio.run(run()) + + +def test_periodic_cancel_during_final_snapshot_retains_root_owner( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + body_completed = threading.Event() + snapshot_started = threading.Event() + release_snapshot = threading.Event() + snapshot = engine.coordinator.snapshot + + async def host(*_args, **_kwargs): + body_completed.set() + + def block_final_snapshot(): + rows = snapshot() + if (body_completed.is_set() and len(rows) == 1 + and next(iter(rows.values())).get("kind") == "background"): + snapshot_started.set() + assert release_snapshot.wait(5), "test did not release final snapshot" + return rows + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + monkeypatch.setattr(engine.coordinator, "snapshot", block_final_snapshot) + task = asyncio.create_task(vpc.reconcile("vpc-hk-1")) + try: + assert await asyncio.to_thread(snapshot_started.wait, 5) + task.cancel() + finally: + release_snapshot.set() + with pytest.raises(asyncio.CancelledError): + await task + + rows = snapshot() + assert len(rows) == 1 + row = next(iter(rows.values())) + assert row["state"] == "uncertain" + assert row["kind"] == "background" + assert row["operation_id"] == OPERATION_ID + assert row["metadata"] == identity(plan) + + asyncio.run(run()) + + +def test_periodic_cancel_after_committed_root_release_keeps_confirmed_settlement( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, _target = await seed(engine, store) + release = engine.coordinator.release + committed = threading.Event() + resume = threading.Event() + + async def host(*_args, **_kwargs): + return None + + def commit_then_block(target): + result = release(target) + kind = (target.get("kind") if isinstance(target, dict) + else target.kind) + if kind == "background" and result: + committed.set() + assert resume.wait(5), "test did not release committed root" + return result + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + monkeypatch.setattr(engine.coordinator, "release", commit_then_block) + task = asyncio.create_task(vpc.reconcile("vpc-hk-1")) + try: + assert await asyncio.to_thread(committed.wait, 5) + task.cancel() + finally: + resume.set() + with pytest.raises(asyncio.CancelledError): + await task + + # The exact release CAS committed before cancellation. Recreating + # the deleted owner here would turn a confirmed settlement into a + # false uncertainty fence. + assert engine.coordinator.snapshot() == {} + + asyncio.run(run()) + + +def test_periodic_failed_final_release_falls_back_to_uncertain_owner( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + compare = engine.coordinator.store.compare_and_set + fail_release = True + + def disconnect_before_release(kind, key, expected, value): + nonlocal fail_release + before = json.loads(expected) + after = json.loads(value) + background = [row for row in before.get("claims", {}).values() + if row.get("kind") == "background"] + if fail_release and len(background) == 1 and not after.get("claims"): + fail_release = False + raise RuntimeError("background release did not commit") + return compare(kind, key, expected, value) + + async def host(*_args, **_kwargs): + return None + + monkeypatch.setattr( + engine.coordinator.store, "compare_and_set", disconnect_before_release) + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + with pytest.raises(CoordinationError) as raised: + await vpc.reconcile("vpc-hk-1") + assert raised.value.error_code == "coordination_unavailable" + + rows = engine.coordinator.snapshot() + assert len(rows) == 1 + row = next(iter(rows.values())) + assert row["state"] == "uncertain" + assert row["kind"] == "background" + assert row["operation_id"] == OPERATION_ID + assert row["metadata"] == identity(plan) + + asyncio.run(run()) + + +def test_periodic_active_child_leak_retains_root_owner( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + leaked = None + + async def host(*_args, **_kwargs): + nonlocal leaked + context = current_execution() + leaked = engine.coordinator.claim( + operation_id=OPERATION_ID, + resources=("test:leaked-child",), + execution_id=context.claim.execution_id, + kind="resource", + metadata=identity(plan), + ) + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + with pytest.raises(CoordinationError) as raised: + await vpc.reconcile("vpc-hk-1") + assert raised.value.error_code == "coordination_uncertain" + + rows = engine.coordinator.snapshot() + assert len(rows) == 2 + assert rows[leaked.token]["state"] == "uncertain" + roots = [row for token, row in rows.items() if token != leaked.token] + assert len(roots) == 1 + assert roots[0]["state"] == "uncertain" + assert roots[0]["kind"] == "background" + assert roots[0]["operation_id"] == OPERATION_ID + assert roots[0]["metadata"] == identity(plan) + + asyncio.run(run()) + + +def test_periodic_deterministic_failure_with_child_is_upgraded_to_uncertain( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + leaked = None + + async def host(*_args, **_kwargs): + nonlocal leaked + context = current_execution() + leaked = engine.coordinator.claim( + operation_id=OPERATION_ID, + resources=("test:failed-child",), + execution_id=context.claim.execution_id, + kind="resource", + metadata=identity(plan), + ) + raise VpcError("deterministic failure", 409, + "vpc_revision_conflict") + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + with pytest.raises(CoordinationError) as raised: + await vpc.reconcile("vpc-hk-1") + assert raised.value.error_code == "coordination_uncertain" + rows = engine.coordinator.snapshot() + assert len(rows) == 2 + assert all(row["state"] == "uncertain" for row in rows.values()) + assert rows[leaked.token]["kind"] == "resource" + + asyncio.run(run()) + + +def test_periodic_child_retain_failure_retries_after_root_fence( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + leaked = None + child_retain_calls = 0 + retain = engine.coordinator.retain + + async def host(*_args, **_kwargs): + nonlocal leaked + context = current_execution() + leaked = engine.coordinator.claim( + operation_id=OPERATION_ID, + resources=("test:retry-child-retain",), + execution_id=context.claim.execution_id, + kind="resource", + metadata=identity(plan), + ) + + def fail_first_child_retain(target, *args, **kwargs): + nonlocal child_retain_calls + token = (target.token if hasattr(target, "token") + else target.get("token")) + if leaked is not None and token == leaked.token: + child_retain_calls += 1 + if child_retain_calls == 1: + raise CoordinationError( + "coordination_unavailable", 503) + return retain(target, *args, **kwargs) + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + monkeypatch.setattr( + engine.coordinator, "retain", fail_first_child_retain) + with pytest.raises(CoordinationError) as raised: + await vpc.reconcile("vpc-hk-1") + assert raised.value.error_code == "coordination_uncertain" + assert child_retain_calls == 2 + rows = engine.coordinator.snapshot() + assert len(rows) == 2 + assert all(row["state"] == "uncertain" for row in rows.values()) + + asyncio.run(run()) + + +def test_periodic_cancel_during_child_retain_fences_all_children_and_root( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + leaked = [] + blocked = False + entered = threading.Event() + resume = threading.Event() + retain = engine.coordinator.retain + + async def host(*_args, **_kwargs): + context = current_execution() + for suffix in ("one", "two"): + leaked.append(engine.coordinator.claim( + operation_id=OPERATION_ID, + resources=(f"test:cancel-child-retain-{suffix}",), + execution_id=context.claim.execution_id, + kind="resource", + metadata=identity(plan), + )) + + def commit_then_block(target, *args, **kwargs): + nonlocal blocked + token = (target.token if hasattr(target, "token") + else target.get("token")) + result = retain(target, *args, **kwargs) + if leaked and token == leaked[0].token and not blocked: + blocked = True + entered.set() + assert resume.wait(5), "test did not release child retain" + return result + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + monkeypatch.setattr(engine.coordinator, "retain", commit_then_block) + task = asyncio.create_task(vpc.reconcile("vpc-hk-1")) + try: + assert await asyncio.to_thread(entered.wait, 5) + task.cancel() + finally: + resume.set() + with pytest.raises(asyncio.CancelledError): + await task + + rows = engine.coordinator.snapshot() + assert len(rows) == 3 + assert all(rows[item.token]["state"] == "uncertain" + for item in leaked) + assert all(row["state"] == "uncertain" for row in rows.values()) + + asyncio.run(run()) + + +def test_periodic_unknown_execution_never_returns_success( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, _target = await seed(engine, store) + + async def host(*_args, **_kwargs): + mark_execution_unknown() + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + with pytest.raises(CoordinationError) as raised: + await vpc.reconcile("vpc-hk-1") + assert raised.value.error_code == "coordination_uncertain" + rows = engine.coordinator.snapshot() + assert len(rows) == 1 + assert next(iter(rows.values()))["state"] == "uncertain" + + asyncio.run(run()) + + +def test_http_active_child_leak_retains_root_owner( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + leaked = None + + async def host(*_args, **_kwargs): + nonlocal leaked + context = current_execution() + leaked = engine.coordinator.claim( + operation_id=OPERATION_ID, + resources=("test:http-leaked-child",), + execution_id=context.claim.execution_id, + kind="resource", + metadata=identity(plan), + ) + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + status, result = await request( + engine, "POST", "/internal/v1/vpcs/vpc-hk-1/reconcile", {}) + assert status == 503, result + assert result["error_code"] == "coordination_uncertain" + + rows = engine.coordinator.snapshot() + assert len(rows) == 2 + assert rows[leaked.token]["state"] == "uncertain" + roots = [row for token, row in rows.items() if token != leaked.token] + assert len(roots) == 1 + assert roots[0]["state"] == "uncertain" + assert roots[0]["kind"] == "request" + assert roots[0]["operation_id"] == OPERATION_ID + assert roots[0]["metadata"] == identity(plan) + + asyncio.run(run()) + + +def test_http_child_retain_failure_still_retains_root_owner( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + leaked = None + child_retain_calls = 0 + retain = engine.coordinator.retain + + async def host(*_args, **_kwargs): + nonlocal leaked + context = current_execution() + leaked = engine.coordinator.claim( + operation_id=OPERATION_ID, + resources=("test:http-child-retain-failure",), + execution_id=context.claim.execution_id, + kind="resource", + metadata=identity(plan), + ) + + def fail_child_retain(claim, *args, **kwargs): + nonlocal child_retain_calls + token = (claim.token if hasattr(claim, "token") else + claim.get("token")) + if leaked is not None and token == leaked.token: + child_retain_calls += 1 + if child_retain_calls == 1: + raise CoordinationError( + "coordination_unavailable", 503) + return retain(claim, *args, **kwargs) + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + monkeypatch.setattr(engine.coordinator, "retain", fail_child_retain) + status, result = await request( + engine, "POST", "/internal/v1/vpcs/vpc-hk-1/reconcile", {}) + assert status == 503, result + + rows = engine.coordinator.snapshot() + assert len(rows) == 2 + assert child_retain_calls == 2 + assert rows[leaked.token]["state"] == "uncertain" + roots = [row for token, row in rows.items() if token != leaked.token] + assert len(roots) == 1 + assert roots[0]["state"] == "uncertain" + assert roots[0]["operation_id"] == OPERATION_ID + assert roots[0]["metadata"] == identity(plan) + + asyncio.run(run()) + + +def test_http_unknown_execution_without_residual_returns_503( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, _target = await seed(engine, store) + + async def host(*_args, **_kwargs): + mark_execution_unknown() + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + status, result = await request( + engine, "POST", "/internal/v1/vpcs/vpc-hk-1/reconcile", {}) + assert status == 503, result + assert result["error_code"] == "coordination_uncertain" + rows = engine.coordinator.snapshot() + assert len(rows) == 1 + root = next(iter(rows.values())) + assert root["state"] == "uncertain" + assert root["operation_id"] == OPERATION_ID + + asyncio.run(run()) + + +def test_http_lost_child_release_reply_with_absent_token_returns_503( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, _target = await seed(engine, store) + release = engine.coordinator.release + lost_reply = False + + async def host(*_args, **_kwargs): + return None + + def commit_child_release_then_disconnect(claim): + nonlocal lost_reply + released = release(claim) + if claim.kind == "resource" and not lost_reply: + lost_reply = True + mark_execution_unknown() + raise CoordinationError( + "coordination_unavailable", 503) + return released + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + monkeypatch.setattr( + engine.coordinator, "release", + commit_child_release_then_disconnect) + status, result = await request( + engine, "POST", "/internal/v1/vpcs/vpc-hk-1/reconcile", {}) + assert status == 503, result + assert result["error_code"] == "coordination_uncertain" + assert lost_reply is True + rows = engine.coordinator.snapshot() + assert len(rows) == 1 + root = next(iter(rows.values())) + assert root["kind"] == "request" + assert root["state"] == "uncertain" + + asyncio.run(run()) + + +def test_http_exception_retains_residual_child_and_root( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + leaked = None + + async def host(*_args, **_kwargs): + nonlocal leaked + context = current_execution() + leaked = engine.coordinator.claim( + operation_id=OPERATION_ID, + resources=("test:http-exception-child",), + execution_id=context.claim.execution_id, + kind="resource", + metadata=identity(plan), + ) + raise VpcError("deterministic failure", 409, + "vpc_revision_conflict") + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + status, result = await request( + engine, "POST", "/internal/v1/vpcs/vpc-hk-1/reconcile", {}) + assert status == 503, result + assert result["error_code"] == "coordination_uncertain" + rows = engine.coordinator.snapshot() + assert len(rows) == 2 + assert rows[leaked.token]["state"] == "uncertain" + roots = [row for token, row in rows.items() + if token != leaked.token] + assert len(roots) == 1 + assert roots[0]["state"] == "uncertain" + + asyncio.run(run()) + + +def test_http_unhandled_exception_retains_residual_child_and_root( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, target = await seed(engine, store) + plan = parse_vpc_plan(target) + leaked = None + + async def host(*_args, **_kwargs): + nonlocal leaked + context = current_execution() + leaked = engine.coordinator.claim( + operation_id=OPERATION_ID, + resources=("test:http-unhandled-exception-child",), + execution_id=context.claim.execution_id, + kind="resource", + metadata=identity(plan), + ) + raise RuntimeError("unhandled handler failure") + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + status, result = await request( + engine, "POST", "/internal/v1/vpcs/vpc-hk-1/reconcile", {}) + assert status == 503, result + assert result["error_code"] == "coordination_uncertain" + rows = engine.coordinator.snapshot() + # The exception escapes before the nested catalog/member locks can + # finish, so every claim from this execution must be fenced. + assert len(rows) == 5 + assert rows[leaked.token]["state"] == "uncertain" + assert all(row["state"] == "uncertain" + and row["operation_id"] == OPERATION_ID + and row["metadata"] == identity(plan) + for row in rows.values()) + roots = [row for row in rows.values() + if row["kind"] == "request"] + assert len(roots) == 1 + + asyncio.run(run()) + + +def test_http_false_root_release_is_read_back_and_returns_503( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, _target = await seed(engine, store) + release = engine.coordinator.release + failed = False + + async def host(*_args, **_kwargs): + return None + + def false_root_release(claim): + nonlocal failed + kind = (claim.kind if hasattr(claim, "kind") else + claim.get("kind")) + if kind == "request" and not failed: + failed = True + return False + return release(claim) + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + monkeypatch.setattr( + engine.coordinator, "release", false_root_release) + status, result = await request( + engine, "POST", "/internal/v1/vpcs/vpc-hk-1/reconcile", {}) + assert status == 503, result + assert result["error_code"] == "coordination_uncertain" + assert failed is True + rows = engine.coordinator.snapshot() + assert len(rows) == 1 + root = next(iter(rows.values())) + assert root["state"] == "uncertain" + assert root["operation_id"] == OPERATION_ID + + asyncio.run(run()) + + +def test_http_lost_root_release_reply_reads_absence_and_returns_503( + tmp_path, monkeypatch): + async def run(): + async with running_engine(tmp_path) as (engine, store, _services): + vpc, _target = await seed(engine, store) + release = engine.coordinator.release + lost_reply = False + + async def host(*_args, **_kwargs): + return None + + def commit_root_release_then_disconnect(claim): + nonlocal lost_reply + released = release(claim) + kind = (claim.kind if hasattr(claim, "kind") else + claim.get("kind")) + if kind == "request" and not lost_reply: + lost_reply = True + raise CoordinationError( + "coordination_unavailable", 503) + return released + + monkeypatch.setattr(vpc, "_reconcile_dataplane_unlocked", host) + monkeypatch.setattr( + engine.coordinator, "release", + commit_root_release_then_disconnect) + status, result = await request( + engine, "POST", "/internal/v1/vpcs/vpc-hk-1/reconcile", {}) + assert status == 503, result + assert result["error_code"] == "coordination_uncertain" + assert lost_reply is True + assert engine.coordinator.snapshot() == {} + + asyncio.run(run()) diff --git a/engine/tests/test_vpc_state_overlay.py b/engine/tests/test_vpc_state_overlay.py index 81a20f62..1ab24422 100644 --- a/engine/tests/test_vpc_state_overlay.py +++ b/engine/tests/test_vpc_state_overlay.py @@ -13,6 +13,8 @@ import virsh_panel.vpc_dataplane as dataplane_module from virsh_panel.shell_contracts import WG_ALLOWED_IPS_SH +OTHER_WG_KEY = "AgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgI=" + def _loader_from_script(script, *, observe=False): start = script.index('VPC_STATE_DIR="/etc/virt-cloud/vpc-dataplane"') @@ -55,6 +57,19 @@ def _state_text(plan, *, cidr_after_routes=False): return text +def _legacy_aggregate_state_text(plan, *, cidr_after_routes=False): + """Render the pre-migration state shape with full Fabric peer closure.""" + lines = _state_text( + plan, cidr_after_routes=cidr_after_routes).splitlines() + lines = [line for line in lines if not line.startswith('peer|')] + for peer in plan.peers: + if peer.tenant_allowed_ips: + lines.append( + f"peer|{peer.host_id}|{peer.public_key}|" + + ','.join(peer.tenant_allowed_ips)) + return '\n'.join(lines) + '\n' + + def _with_global_peer_tenants(plan, *extra_ips): """Use the same trusted peer closure that the coordinator supplies.""" payload = plan.vpc.canonical() @@ -114,6 +129,68 @@ printf 'state=%s\n' "$VPC_STATE_STATUS" capture_output=True, check=False) +def _run_observe_wg_contract( + tmp_path, plan, texts, actual_by_key, *, peer_read_status=0): + """Run the rendered observer's exact state-union/AllowedIPs block.""" + state_dir = tmp_path / 'states' + state_dir.mkdir(mode=0o700) + for index, state_text in enumerate(texts, 1): + state_file = state_dir / f'{index:011x}.state' + state_file.write_text(state_text) + state_file.chmod(0o600) + wg_rows = tmp_path / 'wg-rows' + wg_rows.write_text(''.join( + f'{key}\t{allowed}\n' for key, allowed in actual_by_key.items())) + + script = dataplane_module.render_observe_script(plan) + block_start = script.index('vpc_state_loaded=no') + block_end = script.index('fabric_nft_state=no', block_start) + harness = '\n'.join(( + 'set -eu', + _loader_from_script(script, observe=True), + WG_ALLOWED_IPS_SH, + 'VPC_STATE_DIR=' + shlex.quote(str(state_dir)), + 'VPC_PEER_FILE=' + shlex.quote(str(tmp_path / 'peers')), + 'VPC_ROUTE_FILE=' + shlex.quote(str(tmp_path / 'routes')), + 'VPC_TENANT_FILE=' + shlex.quote(str(tmp_path / 'tenants')), + 'VPC_WG_ROWS=' + shlex.quote(str(wg_rows)), + 'VPC_WG_PEER_READ_STATUS=' + str(peer_read_status), + 'FIREWALL_MODE=native_nftables', + r''' +vc_fixture_stat() { + [ "$1" = -c ] || return 9 + case "$2" in + '%u:%a') + if [ "$3" = "$VPC_STATE_DIR" ]; then printf '0:700\n' + elif [ "${3%/*}" = "$VPC_STATE_DIR" ] && [ -f "$3" ]; then + printf '0:600\n' + else return 9; fi ;; + '%s') [ "${3%/*}" = "$VPC_STATE_DIR" ] && [ -f "$3" ] || return 9 + wc -c < "$3" | tr -d ' ' ;; + *) return 9 ;; + esac +} +vc_fixture_wg() { + [ "$1" = show ] && [ "$2" = wg-fabric0 ] && [ "$#" = 3 ] || return 9 + case "$3" in + allowed-ips) cat "$VPC_WG_ROWS" ;; + peers) + [ "$VPC_WG_PEER_READ_STATUS" = 0 ] || + return "$VPC_WG_PEER_READ_STATUS" + cut -f1 "$VPC_WG_ROWS" ;; + *) return 9 ;; + esac +} +STAT=vc_fixture_stat +WG=vc_fixture_wg +WG_IFACE=wg-fabric0 +'''.strip(), + script[block_start:block_end], + )) + return subprocess.run(['/bin/sh', '-c', harness], text=True, + capture_output=True, check=False) + + @pytest.mark.parametrize('mode', ['apply', 'observe']) @pytest.mark.parametrize('reverse', [False, True]) @pytest.mark.parametrize('cidr_after_routes', [False, True]) @@ -163,30 +240,32 @@ def test_global_peer_tenants_join_routes_once_in_the_fabric_address_universe( assert result.stdout == 'state=owned\n' route_ips = {row['destination_cidr'] for plan in plans for row in dataplane_module._routes(plan)} - expected = sorted(route_ips | {extra}) + expected = sorted(route_ips) actual = (tmp_path / 'tenants').read_text().splitlines() assert sorted(actual) == expected assert len(actual) == len(set(actual)) expected_peers = sorted( - peer.public_key + '|' + address - for plan in plans for peer in plan.peers - for address in peer.tenant_allowed_ips) + plan.peers[0].public_key + '|' + row['destination_cidr'] + for plan in plans for row in dataplane_module._routes(plan) + if row['device'] == 'wg-fabric0') assert sorted((tmp_path / 'peers').read_text().splitlines()) == expected_peers - # A global peer authorization contributes no route in this VPC's table. - # It must not weaken the independent per-record route/CIDR contract. + # A trusted aggregate entry contributes no durable route, peer, or nft + # authorization. Runtime convergence still checks it through _allowed(). routes = (tmp_path / 'routes').read_text().splitlines() assert {row.split('|', 1)[0] for row in routes} == route_ips assert extra not in route_ips + assert extra not in actual + assert all(not row.endswith('|' + extra) + for row in (tmp_path / 'peers').read_text().splitlines()) @pytest.mark.parametrize('mode', ['apply', 'observe']) @pytest.mark.parametrize('cidr_after_routes', [False, True]) -def test_peer_only_state_keeps_its_global_tenants_without_local_routes( +def test_peer_only_state_does_not_authorize_aggregate_without_owned_routes( tmp_path, mode, cidr_after_routes): carrier = _with_global_peer_tenants(_plan(), '10.46.0.21/32') - # The loader supports a retained peer-only record after local members and - # routes disappear. Render that record with the real state producer; - # extract the loader from an ordinary participating-host transaction. + # A retained record may outlive all local routes. Render it with the new + # producer and extract the loader from a participating-host transaction. peer_only = replace(carrier, members=()) text = _state_text(peer_only, cidr_after_routes=cidr_after_routes) assert '\nroute|' not in text and '\nlocal_neighbor|' not in text @@ -194,25 +273,264 @@ def test_peer_only_state_keeps_its_global_tenants_without_local_routes( assert result.returncode == 0, result.stderr assert result.stdout == 'state=owned\n' assert (tmp_path / 'routes').read_text() == '' - actual = (tmp_path / 'tenants').read_text().splitlines() - assert sorted(actual) == ['10.44.0.11/32', '10.46.0.21/32'] - assert len(actual) == len(set(actual)) + assert (tmp_path / 'peers').read_text() == '' + assert (tmp_path / 'tenants').read_text() == '' @pytest.mark.parametrize('mode', ['apply', 'observe']) @pytest.mark.parametrize('reverse', [False, True]) @pytest.mark.parametrize('cidr_after_routes', [False, True]) -def test_global_tenant_union_still_rejects_duplicate_peer_ownership( +def test_identical_global_peer_closure_is_deduplicated_across_vpc_states( tmp_path, mode, reverse, cidr_after_routes): first, second = _two_vpc_plans() - first = _with_global_peer_tenants(first, '10.46.0.21/32') - second = _with_global_peer_tenants(second, '10.46.0.21/32') - texts = [_state_text(plan, cidr_after_routes=cidr_after_routes) + aggregate = ( + '10.44.0.11/32', '10.45.0.11/32', '10.46.0.21/32', + ) + first = _with_global_peer_tenants(first, *aggregate) + second = _with_global_peer_tenants(second, *aggregate) + assert first.peers[0].tenant_allowed_ips == aggregate + assert second.peers[0].tenant_allowed_ips == aggregate + texts = [_legacy_aggregate_state_text( + plan, cidr_after_routes=cidr_after_routes) for plan in (first, second)] result = _run_multi_vpc_loader( tmp_path, first, texts[::-1] if reverse else texts, mode) - assert result.returncode == (1 if mode == 'observe' else 71), result.stderr - assert 'state=owned' not in result.stdout + assert result.returncode == 0, result.stderr + assert result.stdout == 'state=owned\n' + key = first.peers[0].public_key + peer_rows = (tmp_path / 'peers').read_text().splitlines() + route_backed = aggregate[:2] + assert sorted(peer_rows) == [key + '|' + address + for address in route_backed] + assert len(peer_rows) == len(set(peer_rows)) + route_ips = { + row['destination_cidr'] + for plan in (first, second) + for row in dataplane_module._routes(plan) + } + tenant_ips = (tmp_path / 'tenants').read_text().splitlines() + assert tenant_ips == sorted(route_ips) + assert len(tenant_ips) == len(set(tenant_ips)) + assert all(not row.endswith('|' + aggregate[2]) for row in peer_rows) + assert aggregate[2] not in tenant_ips + + +def test_new_state_persists_member_and_service_routes_not_fabric_aggregate(): + base = _plan() + payload = base.vpc.canonical() + payload['routes'] = [{ + 'route_id': 'service-b', + 'destination_cidr': '10.44.0.100/32', + 'next_hop_member_id': 'member-b', + 'metric': 20, + }] + tenants = ('10.44.0.100/32', '10.44.0.11/32', '10.46.0.21/32') + peer = base.peers[0] + payload['fabric_peers'] = [{ + 'host_id': peer.host_id, + 'public_key': peer.public_key, + 'overlay_ip': peer.overlay_cidr, + 'tenant_allowed_ips': list(tenants), + }] + plan = dataplane_module.parse_vpc_dataplane_plan( + payload, base.host_id, fabric_plan_digest=base.fabric_plan_digest, + trusted_tenant_allowed_ips={peer.host_id: tenants}) + state = _state_text(plan) + key = plan.peers[0].public_key + assert (f'peer|host-b|{key}|' + '10.44.0.100/32,10.44.0.11/32\n') in state + assert '10.46.0.21/32' not in state + assert set(dataplane_module._allowed(plan, plan.peers[0]).split(',')) == { + plan.peers[0].overlay_cidr, + '10.44.0.100/32', + '10.44.0.11/32', + '10.46.0.21/32', + } + + +@pytest.mark.parametrize('mode', ['apply', 'observe']) +@pytest.mark.parametrize('reverse', [False, True]) +@pytest.mark.parametrize('cidr_after_routes', [False, True]) +def test_legacy_and_new_state_mix_keeps_unique_route_backed_peer_rows( + tmp_path, mode, reverse, cidr_after_routes): + first, second = _two_vpc_plans() + aggregate = ( + '10.44.0.11/32', '10.45.0.11/32', '10.46.0.21/32', + ) + first = _with_global_peer_tenants(first, *aggregate) + second = _with_global_peer_tenants(second, *aggregate) + texts = [ + _legacy_aggregate_state_text( + first, cidr_after_routes=cidr_after_routes), + _state_text(second, cidr_after_routes=cidr_after_routes), + ] + result = _run_multi_vpc_loader( + tmp_path, first, texts[::-1] if reverse else texts, mode) + assert result.returncode == 0, result.stderr + key = first.peers[0].public_key + assert sorted((tmp_path / 'peers').read_text().splitlines()) == [ + key + '|10.44.0.11/32', key + '|10.45.0.11/32', + ] + assert (tmp_path / 'tenants').read_text().splitlines() == [ + '10.44.0.10/32', '10.44.0.11/32', + '10.45.0.10/32', '10.45.0.11/32', + ] + + +@pytest.mark.parametrize('mode', ['apply', 'observe']) +def test_legacy_aggregate_withdraws_deleted_vpc_route_ownership( + tmp_path, mode): + first, second = _two_vpc_plans() + aggregate = ('10.44.0.11/32', '10.45.0.11/32') + first = _with_global_peer_tenants(first, *aggregate) + second = _with_global_peer_tenants(second, *aggregate) + legacy = [_legacy_aggregate_state_text(plan) + for plan in (first, second)] + + before = tmp_path / 'before' + before.mkdir() + loaded = _run_multi_vpc_loader(before, first, legacy, mode) + assert loaded.returncode == 0, loaded.stderr + key = first.peers[0].public_key + assert sorted((before / 'peers').read_text().splitlines()) == [ + key + '|10.44.0.11/32', key + '|10.45.0.11/32', + ] + assert (before / 'tenants').read_text().splitlines() == [ + '10.44.0.10/32', '10.44.0.11/32', + '10.45.0.10/32', '10.45.0.11/32', + ] + + after = tmp_path / 'after' + after.mkdir() + removed = _run_multi_vpc_loader(after, first, [legacy[0]], mode) + assert removed.returncode == 0, removed.stderr + assert (after / 'peers').read_text() == key + '|10.44.0.11/32\n' + assert (after / 'tenants').read_text().splitlines() == [ + '10.44.0.10/32', '10.44.0.11/32', + ] + + +def test_observer_uses_only_local_route_backed_state_for_interleaved_topology( + tmp_path): + # Fabric A/B/C has VPC1 on A+B and VPC2 on B+C. The catalog aggregate for + # peer B includes both B addresses, but host A has only VPC1 state and must + # therefore require only overlay(B)+VPC1(B). + first, _second = _two_vpc_plans() + catalog_view = _with_global_peer_tenants(first, '10.45.0.11/32') + peer = catalog_view.peers[0] + result = _run_observe_wg_contract( + tmp_path, catalog_view, [_state_text(first)], { + peer.public_key: peer.overlay_cidr + ',10.44.0.11/32', + }) + assert result.returncode == 0, result.stderr + assert (f'__VC_VPC_WG_STATE__={peer.host_id}|owned\n' + in result.stdout) + allowed = next( + line for line in result.stdout.splitlines() + if line.startswith('__VC_VPC_WG_ALLOWED__=')) + assert '10.44.0.11/32' in allowed + assert '10.45.0.11/32' not in allowed + + +def test_observer_exact_union_withdraws_deleted_same_peer_state(tmp_path): + first, second = _two_vpc_plans() + aggregate = ('10.44.0.11/32', '10.45.0.11/32') + observed_plan = _with_global_peer_tenants(first, *aggregate) + peer = observed_plan.peers[0] + union = peer.overlay_cidr + ',' + ','.join(aggregate) + + before = tmp_path / 'before' + before.mkdir() + combined = _run_observe_wg_contract( + before, observed_plan, [_state_text(first), _state_text(second)], + {peer.public_key: union}) + assert combined.returncode == 0, combined.stderr + assert f'__VC_VPC_WG_STATE__={peer.host_id}|owned' in combined.stdout + + stale_dir = tmp_path / 'stale-after-delete' + stale_dir.mkdir() + stale = _run_observe_wg_contract( + stale_dir, observed_plan, [_state_text(first)], + {peer.public_key: union}) + assert stale.returncode == 0, stale.stderr + assert f'__VC_VPC_WG_STATE__={peer.host_id}|missing' in stale.stdout + + exact_dir = tmp_path / 'exact-after-delete' + exact_dir.mkdir() + exact = _run_observe_wg_contract( + exact_dir, observed_plan, [_state_text(first)], { + peer.public_key: peer.overlay_cidr + ',10.44.0.11/32', + }) + assert exact.returncode == 0, exact.stderr + assert f'__VC_VPC_WG_STATE__={peer.host_id}|owned' in exact.stdout + + +def test_observer_rejects_stale_tenant_on_authoritative_non_vpc_peer( + tmp_path): + first, second = _two_vpc_plans() + first_key = first.peers[0].public_key + second = replace(second, peers=(replace( + second.peers[0], public_key=OTHER_WG_KEY, + overlay_ip='100.64.0.3'),)) + observed = replace( + first, + fabric_peer_keys=frozenset({first_key, OTHER_WG_KEY}), + fabric_peer_overlays=frozenset({ + (first_key, '100.64.0.2/32'), + (OTHER_WG_KEY, '100.64.0.3/32'), + }), + ) + rows = { + first_key: '100.64.0.2/32,10.44.0.11/32', + OTHER_WG_KEY: '100.64.0.3/32,10.45.0.11/32', + } + + exact_dir = tmp_path / 'exact-closure' + exact_dir.mkdir() + exact = _run_observe_wg_contract( + exact_dir, observed, [_state_text(first), _state_text(second)], rows) + assert exact.returncode == 0, exact.stderr + assert '__VC_VPC_WG_PEER_SET__=exact\n' in exact.stdout + + wrong_overlay_dir = tmp_path / 'wrong-overlay-closure' + wrong_overlay_dir.mkdir() + wrong_overlay = _run_observe_wg_contract( + wrong_overlay_dir, observed, + [_state_text(first), _state_text(second)], { + **rows, + OTHER_WG_KEY: '100.64.0.4/32,10.45.0.11/32', + }) + assert wrong_overlay.returncode == 0, wrong_overlay.stderr + assert '__VC_VPC_WG_PEER_SET__=missing\n' in wrong_overlay.stdout + + # VPC2's final route/state was withdrawn while its host remains a valid + # Fabric peer. The stale AllowedIP must invalidate VPC1's host readback too; + # otherwise WireGuard can still route VPC2's deleted address to that key. + stale_dir = tmp_path / 'stale-closure' + stale_dir.mkdir() + stale = _run_observe_wg_contract( + stale_dir, observed, [_state_text(first)], rows) + assert stale.returncode == 0, stale.stderr + assert '__VC_VPC_WG_PEER_SET__=missing\n' in stale.stdout + assert '__VC_VPC_WG_PEER_SET__=exact\n' not in stale.stdout + + +def test_observer_empty_authoritative_peer_set_requires_successful_readback( + tmp_path): + base = _plan() + plan = replace( + base, + members=tuple(member for member in base.members + if member.host_id == base.host_id), + peers=(), + fabric_peer_keys=frozenset(), + fabric_peer_overlays=frozenset(), + ) + result = _run_observe_wg_contract( + tmp_path, plan, [_state_text(plan)], {}, peer_read_status=1) + assert result.returncode == 0, result.stderr + assert '__VC_VPC_WG_PEER_SET__=missing\n' in result.stdout + assert '__VC_VPC_WG_PEER_SET__=exact\n' not in result.stdout @pytest.mark.parametrize('mode', ['apply', 'observe']) @@ -264,7 +582,7 @@ elif args[:2]==['-c','-f'] or args[:1]==['-f']: assert len(values)==len(set(values)),values (root/('checked.batch' if args[0]=='-c' else 'applied.batch')).write_text(batch) if args[0]!='-c': - if MUTATION=='missing':values.remove('10.46.0.21/32') + if MUTATION=='missing':values.remove('10.44.0.11/32') elif MUTATION=='extra':values.append('10.47.0.99/32') (root/'live-set.json').write_text(json.dumps(sorted(values))) elif args==['list','set','inet','virt_cloud_wg_fabric','vpc_tenant_ips']: @@ -304,7 +622,7 @@ def test_global_tenant_refresh_and_fabric_observer_share_the_exact_universe( result = _run_multi_vpc_loader( tmp_path, carrier, [_state_text(carrier)], mode, extra_script=script) assert result.returncode == 0, result.stderr - wanted = ['10.44.0.10/32', '10.44.0.11/32', '10.46.0.21/32'] + wanted = ['10.44.0.10/32', '10.44.0.11/32'] assert (tmp_path / 'checked.batch').read_text() == ( tmp_path / 'applied.batch').read_text() matches = mutation == 'none' @@ -369,13 +687,14 @@ def test_multi_vpc_state_load_rejects_routes_outside_the_owning_record( @pytest.mark.parametrize('mode', ['apply', 'observe']) @pytest.mark.parametrize('reverse', [False, True]) -def test_multi_vpc_state_load_rejects_duplicate_peer_ips_across_records( +def test_multi_vpc_state_load_rejects_one_peer_ip_with_different_keys( tmp_path, mode, reverse): first, second = _two_vpc_plans() key = first.peers[0].public_key duplicate = _state_text(second).replace( 'peer|' + second.peers[0].host_id + '|' + key + '|10.45.0.11/32', - 'peer|' + second.peers[0].host_id + '|' + key + '|10.44.0.11/32') + 'peer|' + second.peers[0].host_id + '|' + OTHER_WG_KEY + + '|10.44.0.11/32,10.45.0.11/32') texts = [_state_text(first), duplicate] result = _run_multi_vpc_loader( tmp_path, first, texts[::-1] if reverse else texts, mode) @@ -383,6 +702,46 @@ def test_multi_vpc_state_load_rejects_duplicate_peer_ips_across_records( assert 'state=owned' not in result.stdout +@pytest.mark.parametrize('mode', ['apply', 'observe']) +def test_vpc_state_load_rejects_duplicate_peer_ip_within_one_state( + tmp_path, mode): + plan = _plan() + text = _state_text(plan) + text += next(line for line in text.splitlines() + if line.startswith('peer|')) + '\n' + result = _run_multi_vpc_loader(tmp_path, plan, [text], mode) + assert result.returncode == (1 if mode == 'observe' else 71), result.stderr + assert 'state=owned' not in result.stdout + + +@pytest.mark.parametrize('mode', ['apply', 'observe']) +def test_vpc_state_load_requires_one_key_for_every_wireguard_route( + tmp_path, mode): + plan = _plan() + text = '\n'.join(line for line in _state_text(plan).splitlines() + if not line.startswith('peer|')) + '\n' + result = _run_multi_vpc_loader(tmp_path, plan, [text], mode) + assert result.returncode == (1 if mode == 'observe' else 71), result.stderr + assert 'state=owned' not in result.stdout + + +@pytest.mark.parametrize('mode', ['apply', 'observe']) +@pytest.mark.parametrize('reverse', [False, True]) +def test_legacy_aggregate_cannot_mask_missing_peer_in_route_owner( + tmp_path, mode, reverse): + first, second = _two_vpc_plans() + broken_owner = '\n'.join( + line for line in _state_text(first).splitlines() + if not line.startswith('peer|')) + '\n' + legacy_carrier = _legacy_aggregate_state_text( + _with_global_peer_tenants(second, '10.44.0.11/32')) + texts = [broken_owner, legacy_carrier] + result = _run_multi_vpc_loader( + tmp_path, first, texts[::-1] if reverse else texts, mode) + assert result.returncode == (1 if mode == 'observe' else 71), result.stderr + assert 'state=owned' not in result.stdout + + @pytest.mark.parametrize('peer_mode', ['none', 'valid', 'duplicate_ip', 'public_ip']) def test_full_vpc_state_load_handles_peers_under_nounset(tmp_path, peer_mode): plan = _plan() @@ -432,14 +791,11 @@ printf 'state=%s\n' "$VPC_STATE_STATUS" )) result = subprocess.run(['/bin/sh', '-c', harness], text=True, capture_output=True, check=False) - if peer_mode in ('none', 'valid'): + if peer_mode == 'valid': assert result.returncode == 0, result.stderr assert result.stdout == 'state=owned\n' peers = (tmp_path / 'peers').read_text() - if peer_mode == 'none': - assert peers == '' - else: - assert peers == plan.peers[0].public_key + '|10.44.0.11/32\n' + assert peers == plan.peers[0].public_key + '|10.44.0.11/32\n' else: assert result.returncode != 0 assert 'state=owned' not in result.stdout @@ -471,17 +827,19 @@ def test_vpc_apply_and_remove_merge_actual_full_fabric_overlay( plan.peers[0], overlay_ip=expected_overlay.removesuffix('/32')),)) if transaction == 'apply': script = dataplane_module.render_apply_script(plan) - start = script.index('while IFS=\'|\' read -r peer_host key allowed; do') + start = script.index('expected_fabric_peer_keys=') end = script.index('# The shared Fabric quarantine table', start) else: script = dataplane_module.render_remove_script(replace( plan, members=tuple(m for m in plan.members if m.host_id != plan.host_id))) - start = script.index('for key in $("$WG" show "$WG_IFACE" peers') + start = script.index('expected_fabric_peer_keys=') end = script.index('vc_vpc_fabric_nft_live_ready || exit 74', start) merge = script[start:end] peer_file = tmp_path / 'peers' peer_file.write_text(plan.peers[0].public_key + '|10.44.0.11/32\n') + actual_file = tmp_path / 'actual' + actual_file.write_text(actual) mutations = tmp_path / 'mutations' mutations.write_text('') harness = '\n'.join(( @@ -489,7 +847,7 @@ def test_vpc_apply_and_remove_merge_actual_full_fabric_overlay( _loader_from_script(script), WG_ALLOWED_IPS_SH, 'VPC_PEER_FILE=' + shlex.quote(str(peer_file)), - 'VC_ACTUAL=' + shlex.quote(actual), + 'VC_ACTUAL_FILE=' + shlex.quote(str(actual_file)), 'VC_KEY=' + shlex.quote(plan.peers[0].public_key), 'VC_MUTATIONS=' + shlex.quote(str(mutations)), 'WG_IFACE=wg-fabric0', @@ -498,10 +856,11 @@ vc_fixture_wg() { if [ "$*" = 'show wg-fabric0 peers' ]; then printf '%s\n' "$VC_KEY" elif [ "$*" = 'show wg-fabric0 allowed-ips' ]; then - printf '%s\t%s\n' "$VC_KEY" "$VC_ACTUAL" + printf '%s\t%s\n' "$VC_KEY" "$(cat "$VC_ACTUAL_FILE")" elif [ "$1" = set ] && [ "$2" = wg-fabric0 ] && [ "$3" = peer ] && [ "$4" = "$VC_KEY" ] && [ "$5" = allowed-ips ]; then + printf '%s' "$6" > "$VC_ACTUAL_FILE" printf '%s\n' "$6" >> "$VC_MUTATIONS" else return 9; fi } @@ -519,6 +878,143 @@ WG=vc_fixture_wg assert mutations.read_text() == '' +def test_vpc_apply_converges_full_authoritative_fabric_peer_closure(tmp_path): + base = _plan() + first_key = base.peers[0].public_key + plan = replace( + base, + fabric_peer_keys=frozenset({first_key, OTHER_WG_KEY}), + fabric_peer_overlays=frozenset({ + (first_key, '100.64.0.2/32'), + (OTHER_WG_KEY, '100.64.0.3/32'), + }), + ) + script = dataplane_module.render_apply_script(plan) + block_start = script.index( + 'expected_fabric_peer_keys=', + script.index('VC_VPC_APPLY_PHASE=wireguard')) + block_end = script.index( + '# The shared Fabric quarantine table', block_start) + + peer_file = tmp_path / 'peers' + peer_file.write_text(first_key + '|10.44.0.11/32\n') + wg_rows = tmp_path / 'wg-rows' + wg_rows.write_text( + first_key + '\t100.64.0.2/32,10.44.0.11/32\n' + + OTHER_WG_KEY + '\t100.64.0.4/32,10.45.0.11/32\n') + mutations = tmp_path / 'mutations' + mutations.write_text('') + harness = '\n'.join(( + 'set -eu', + _loader_from_script(script), + WG_ALLOWED_IPS_SH, + 'VPC_PEER_FILE=' + shlex.quote(str(peer_file)), + 'VC_WG_ROWS=' + shlex.quote(str(wg_rows)), + 'VC_WG_MUTATIONS=' + shlex.quote(str(mutations)), + 'WG_IFACE=wg-fabric0', + r''' +vc_fixture_wg() { + if [ "$*" = 'show wg-fabric0 peers' ]; then + cut -f1 "$VC_WG_ROWS" + elif [ "$*" = 'show wg-fabric0 allowed-ips' ]; then + cat "$VC_WG_ROWS" + elif [ "$1" = set ] && [ "$2" = wg-fabric0 ] && + [ "$3" = peer ] && [ "$5" = allowed-ips ] && [ "$#" = 6 ]; then + awk -F '\t' -v OFS='\t' -v wanted="$4" -v allowed="$6" ' + $1 == wanted {$2=allowed; found++} {print} + END {exit found == 1 ? 0 : 1} + ' "$VC_WG_ROWS" > "$VC_WG_ROWS.next" || return 9 + mv -f "$VC_WG_ROWS.next" "$VC_WG_ROWS" || return 9 + printf '%s|%s\n' "$4" "$6" >> "$VC_WG_MUTATIONS" + else return 9; fi +} +WG=vc_fixture_wg +'''.strip(), + script[block_start:block_end], + )) + + result = subprocess.run(['/bin/sh', '-c', harness], text=True, + capture_output=True, check=False) + assert result.returncode == 0, result.stderr + assert wg_rows.read_text().splitlines() == [ + first_key + '\t100.64.0.2/32,10.44.0.11/32', + OTHER_WG_KEY + '\t100.64.0.3/32', + ] + assert mutations.read_text().splitlines() == [ + first_key + '|100.64.0.2/32,10.44.0.11/32', + OTHER_WG_KEY + '|100.64.0.3/32', + ] + + +def test_vpc_remove_converges_authoritative_closure_for_retiring_host( + tmp_path): + base = _plan() + first_key = base.peers[0].public_key + plan = replace( + base, + members=tuple(member for member in base.members + if member.host_id != base.host_id), + fabric_peer_keys=frozenset({first_key, OTHER_WG_KEY}), + fabric_peer_overlays=frozenset({ + (first_key, '100.64.0.2/32'), + (OTHER_WG_KEY, '100.64.0.3/32'), + }), + ) + script = dataplane_module.render_remove_script(plan) + block_start = script.index('expected_fabric_peer_keys=') + block_end = script.index( + 'vc_vpc_fabric_nft_live_ready || exit 74', block_start) + + peer_file = tmp_path / 'peers' + peer_file.write_text(OTHER_WG_KEY + '|10.45.0.11/32\n') + wg_rows = tmp_path / 'wg-rows' + wg_rows.write_text( + first_key + '\t100.64.0.8/32,10.44.0.11/32\n' + + OTHER_WG_KEY + '\t100.64.0.9/32,10.45.0.11/32\n') + mutations = tmp_path / 'mutations' + mutations.write_text('') + harness = '\n'.join(( + 'set -eu', + _loader_from_script(script), + WG_ALLOWED_IPS_SH, + 'VPC_PEER_FILE=' + shlex.quote(str(peer_file)), + 'VC_WG_ROWS=' + shlex.quote(str(wg_rows)), + 'VC_WG_MUTATIONS=' + shlex.quote(str(mutations)), + 'WG_IFACE=wg-fabric0', + r''' +vc_fixture_wg() { + if [ "$*" = 'show wg-fabric0 peers' ]; then + cut -f1 "$VC_WG_ROWS" + elif [ "$*" = 'show wg-fabric0 allowed-ips' ]; then + cat "$VC_WG_ROWS" + elif [ "$1" = set ] && [ "$2" = wg-fabric0 ] && + [ "$3" = peer ] && [ "$5" = allowed-ips ] && [ "$#" = 6 ]; then + awk -F '\t' -v OFS='\t' -v wanted="$4" -v allowed="$6" ' + $1 == wanted {$2=allowed; found++} {print} + END {exit found == 1 ? 0 : 1} + ' "$VC_WG_ROWS" > "$VC_WG_ROWS.next" || return 9 + mv -f "$VC_WG_ROWS.next" "$VC_WG_ROWS" || return 9 + printf '%s|%s\n' "$4" "$6" >> "$VC_WG_MUTATIONS" + else return 9; fi +} +WG=vc_fixture_wg +'''.strip(), + script[block_start:block_end], + )) + + result = subprocess.run(['/bin/sh', '-c', harness], text=True, + capture_output=True, check=False) + assert result.returncode == 0, result.stderr + assert wg_rows.read_text().splitlines() == [ + first_key + '\t100.64.0.2/32', + OTHER_WG_KEY + '\t100.64.0.3/32,10.45.0.11/32', + ] + assert mutations.read_text().splitlines() == [ + first_key + '|100.64.0.2/32', + OTHER_WG_KEY + '|100.64.0.3/32,10.45.0.11/32', + ] + + @pytest.mark.parametrize('malformed', [ '100.99.106.2/32\n100.64.0.2/32', '100.99.106.2/32 10.44.0.99/32', diff --git a/engine/tests/test_wg_allowed_readback.py b/engine/tests/test_wg_allowed_readback.py index e1b55eb0..8b0d484e 100644 --- a/engine/tests/test_wg_allowed_readback.py +++ b/engine/tests/test_wg_allowed_readback.py @@ -113,18 +113,20 @@ def test_generated_production_paths_read_complete_unordered_peer_rows( body = '' if operation == 'apply': script = dataplane.render_apply_script(plan) - start = script.index("while IFS='|' read -r peer_host key allowed; do") + start = script.index( + 'expected_fabric_peer_keys=', + script.index('VC_VPC_APPLY_PHASE=wireguard')) end = script.index('# The shared Fabric quarantine table', start) body = script[start:end] elif operation == 'remove': script = dataplane.render_remove_script(replace( plan, members=tuple(m for m in plan.members if m.host_id != plan.host_id))) - start = script.index('for key in $("$WG" show "$WG_IFACE" peers') + start = script.index('expected_fabric_peer_keys=') end = script.index('vc_vpc_fabric_nft_live_ready || exit 74', start) body = script[start:end] elif operation == 'vpc_observe': script = dataplane.render_observe_script(plan) - start = script.index("while IFS='|' read -r peer_host key expected_allowed; do") + start = script.index('vpc_state_loaded=no') end = script.index('expected_fabric_peer_keys=', start) body = script[start:end] else: @@ -136,6 +138,10 @@ def test_generated_production_paths_read_complete_unordered_peer_rows( if operation in ('apply', 'remove'): body = ('vpc_state_allowed_for_peer() { printf "10.44.0.11/32,10.44.0.99/32"; }\n' + body) + elif operation == 'vpc_observe': + body = ('vpc_state_load() { VPC_STATE_STATUS=owned; }\n' + 'vpc_state_allowed_for_peer() { printf "10.44.0.11/32,10.44.0.99/32"; }\n' + + body) result = run_shell(body, rows, mutation_file=mutations) assert result.returncode == 0, result.stderr if operation in ('apply', 'remove'): @@ -173,22 +179,25 @@ def test_generated_paths_never_accept_partial_or_failed_readback( mutations = tmp_path / 'mutations' if operation == 'apply': script = dataplane.render_apply_script(plan) - start = script.index("while IFS='|' read -r peer_host key allowed; do") + start = script.index( + 'expected_fabric_peer_keys=', + script.index('VC_VPC_APPLY_PHASE=wireguard')) end = script.index('# The shared Fabric quarantine table', start) elif operation == 'remove': script = dataplane.render_remove_script(replace( plan, members=tuple(m for m in plan.members if m.host_id != plan.host_id))) - start = script.index('for key in $("$WG" show "$WG_IFACE" peers') + start = script.index('expected_fabric_peer_keys=') end = script.index('vc_vpc_fabric_nft_live_ready || exit 74', start) elif operation == 'vpc_observe': script = dataplane.render_observe_script(plan) - start = script.index("while IFS='|' read -r peer_host key expected_allowed; do") + start = script.index('vpc_state_loaded=no') end = script.index('expected_fabric_peer_keys=', start) else: script = fabric._observe_script((), expected_digest=FABRIC_DIGEST) start = script.index(' $WG show "$IFACE" peers 2>/dev/null | while') end = script.index(' $IP -N -o route show', start) - body = ('vpc_state_allowed_for_peer() { printf "10.44.0.11/32"; }\n' + body = ('vpc_state_load() { VPC_STATE_STATUS=owned; }\n' + 'vpc_state_allowed_for_peer() { printf "10.44.0.11/32,10.44.0.99/32"; }\n' + script[start:end]) result = run_shell(body, rows, mutation_file=mutations, command_status=1 if scenario == 'command_failed' else 0) diff --git a/engine/virsh_panel/coordination.py b/engine/virsh_panel/coordination.py index 9f4b528a..708fbd9f 100644 --- a/engine/virsh_panel/coordination.py +++ b/engine/virsh_panel/coordination.py @@ -11,6 +11,7 @@ from __future__ import annotations from contextlib import contextmanager from contextvars import ContextVar from dataclasses import dataclass, field +import hashlib import json import re import secrets @@ -24,6 +25,8 @@ RECORD_KEY = "admission-v1" _IDENTIFIER = re.compile(r"[A-Za-z0-9_.:-]{1,191}\Z") _TOKEN = re.compile(r"[A-Za-z0-9_-]{16,128}\Z") _KIND = re.compile(r"[a-z][a-z0-9_-]{0,63}\Z") +_EXPLICIT_VPC_RECONCILE_METADATA = {"explicit_vpc_reconcile": True} +_RESOLVE_AND_BIND_RECEIPT_KIND = "resolve_and_bind_v1" _UNKNOWN_CODES = frozenset({ "libvirt_execution_uncertain", "libvirt_execution_blocked", "coordination_uncertain", "operation_uncertain", @@ -117,6 +120,18 @@ def _resources(values) -> tuple[str, ...]: return result +def _digest(value: Any) -> str: + return hashlib.sha256(_encode(value).encode("ascii")).hexdigest() + + +def _claim_envelope(record: Mapping[str, Any]) -> dict[str, Any]: + """Return the immutable admission identity used by handoff receipts.""" + return {key: record.get(key) for key in ( + "token", "execution_id", "owner_id", "operation_id", "resources", + "maintenance", "kind", "metadata", + )} + + class DurableCoordinator: def __init__(self, store, *, owner_id: str | None = None, cas_attempts: int = 32, max_claims: int = 1024): @@ -356,6 +371,119 @@ class DurableCoordinator: def snapshot(self) -> dict[str, dict]: return self._read()[1]["claims"] + def bind_operation(self, claim: Claim, operation_id: str, + metadata: Mapping[str, Any]) -> Claim: + """Bind a validated operation envelope before its first resource claim. + + Identity, ownership and execution evidence stay unchanged. An existing + binding is immutable, and another execution's same operation remains + fenced even when its resource set is empty. + """ + if not isinstance(operation_id, str) or not _IDENTIFIER.fullmatch(operation_id): + raise ValueError("Invalid coordination operation") + if not isinstance(metadata, Mapping): + raise ValueError("Invalid coordination metadata") + additions = json.loads(_encode(dict(metadata))) + + def mutate(document): + current = self._owned(document, claim) + if current is None: + raise CoordinationError("coordination_unavailable", 503) + if current["state"] != "active": + raise CoordinationClaimRejected("coordination_uncertain", 503) + if (current["operation_id"] not in {None, operation_id} + or any(key in current["metadata"] and current["metadata"][key] != value + for key, value in additions.items())): + raise CoordinationClaimRejected("coordination_busy", 409) + for existing in document["claims"].values(): + if existing["token"] == current["token"]: + continue + if existing["maintenance"] or ( + (existing["state"] == "uncertain" + or existing["execution_id"] != current["execution_id"]) + and existing["operation_id"] == operation_id + ): + unknown = existing["state"] == "uncertain" + raise CoordinationClaimRejected( + "coordination_uncertain" if unknown else "coordination_busy", + 503 if unknown else 409) + combined = {**current["metadata"], **additions} + if len(_encode(combined)) > 8192: + raise ValueError("Coordination metadata is too large") + changed = current["operation_id"] != operation_id or combined != current["metadata"] + current.update(operation_id=operation_id, metadata=combined, + updated_at=int(time.time())) + return changed, Claim(current["token"], current["execution_id"], current["owner_id"], + operation_id, tuple(current["resources"]), current["kind"], + json.loads(_encode(combined))) + + return self._change(mutate) + + def bind_explicit_operation(self, claim: Claim | Mapping[str, Any], + operation_id: str, + metadata: Mapping[str, Any]) -> Claim: + """Consume the one-shot VPC reconcile capability while binding. + + This stricter sibling of :meth:`bind_operation` is used only by an + explicitly authorized VPC request. The marker is replaced by the + signed target identity in the same CAS, so nested resource claims + cannot inherit a capability that was meant only for the HTTP root. + """ + replacement_identity = _identity(claim) + if not isinstance(operation_id, str) or not _IDENTIFIER.fullmatch(operation_id): + raise ValueError("Invalid coordination operation") + if not isinstance(metadata, Mapping): + raise ValueError("Invalid coordination metadata") + target_metadata = json.loads(_encode(dict(metadata))) + if len(_encode(target_metadata)) > 8192: + raise ValueError("Coordination metadata is too large") + + def claim_value(name, default=None): + if isinstance(claim, Mapping): + return claim.get(name, default) + return getattr(claim, name, default) + + if (replacement_identity[2] != self.owner_id + or claim_value("operation_id") is not None + or tuple(claim_value("resources", ())) != () + or claim_value("kind") != "request" + or claim_value("metadata", {}) != _EXPLICIT_VPC_RECONCILE_METADATA): + raise ValueError("Invalid explicit reconciliation root") + + def mutate(document): + current = self._owned(document, claim) + if (current is None + or current["state"] != "active" + or current["maintenance"] + or current["operation_id"] is not None + or current["resources"] != [] + or current["kind"] != "request" + or current["metadata"] != _EXPLICIT_VPC_RECONCILE_METADATA): + raise CoordinationClaimRejected("coordination_uncertain", 503) + conflicts = [ + existing for existing in document["claims"].values() + if existing["token"] != current["token"] + and (existing["execution_id"] == current["execution_id"] + or existing["operation_id"] == operation_id) + ] + if conflicts: + unknown = any(existing["state"] == "uncertain" + for existing in conflicts) + raise CoordinationClaimRejected( + "coordination_uncertain" if unknown else "coordination_busy", + 503 if unknown else 409) + current.update( + operation_id=operation_id, + metadata=json.loads(_encode(target_metadata)), + updated_at=int(time.time()), + ) + return True, Claim( + current["token"], current["execution_id"], current["owner_id"], + operation_id, (), "request", + json.loads(_encode(target_metadata))) + + return self._change(mutate) + def operation_claim(self, operation_id: str) -> dict | None: matches = [claim for claim in self.snapshot().values() if claim["operation_id"] == operation_id] @@ -448,6 +576,208 @@ class DurableCoordinator: return self._change(mutate) + def resolve_and_bind(self, claim: Mapping[str, Any], replacement: Claim | Mapping[str, Any], + evidence: str, *, operation_id: str, + metadata: Mapping[str, Any]) -> Claim: + """Atomically retire an exact uncertain owner into an active request. + + The replacement must already be an active request owned by this + coordinator. Its token is preserved, so an HTTP request can take over + an explicitly reconciled operation without a release/admission gap. + """ + if not isinstance(evidence, str) or not evidence.strip() or len(evidence) > 1024: + raise ValueError("Explicit reconciliation evidence is required") + _identity(claim) + replacement_identity = _identity(replacement) + if not isinstance(operation_id, str) or not _IDENTIFIER.fullmatch(operation_id): + raise ValueError("Invalid coordination operation") + if not isinstance(metadata, Mapping): + raise ValueError("Invalid coordination metadata") + target_metadata = json.loads(_encode(dict(metadata))) + if len(_encode(target_metadata)) > 8192: + raise ValueError("Coordination metadata is too large") + + def replacement_value(name, default=None): + if isinstance(replacement, Mapping): + return replacement.get(name, default) + return getattr(replacement, name, default) + + if (replacement_identity[2] != self.owner_id + or replacement_value("operation_id") is not None + or tuple(replacement_value("resources", ())) != () + or replacement_value("kind") != "request" + or replacement_value("metadata", {}) + != _EXPLICIT_VPC_RECONCILE_METADATA): + raise ValueError("Invalid explicit reconciliation root") + + def mutate(document): + original = self._owned(document, claim) + current = document["claims"].get(replacement_identity[0]) + if (original is None or original != dict(claim) + or original["state"] != "uncertain" + or original["operation_id"] != operation_id + or original["resources"] != [] + or original["maintenance"] + or original["kind"] not in {"background", "request"} + or original["metadata"] != target_metadata + or current is None + or (current["execution_id"], current["owner_id"]) + != replacement_identity[1:] + or current["owner_id"] != self.owner_id + or current["token"] == original["token"] + or current["state"] != "active" + or current["maintenance"] + or current["kind"] != "request" + or current["operation_id"] is not None + or current["resources"] != [] + or current["metadata"] + != _EXPLICIT_VPC_RECONCILE_METADATA): + raise CoordinationClaimRejected("coordination_uncertain", 503) + conflicts = [ + existing for existing in document["claims"].values() + if existing["token"] not in {original["token"], current["token"]} + and (existing["execution_id"] in { + original["execution_id"], current["execution_id"]} + or existing["operation_id"] == operation_id) + ] + if conflicts: + unknown = any(existing["state"] == "uncertain" + for existing in conflicts) + raise CoordinationClaimRejected( + "coordination_uncertain" if unknown else "coordination_busy", + 503 if unknown else 409) + now = int(time.time()) + current.update( + operation_id=operation_id, resources=[], + metadata=json.loads(_encode(target_metadata)), updated_at=now) + receipt = { + "kind": _RESOLVE_AND_BIND_RECEIPT_KIND, + "token": original["token"], + "operation_id": operation_id, + "resolved_by": self.owner_id, + "resolved_at": now, + "evidence": evidence.strip(), + "source_claim_sha256": _digest(original), + "replacement_token": current["token"], + "replacement_identity_sha256": _digest( + _claim_envelope(current)), + } + receipts = document.setdefault("resolutions", []) + receipts.append(receipt) + document["resolutions"] = receipts[-100:] + del document["claims"][original["token"]] + return True, Claim( + current["token"], current["execution_id"], current["owner_id"], + operation_id, (), "request", + json.loads(_encode(target_metadata))) + + return self._change(mutate) + + def classify_resolve_and_bind( + self, claim: Mapping[str, Any], replacement: Claim | Mapping[str, Any], + evidence: str, *, operation_id: str, + metadata: Mapping[str, Any]) -> tuple[str, dict[str, Any] | None]: + """Classify a lost handoff response from one exact durable snapshot. + + ``committed`` requires the source to be gone, the request to carry the + exact bound VPC identity, and one exact receipt written by the same CAS. + ``uncommitted`` requires both original rows to remain unchanged and no + matching receipt. Every mixed state is deliberately unclassifiable. + """ + if not isinstance(evidence, str) or not evidence.strip() or len(evidence) > 1024: + raise ValueError("Explicit reconciliation evidence is required") + _identity(claim) + replacement_identity = _identity(replacement) + if not isinstance(operation_id, str) or not _IDENTIFIER.fullmatch(operation_id): + raise ValueError("Invalid coordination operation") + if not isinstance(metadata, Mapping): + raise ValueError("Invalid coordination metadata") + target_metadata = json.loads(_encode(dict(metadata))) + if len(_encode(target_metadata)) > 8192: + raise ValueError("Coordination metadata is too large") + + def replacement_value(name, default=None): + if isinstance(replacement, Mapping): + return replacement.get(name, default) + return getattr(replacement, name, default) + + if (replacement_identity[2] != self.owner_id + or replacement_value("operation_id") is not None + or tuple(replacement_value("resources", ())) != () + or replacement_value("kind") != "request" + or replacement_value("metadata", {}) + != _EXPLICIT_VPC_RECONCILE_METADATA): + raise ValueError("Invalid explicit reconciliation root") + + _raw, document = self._read() + original = document["claims"].get(claim["token"]) + current = document["claims"].get(replacement_identity[0]) + other_conflict = any( + existing["token"] not in {claim["token"], replacement_identity[0]} + and (existing["execution_id"] in { + claim["execution_id"], replacement_identity[1]} + or existing["operation_id"] == operation_id) + for existing in document["claims"].values() + ) + anonymous = bool( + current is not None + and (current["execution_id"], current["owner_id"]) + == replacement_identity[1:] + and current["state"] == "active" + and current["maintenance"] is False + and current["operation_id"] is None + and current["resources"] == [] + and current["kind"] == "request" + and current["metadata"] == _EXPLICIT_VPC_RECONCILE_METADATA) + bound = bool( + current is not None + and (current["execution_id"], current["owner_id"]) + == replacement_identity[1:] + and current["state"] in {"active", "uncertain"} + and current["maintenance"] is False + and current["operation_id"] == operation_id + and current["resources"] == [] + and current["kind"] == "request" + and current["metadata"] == target_metadata) + expected_receipt = { + "kind": _RESOLVE_AND_BIND_RECEIPT_KIND, + "token": claim["token"], + "operation_id": operation_id, + "resolved_by": self.owner_id, + "evidence": evidence.strip(), + "source_claim_sha256": _digest(dict(claim)), + "replacement_token": replacement_identity[0], + "replacement_identity_sha256": ( + _digest(_claim_envelope(current)) if bound else None), + } + + def receipt_matches(receipt) -> bool: + return bool( + isinstance(receipt, dict) + and set(receipt) == {*expected_receipt, "resolved_at"} + and isinstance(receipt.get("resolved_at"), int) + and not isinstance(receipt.get("resolved_at"), bool) + and receipt["resolved_at"] > 0 + and all(receipt.get(key) == value + for key, value in expected_receipt.items())) + + related_receipts = [ + receipt for receipt in document.get("resolutions", []) + if (isinstance(receipt, dict) + and (receipt.get("token") == claim["token"] + or receipt.get("replacement_token") + == replacement_identity[0])) + ] + matching_receipts = [receipt for receipt in related_receipts + if receipt_matches(receipt)] + if (original is None and bound and not other_conflict + and len(matching_receipts) == len(related_receipts) == 1): + return "committed", current + if (original == dict(claim) and anonymous and not other_conflict + and not related_receipts): + return "uncommitted", current + return "mixed", current + def configure_default_coordinator(coordinator: DurableCoordinator | None) -> None: global _default_coordinator diff --git a/engine/virsh_panel/engine_security.py b/engine/virsh_panel/engine_security.py index b280feb2..373d46ef 100644 --- a/engine/virsh_panel/engine_security.py +++ b/engine/virsh_panel/engine_security.py @@ -66,7 +66,7 @@ def parse_actor(encoded: str) -> dict: raise EngineAuthError() if value["is_super_admin"] and not value["is_admin"]: raise EngineAuthError() - for key in ("can_provision", "system_cleanup"): + for key in ("can_provision", "system_cleanup", "vpc_reconcile_explicit"): if key in value and not isinstance(value[key], bool): raise EngineAuthError() if "operation_id" in value and not re.fullmatch(r"[A-Za-z0-9_-]{8,80}", str(value["operation_id"])): diff --git a/engine/virsh_panel/fabric.py b/engine/virsh_panel/fabric.py index 225bc626..e8732b81 100644 --- a/engine/virsh_panel/fabric.py +++ b/engine/virsh_panel/fabric.py @@ -395,7 +395,7 @@ vpc_state_load() { : > "$VPC_TENANT_FILE" || vpc_state_fail || return 1 chmod 600 "$VPC_TENANT_FILE" || vpc_state_fail || return 1 vpc_state_files=0 - vpc_state_seen_ips= + vpc_state_seen_peer_records= vpc_state_seen_route_ips= for vpc_state_file in "$VPC_STATE_DIR"/*.state; do [ -e "$vpc_state_file" ] || [ -L "$vpc_state_file" ] || continue @@ -425,6 +425,7 @@ vpc_state_load() { vpc_state_firewall_mode= vpc_state_table= vpc_state_bridge= + vpc_state_file_seen_ips= while IFS='|' read -r vpc_kind vpc_a vpc_b vpc_c vpc_d; do case "$vpc_kind" in owner) @@ -540,19 +541,27 @@ vpc_state_load() { vpc_state_allowed "$vpc_ip" || { IFS=$vpc_old_ifs; vpc_state_fail || return 1; } vpc_state_private "$vpc_ip" || { IFS=$vpc_old_ifs; vpc_state_fail || return 1; } vpc_state_in_fabric "$vpc_ip" && { IFS=$vpc_old_ifs; vpc_state_fail || return 1; } - case " $vpc_state_seen_ips " in + case " $vpc_state_file_seen_ips " in *" $vpc_ip "*) IFS=$vpc_old_ifs; vpc_state_fail || return 1 ;; esac - vpc_state_seen_ips="$vpc_state_seen_ips $vpc_ip" - printf '%s|%s\n' "$vpc_b" "$vpc_ip" >> "$VPC_PEER_FILE" || { - IFS=$vpc_old_ifs; vpc_state_fail || return 1; - } - # A trusted peer snapshot may include another VPC on - # this Fabric. The shared nft set must cover the same - # route/peer universe that the observer verifies. - printf '%s\n' "$vpc_ip" >> "$VPC_TENANT_FILE" || { - IFS=$vpc_old_ifs; vpc_state_fail || return 1; - } + vpc_state_file_seen_ips="$vpc_state_file_seen_ips $vpc_ip" + vpc_peer_record="$vpc_ip|$vpc_b" + case " $vpc_state_seen_peer_records " in + *" $vpc_peer_record "*) : ;; + *" $vpc_ip|"*) + IFS=$vpc_old_ifs + vpc_state_fail || return 1 ;; + *) + vpc_state_seen_peer_records="$vpc_state_seen_peer_records $vpc_peer_record" + printf '%s|%s\n' "$vpc_b" "$vpc_ip" >> "$VPC_PEER_FILE" || { + IFS=$vpc_old_ifs; vpc_state_fail || return 1; + } + # Keep validated legacy aggregate entries as + # candidates only. After every state is parsed, + # live wg-fabric0 routes select the exact peer + # contribution that remains authorized. + : ;; + esac done IFS=$vpc_old_ifs ;; '') : ;; @@ -590,11 +599,48 @@ vpc_state_load() { [ -z "$vpc_route_extra" ] || vpc_state_fail || return 1 vpc_state_cidr_contains "$vpc_route_prefix" "$vpc_state_cidr_value" || vpc_state_fail || return 1 + if [ "$vpc_route_dev" = wg-fabric0 ]; then + case " $vpc_state_file_seen_ips " in + *" $vpc_route_prefix "*) : ;; + *) vpc_state_fail || return 1 ;; + esac + fi done < "$vpc_state_file" done - # Route and peer records naturally overlap. Deduplicate only the derived - # tenant universe; the independent route/peer conflict checks above must - # still reject ambiguous source records before any nft refresh. + # Older state writers copied the complete Fabric peer aggregate into each + # VPC record. Validate those entries above, but authorize only IPs that + # still have one globally unique wg-fabric0 route after all records have + # passed their own CIDR checks. This preserves mixed-version reads while + # deleting one state immediately withdraws its stale aggregate entries. + vpc_peer_final="$VPC_PEER_FILE.route-backed" + rm -f "$vpc_peer_final" 2>/dev/null || vpc_state_fail || return 1 + : > "$vpc_peer_final" || vpc_state_fail || return 1 + chmod 600 "$vpc_peer_final" || { + rm -f "$vpc_peer_final"; vpc_state_fail || return 1; + } + if ! awk -F'|' ' + stage != "routes" { + if (NF != 2 || $1 == "" || $2 == "" || candidate[$2] != "") bad=1 + else candidate[$2]=$1 + next + } + { + if (NF != 5 || $1 == "" || $2 == "") {bad=1; next} + if ($2 == "wg-fabric0") { + if (!($1 in candidate)) bad=1 + else print candidate[$1] "|" $1 + } + } + END {exit bad ? 1 : 0} + ' "$VPC_PEER_FILE" stage=routes "$VPC_ROUTE_FILE" > "$vpc_peer_final"; then + rm -f "$vpc_peer_final" + vpc_state_fail || return 1 + fi + mv -f "$vpc_peer_final" "$VPC_PEER_FILE" || { + rm -f "$vpc_peer_final"; vpc_state_fail || return 1; + } + # The Fabric nft set covers both ends of routed traffic, including local + # bridge routes. Aggregate-only peer entries never enter this file. LC_ALL=C sort -u -o "$VPC_TENANT_FILE" "$VPC_TENANT_FILE" || vpc_state_fail || return 1 if [ "$vpc_state_files" -gt 0 ]; then VPC_STATE_STATUS=owned; fi return 0 @@ -2774,7 +2820,7 @@ _BOOT_HELPER = r'''#!/bin/sh PLAN_DIGEST="__VC_FABRIC_DIGEST__" # libvirt-startup-independent-v1 # vpc-local-neighbor-contract-v1 -# vpc-state-route-scope-contract-v3 +# vpc-state-route-scope-contract-v4 EXPECTED_DIGEST="$PLAN_DIGEST" EXPECTED_NFT_RULES_SHA256="__VC_FABRIC_NFT_RULES_SHA256__" DEPLOYMENT_MODE="__VC_FABRIC_MODE__" @@ -8430,7 +8476,7 @@ artifact_state() {{ if [ "$artifact_kind" = helper ]; then grep -Fqx -- '# vpc-local-neighbor-contract-v1' "$artifact_path" || {{ printf '%s' foreign; return; }} - grep -Fqx -- '# vpc-state-route-scope-contract-v3' "$artifact_path" || + grep -Fqx -- '# vpc-state-route-scope-contract-v4' "$artifact_path" || {{ printf '%s' foreign; return; }} fi if [ "$artifact_kind" = helper ] && diff --git a/engine/virsh_panel/internal.py b/engine/virsh_panel/internal.py index 2525f06b..ed29d7a9 100644 --- a/engine/virsh_panel/internal.py +++ b/engine/virsh_panel/internal.py @@ -48,10 +48,10 @@ from .engine_context import (EngineComponents, HostStore, LogStore, MainStore, TRUSTED_ACTOR, actor_session) from .engine_security import (EngineAuthError, authenticate, decode_secret, signed_headers) -from .fabric import FabricCoordinator, FabricError +from .fabric import FabricCoordinator, FabricError, _await_mutation_completion from .libvirt_rpc import EXECUTION_ERROR_CODES as LIBVIRT_EXECUTION_ERROR_CODES from .coordination import ( - DurableCoordinator, CoordinationError, CoordinationClaimRejected, + Claim, DurableCoordinator, CoordinationError, CoordinationClaimRejected, current_execution, execution_context, configure_default_coordinator, update_current_claim, _uncertain_exception, connection_failure_before_write, execution_failure_before_write, @@ -419,43 +419,181 @@ class _DurableAsyncLock(asyncio.Lock): self.coordinator = coordinator self.resources = resources + def _matches_expected_claim(self, record, *, token=None): + return (isinstance(record, dict) + and (token is None or record.get("token") == token) + and record.get("execution_id") == self._claim_execution_id + and record.get("owner_id") == self.coordinator.owner_id + and record.get("operation_id") == self._claim_operation_id + and tuple(record.get("resources", ())) == self._claim_resources + and record.get("kind") == "resource" + and record.get("metadata") == self._claim_metadata + and record.get("maintenance") is False + and record.get("state") in {"active", "uncertain"}) + + async def _coordination_call(self, function, *args, **kwargs): + return await _await_mutation_completion( + asyncio.to_thread(function, *args, **kwargs)) + + async def _recover_expected_claim(self, *, token=None): + rows = await self._coordination_call(self.coordinator.snapshot) + matches = [record for record in rows.values() + if self._matches_expected_claim(record, token=token)] + if len(matches) > 1: + raise CoordinationError("coordination_unavailable", 503) + if not matches: + return None, None + record = matches[0] + claim = Claim(record["token"], record["execution_id"], + record["owner_id"], record["operation_id"], + tuple(record["resources"]), record["kind"], + json.loads(json.dumps(record["metadata"]))) + return claim, record["state"] + + async def _retain_expected_claim(self): + token = self.claim.token if self.claim is not None else None + claim, state = await self._recover_expected_claim(token=token) + if claim is None: + return False + self.claim = claim + if state == "uncertain": + return True + try: + retained = await self._coordination_call( + self.coordinator.retain, claim) + except BaseException: + # Retain is idempotent. A lost CAS response is confirmed from the + # exact immutable child identity before deciding whether to retry. + recovered, recovered_state = await self._recover_expected_claim( + token=claim.token) + if recovered is None: + return False + self.claim = recovered + if recovered_state == "uncertain": + return True + retained = await self._coordination_call( + self.coordinator.retain, recovered) + if retained: + return True + recovered, recovered_state = await self._recover_expected_claim( + token=claim.token) + return recovered is not None and recovered_state == "uncertain" + + async def _release_or_retain_expected_claim(self): + try: + released = await self._coordination_call( + self.coordinator.release, self.claim) + except BaseException: + # Never replay an unconfirmed release. If it did not commit, turn + # the exact still-owned child into an uncertainty fence instead. + recovered, _state = await self._recover_expected_claim( + token=self.claim.token) + if recovered is None: + return + self.claim = recovered + if not await self._retain_expected_claim(): + raise CoordinationError("coordination_unavailable", 503) + raise + if released: + return + recovered, _state = await self._recover_expected_claim( + token=self.claim.token) + if recovered is None: + return + self.claim = recovered + if not await self._retain_expected_claim(): + raise CoordinationError("coordination_unavailable", 503) + raise CoordinationError("coordination_uncertain", 503) + + async def _claim_and_publish(self): + try: + self.claim = await self._coordination_call( + self.coordinator.claim, resources=self._claim_resources, + operation_id=self._claim_operation_id, + execution_id=self._claim_execution_id, + kind="resource", metadata=self._claim_metadata) + except CoordinationClaimRejected: + # A confirmed rejection proves this execution installed no child. + raise + except BaseException: + # A failed/lost claim response can follow a committed CAS. Recover + # only one exact child; a broader match could seize another task. + recovered, _state = await self._recover_expected_claim() + if recovered is not None: + self._claim_response_unknown = True + self.claim = recovered + if not await self._retain_expected_claim(): + raise CoordinationError("coordination_unavailable", 503) + raise + async def __aenter__(self): - await self.acquire() context = current_execution() + claim_operation_id = context.claim.operation_id if context else None + claim_execution_id = (context.claim.execution_id if context else + secrets.token_hex(32)) + while claim_execution_id == self.coordinator.owner_id: + claim_execution_id = secrets.token_hex(32) + claim_resources = tuple(sorted(set(self.resources))) + claim_metadata = json.loads(json.dumps( + context.claim.metadata if context else {}, ensure_ascii=True, + allow_nan=False, separators=(",", ":"), sort_keys=True)) + await self.acquire() try: - self.claim = await asyncio.to_thread( - self.coordinator.claim, resources=self.resources, - operation_id=context.claim.operation_id if context else None, - execution_id=context.claim.execution_id if context else None, - kind="resource", metadata=context.claim.metadata if context else None) + self.claim = None + self.context = None + self._claim_response_unknown = False + self._claim_operation_id = claim_operation_id + self._claim_execution_id = claim_execution_id + self._claim_resources = claim_resources + self._claim_metadata = claim_metadata + # Publish self.claim inside the shielded child so caller + # cancellation cannot lose a successfully returned admission. + await _await_mutation_completion(self._claim_and_publish()) + self.context = execution_context(self.coordinator, self.claim) + self.context.__enter__() except BaseException: - self.release() + try: + if self.claim is not None: + settlement = (self._retain_expected_claim() + if self._claim_response_unknown else + self._release_or_retain_expected_claim()) + settled = await _await_mutation_completion(settlement) + if self._claim_response_unknown and not settled: + raise CoordinationError("coordination_unavailable", 503) + finally: + self.release() raise - self.context = execution_context(self.coordinator, self.claim) - self.context.__enter__() return self + async def _settle(self, exc): + not_started = False + if (exc is not None and self.claim.operation_id + and self.claim.metadata.get("job_kind") != "vpc_operation"): + try: + raw = await self._coordination_call( + self.coordinator.store.get, "operation", + self.claim.operation_id) + record = json.loads(raw) if raw else {} + not_started = record.get("mutation_started") is False + except Exception: + pass + context = current_execution() + if context is not None and context.execution_unknown.is_set(): + not_started = False + if exc is not None and _uncertain_exception(exc) and not not_started: + if not await self._retain_expected_claim(): + raise CoordinationError("coordination_unavailable", 503) + return + await self._release_or_retain_expected_claim() + async def __aexit__(self, exc_type, exc, tb): try: - not_started = False - if exc is not None and self.claim.operation_id: - try: - raw = await asyncio.to_thread( - self.coordinator.store.get, "operation", self.claim.operation_id) - record = json.loads(raw) if raw else {} - not_started = record.get("mutation_started") is False - except Exception: - pass - if current_execution().execution_unknown.is_set(): - not_started = False - if exc is not None and _uncertain_exception(exc) and not not_started: - await asyncio.to_thread(self.coordinator.retain, self.claim) - else: - if not await asyncio.to_thread(self.coordinator.release, self.claim): - raise CoordinationError("coordination_uncertain", 503) + await _await_mutation_completion(self._settle(exc)) finally: - self.context.__exit__(exc_type, exc, tb) - self.release() + try: + self.context.__exit__(exc_type, exc, tb) + finally: + self.release() class EngineApplication: @@ -490,6 +628,7 @@ class EngineApplication: store, self.fabric, self.registry, vm_lock=lambda host_id, vm_id: self._target_lock(host_id, vm_id), catalog_lock=self._network_catalog_mutation_lock, + coordinator=self.coordinator, ) self.introspection = introspection project_dir = Path(config.get("project_dir", Path(__file__).resolve().parents[2])) @@ -874,6 +1013,9 @@ class EngineApplication: method == "POST" and path == "/internal/v1/hosts/scan") fabric_admission_request = bool(method == "POST" and re.fullmatch( r"/internal/v1/fabric/(?:apply|[A-Za-z0-9][A-Za-z0-9_.-]{0,63}/reconcile)", path)) + vpc_write_request = bool( + method != "GET" and (path == "/internal/v1/vpcs/apply" + or path.startswith("/internal/v1/vpcs/"))) async def execute(): await asyncio.to_thread(self._fence_legacy_operations) @@ -882,7 +1024,11 @@ class EngineApplication: claim = await asyncio.to_thread( self.coordinator.claim, operation_id=op_id if isinstance(op_id, str) and OP_ID.fullmatch(op_id) else None, - resources=() if cancel_queued_snapshot else resources, kind="request") + resources=() if cancel_queued_snapshot else resources, kind="request", + metadata={"explicit_vpc_reconcile": True} + if (vpc_write_request + and delegated.get("vpc_reconcile_explicit") is True) + else None) except CoordinationClaimRejected as exc: if (type(exc) is not CoordinationClaimRejected or not fabric_admission_request or exc.error_code not in {"coordination_busy", "coordination_uncertain"}): @@ -894,9 +1040,118 @@ class EngineApplication: "error": str(exc), "error_code": exc.error_code}) return messages dispatch_started = False + vpc_finalized = False messages = [] async def capture(message): messages.append(message) + + def same_claim_identity(expected, current): + fields = ( + "token", "execution_id", "owner_id", "operation_id", + "resources", "maintenance", "kind", "metadata", + "created_at", + ) + return (isinstance(current, dict) + and all(current.get(field) == expected.get(field) + for field in fields)) + + async def exact_current(expected): + rows = await asyncio.to_thread(self.coordinator.snapshot) + current = rows.get(expected["token"]) + if current is not None and not same_claim_identity( + expected, current): + raise CoordinationError("coordination_unavailable", 503) + return current + + async def retain_exact(expected, *, allow_absent): + """Idempotently retain one snapshot owner, with readback/retry.""" + current = expected + last_error = None + for _attempt in range(2): + if current["state"] == "uncertain": + return True + try: + if await asyncio.to_thread( + self.coordinator.retain, current): + return True + except BaseException as retain_error: + last_error = retain_error + current = await exact_current(expected) + if current is None: + if allow_absent: + return False + raise CoordinationError( + "coordination_unavailable", 503) from last_error + if current["state"] == "uncertain": + return True + raise CoordinationError( + "coordination_unavailable", 503) from last_error + + async def release_exact_root(expected): + """Release once, then prove absence or retain the exact root.""" + try: + if await asyncio.to_thread( + self.coordinator.release, expected): + return True + except BaseException: + pass + current = await exact_current(expected) + if current is None: + # The CAS may have committed, but a false/lost response is + # still an unconfirmed VPC final result and must become 503. + return False + await retain_exact(current, allow_absent=False) + return False + + async def finalize_vpc_execution(*, keep_root): + """Settle one VPC root and all children without a replay gap.""" + rows = await asyncio.to_thread(self.coordinator.snapshot) + root = rows.get(claim.token) + if root is not None and root["execution_id"] != claim.execution_id: + raise CoordinationError("coordination_unavailable", 503) + execution_claims = { + token: row for token, row in rows.items() + if row["execution_id"] == claim.execution_id} + children = { + token: row for token, row in execution_claims.items() + if token != claim.token} + execution_unknown = current_execution().execution_unknown.is_set() + residual = bool(children) or any( + row["state"] == "uncertain" + for row in execution_claims.values()) + forced_uncertain = execution_unknown or residual + if not (keep_root or forced_uncertain): + if root is None: + return False, True + released = await release_exact_root(root) + return released, not released + + # Fence the root first. A child retain failure can then be + # retried without ever making this execution look complete. + root_error = None + if root is None: + root_error = CoordinationError( + "coordination_unavailable", 503) + else: + try: + await retain_exact(root, allow_absent=False) + except BaseException as exc: + root_error = exc + + child_error = None + for row in children.values(): + if row["state"] != "active": + continue + try: + await retain_exact(row, allow_absent=True) + except BaseException as exc: + child_error = child_error or exc + if root_error is not None: + raise root_error + if child_error is not None: + raise child_error + return False, forced_uncertain + with execution_context(self.coordinator, claim, connection_failure_safe=libvirt_only, tracked_mutations=bool(operation_request)): @@ -938,7 +1193,8 @@ class EngineApplication: if item["type"] == "http.response.start"), 503) raw = await asyncio.to_thread(self.store.get, "operation", op_id) if op_id else None record = json.loads(raw) if raw else {} - current = (await asyncio.to_thread(self.coordinator.snapshot)).get(claim.token, {}) + claims = await asyncio.to_thread(self.coordinator.snapshot) + current = claims.get(claim.token, {}) response_error_code = None if status >= 500: with suppress(TypeError, ValueError, AttributeError): @@ -955,8 +1211,20 @@ class EngineApplication: and bool(record.get("cleanup_pending"))) if current_execution().execution_unknown.is_set(): safe_failure = False - if (current_execution().execution_unknown.is_set() - or (status >= 500 or record.get("status") == "uncertain") and not safe_failure): + keep_root = ( + current_execution().execution_unknown.is_set() + or (status >= 500 + or record.get("status") == "uncertain") + and not safe_failure) + if vpc_write_request: + released, forced_uncertain = ( + await _await_mutation_completion( + finalize_vpc_execution(keep_root=keep_root))) + vpc_finalized = True + if not released and (status < 500 or forced_uncertain): + raise CoordinationError( + "coordination_uncertain", 503) + elif keep_root: await asyncio.to_thread(self.coordinator.retain, claim) else: await asyncio.to_thread(self.coordinator.release, claim) @@ -999,7 +1267,28 @@ class EngineApplication: mark_execution_unknown() if current_execution().execution_unknown.is_set(): not_started = False - if _uncertain_exception(exc) and not not_started and dispatch_started: + keep_root = (_uncertain_exception(exc) + and not not_started and dispatch_started) + if (vpc_write_request and dispatch_started + and not vpc_finalized): + settlement_error = None + released = False + forced_uncertain = False + try: + released, forced_uncertain = ( + await _await_mutation_completion( + finalize_vpc_execution( + keep_root=keep_root))) + except BaseException as final_error: + settlement_error = final_error + if isinstance(exc, asyncio.CancelledError): + raise + if settlement_error is not None: + raise settlement_error from exc + if not released and (keep_root or forced_uncertain): + raise CoordinationError( + "coordination_uncertain", 503) from exc + elif keep_root: await asyncio.to_thread(self.coordinator.retain, claim) else: await asyncio.to_thread(self.coordinator.release, claim) diff --git a/engine/virsh_panel/vpc.py b/engine/virsh_panel/vpc.py index c2bcdf44..d1b39da2 100644 --- a/engine/virsh_panel/vpc.py +++ b/engine/virsh_panel/vpc.py @@ -17,12 +17,16 @@ import hashlib import ipaddress import json import re +import secrets import threading import time from dataclasses import dataclass, replace from typing import Any, Collection, Mapping, Sequence -from .coordination import CoordinationAdmissionRejected, CoordinationError +from .coordination import ( + CoordinationAdmissionRejected, CoordinationClaimRejected, CoordinationError, + current_execution, execution_context, _uncertain_exception, +) from .fabric import ( FabricCoordinator, FabricError, @@ -1015,10 +1019,12 @@ class VpcCoordinator: def __init__(self, store: Any, fabric: Any, registry: Any | None = None, vm_lock: Any | None = None, - catalog_lock: Any | None = None): + catalog_lock: Any | None = None, + coordinator: Any | None = None): self.store = store self.fabric = fabric self._catalog_lock_provider = catalog_lock + self._coordinator = coordinator # The registry is optional for pure control-plane/unit callers. The # production EngineApplication supplies it so a VPC revision can only # become active after host bridge/WireGuard/nft and libvirt readback. @@ -1037,6 +1043,540 @@ class VpcCoordinator: self._fallback_catalog_lock: Any | None = None self._reconcile_wakeup: Any | None = None + @staticmethod + def _operation_identity(plan: VpcPlan) -> dict[str, Any]: + return {"job_kind": "vpc_operation", "vpc_id": plan.vpc_id, + "desired_revision": plan.desired_revision, + "desired_digest": plan.desired_digest} + + def _retained_operation_target_matches(self, plan: VpcPlan) -> bool: + """Revalidate a retained root against the durable target under its lock.""" + receipt = self._load_delete_receipt(plan.vpc_id) + if receipt is not None: + return bool( + receipt["operation_id"] == plan.operation_id + and receipt["desired_revision"] == plan.desired_revision + and receipt["desired_digest"] == plan.desired_digest) + record = self._load("vpc_desired", plan.vpc_id) + if record is None: + return False + durable = _plan_from_record(record) + return bool( + durable.operation_id == plan.operation_id + and durable.desired_revision == plan.desired_revision + and durable.desired_digest == plan.desired_digest + and durable.canonical() == plan.canonical()) + + @asynccontextmanager + async def _operation_scope(self, plan: VpcPlan): + """Carry one validated target through HTTP and periodic resource claims.""" + parent = current_execution() + coordinator = parent.coordinator if parent is not None else self._coordinator + if coordinator is None: + yield # Pure control-plane embeddings have no durable owner. + return + metadata = self._operation_identity(plan) + explicit_marker = {"explicit_vpc_reconcile": True} + handoff_evidence = ( + f"vpc-explicit-reconcile-v1 operation={plan.operation_id} " + f"vpc={plan.vpc_id} revision={plan.desired_revision} " + f"digest={plan.desired_digest}; exact childless retained root") + claim = None + background_execution_id = None + handoff_original = None + handoff_attempted = False + explicit_bind_attempted = False + + def claim_value(value, name: str, default: Any = None) -> Any: + if isinstance(value, Mapping): + return value.get(name, default) + return getattr(value, name, default) + + def exact_parent(record: Mapping[str, Any] | None, *, bound: bool) -> bool: + if parent is None or not isinstance(record, Mapping): + return False + expected_metadata = metadata if bound else explicit_marker + expected_operation = plan.operation_id if bound else None + return bool( + record.get("token") == parent.claim.token + and record.get("execution_id") == parent.claim.execution_id + and record.get("owner_id") == parent.claim.owner_id + and record.get("operation_id") == expected_operation + and record.get("resources") == [] + and record.get("kind") == "request" + and record.get("metadata") == expected_metadata + and record.get("maintenance") is False + and record.get("state") in {"active", "uncertain"}) + + async def retained_root_for_explicit_reconcile(): + """Return only one exact, childless VPC root eligible for handoff.""" + if parent is None: + return None + marker_present = "explicit_vpc_reconcile" in parent.claim.metadata + explicit_root = bool( + parent.claim.operation_id is None + and tuple(parent.claim.resources) == () + and parent.claim.kind == "request" + and parent.claim.metadata == explicit_marker) + if not explicit_root: + if marker_present: + raise CoordinationError("coordination_unavailable", 503) + return None + rows = await _await_mutation_completion( + asyncio.to_thread(coordinator.snapshot)) + current = rows.get(parent.claim.token) + if not exact_parent(current, bound=False) or current["state"] != "active": + raise CoordinationError("coordination_unavailable", 503) + if not self._retained_operation_target_matches(plan): + return None + eligible = [] + for row in rows.values(): + if (row.get("token") != parent.claim.token + and row.get("state") == "uncertain" + and row.get("kind") in {"background", "request"} + and row.get("operation_id") == plan.operation_id + and row.get("resources") == [] + and row.get("maintenance") is False + and row.get("metadata") == metadata): + eligible.append(row) + if len(eligible) != 1: + return None + original = eligible[0] + if any( + row.get("token") not in {original["token"], parent.claim.token} + and (row.get("execution_id") in { + original["execution_id"], parent.claim.execution_id} + or row.get("operation_id") == plan.operation_id) + for row in rows.values() + ): + return None + return original + + async def admit(): + nonlocal claim, background_execution_id, handoff_attempted + nonlocal handoff_original, explicit_bind_attempted + if parent is not None: + handoff_original = await retained_root_for_explicit_reconcile() + if handoff_original is not None: + handoff_attempted = True + claim = await asyncio.to_thread( + coordinator.resolve_and_bind, handoff_original, parent.claim, + handoff_evidence, + operation_id=plan.operation_id, metadata=metadata) + return + explicit_bind_attempted = parent.claim.metadata == explicit_marker + binder = (coordinator.bind_explicit_operation + if explicit_bind_attempted else coordinator.bind_operation) + claim = await asyncio.to_thread( + binder, parent.claim, plan.operation_id, metadata) + else: + background_execution_id = secrets.token_hex(32) + claim = await asyncio.to_thread( + coordinator.claim, operation_id=plan.operation_id, + execution_id=background_execution_id, + kind="background", metadata=metadata) + + def root_value(name: str, default: Any = None) -> Any: + return claim_value(claim, name, default) + + def same_root(record: Mapping[str, Any] | None) -> bool: + return bool( + isinstance(record, Mapping) + and record.get("token") == root_value("token") + and record.get("execution_id") == root_value("execution_id") + and record.get("owner_id") == root_value("owner_id") + and record.get("operation_id") == root_value("operation_id") + and tuple(record.get("resources", ())) + == tuple(root_value("resources", ())) + and record.get("kind") == root_value("kind") + and record.get("metadata") == root_value("metadata", {}) + and record.get("maintenance") + == root_value("maintenance", False)) + + async def read_root() -> Mapping[str, Any] | None: + rows = await _await_mutation_completion( + asyncio.to_thread(coordinator.snapshot)) + current = rows.get(root_value("token")) + if current is not None and not same_root(current): + raise CoordinationError("coordination_unavailable", 503) + return current + + async def retain_root_exact() -> None: + try: + retained = await _await_mutation_completion( + asyncio.to_thread(coordinator.retain, claim)) + except BaseException as first_error: + current = await read_root() + if current is None: + raise first_error + if current.get("state") == "uncertain": + if isinstance(first_error, asyncio.CancelledError): + raise first_error + return + if current.get("state") != "active": + raise CoordinationError( + "coordination_unavailable", 503) from first_error + try: + retained = await _await_mutation_completion( + asyncio.to_thread(coordinator.retain, current)) + except BaseException as retry_error: + final = await read_root() + if final is not None and final.get("state") == "uncertain": + if isinstance(retry_error, asyncio.CancelledError): + raise retry_error + if isinstance(first_error, asyncio.CancelledError): + raise first_error + return + raise retry_error from first_error + if isinstance(first_error, asyncio.CancelledError): + raise first_error + if not retained: + current = await read_root() + if current is not None and current.get("state") == "uncertain": + return + raise CoordinationError("coordination_uncertain", 503) + + async def settle_background(keep: bool) -> None: + if keep: + await retain_root_exact() + return + try: + released = await _await_mutation_completion( + asyncio.to_thread(coordinator.release, claim)) + except BaseException as release_error: + current = await read_root() + if current is not None: + try: + await retain_root_exact() + except BaseException as retain_error: + raise retain_error from release_error + raise + if not released: + # A false release result is not a successful VPC settlement. + # The caller must reconcile even when a concurrent read shows + # that the exact root has disappeared. + raise CoordinationError("coordination_uncertain", 503) + + async def retain_residual(row: Mapping[str, Any]) -> None: + try: + retained = await _await_mutation_completion( + asyncio.to_thread(coordinator.retain, row)) + except BaseException as first_error: + # Fence the root before trying to classify a child settlement + # whose CAS response may have been lost. + try: + await settle_background(True) + recovered_rows = await _await_mutation_completion( + asyncio.to_thread(coordinator.snapshot)) + except BaseException as fence_error: + raise fence_error from first_error + current = recovered_rows.get(row["token"]) + if current is None: + if isinstance(first_error, asyncio.CancelledError): + raise first_error + return + identity_keys = ( + "token", "execution_id", "owner_id", "operation_id", + "resources", "kind", "metadata", "maintenance", + ) + if any(current.get(key) != row.get(key) + for key in identity_keys): + raise CoordinationError( + "coordination_unavailable", 503) from first_error + if current.get("state") == "uncertain": + if isinstance(first_error, asyncio.CancelledError): + raise first_error + return + if current.get("state") != "active": + raise CoordinationError( + "coordination_unavailable", 503) from first_error + try: + retained = await _await_mutation_completion( + asyncio.to_thread(coordinator.retain, current)) + except BaseException as retry_error: + try: + final_rows = await _await_mutation_completion( + asyncio.to_thread(coordinator.snapshot)) + except BaseException as readback_error: + raise readback_error from retry_error + final = final_rows.get(row["token"]) + if final is None or final.get("state") == "uncertain": + if isinstance(retry_error, asyncio.CancelledError): + raise retry_error + if isinstance(first_error, asyncio.CancelledError): + raise first_error + return + raise retry_error from first_error + if isinstance(first_error, asyncio.CancelledError): + raise first_error + if not retained: + # A definite false means the exact child disappeared before + # settlement. The root still remains uncertain because the + # final snapshot observed an unsettled execution. + return + + async def finalize_background( + force_keep: bool, cancellation_seen: asyncio.Event) -> bool: + try: + rows = await _await_mutation_completion( + asyncio.to_thread(coordinator.snapshot)) + except BaseException: + # Final readback/cancellation can race after host work but + # before root settlement. Preserve the root before returning + # the readback failure. + await settle_background(True) + raise + residual = [ + row for row in rows.values() + if (row["execution_id"] == claim.execution_id + and row["token"] != claim.token) + ] + for row in residual: + if row["state"] == "active": + await retain_residual(row) + unknown = (force_keep + or cancellation_seen.is_set() + or current_execution().execution_unknown.is_set() + or bool(residual) + or any( + row["execution_id"] == claim.execution_id + and row["state"] == "uncertain" + for row in rows.values())) + await settle_background(unknown) + return unknown + + async def drain_background_finalizer(force_keep: bool) -> bool: + """Finish every owner settlement before propagating cancellation.""" + cancellation_seen = asyncio.Event() + task = asyncio.create_task( + finalize_background(force_keep, cancellation_seen)) + caller_cancelled = False + while True: + try: + result = await asyncio.shield(task) + except asyncio.CancelledError: + if task.cancelled(): + raise + caller_cancelled = True + cancellation_seen.set() + if not task.done(): + continue + try: + result = task.result() + except BaseException as exc: + raise asyncio.CancelledError() from exc + except BaseException as exc: + if caller_cancelled: + raise asyncio.CancelledError() from exc + raise + if caller_cancelled: + raise asyncio.CancelledError() + return result + + async def recover_background_admission() -> None: + """Fence a background owner whose claim response was interrupted.""" + nonlocal claim + if claim is None and background_execution_id is not None: + # A CAS response can be lost after the background claim was + # durably installed. Recover only this exact execution; an + # operation id alone could name another live worker. No host + # work has been admitted yet, so retaining the recovered row + # records the uncertain database outcome without replaying it. + rows = await _await_mutation_completion( + asyncio.to_thread(coordinator.snapshot)) + recovered = [row for row in rows.values() + if (row.get("execution_id") == background_execution_id + and row.get("owner_id") == coordinator.owner_id + and row.get("operation_id") == plan.operation_id + and row.get("resources") == [] + and row.get("kind") == "background" + and row.get("metadata") == metadata + and row.get("maintenance") is False)] + if len(recovered) > 1: + raise CoordinationError("coordination_unavailable", 503) + if recovered: + claim = recovered[0] + if claim is not None: + await retain_root_exact() + + async def recover_parent_handoff() -> None: + """Fence an atomically rebound request after a lost CAS response.""" + nonlocal claim + if parent is None or not handoff_attempted or handoff_original is None: + return + outcome, current = await _await_mutation_completion( + asyncio.to_thread( + coordinator.classify_resolve_and_bind, + handoff_original, parent.claim, handoff_evidence, + operation_id=plan.operation_id, metadata=metadata)) + if outcome == "uncommitted": + try: + claim = await _await_mutation_completion( + asyncio.to_thread( + coordinator.resolve_and_bind, + handoff_original, parent.claim, handoff_evidence, + operation_id=plan.operation_id, metadata=metadata)) + except BaseException as retry_error: + outcome, current = await _await_mutation_completion( + asyncio.to_thread( + coordinator.classify_resolve_and_bind, + handoff_original, parent.claim, handoff_evidence, + operation_id=plan.operation_id, metadata=metadata)) + if outcome != "committed": + if (isinstance(current, Mapping) + and current.get("token") == parent.claim.token + and current.get("execution_id") + == parent.claim.execution_id + and current.get("owner_id") + == parent.claim.owner_id): + claim = current + await retain_root_exact() + raise CoordinationError( + "coordination_unavailable", 503) from retry_error + else: + await retain_root_exact() + return + if outcome == "committed" and exact_parent(current, bound=True): + claim = current + await retain_root_exact() + return + if (isinstance(current, Mapping) + and current.get("token") == parent.claim.token + and current.get("execution_id") == parent.claim.execution_id + and current.get("owner_id") == parent.claim.owner_id): + claim = current + await retain_root_exact() + raise CoordinationError("coordination_unavailable", 503) + + async def classify_explicit_bind(): + rows = await _await_mutation_completion( + asyncio.to_thread(coordinator.snapshot)) + current = rows.get(parent.claim.token) + conflicts = any( + row.get("token") != parent.claim.token + and (row.get("execution_id") == parent.claim.execution_id + or row.get("operation_id") == plan.operation_id) + for row in rows.values()) + if not conflicts and exact_parent(current, bound=True): + return "committed", current + if (not conflicts and exact_parent(current, bound=False) + and current.get("state") == "active"): + return "uncommitted", current + return "mixed", current + + async def recover_explicit_bind() -> None: + """Classify/retry a marker-consuming bind without losing its root.""" + nonlocal claim + if parent is None or not explicit_bind_attempted: + return + + outcome, current = await classify_explicit_bind() + if outcome == "uncommitted": + try: + claim = await _await_mutation_completion( + asyncio.to_thread( + coordinator.bind_explicit_operation, + parent.claim, plan.operation_id, metadata)) + except BaseException as retry_error: + outcome, current = await classify_explicit_bind() + if outcome != "committed": + if (isinstance(current, Mapping) + and current.get("token") == parent.claim.token + and current.get("execution_id") + == parent.claim.execution_id + and current.get("owner_id") + == parent.claim.owner_id): + claim = current + await retain_root_exact() + raise CoordinationError( + "coordination_unavailable", 503) from retry_error + else: + await retain_root_exact() + return + if outcome == "committed" and exact_parent(current, bound=True): + claim = current + await retain_root_exact() + return + if (isinstance(current, Mapping) + and current.get("token") == parent.claim.token + and current.get("execution_id") == parent.claim.execution_id + and current.get("owner_id") == parent.claim.owner_id): + claim = current + await retain_root_exact() + raise CoordinationError("coordination_unavailable", 503) + + try: + await _await_mutation_completion(admit()) + except CoordinationClaimRejected as exc: + if handoff_attempted: + outcome, current = await _await_mutation_completion( + asyncio.to_thread( + coordinator.classify_resolve_and_bind, + handoff_original, parent.claim, handoff_evidence, + operation_id=plan.operation_id, metadata=metadata)) + if outcome == "committed" and exact_parent(current, bound=True): + claim = current + await _await_mutation_completion(retain_root_exact()) + raise CoordinationError("coordination_uncertain", 503) from exc + elif explicit_bind_attempted: + outcome, current = await classify_explicit_bind() + if outcome == "committed": + claim = current + await _await_mutation_completion(retain_root_exact()) + raise CoordinationError("coordination_uncertain", 503) from exc + # Only a proven binding/admission rejection is side-effect free. + # Unavailable storage and existing execution uncertainty retain + # the normal conservative handling. + raise CoordinationAdmissionRejected(exc.error_code, exc.status) from exc + except BaseException as exc: + # Cancellation can arrive after CAS committed but before admit + # returns. Preserve the known background owner before losing its + # handle; HTTP ownership is settled by the outer request guard. + if parent is None: + try: + await _await_mutation_completion( + recover_background_admission()) + except BaseException as settle_exc: + raise settle_exc from exc + elif handoff_attempted: + try: + await _await_mutation_completion(recover_parent_handoff()) + except BaseException as settle_exc: + raise settle_exc from exc + elif explicit_bind_attempted: + try: + await _await_mutation_completion(recover_explicit_bind()) + except BaseException as settle_exc: + raise settle_exc from exc + raise + with execution_context(coordinator, claim): + try: + yield + except BaseException as exc: + if parent is None: + try: + unknown = await drain_background_finalizer( + _uncertain_exception(exc)) + except BaseException as settle_exc: + raise settle_exc from exc + if unknown and not isinstance(exc, asyncio.CancelledError): + raise CoordinationError( + "coordination_uncertain", 503) from exc + raise + else: + if parent is None: + unknown = await drain_background_finalizer(False) + if unknown: + raise CoordinationError("coordination_uncertain", 503) + + def _check_operation_identity(self, plan: VpcPlan) -> None: + context = current_execution() + if (context is not None and context.claim.metadata.get("job_kind") == "vpc_operation" + and (context.claim.operation_id != plan.operation_id + or any(context.claim.metadata.get(key) != value + for key, value in self._operation_identity(plan).items()))): + raise VpcError("VPC 目标已由其他操作替代", 409, "vpc_revision_conflict") + @staticmethod def _key(vpc_id: str, child_id: str) -> str: return f"{vpc_id}.{child_id}" @@ -1889,6 +2429,7 @@ class VpcCoordinator: fabric_public_keys: Mapping[str, str] | None = None, fabric_plan_digest: str | None = None, private_link_states: Mapping[str, str] | None = None, + fabric_peer_overlays: Mapping[str, str] | None = None, request_deadline: float | None = None, previous_host: Mapping[str, Any] | None = None, ) -> dict[str, Any]: @@ -1917,10 +2458,32 @@ class VpcCoordinator: key for peer_host, key in fabric_public_keys.items() if peer_host != host_id ] + authoritative_peer_overlays = None + if fabric_peer_overlays is not None: + if not isinstance(fabric_peer_overlays, Mapping): + raise VpcError("Fabric peer overlay 闭集无效", 500, + "vpc_fabric_record_corrupt") + local_public_key = ( + fabric_public_keys.get(host_id) + if fabric_public_keys is not None else None) + if fabric_peer_overlays and not isinstance(local_public_key, str): + raise VpcError("Fabric 本机公钥记录不完整", 500, + "vpc_fabric_record_corrupt") + authoritative_peer_overlays = { + key: overlay + for key, overlay in fabric_peer_overlays.items() + if key != local_public_key + } + if (authoritative_peer_keys is not None + and set(authoritative_peer_overlays) + != set(authoritative_peer_keys)): + raise VpcError("Fabric 公钥与 overlay 闭集不一致", 500, + "vpc_fabric_record_corrupt") dataplane = parse_vpc_dataplane_plan( payload, host_id, trusted_tenant_allowed_ips=trusted_peer_tenant_ips, fabric_peer_keys=authoritative_peer_keys, + fabric_peer_overlays=authoritative_peer_overlays, fabric_plan_digest=fabric_plan_digest) # A root-delete plan extends the canonical digest with its signed # retiring-member manifest. The host projection intentionally @@ -1932,10 +2495,33 @@ class VpcCoordinator: # plan. Construct a typed empty-local plan directly so the # removal transaction can validate the deterministic VPC # resources without reintroducing the old host into Fabric. + authoritative_peer_keys = None + authoritative_peer_overlays = None + if fabric_public_keys is not None: + authoritative_peer_keys = frozenset( + key for peer_host, key in fabric_public_keys.items() + if peer_host != host_id) + if fabric_peer_overlays is not None: + local_public_key = ( + fabric_public_keys.get(host_id) + if fabric_public_keys is not None else None) + if fabric_peer_overlays and not isinstance(local_public_key, str): + raise VpcError("Fabric 本机公钥记录不完整", 500, + "vpc_fabric_record_corrupt") + authoritative_peer_overlays = frozenset( + (key, overlay) + for key, overlay in fabric_peer_overlays.items() + if key != local_public_key) + if (authoritative_peer_keys is not None + and {key for key, _overlay in authoritative_peer_overlays} + != authoritative_peer_keys): + raise VpcError("Fabric 公钥与 overlay 闭集不一致", 500, + "vpc_fabric_record_corrupt") dataplane = VpcDataplanePlan( plan.vpc, host_id, tuple(plan.members), tuple(), resource_names(plan.vpc_id), - fabric_plan_digest) + fabric_plan_digest, authoritative_peer_keys, + authoritative_peer_overlays) if request_deadline is None: request_deadline = ( time.monotonic() + VPC_REQUEST_DEADLINE_SECONDS) @@ -2609,6 +3195,13 @@ class VpcCoordinator: if any(not isinstance(key, str) for key in fabric_public_keys.values()): raise VpcError("Fabric 公钥记录不完整", 500, "vpc_fabric_record_corrupt") + fabric_peer_overlays = { + str(fabric_public_keys[node.host_id]): node.overlay_ip + "/32" + for node in full_fabric_plan.nodes + } + if len(fabric_peer_overlays) != len(full_fabric_plan.nodes): + raise VpcError("Fabric 公钥与 overlay 闭集重复", 500, + "vpc_fabric_record_corrupt") fabric_plan = full_fabric_plan # ``record`` is the execution closure for this pass. Root-delete # retries deliberately project it down to hosts that do not yet @@ -2673,6 +3266,7 @@ class VpcCoordinator: profile, all_peers, previous_plan, host.get("retiring_members", []), fabric_public_keys, fabric_plan_digest, private_link_states, + fabric_peer_overlays=fabric_peer_overlays, request_deadline=request_deadline, previous_host=host) if time.monotonic() >= request_deadline: @@ -3722,13 +4316,13 @@ class VpcCoordinator: # the global order per-VPC -> catalog, and do not release either lock # while a cancelled coroutine still has a blocking host mutation or # durable receipt commit in flight. - async with self._lock(vpc_id), _pre_dataplane_lock( + async with self._lock(vpc_id), self._operation_scope(plan), _pre_dataplane_lock( self._catalog_mutation_lock()): return await _await_mutation_completion(delete_locked()) async def apply(self, value: Mapping[str, Any]) -> dict[str, Any]: plan = parse_vpc_plan(value) - async with self._lock(plan.vpc_id), _pre_dataplane_lock( + async with self._lock(plan.vpc_id), self._operation_scope(plan), _pre_dataplane_lock( self._catalog_mutation_lock()): return await _await_mutation_completion( self._apply_locked(plan)) @@ -3763,6 +4357,9 @@ class VpcCoordinator: if old_digest != plan.desired_digest: raise VpcError("相同 desired_revision 的 VPC 计划已存在", 409, "vpc_revision_conflict") + if plan.operation_id != previous_plan.operation_id: + raise VpcError("相同 VPC 目标已绑定其他操作编号", 409, + "vpc_revision_conflict") # Preserve exact host absence proofs monotonically. A # partial pass may have removed host A while host B was # unavailable; replay must retry only B instead of making @@ -4551,6 +5148,7 @@ class VpcCoordinator: if current.get("desired_digest") != plan.desired_digest: raise VpcError("VPC 计划摘要与持久化记录不一致", 500, "vpc_record_corrupt") + self._check_operation_identity(plan) gate = self._fabric_gate(plan) # Reconcile is the durable retry path after a host response was # lost or the engine restarted. Exact active hosts return without @@ -4562,10 +5160,18 @@ class VpcCoordinator: self._save_children(current) return self._public(current), None - async with self._lock(vpc_id), _pre_dataplane_lock( - self._catalog_mutation_lock()): - result, delete_payload = await _await_mutation_completion( - reconcile_locked()) + async with self._lock(vpc_id): + identity = self._load("vpc_desired", vpc_id) + if identity is None: + raise VpcError("VPC 不存在", 404, "vpc_not_found") + if self._delete_marker(identity) is not None: + result, delete_payload = None, self._delete_payload_from_record(identity) + else: + plan = _plan_from_record(identity) + async with self._operation_scope(plan), _pre_dataplane_lock( + self._catalog_mutation_lock()): + result, delete_payload = await _await_mutation_completion( + reconcile_locked()) # The locked recheck may observe a delete target that was installed # after the optimistic read. Release the non-reentrant lock before # entering ``delete``, which acquires the same per-VPC and catalog diff --git a/engine/virsh_panel/vpc_dataplane.py b/engine/virsh_panel/vpc_dataplane.py index b2ef9c7f..258c8d13 100644 --- a/engine/virsh_panel/vpc_dataplane.py +++ b/engine/virsh_panel/vpc_dataplane.py @@ -65,6 +65,49 @@ _APPLY_RECEIPT_LIMIT = 1024 _APPLY_RECEIPTS: OrderedDict[tuple[str, ...], tuple[str, str]] = OrderedDict() _APPLY_RECEIPT_LOCK = Lock() +# The synchronous VPC SSH boundary intentionally discards arbitrary remote +# stderr because host tools may print paths or configuration. Keep one +# closed, host-local journal record instead so an apply failure can be placed +# before or after network activation without exposing command output. +VPC_APPLY_SAFE_LOG_SH = r''' +LOGGER= +for vc_vpc_logger_path in /usr/bin/logger /bin/logger /usr/sbin/logger /sbin/logger; do + if [ -x "$vc_vpc_logger_path" ]; then LOGGER=$vc_vpc_logger_path; break; fi +done +VC_VPC_APPLY_PHASE=entry +vc_vpc_apply_failure_log() { + vc_vpc_log_status=$1 + case "$vc_vpc_log_status" in ''|*[!0-9]*) vc_vpc_log_status=1 ;; esac + [ "$vc_vpc_log_status" -le 255 ] 2>/dev/null || vc_vpc_log_status=1 + [ "$vc_vpc_log_status" != 0 ] || return 0 + case "${VC_VPC_APPLY_PHASE:-}" in + entry|tools|preflight|transaction_snapshot|quarantine|firewall|bridge|persistence|activation|post_activation_sysctl|runtime|proxy|neighbors|routes|policy_rules|fabric_acl|wireguard|final_readback|commit|rollback) + vc_vpc_log_phase=$VC_VPC_APPLY_PHASE ;; + *) vc_vpc_log_phase=unknown ;; + esac + [ -n "${LOGGER:-}" ] || return 0 + "$LOGGER" -t virt-cloud-vpc -- \ + "apply_state=failed apply_phase=$vc_vpc_log_phase exit_status=$vc_vpc_log_status" \ + >/dev/null 2>&1 || true + return 0 +} +vc_vpc_apply_log_exit() { + vc_vpc_log_exit_status=$? + trap - EXIT + vc_vpc_apply_failure_log "$vc_vpc_log_exit_status" || true + exit "$vc_vpc_log_exit_status" +} +vc_vpc_apply_signal_exit() { + vc_vpc_signal_status=$1 + trap - HUP INT TERM + exit "$vc_vpc_signal_status" +} +trap vc_vpc_apply_log_exit EXIT +trap 'vc_vpc_apply_signal_exit 129' HUP +trap 'vc_vpc_apply_signal_exit 130' INT +trap 'vc_vpc_apply_signal_exit 143' TERM +'''.strip() + # iproute2 prints host routes without /32 and suppresses dev when its query # already selected that device. Infer only the explicitly supplied filter; # unfiltered inventories must still contain a unique output device. A private @@ -243,7 +286,7 @@ vc_vpc_neighbor_helper_ready() { [ "$(sed -n '3p' "$vc_ln_helper")" = "# fabric-plan-digest=$EXPECTED_DIGEST" ] && [ "$(sed -n '4p' "$vc_ln_helper")" = "PLAN_DIGEST=\"$EXPECTED_DIGEST\"" ] && [ "$(sed -n '6p' "$vc_ln_helper")" = '# vpc-local-neighbor-contract-v1' ] && - [ "$(sed -n '7p' "$vc_ln_helper")" = '# vpc-state-route-scope-contract-v3' ] + [ "$(sed -n '7p' "$vc_ln_helper")" = '# vpc-state-route-scope-contract-v4' ] } vc_vpc_neighbors_snapshot() { vc_ln_snapshot=$(vc_local_neighbor_records "$BRIDGE") || return 1 @@ -933,6 +976,11 @@ class VpcDataplanePlan: # separate from ``peers`` because one VPC can use only a subset of the # peers present on a shared Fabric. fabric_peer_keys: frozenset[str] | None = None + # Authoritative public-key -> overlay /32 closure from the complete + # engine-validated Fabric plan. The key-only field remains for legacy + # callers, but only this mapping can prove and converge cryptokey routing + # for peers outside this VPC's participant set. + fabric_peer_overlays: frozenset[tuple[str, str]] | None = None def __post_init__(self) -> None: _fabric_plan_digest(self.fabric_plan_digest) @@ -1033,6 +1081,23 @@ def _key(value: Any) -> str: return value +def _fabric_overlay_cidr(value: Any) -> str: + try: + interface = ipaddress.ip_interface(value) + except (TypeError, ValueError) as exc: + raise VpcDataplaneError("Fabric peer overlay 地址无效", 409, + "vpc_fabric_peer_set_invalid") from exc + fabric_overlay = ipaddress.IPv4Network("100.64.0.0/10") + if (not isinstance(interface, ipaddress.IPv4Interface) + or interface.network.prefixlen != 32 + or interface.ip not in fabric_overlay + or interface.ip.is_loopback + or interface.ip.is_link_local): + raise VpcDataplaneError("Fabric peer overlay 地址无效", 409, + "vpc_fabric_peer_set_invalid") + return f"{interface.ip}/32" + + def _mac(value: Any, label: str = "私网 MAC") -> str: if not isinstance(value, str) or _MAC.fullmatch(value) is None: raise VpcDataplaneError(f"{label}无效") @@ -1252,6 +1317,7 @@ def parse_vpc_dataplane_plan( fabric_peers: Sequence[Mapping[str, Any]] | None = None, trusted_tenant_allowed_ips: Mapping[str, Sequence[str]] | None = None, fabric_peer_keys: Sequence[str] | None = None, + fabric_peer_overlays: Mapping[str, str] | None = None, fabric_plan_digest: str | None = None, ) -> VpcDataplanePlan: """Parse existing VPC DTO plus an optional engine-only Fabric snapshot.""" @@ -1337,9 +1403,65 @@ def parse_vpc_dataplane_plan( raise VpcDataplaneError("VPC peer 不属于 Fabric 公钥闭集", 409, "vpc_fabric_peer_set_invalid") authoritative_keys = frozenset(canonical_keys) + authoritative_overlays: frozenset[tuple[str, str]] | None = None + if fabric_peer_overlays is not None: + if (not isinstance(fabric_peer_overlays, Mapping) + or len(fabric_peer_overlays) > 64): + raise VpcDataplaneError("Fabric peer overlay 闭集无效", 409, + "vpc_fabric_peer_set_invalid") + canonical_overlays: dict[str, str] = {} + seen_overlay_cidrs: set[str] = set() + for raw_key, raw_overlay in fabric_peer_overlays.items(): + key = _key(raw_key) + overlay_cidr = _fabric_overlay_cidr(raw_overlay) + if key in canonical_overlays or overlay_cidr in seen_overlay_cidrs: + raise VpcDataplaneError("Fabric peer overlay 闭集重复", 409, + "vpc_fabric_peer_set_invalid") + canonical_overlays[key] = overlay_cidr + seen_overlay_cidrs.add(overlay_cidr) + overlay_keys = frozenset(canonical_overlays) + if authoritative_keys is not None and overlay_keys != authoritative_keys: + raise VpcDataplaneError("Fabric peer 公钥与 overlay 闭集不一致", 409, + "vpc_fabric_peer_set_invalid") + for peer in peers: + if canonical_overlays.get(peer.public_key) != peer.overlay_cidr: + raise VpcDataplaneError("VPC peer overlay 不属于 Fabric 闭集", 409, + "vpc_fabric_peer_set_invalid") + authoritative_keys = overlay_keys + authoritative_overlays = frozenset(canonical_overlays.items()) return VpcDataplanePlan( vpc, host_id, tuple(members), peers, resource_names(vpc.vpc_id), - fabric_plan_digest, authoritative_keys) + fabric_plan_digest, authoritative_keys, authoritative_overlays) + + +def _authoritative_fabric_peer_overlays( + plan: VpcDataplanePlan, +) -> dict[str, str] | None: + """Validate and materialize the engine-only full Fabric peer closure.""" + if plan.fabric_peer_overlays is None: + return None + mapping: dict[str, str] = {} + seen_overlays: set[str] = set() + for row in plan.fabric_peer_overlays: + if not isinstance(row, tuple) or len(row) != 2: + raise VpcDataplaneError("Fabric peer overlay 闭集无效", 409, + "vpc_fabric_peer_set_invalid") + key = _key(row[0]) + overlay = _fabric_overlay_cidr(row[1]) + if key in mapping or overlay in seen_overlays: + raise VpcDataplaneError("Fabric peer overlay 闭集重复", 409, + "vpc_fabric_peer_set_invalid") + mapping[key] = overlay + seen_overlays.add(overlay) + if (plan.fabric_peer_keys is None + or frozenset(mapping) != plan.fabric_peer_keys): + raise VpcDataplaneError("Fabric peer 公钥与 overlay 闭集不一致", 409, + "vpc_fabric_peer_set_invalid") + for peer in plan.peers: + if mapping.get(peer.public_key) != peer.overlay_cidr: + raise VpcDataplaneError("VPC peer overlay 不属于 Fabric 闭集", 409, + "vpc_fabric_peer_set_invalid") + return dict(sorted(mapping.items())) def _routes(plan: VpcDataplanePlan) -> tuple[dict[str, Any], ...]: @@ -1674,11 +1796,18 @@ def _state( metric=route.get("metric", 100))) for member in sorted(plan.local_members, key=lambda item: item.private_ip): lines.append(f"local_neighbor|{member.private_ip}|{member.private_mac}") + wg_route_ips = { + str(route["destination_cidr"]) + for route in routes if route["device"] == FABRIC_INTERFACE + } + owned_peer_ips = _expected_peer_tenant_ips(plan.vpc) for peer in plan.peers: - ips = sorted(set( - [m.private_ip + "/32" for m in plan.remote_members - if m.host_id == peer.host_id] - + list(peer.tenant_allowed_ips))) + # Persist only this VPC's route-backed contribution. The authenticated + # aggregate remains on ``peer.tenant_allowed_ips`` for live WireGuard + # convergence/readback, while old aggregate state is filtered by the + # loader during a mixed-version rollout. + ips = sorted(set(owned_peer_ips.get(peer.host_id, frozenset())) + & wg_route_ips) if ips: lines.append(f"peer|{peer.host_id}|{peer.public_key}|{','.join(ips)}") return "\n".join(lines) + "\n" @@ -1860,9 +1989,36 @@ def _render_apply_script_v2( proxy_lines = "\n".join( str(ipaddress.ip_network(item["destination_cidr"], strict=True).network_address) for item in routes) + peers_by_key = {peer.public_key: peer for peer in plan.peers} + authoritative_peer_overlays = _authoritative_fabric_peer_overlays(plan) + if (authoritative_peer_overlays is None + and plan.fabric_peer_keys is not None + and not plan.fabric_peer_keys.issubset(peers_by_key)): + raise VpcDataplaneError( + "完整 Fabric peer 收敛缺少权威 overlay 闭集", 409, + "vpc_fabric_peer_set_invalid") + # A VPC revision can stop using one Fabric peer while the peer itself + # remains part of the host's WireGuard Fabric. Converge that full + # authoritative closure after rewriting durable state, otherwise the + # removed peer retains the withdrawn tenant /32 indefinitely. Direct + # parser callers without the engine-only closure retain the historical + # current-VPC scope; production plans always provide the full key-to-overlay + # closure. + apply_fabric_peer_keys = tuple(sorted( + authoritative_peer_overlays + if authoritative_peer_overlays is not None + else (plan.fabric_peer_keys + if plan.fabric_peer_keys is not None else peers_by_key) + )) peer_lines = "\n".join( - f"{peer.host_id}|{peer.public_key}|{_allowed(plan, peer)}" - for peer in plan.peers) + f"{peers_by_key[key].host_id if key in peers_by_key else '-'}|" + f"{key}|{authoritative_peer_overlays[key]}" + if authoritative_peer_overlays is not None + else f"{peers_by_key[key].host_id}|{key}|{peers_by_key[key].overlay_cidr}" + for key in apply_fabric_peer_keys) + apply_fabric_peer_key_words = " ".join(apply_fabric_peer_keys) + apply_fabric_peer_keys_provided = ( + "yes" if authoritative_peer_overlays is not None else "no") state = _state(plan, routes, firewall_mode) # Local service /32s use a guest address as the next hop on an otherwise # unnumbered private bridge. Linux therefore needs the explicit `onlink` @@ -1965,6 +2121,7 @@ set -eu umask 077 PATH=/usr/sbin:/usr/bin:/sbin:/bin export PATH +{VPC_APPLY_SAFE_LOG_SH} IP=$(for p in /usr/sbin/ip /usr/bin/ip /sbin/ip /bin/ip; do [ -x "$p" ] && {{ printf '%s' "$p"; break; }}; done) WG=$(for p in /usr/bin/wg /usr/sbin/wg /bin/wg /sbin/wg; do [ -x "$p" ] && {{ printf '%s' "$p"; break; }}; done) NFT=$(for p in /usr/sbin/nft /usr/bin/nft /sbin/nft /bin/nft; do [ -x "$p" ] && {{ printf '%s' "$p"; break; }}; done) @@ -2009,12 +2166,14 @@ nft_plan_owned() {{ [ "$(printf '%s\\n' "$nft_plan_text" | grep -Foc 'comment "virt-cloud-wg-fabric-v1:'"$nft_plan_digest"'"' || true)" = 8 ] }} +VC_VPC_APPLY_PHASE=tools [ -n "$IP" ] && [ -n "$WG" ] && [ -n "$NFT" ] && [ -n "$FLOCK" ] && [ -n "$STAT" ] && [ -n "$CMP" ] && [ -n "$SYSCTL" ] && [ -n "$BRIDGE_CMD" ] && [ -n "$SYSTEMCTL" ] || exit 70 if [ "$MODE" = networkmanager_nmcli ]; then NMCLI=$(for p in /usr/bin/nmcli /usr/sbin/nmcli /bin/nmcli /sbin/nmcli; do [ -x "$p" ] && {{ printf '%s' "$p"; break; }}; done) [ -n "$NMCLI" ] || exit 70 fi [ "$FIREWALL_MODE" = native_nftables ] || exit 71 +VC_VPC_APPLY_PHASE=preflight if [ -L /run/virt-cloud ] || {{ [ -e /run/virt-cloud ] && [ ! -d /run/virt-cloud ]; }}; then exit 71; fi if [ -L /etc/virt-cloud ] || {{ [ -e /etc/virt-cloud ] && [ ! -d /etc/virt-cloud ]; }}; then exit 71; fi if [ -L {state_dir} ] || {{ [ -e {state_dir} ] && [ ! -d {state_dir} ]; }}; then exit 71; fi @@ -2280,13 +2439,17 @@ fi vc_ip_rule_scope_check {_q(plan.vpc.cidr)} "$ROUTE_TABLE" "$RULE_PRIORITY" "$WG_IFACE" managed 73 73 || exit $? # Read-only preflight leaves the original ACL and link intact. The journal # must contain the runtime baseline before quarantine can flush kernel state. +VC_VPC_APPLY_PHASE=transaction_snapshot mkdir -m 700 "$TXN" vc_vpc_write_native_transaction_mode || exit 79 printf '%s\\n' '{DATAPLANE_OWNER_MARKER}:{plan.vpc.vpc_id}:{plan.host_id}:{plan.desired_revision}:{plan.desired_digest}' > "$TXN/inflight" chmod 600 "$TXN/inflight" printf '%s\\n' preparing > "$TXN/state"; chmod 600 "$TXN/state" rollback() {{ - status=$?; [ "$COMMITTED" = yes ] && exit "$status"; trap - EXIT HUP INT TERM + status=$? + trap - EXIT HUP INT TERM + vc_vpc_apply_failure_log "$status" || true + [ "$COMMITTED" = yes ] && exit "$status" set +e if [ "$MUTATION_STARTED" = no ]; then # Snapshot/preflight failure has not touched any dataplane resource. @@ -2679,10 +2842,17 @@ rollback() {{ fi "$IP" link set dev "$BRIDGE" down >/dev/null 2>&1 || true fi - if [ "$ROLLBACK_FAILED" = yes ]; then printf '%s\\n' rollback_failed > "$TXN/state"; chmod 600 "$TXN/state"; exit 79; fi + if [ "$ROLLBACK_FAILED" = yes ]; then + VC_VPC_APPLY_PHASE=rollback + vc_vpc_apply_failure_log 79 || true + printf '%s\\n' rollback_failed > "$TXN/state"; chmod 600 "$TXN/state"; exit 79 + fi rm -rf "$TXN"; exit "$status" }} trap rollback EXIT HUP INT TERM +trap 'vc_vpc_apply_signal_exit 129' HUP +trap 'vc_vpc_apply_signal_exit 130' INT +trap 'vc_vpc_apply_signal_exit 143' TERM if [ -f "$OWNER_FILE" ]; then [ ! -L "$OWNER_FILE" ] && grep -Fqx "$OWNER_MARKER" "$OWNER_FILE" || exit 71 cp -p "$OWNER_FILE" "$TXN/owner.before" @@ -2734,6 +2904,7 @@ if [ "$BRIDGE_TOPOLOGY_CAPTURED" = yes ]; then # Kernel DOWN and NM unmanaged can flush proxy neighbours and routes. The # original ACL, owner, manager, topology and runtime are now journaled. vc_vpc_bridge_topology_matches || exit 72 + VC_VPC_APPLY_PHASE=quarantine MUTATION_STARTED=yes RUNTIME_QUARANTINED=yes if [ "$MODE" = networkmanager_nmcli ]; then @@ -2795,8 +2966,10 @@ vc_vpc_nft_matches_snapshot "$TXN/nft.desired" || exit 72 }} # Install and exactly read back the raw tenant guard before restoring link # administration in either persistence mode. +VC_VPC_APPLY_PHASE=firewall MUTATION_STARTED=yes vc_vpc_install_rules +VC_VPC_APPLY_PHASE=bridge if "$IP" link show dev "$BRIDGE" >/dev/null 2>&1; then "$IP" -d link show dev "$BRIDGE" | grep -Eq '(^|[[:space:]])bridge[[:space:]]+(forward_delay|stp_state|hello_time|ageing_time)([[:space:]]|$)' || exit 71 [ -f "$OWNER_FILE" ] || exit 71 @@ -2884,6 +3057,7 @@ for vc_runtime_key in proxy_arp proxy_arp_pvlan; do cat "$vc_runtime_path" > "$TXN/$vc_runtime_key.before" || exit 72 fi done +VC_VPC_APPLY_PHASE=persistence if [ "$MODE" = networkmanager_nmcli ]; then [ -n "$NMCLI" ] || exit 70 NM_PROFILE_UUID= @@ -2910,11 +3084,13 @@ if [ "$MODE" = networkmanager_nmcli ]; then vc_nm_device_release || exit 72 NM_DEVICE_QUARANTINED=no fi + VC_VPC_APPLY_PHASE=activation if [ "$NM_PRESERVE_ACTIVE_BRIDGE" = yes ]; then vc_nm_preserve_bridge_ready || exit 72 else "$NMCLI" connection up uuid "$NM_PROFILE_UUID" >/dev/null fi + VC_VPC_APPLY_PHASE=post_activation_sysctl vc_vpc_rp_filter_set || exit 72 else [ -x "$IFUP" ] && [ -x "$IFQUERY" ] && [ -x "$IFDOWN" ] || exit 70 @@ -2949,9 +3125,12 @@ else # have acquired a libvirt tap before a later step fails. The rollback path # below inspects ports first, then deletes only an empty newly-created # bridge or restores an empty existing bridge to its sampled DOWN state. + VC_VPC_APPLY_PHASE=activation VC_FABRIC_APPLYING=1 "$IFUP" --force "$BRIDGE" >/dev/null 2>&1 || exit 72 + VC_VPC_APPLY_PHASE=post_activation_sysctl vc_vpc_rp_filter_set || exit 72 fi +VC_VPC_APPLY_PHASE=runtime if [ "$BRIDGE_TOPOLOGY_CAPTURED" = yes ]; then vc_vpc_bridge_topology_matches || exit 72 fi @@ -2961,6 +3140,7 @@ printf '%s\\n' 0 > "/proc/sys/net/ipv4/conf/$BRIDGE/proxy_arp" || exit 72 PROXY_ARP_PVLAN_CHANGED=yes printf '%s\\n' 1 > "/proc/sys/net/ipv4/conf/$BRIDGE/proxy_arp_pvlan" || exit 72 [ "$(cat "/proc/sys/net/ipv4/conf/$BRIDGE/proxy_arp_pvlan" 2>/dev/null)" = 1 ] || exit 72 +VC_VPC_APPLY_PHASE=proxy vc_vpc_proxy_snapshot "$TXN/proxy.activation.current" || exit 72 vc_vpc_runtime_subset "$TXN/proxy.activation.current" "$TXN/proxy.before" || exit 71 PROXY_ENTRIES_CHANGED=yes @@ -2989,7 +3169,9 @@ cat <<'{proxy_marker}' | LC_ALL=C sort -u > "$TXN/proxy.desired" {proxy_marker} "$CMP" -s "$TXN/proxy.after" "$TXN/proxy.desired" || exit 72 # Restore exact member bindings after the manager may have cleared them. +VC_VPC_APPLY_PHASE=neighbors vc_vpc_neighbors_reconcile || exit 72 +VC_VPC_APPLY_PHASE=routes ROUTES_CHANGED=yes cat > "$TXN/routes.desired" <<'{route_marker}' {route_lines} @@ -3039,6 +3221,7 @@ done <<'{route_marker}' # Recheck the complete RPDB scope after route convergence, immediately before # adding missing rules. An out-of-band rule created during the transaction # must force rollback rather than make a successful apply read back degraded. +VC_VPC_APPLY_PHASE=policy_rules # vc-policy-rule-preflight-begin : > "$TXN/rules.before" : > "$TXN/rules.attempted" @@ -3052,6 +3235,7 @@ vc_ip_rule_preflight "$WG_RULE_PRIORITY" "$WG_IFACE" {_q(plan.vpc.cidr)} "$ROUTE vc_ip_rule_ensure "$RULE_PRIORITY" - {_q(plan.vpc.cidr)} "$ROUTE_TABLE" 73 73 "$TXN/rules.attempted" "$TXN/rules.created" || exit $? vc_ip_rule_ensure "$WG_RULE_PRIORITY" "$WG_IFACE" {_q(plan.vpc.cidr)} "$ROUTE_TABLE" 73 73 "$TXN/rules.attempted" "$TXN/rules.created" || exit $? # vc-policy-rule-ensure-end +VC_VPC_APPLY_PHASE=fabric_acl cat > "$STATE_FILE" <<'{state_marker}' {state.rstrip()} {state_marker} @@ -3068,23 +3252,54 @@ vc_nft_table_snapshot inet virt_cloud_wg_fabric \ [ ! -e "$TXN/fabric.nft.desired.absent" ] || exit 74 vc_nft_table_matches_snapshot inet virt_cloud_wg_fabric \ "$TXN/fabric.nft.desired" || exit 74 -while IFS='|' read -r peer_host key allowed; do +VC_VPC_APPLY_PHASE=wireguard +expected_fabric_peer_keys={_q(apply_fabric_peer_key_words)} +if [ "{apply_fabric_peer_keys_provided}" = yes ]; then + live_fabric_peer_keys=$("$WG" show "$WG_IFACE" peers 2>/dev/null) || exit 74 + for live_peer_key in $live_fabric_peer_keys; do + case " $expected_fabric_peer_keys " in + *" $live_peer_key "*) : ;; + *) exit 74 ;; + esac + done + for expected_peer_key in $expected_fabric_peer_keys; do + printf '%s\n' "$live_fabric_peer_keys" | + grep -Fqx "$expected_peer_key" || exit 74 + done +fi +while IFS='|' read -r peer_host key expected_overlay; do [ -n "$peer_host" ] || continue "$WG" show "$WG_IFACE" peers | grep -Fqx "$key" || exit 74 - case ",$allowed," in *',0.0.0.0/0,'*|*',0.0.0.0/1,'*|*',128.0.0.0/1,'* ) exit 74 ;; esac + case ",$expected_overlay," in *',0.0.0.0/0,'*|*',0.0.0.0/1,'*|*',128.0.0.0/1,'* ) exit 74 ;; esac # Fabric has one AllowedIPs list per public key. Merge the complete, # validated tenant set from every panel-owned VPC state file while keeping # the peer's single overlay /32. Never replace the list with this VPC's # partial view: doing so would black-hole another concurrently active VPC. merge_actual=$(vc_wg_allowed_for_peer "$WG_IFACE" "$key") || exit 74 - merge_overlay=$(vpc_state_overlay_from_allowed "$merge_actual") || exit 74 + vpc_state_overlay_from_allowed "$merge_actual" >/dev/null || exit 74 merge_extra=$(vpc_state_allowed_for_peer "$key") || exit 74 - merge_allowed="$merge_overlay" + merge_allowed="$expected_overlay" [ -z "$merge_extra" ] || merge_allowed="$merge_allowed,$merge_extra" "$WG" set "$WG_IFACE" peer "$key" allowed-ips "$merge_allowed" || exit 74 done <<'{peer_marker}' {peer_lines} {peer_marker} +# `wg set` success is not an exact-state receipt. Re-read every authoritative +# peer and prove its Fabric overlay and route-backed tenant union exactly. +while IFS='|' read -r final_peer_host final_peer_key final_overlay; do + [ -n "$final_peer_host" ] || continue + final_actual=$(vc_wg_allowed_for_peer "$WG_IFACE" "$final_peer_key") || exit 74 + vpc_state_overlay_from_allowed "$final_actual" >/dev/null || exit 74 + final_extra=$(vpc_state_allowed_for_peer "$final_peer_key") || exit 74 + final_expected="$final_overlay" + [ -z "$final_extra" ] || final_expected="$final_expected,$final_extra" + final_actual_set=$(printf '%s\n' "$final_actual" | tr ',' '\n' | awk 'NF' | LC_ALL=C sort -u) + final_expected_set=$(printf '%s\n' "$final_expected" | tr ',' '\n' | awk 'NF' | LC_ALL=C sort -u) + [ -n "$final_actual_set" ] && + [ "$final_actual_set" = "$final_expected_set" ] || exit 74 +done <<'{peer_marker}' +{peer_lines} +{peer_marker} # The shared Fabric quarantine table is the last enforcement boundary before # a VPC bridge is admitted. Reparse the complete stateless grammar, # bind every owner comment to the engine-authenticated full Fabric digest, @@ -3092,6 +3307,7 @@ done <<'{peer_marker}' # prove the whole table is still the post-refresh snapshot. A retained # comment with altered drop semantics, a range element, or a late mutation # therefore fails closed. +VC_VPC_APPLY_PHASE=final_readback vc_vpc_fabric_nft_live_ready || exit 74 vpc_state_nft_set_matches || exit 74 vc_nft_table_matches_snapshot inet virt_cloud_wg_fabric \ @@ -3105,6 +3321,7 @@ vc_ln_final_expected=$(cat "$TXN/neighbors.desired") || exit 74 vc_ln_final_actual=$(vc_local_neighbor_records "$BRIDGE") || exit 74 vc_ln_final_wanted=$(printf '%s\\n' "$vc_ln_final_expected" | awk 'NF {{print $0 "|PERMANENT"}}') || exit 74 [ "$vc_ln_final_actual" = "$vc_ln_final_wanted" ] || exit 74 +VC_VPC_APPLY_PHASE=commit rm -f "$VPC_PEER_FILE" "$VPC_ROUTE_FILE" "$VPC_TENANT_FILE" "$VPC_TENANT_FILE.nft" printf '%s\\n' ready > "$TXN/state"; chmod 600 "$TXN/state" printf '%s\\n' '__VC_VPC_APPLIED__=yes' @@ -3160,6 +3377,15 @@ def render_remove_script( fragment = f"/etc/network/interfaces.d/virt-cloud-vpc-{r.token}" sysctl_file = f"/etc/sysctl.d/99-virt-cloud-vpc-{r.token}.conf" proxy_file = f"/etc/virt-cloud/vpc-dataplane/{r.token}.proxy" + authoritative_peer_overlays = _authoritative_fabric_peer_overlays(plan) + remove_peer_lines = "\n".join( + f"{key}|{overlay}" + for key, overlay in (authoritative_peer_overlays or {}).items()) + remove_peer_key_words = " ".join(authoritative_peer_overlays or ()) + remove_peer_marker = "__VC_VPC_REMOVE_FABRIC_PEERS_" + hashlib.sha256( + remove_peer_lines.encode()).hexdigest()[:12] + remove_peer_closure_provided = ( + "yes" if authoritative_peer_overlays is not None else "no") state_loader = _VPC_STATE_LOADER.replace( "__VC_FABRIC_HOST_ID__", plan.host_id).replace( "__VC_FABRIC_ID__", plan.vpc.fabric_id).replace( @@ -3877,13 +4103,57 @@ vc_nft_table_snapshot inet virt_cloud_wg_fabric \ [ ! -e "$TXN/fabric.nft.desired.absent" ] || exit 74 vc_nft_table_matches_snapshot inet virt_cloud_wg_fabric \ "$TXN/fabric.nft.desired" || exit 74 -for key in $("$WG" show "$WG_IFACE" peers 2>/dev/null); do - actual=$(vc_wg_allowed_for_peer "$WG_IFACE" "$key") || exit 74 - overlay=$(vpc_state_overlay_from_allowed "$actual") || exit 74 - extra=$(vpc_state_allowed_for_peer "$key") || exit 74 - merged="$overlay"; [ -z "$extra" ] || merged="$merged,$extra" - "$WG" set "$WG_IFACE" peer "$key" allowed-ips "$merged" || exit 74 -done +expected_fabric_peer_keys={_q(remove_peer_key_words)} +if [ "{remove_peer_closure_provided}" = yes ]; then + live_fabric_peer_keys=$("$WG" show "$WG_IFACE" peers 2>/dev/null) || exit 74 + for live_peer_key in $live_fabric_peer_keys; do + case " $expected_fabric_peer_keys " in + *" $live_peer_key "*) : ;; + *) exit 74 ;; + esac + done + for expected_peer_key in $expected_fabric_peer_keys; do + printf '%s\n' "$live_fabric_peer_keys" | + grep -Fqx "$expected_peer_key" || exit 74 + done + while IFS='|' read -r key expected_overlay; do + [ -n "$key" ] || continue + actual=$(vc_wg_allowed_for_peer "$WG_IFACE" "$key") || exit 74 + vpc_state_overlay_from_allowed "$actual" >/dev/null || exit 74 + extra=$(vpc_state_allowed_for_peer "$key") || exit 74 + merged="$expected_overlay"; [ -z "$extra" ] || merged="$merged,$extra" + "$WG" set "$WG_IFACE" peer "$key" allowed-ips "$merged" || exit 74 + done <<'{remove_peer_marker}' +{remove_peer_lines} +{remove_peer_marker} +else + # Legacy direct callers lack a full authenticated overlay mapping. Keep + # their historical route-withdrawal behavior without treating it as an + # authoritative Fabric closure receipt. + for key in $("$WG" show "$WG_IFACE" peers 2>/dev/null); do + actual=$(vc_wg_allowed_for_peer "$WG_IFACE" "$key") || exit 74 + overlay=$(vpc_state_overlay_from_allowed "$actual") || exit 74 + extra=$(vpc_state_allowed_for_peer "$key") || exit 74 + merged="$overlay"; [ -z "$extra" ] || merged="$merged,$extra" + "$WG" set "$WG_IFACE" peer "$key" allowed-ips "$merged" || exit 74 + done +fi +if [ "{remove_peer_closure_provided}" = yes ]; then + while IFS='|' read -r final_peer_key final_overlay; do + [ -n "$final_peer_key" ] || continue + final_actual=$(vc_wg_allowed_for_peer "$WG_IFACE" "$final_peer_key") || exit 74 + vpc_state_overlay_from_allowed "$final_actual" >/dev/null || exit 74 + final_extra=$(vpc_state_allowed_for_peer "$final_peer_key") || exit 74 + final_expected="$final_overlay" + [ -z "$final_extra" ] || final_expected="$final_expected,$final_extra" + final_actual_set=$(printf '%s\n' "$final_actual" | tr ',' '\n' | awk 'NF' | LC_ALL=C sort -u) + final_expected_set=$(printf '%s\n' "$final_expected" | tr ',' '\n' | awk 'NF' | LC_ALL=C sort -u) + [ -n "$final_actual_set" ] && + [ "$final_actual_set" = "$final_expected_set" ] || exit 74 + done <<'{remove_peer_marker}' +{remove_peer_lines} +{remove_peer_marker} +fi vc_vpc_fabric_nft_live_ready || exit 74 vpc_state_nft_set_matches || exit 74 vc_nft_table_matches_snapshot inet virt_cloud_wg_fabric \ @@ -3953,15 +4223,25 @@ def render_observe_script( f"{x['destination_cidr']}|{x['device']}|{x.get('via') or '-'}|{x.get('metric', 100)}" for x in routes) peer_lines = "\n".join( - f"{p.host_id}|{p.public_key}|{_allowed(plan, p)}" + # The host's validated state files, rather than the control-plane + # catalog aggregate, own the tenant half of AllowedIPs. The current + # plan still supplies the authenticated peer identity and overlay /32. + f"{p.host_id}|{p.public_key}|{p.overlay_cidr}" for p in plan.peers) - fabric_peer_key_words = " ".join(sorted(plan.fabric_peer_keys or ())) + authoritative_peer_overlays = _authoritative_fabric_peer_overlays(plan) + fabric_peer_overlay_lines = "\n".join( + f"{key}|{overlay}" + for key, overlay in (authoritative_peer_overlays or {}).items()) + fabric_peer_key_words = " ".join(authoritative_peer_overlays or ()) fabric_peer_key_arg = _q(fabric_peer_key_words) - fabric_peer_keys_provided = "yes" if plan.fabric_peer_keys is not None else "no" + fabric_peer_keys_provided = ( + "yes" if authoritative_peer_overlays is not None else "no") route_marker = "__VC_VPC_OBS_ROUTES_" + hashlib.sha256( route_lines.encode()).hexdigest()[:12] peer_marker = "__VC_VPC_OBS_PEERS_" + hashlib.sha256( peer_lines.encode()).hexdigest()[:12] + fabric_peer_marker = "__VC_VPC_OBS_FABRIC_PEERS_" + hashlib.sha256( + fabric_peer_overlay_lines.encode()).hexdigest()[:12] state_loader = _VPC_STATE_LOADER.replace( "__VC_FABRIC_HOST_ID__", plan.host_id).replace( "__VC_FABRIC_ID__", plan.vpc.fabric_id).replace( @@ -4114,7 +4394,7 @@ vc_vpc_observe_cleanup() {{ trap 'vc_vpc_observe_cleanup' EXIT trap 'exit 79' HUP INT TERM vc_vpc_fabric_observe_ready() {{ - vpc_state_load || return 1 + [ "$VPC_STATE_STATUS" = owned ] || return 1 vc_nft_table_snapshot inet virt_cloud_wg_fabric \ "$FABRIC_NFT_SNAPSHOT" || return 1 [ -f "$FABRIC_NFT_SNAPSHOT" ] && @@ -4383,18 +4663,30 @@ route_extra=$(printf '%s\\n' "$route_records" | awk -F'|' '{{ }} END {{ if (count != {len(routes)}) extra=1; print extra ? "yes" : "no" }}') printf '%s\\n' "__VC_VPC_ROUTE_EXTRA__=$route_extra" {route_checks} -while IFS='|' read -r peer_host key expected_allowed; do +vpc_state_loaded=no +if vpc_state_load && [ "$VPC_STATE_STATUS" = owned ]; then + vpc_state_loaded=yes +fi +while IFS='|' read -r peer_host key expected_overlay; do [ -n "$peer_host" ] || continue actual=$(vc_wg_allowed_for_peer "$WG_IFACE" "$key") || actual=invalid - # ``wg show`` does not promise AllowedIPs order. Canonicalize both - # normalized lists before comparing, and retain the complete set in - # the marker. The Python adapter projects this to this VPC's tenant /32s - # after proving that no extra or missing entry exists. + expected_allowed=$expected_overlay + state=missing + if [ "$vpc_state_loaded" = yes ]; then + state_allowed=$(vpc_state_allowed_for_peer "$key") || state_allowed=invalid + if [ "$state_allowed" != invalid ] && [ -n "$state_allowed" ]; then + expected_allowed="$expected_allowed,$state_allowed" + fi + fi + # ``wg show`` does not promise AllowedIPs order. Canonicalize the live set + # and the exact overlay + route-backed union derived by the shared state + # loader. The Python adapter separately proves this VPC's required routes + # are a subset before projecting its own tenant /32s. expected_set=$(printf '%s\\n' "$expected_allowed" | tr ',' '\\n' | awk 'NF' | LC_ALL=C sort -u) actual_set=$(printf '%s\\n' "$actual" | tr ',' '\\n' | awk 'NF' | LC_ALL=C sort -u) allowed=$(printf '%s\\n' "$actual_set" | awk 'BEGIN {{ ORS=""; sep="" }} NF {{ printf "%s%s", sep, $0; sep="," }}') - state=missing - if [ -n "$actual_set" ] && [ "$actual_set" = "$expected_set" ]; then + if [ "$vpc_state_loaded" = yes ] && [ "$state_allowed" != invalid ] && + [ -n "$actual_set" ] && [ "$actual_set" = "$expected_set" ]; then state=owned else allowed=missing @@ -4407,8 +4699,13 @@ done <<'{peer_marker}' expected_fabric_peer_keys={fabric_peer_key_arg} wg_peer_set=unverified if [ "{fabric_peer_keys_provided}" = yes ]; then - live_fabric_peer_keys=$("$WG" show "$WG_IFACE" peers 2>/dev/null || true) wg_peer_set=exact + if live_fabric_peer_keys=$("$WG" show "$WG_IFACE" peers 2>/dev/null); then + : + else + live_fabric_peer_keys= + wg_peer_set=missing + fi for live_peer_key in $live_fabric_peer_keys; do case " $expected_fabric_peer_keys " in *" $live_peer_key "*) : ;; @@ -4418,6 +4715,41 @@ if [ "{fabric_peer_keys_provided}" = yes ]; then for expected_peer_key in $expected_fabric_peer_keys; do printf '%s\n' "$live_fabric_peer_keys" | grep -Fqx "$expected_peer_key" || wg_peer_set=missing done + # Key presence alone cannot prove the shared cryptokey-routing boundary. + # A VPC revision may have withdrawn the final route for a peer that remains + # in the Fabric. Require every authoritative key to carry exactly one live + # overlay /32 plus only the route-backed tenant union from durable host + # state, including peers outside this VPC's current participant set. + if [ "$wg_peer_set" = exact ] && [ "$vpc_state_loaded" = yes ]; then + while IFS='|' read -r expected_peer_key expected_overlay; do + [ -n "$expected_peer_key" ] || continue + closure_actual=$(vc_wg_allowed_for_peer "$WG_IFACE" "$expected_peer_key") || {{ + wg_peer_set=missing + break + }} + vpc_state_overlay_from_allowed "$closure_actual" >/dev/null || {{ + wg_peer_set=missing + break + }} + closure_extra=$(vpc_state_allowed_for_peer "$expected_peer_key") || {{ + wg_peer_set=missing + break + }} + closure_expected="$expected_overlay" + [ -z "$closure_extra" ] || closure_expected="$closure_expected,$closure_extra" + closure_actual_set=$(printf '%s\n' "$closure_actual" | tr ',' '\n' | awk 'NF' | LC_ALL=C sort -u) + closure_expected_set=$(printf '%s\n' "$closure_expected" | tr ',' '\n' | awk 'NF' | LC_ALL=C sort -u) + if [ -z "$closure_actual_set" ] || + [ "$closure_actual_set" != "$closure_expected_set" ]; then + wg_peer_set=missing + break + fi + done <<'{fabric_peer_marker}' +{fabric_peer_overlay_lines} +{fabric_peer_marker} + elif [ "$wg_peer_set" = exact ]; then + wg_peer_set=missing + fi fi printf '%s\n' "__VC_VPC_WG_PEER_SET__=$wg_peer_set" fabric_nft_state=no @@ -4998,41 +5330,39 @@ class VpcPrivateHostAdapter: x.get("via"), str(x.get("metric", 100))) for x in parsed.get("routes", []) } - expected_wg_full = { - # The live `wg show` value includes the peer's overlay /32 as the - # first AllowedIP. Compare the complete cryptokey-routing set; - # project tenant entries only after that exact check. - p.host_id: (p.public_key, set(_allowed(plan, p).split(","))) - for p in plan.peers - } + wg_allowed_rows = parsed.get("wg_allowed_ips", []) actual_wg_full = { x.get("peer_host_id"): (x.get("public_key"), set(x.get("tenant_allowed_ips", []))) - for x in parsed.get("wg_allowed_ips", []) + for x in wg_allowed_rows } - # The live peer may carry /32s belonging to other VPCs sharing this - # Fabric. Verify the complete aggregate here, then expose only this - # VPC's tenant set to the PHP contract. + # Durable host state owns the complete live tenant union. The shell + # proves that union is exact; here retain the plan-level obligation: + # this VPC's peer key, overlay, members and service routes must all be + # present before the result can be accepted or projected to PHP. + required_tenant = _expected_peer_tenant_ips(plan.vpc) expected_tenant: dict[str, set[str]] = { - p.host_id: {m.private_ip + "/32" for m in plan.remote_members - if m.host_id == p.host_id} + p.host_id: set(required_tenant.get(p.host_id, frozenset())) for p in plan.peers } - members_by_id = {m.member_id: m for m in plan.members} - for route in plan.vpc.routes: - target = (members_by_id.get(route.next_hop_member_id) - if route.next_hop_member_id is not None else None) - target_host = target.host_id if target is not None else route.next_hop_host_id - if target_host in expected_tenant: - expected_tenant[target_host].add(route.destination_cidr) expected_wg = { p.host_id: (p.public_key, expected_tenant.get(p.host_id, set())) for p in plan.peers } - actual_wg = { - host: (key, ips & expected_wg.get(host, (None, set()))[1]) - for host, (key, ips) in actual_wg_full.items() + required_wg = { + p.host_id: (p.public_key, + {p.overlay_cidr} | expected_tenant.get(p.host_id, set())) + for p in plan.peers } + wg_required_verified = ( + len(wg_allowed_rows) == len(actual_wg_full) == len(required_wg) + and set(actual_wg_full) == set(required_wg) + and all( + actual_wg_full[host][0] == key + and required_ips.issubset(actual_wg_full[host][1]) + for host, (key, required_ips) in required_wg.items() + ) + ) members = _member_nics(plan, xml_by_instance) actual_route_state = { (x.get("destination_cidr"), x.get("device"), str(x.get("table")), @@ -5059,6 +5389,14 @@ class VpcPrivateHostAdapter: for x in parsed.get("rules", []) if x.get("state") == "owned" } expected_peer_states = {p.host_id: "owned" for p in plan.peers} + wg_state_marker_count = sum( + line.startswith("__VC_VPC_WG_STATE__=") + for line in output.splitlines() + ) + wg_peer_set_marker_count = sum( + line.startswith("__VC_VPC_WG_PEER_SET__=") + for line in output.splitlines() + ) expected_bridge_nft = "yes" if plan.local_members else "none" host_dataplane_verified = (parsed.get("bridge") == plan.resources.bridge and parsed.get("bridge_owner") == DATAPLANE_OWNER_MARKER @@ -5094,9 +5432,13 @@ class VpcPrivateHostAdapter: and actual_routes == actual_route_state and route_flags_valid and expected_rules == actual_rules - and expected_wg_full == actual_wg_full - and (plan.fabric_peer_keys is None - or parsed.get("wg_peer_set") == "exact") + and wg_required_verified + and ((plan.fabric_peer_keys is None + and plan.fabric_peer_overlays is None) + or (plan.fabric_peer_overlays is not None + and wg_peer_set_marker_count == 1 + and parsed.get("wg_peer_set") == "exact")) + and wg_state_marker_count == len(expected_peer_states) and expected_peer_states == parsed.get("wg_states", {}) and parsed.get("arp_nft") == ("yes" if plan.local_members else "none")) host_config_verified = host_dataplane_verified and all( @@ -5108,7 +5450,7 @@ class VpcPrivateHostAdapter: and parsed.get("firewall_mode") == firewall_mode and parsed.get("firewall_state") == "owned" and parsed.get("state_firewall_mode") == firewall_mode - and plan.fabric_peer_keys is not None + and plan.fabric_peer_overlays is not None and parsed.get("wg_peer_set") == "exact" and parsed.get("helper_contract") == "owned" and parsed.get("bridge_admin") == "up" diff --git a/web/app/Services/Cloud/CloudVpcService.php b/web/app/Services/Cloud/CloudVpcService.php index b8438dea..15e97004 100644 --- a/web/app/Services/Cloud/CloudVpcService.php +++ b/web/app/Services/Cloud/CloudVpcService.php @@ -1098,6 +1098,12 @@ final class CloudVpcService } [$operation, $payload, $runtimeProofRefresh, $previousOutcomePending] = $claim; $leaseToken = (string) $operation->lease_token; + // Only the explicit user reconcile route may transfer an exact, + // childless retained Engine root for this immutable target. Keep the + // capability inside the signed actor and off worker/power requests. + $engineActor = $allowFailedRetry + ? array_replace($actor, ['vpc_reconcile_explicit' => true]) + : $actor; try { // A previous apply may have finished while its PHP receipt was // lost. Recover that exact proof before requiring a stopped guest @@ -1118,7 +1124,7 @@ final class CloudVpcService null, self::ENGINE_TIMEOUT_SECONDS ))->vpc( 'POST', '/internal/v1/vpcs/' . $vpcKey . '/delete', - $actor, $payload, (string) $operation->operation_id + $engineActor, $payload, (string) $operation->operation_id ); $expectedRevision = (int) ($payload['desired_revision'] ?? 0); $expectedDigest = (string) ($payload['desired_digest'] ?? ''); @@ -1146,7 +1152,7 @@ final class CloudVpcService $result = (new CloudEngineClient( null, self::ENGINE_TIMEOUT_SECONDS ))->vpc('POST', '/internal/v1/vpcs/apply', - $actor, $payload, (string) $operation->operation_id); + $engineActor, $payload, (string) $operation->operation_id); $observed = self::applyObservation($operation, $payload, $result); self::recordObserved($vpcKey, (int) ($payload['desired_revision'] ?? 0), (string) ($payload['desired_digest'] ?? ''), $observed, diff --git a/web/tests/CloudVpcFailedReconcileTest.php b/web/tests/CloudVpcFailedReconcileTest.php index 4a213bde..06ca174f 100644 --- a/web/tests/CloudVpcFailedReconcileTest.php +++ b/web/tests/CloudVpcFailedReconcileTest.php @@ -14,7 +14,7 @@ namespace App\Services\Cloud { public function vpc(string $method, string $path, array $actor, ?array $body, ?string $operationId): array { - self::$requests[] = compact('method', 'path', 'body', 'operationId'); + self::$requests[] = compact('method', 'path', 'actor', 'body', 'operationId'); return (self::$respond)($body); } } @@ -63,6 +63,7 @@ foreach (['vpc_create', 'member_attach', 'member_detach'] as $index => $kind) { $after = Capsule::table('cloud_vpc_operations') ->where('operation_id', $vpc['operation_id'])->first(); $assert($request['operationId'] === $vpc['operation_id'] + && ($request['actor']['vpc_reconcile_explicit'] ?? null) === true && $request['body'] === json_decode($before->payload_json, true) && $after->payload_json === $before->payload_json && $after->status === 'uncertain' @@ -297,6 +298,7 @@ $expectCloudError(static fn () => (new \App\Controllers\CloudVpcController())->r ), 502); $assert(count(CloudEngineClient::$requests) === $requestCount + 1 && end(CloudEngineClient::$requests)['operationId'] === $replacementId + && (end(CloudEngineClient::$requests)['actor']['vpc_reconcile_explicit'] ?? null) === true && end(CloudEngineClient::$requests)['body'] === $replacementPayload, 'The explicit user reconcile did not retry the exact failed target.');