You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
virt/docs/TRAFFIC_POOL_ROLLOUT.md

358 lines
16 KiB
Markdown

# Commercial line traffic-pool rollout and recovery runbook
This runbook is the operational boundary for enabling the shared traffic pool.
The accounting contract and FEFO rules remain defined in
[`TRAFFIC_POOL.md`](TRAFFIC_POOL.md); this document covers deployment,
acceptance and recovery only.
## 1. Database and process prerequisites
`php xcat cloudInit` is a fresh-schema installer and validator. It is **not**
an incremental database upgrade command. A fresh deployment may run:
```bash
cd /home/virt-cloud/web
php xcat cloudInit
```
only after importing `web/sql/ss.sql` into a new business database and
configuring a new log database, `enable_log=true` and an empty `db_prefix`.
An installed database must already have the exact schema marker and
fingerprint required by the same source revision.
Do not run `cloudInit` to upgrade an existing installation. An existing
database needs a separately reviewed, one-time migration for the exact source
and database versions. Never make such a database appear current by editing
`cloud_schema_version` or `cloud_schema_fingerprint`.
The deployed panel source, schema and long-lived PHP processes must come from
one release. The following units are required for accounting, network policy
and operation-bound recovery:
- `virt-cloud-lifecycle.service`;
- `virt-cloud-worker.service`;
- at least `virt-cloud-policy-worker@1.service` and
`virt-cloud-policy-worker@2.service`;
- `virt-cloud-recovery-worker.service`.
The source commands behind those units are:
```bash
php xcat cloudLifecycle
php xcat cloudWorker
php xcat cloudWorker --policy-only
php xcat cloudWorker --recovery-only
```
`php xcat cloudLifecycle --once` is available for a bounded non-production
acceptance pass, but it is not a read-only health check: it also advances
billing, backup, traffic and instance lifecycle state.
## 2. Clean identity boundary; no automatic backfill
`traffic_pool_group_id` is the global commercial line identity. A group may
bind region-local lines from Hong Kong, Singapore or any other configured
region, and all of those regions consume one user/group FEFO balance.
Region is coverage metadata only; matching line names or codes do not create
identity.
The line-oriented user and administrator pages require no schema migration and
must not trigger an accounting backfill. Existing data is handled as follows:
| Existing state | Required action |
| --- | --- |
| `disabled`, no durable reference | Bind all intended region-local lines through the revision-checked administrator operation. |
| `shadow` or `enforced` | Preserve the current group and line membership; do not merge groups or rebind lines with SQL. |
| Legacy period with no group identity | Keep legacy per-instance accounting until the period finishes. |
| New purchase or renewal already carrying a group | Keep the frozen identity; a region or host move must not rewrite it. |
The line-to-pool mapping is frozen into a new order and service period. A
renewal inherits the preceding period's identity; it does not read the current
line mapping again. Therefore:
1. create the traffic-pool group and bind all intended network lines before
accepting new orders on those lines;
2. leave already-created old service periods on legacy accounting;
3. do not accept payment or attempt delivery for an old unfinished order that
lacks the required frozen `traffic_pool_group_id` and
`traffic_quota_mode`; let an unpaid invoice expire, or resolve a paid/failed
order through its original task and the existing order/refund workflow
before placing a new order from the current release;
4. do not update an active or scheduled period's `traffic_pool_group_id`;
5. do not edit an order's frozen `spec_json` or manufacture a pool,
entitlement, cursor or ledger event to make an old period appear migrated;
6. do not use renewal as a migration mechanism: a legacy `NULL` identity is
inherited as `NULL`.
If a mapped line still has a pending order or a finite
`scheduled`/`active`/`exhausted` period with a missing or different frozen
identity, readiness reports `line_frozen_identity_mismatch` and enforcement is
blocked. In the test environment, let such periods finish without renewal or
start again with a clean test database. Keep a legacy instance under lifecycle
management until every old scheduled/active/exhausted period is `finished`;
only then may the instance be removed and recreated through the normal product
workflow. Deleting it first is not a shortcut because deleted instances are no
longer advanced by lifecycle while their unfinished periods still block
readiness. An old unfinished order without the complete snapshot must be
expired or safely resolved; provisioning rejects it before Agent dispatch, so
it is not a deliverable legacy order. A previously deleted instance with such
an unfinished period needs a separately reviewed one-time data repair. Do not
repair either condition by an unreviewed SQL backfill.
## 3. Rollout state machine
Every group must advance in order:
```text
disabled -> shadow -> enforced
```
### 3.1 Disabled: configure the commercial identity
Open `/admin/resources?view=traffic`, create a line pool and bind every
region-local network line that represents that commercial line while it is
`disabled`.
- A group needs at least one line before entering shadow mode.
- One line can belong to only one group.
- One group may cover multiple regions; verify both the coverage-region count
and the exact bound-line list before creating orders.
- Line membership can be edited only while the group is disabled.
- Record the group name, ID, revision and bound line IDs in the release log.
- Finish mapping before creating any order, instance or service-period
reference for those lines.
### 3.2 Shadow: build and verify the real ledger
Advance the group to `shadow`. The lifecycle process then performs, in order:
1. collection and persistence of trusted cumulative public-interface traffic;
2. settlement of pending usage into the shared pool;
3. activation, expiry and closing of entitlements;
4. ordinary runtime and per-instance lifecycle reconciliation.
Shadow mode creates the real pool, entitlement, cursor and immutable ledger
records, but it does not add or remove the `traffic_pool_exhausted` network
restriction. Once a service period or pool is durably linked to the group,
shadow mode cannot be returned to disabled; failures must be repaired forward.
Keep the group in shadow for a representative usage and renewal window. At a
minimum, observe several collection intervals, a multi-instance allocation,
an entitlement boundary and repeated lifecycle ticks before enforcement.
### 3.3 Readiness: every blocker must be zero
Do not enable enforcement until the administrator response for the same group
shows all three conditions:
```text
ready = true
can_enforce = true
blockers = []
```
The readiness gate covers:
- `no_network_lines`;
- `line_frozen_identity_mismatch`;
- `pool_attention` and `projection_invalid`;
- `missing_entitlements` and `missing_pools`;
- `period_identity_mismatch`;
- `instance_not_ready` and `legacy_network_unsettled`;
- `usage_unsettled`;
- `cursor_missing` and `cursor_attention`;
- `traffic_sample_stale` (the normal freshness limit is 300 seconds);
- `traffic_scan_failures`;
- `policy_unsettled`;
- `traffic_restriction_present`;
- `network_job_unsettled`.
Fix the underlying sampling, identity, accounting or network-policy condition.
Normal lifecycle/recovery processes may then converge ordinary unsettled
records. `pool_attention` and `cursor_attention` are different: they are not
cleared automatically. They can be inspected and resolved only through the
local, audited break-glass CLI described in section 5.3; there is no HTTP or UI
resolve endpoint. Keep the group in its current safe mode (do not leave
shadow) until the immutable ledger, current counters and exact pool version
pass full verification. Never clear a blocker by deleting evidence or changing
a projection total.
### 3.4 Enforced: irreversible network policy
The administrator must review the empty blocker list, use the latest group
revision and type the complete group name in the confirmation dialog. Only
then advance the group to `enforced`.
Enforced mode is irreversible. It cannot be changed back to shadow or
disabled, and its bound lines cannot be changed. This is intentional: after
multiple instances have consumed shared entitlements, a per-instance balance
cannot be reconstructed safely.
## 4. Acceptance checklist
Complete these checks in shadow where possible, then repeat the network-policy
checks in enforced mode with controlled test instances:
- multiple finite instances owned by one user and group share one FEFO balance;
- instances in different regions but the same commercial line group share that
one FEFO balance;
- different groups and different users never share quota;
- equal line names or codes in different groups remain isolated;
- an unlimited period neither contributes to nor consumes a finite pool;
- deleting one instance does not delete its unexpired paid entitlement;
- period expiry forfeits only that entitlement's unused balance;
- sampled excess is retained as `overage_bytes`, and later quota settles the
debt before exposing usable balance;
- exhausting a pool adds the traffic restriction to every active finite
instance in that user/group;
- new usable quota removes only `traffic_pool_exhausted`, never
`service_expired` or `admin_hold`;
- the user page exposes only enforced groups and never presents shadow data as
an active shared allowance;
- pool or cursor attention prevents an untrusted automatic network restore;
- repeated lifecycle ticks neither double-charge usage nor create duplicate
network mutations;
- an Agent outage or unknown sample fails open for traffic accounting while
ordinary service expiry remains enforceable.
Static and local checks do not replace a controlled Agent/network acceptance
test. Record the release commit, group revision, involved instance IDs, sample
timestamps, task IDs and operation IDs.
## 5. Incident response and forward recovery
### 5.1 Shadow incident
Keep the group in shadow and stop opening new sales on the affected lines
through the existing commercial/scheduling controls. Continue the lifecycle
and recovery processes so expiry and unrelated business state still advance.
Repair the real cause, then wait for the readiness gate to become empty.
Returning shadow to disabled is allowed only before any service period or pool
durably references the group. Do not remove those references to force a
rollback.
### 5.2 Enforced incident
Pause new network mutation dispatch while preserving accounting and recovery:
stop `virt-cloud-worker.service` and every installed
`virt-cloud-policy-worker@N.service`. Stopping only policy workers is
insufficient because the ordinary worker is their fail-safe consumer. Keep
`virt-cloud-lifecycle.service` and `virt-cloud-recovery-worker.service`
running; lifecycle may persist target-state work, while recovery continues to
inspect already-dispatched operations without replaying them. This emergency
pause also stops unrelated ordinary jobs, so record its start and restore the
workers only after the original effects, accounting projection and queued
target state have been reviewed.
Do not stop lifecycle as a way to pause enforcement, and do not attempt a
per-instance rollback. Preserve the immutable ledger and repair or rebuild its
derived projections forward under a separately reviewed procedure.
Never:
- delete or rewrite pool, entitlement, ledger, cursor, restriction or policy
rows to hide an inconsistency;
- reset cumulative instance counters;
- replay `network_suspend` or `network_resume` after a timeout, disconnect,
malformed receipt or empty response;
- clear a policy blocker, release a reservation or issue a refund without the
operation-bound evidence required by the original task.
For an unknown remote effect, retain the original `operation_id` and task.
Let `virt-cloud-recovery-worker.service` inspect and converge that exact
operation. A later explicit operation is allowed only after the original
effect has been resolved and its safety fence has been released by the normal
finalizer.
### 5.3 Attention inspection and audited resolution
The CLI is local to the panel host and does not contact an Agent, create a
CloudJob or directly edit a network restriction. Shell access is the
break-glass authorization boundary. Record the external incident/change ID,
executing host, SSH/sudo operator, reason, pool ID, pool version and UUIDv4 in
the release log.
Always use this sequence:
1. Run the read-only inspection:
```bash
cd /home/virt-cloud/web
php xcat cloudTrafficPoolAttention inspect --pool-id=N
```
2. Repair the real counter or projection cause. Do not edit the pool state,
ledger, cursor totals or restriction rows.
3. Run `inspect` again. Use only this fresh response's `pool_id`, `version`,
`projection_valid`, attention counts and categories. If the pool changes
during inspection, the command fails and must be run again.
4. Generate a new canonical lowercase UUIDv4 and bind it to the external
incident/change record. Build the confirmation from that exact pool ID,
fresh version and UUID.
5. Execute one explicit resolution attempt:
```bash
php xcat cloudTrafficPoolAttention resolve \
--pool-id=N \
--expected-version=V \
--resolution-key=UUID \
--apply \
--confirm=RESOLVE_TRAFFIC_POOL:N:V:UUID
```
The command still fails closed if the version changed, the projection is
invalid, a counter has not caught up, the UUID was already used, or the real
attention cause remains. A successful transaction appends one immutable
`attention_resolved` ledger event and recomputes the projection. It does not
perform a remote network mutation.
If stdout, SSH or the command response is lost, do not repeat `resolve`.
First query the immutable UUID:
```bash
php xcat cloudTrafficPoolAttention inspect-resolution \
--resolution-key=UUID
```
`committed=true` proves which pool and expected version were resolved; do not
submit it again. `committed=false` proves that UUID has no committed resolution,
but it is not permission to retry blindly: run a fresh pool `inspect` and
repeat the exact command with the same UUID only if the pool is still in
attention at the same expected version. Never use one UUID for a different
pool or incident.
For an enforced pool, resolving attention may make the projection trustworthy
and allow the next lifecycle pass to request network restoration. Keep the
ordinary and policy workers paused as described in section 5.2 until the
resolved projection, active restrictions and queued target state have been
reviewed; then restore the workers and observe the original task flow.
### 5.4 Rollback matrix
| State | Operational action |
| --- | --- |
| `disabled`, no durable reference | Edit line mapping or abandon the group. |
| `shadow`, no durable reference | Return to disabled if required. |
| `shadow`, durable period/pool reference | Keep shadow and repair forward. |
| `enforced` | No rollback; preserve the ledger and repair forward. |
| Network operation has unknown effect | Reconcile the original task and `operation_id`; never replay it. |
## 6. Release evidence
Before marking rollout complete, retain:
- the exact Git commit and schema marker/fingerprint used by the fresh
installation or separately approved one-time migration;
- service status and recent logs for lifecycle, ordinary, policy and recovery
workers;
- the commercial line group, coverage regions, exact bound-line mapping and
revision;
- the complete readiness response immediately before enforcement;
- acceptance task IDs and operation IDs;
- any skipped live-host or failure-injection test.
If live Agent/network acceptance was not completed, report only that static
and local tests passed; do not claim the shared traffic pool is production
ready.