16 KiB
Commercial line traffic-pool rollout and recovery runbook
This runbook is the operational boundary for enabling the shared traffic pool.
The accounting contract and FEFO rules remain defined in
TRAFFIC_POOL.md; this document covers deployment,
acceptance and recovery only.
1. Database and process prerequisites
php xcat cloudInit is a fresh-schema installer and validator. It is not
an incremental database upgrade command. A fresh deployment may run:
cd /home/virt-cloud/web
php xcat cloudInit
only after importing web/sql/ss.sql into a new business database and
configuring a new log database, enable_log=true and an empty db_prefix.
An installed database must already have the exact schema marker and
fingerprint required by the same source revision.
Do not run cloudInit to upgrade an existing installation. An existing
database needs a separately reviewed, one-time migration for the exact source
and database versions. Never make such a database appear current by editing
cloud_schema_version or cloud_schema_fingerprint.
The deployed panel source, schema and long-lived PHP processes must come from one release. The following units are required for accounting, network policy and operation-bound recovery:
virt-cloud-lifecycle.service;virt-cloud-worker.service;- at least
virt-cloud-policy-worker@1.serviceandvirt-cloud-policy-worker@2.service; virt-cloud-recovery-worker.service.
The source commands behind those units are:
php xcat cloudLifecycle
php xcat cloudWorker
php xcat cloudWorker --policy-only
php xcat cloudWorker --recovery-only
php xcat cloudLifecycle --once is available for a bounded non-production
acceptance pass, but it is not a read-only health check: it also advances
billing, backup, traffic and instance lifecycle state.
2. Clean identity boundary; no automatic backfill
traffic_pool_group_id is the global commercial line identity. A group may
bind region-local lines from Hong Kong, Singapore or any other configured
region, and all of those regions consume one user/group FEFO balance.
Region is coverage metadata only; matching line names or codes do not create
identity.
The line-oriented user and administrator pages require no schema migration and must not trigger an accounting backfill. Existing data is handled as follows:
| Existing state | Required action |
|---|---|
disabled, no durable reference |
Bind all intended region-local lines through the revision-checked administrator operation. |
shadow or enforced |
Preserve the current group and line membership; do not merge groups or rebind lines with SQL. |
| Legacy period with no group identity | Keep legacy per-instance accounting until the period finishes. |
| New purchase or renewal already carrying a group | Keep the frozen identity; a region or host move must not rewrite it. |
The line-to-pool mapping is frozen into a new order and service period. A renewal inherits the preceding period's identity; it does not read the current line mapping again. Therefore:
- create the traffic-pool group and bind all intended network lines before accepting new orders on those lines;
- leave already-created old service periods on legacy accounting;
- do not accept payment or attempt delivery for an old unfinished order that
lacks the required frozen
traffic_pool_group_idandtraffic_quota_mode; let an unpaid invoice expire, or resolve a paid/failed order through its original task and the existing order/refund workflow before placing a new order from the current release; - do not update an active or scheduled period's
traffic_pool_group_id; - do not edit an order's frozen
spec_jsonor manufacture a pool, entitlement, cursor or ledger event to make an old period appear migrated; - do not use renewal as a migration mechanism: a legacy
NULLidentity is inherited asNULL.
If a mapped line still has a pending order or a finite
scheduled/active/exhausted period with a missing or different frozen
identity, readiness reports line_frozen_identity_mismatch and enforcement is
blocked. In the test environment, let such periods finish without renewal or
start again with a clean test database. Keep a legacy instance under lifecycle
management until every old scheduled/active/exhausted period is finished;
only then may the instance be removed and recreated through the normal product
workflow. Deleting it first is not a shortcut because deleted instances are no
longer advanced by lifecycle while their unfinished periods still block
readiness. An old unfinished order without the complete snapshot must be
expired or safely resolved; provisioning rejects it before Agent dispatch, so
it is not a deliverable legacy order. A previously deleted instance with such
an unfinished period needs a separately reviewed one-time data repair. Do not
repair either condition by an unreviewed SQL backfill.
3. Rollout state machine
Every group must advance in order:
disabled -> shadow -> enforced
3.1 Disabled: configure the commercial identity
Open /admin/resources?view=traffic, create a line pool and bind every
region-local network line that represents that commercial line while it is
disabled.
- A group needs at least one line before entering shadow mode.
- One line can belong to only one group.
- One group may cover multiple regions; verify both the coverage-region count and the exact bound-line list before creating orders.
- Line membership can be edited only while the group is disabled.
- Record the group name, ID, revision and bound line IDs in the release log.
- Finish mapping before creating any order, instance or service-period reference for those lines.
3.2 Shadow: build and verify the real ledger
Advance the group to shadow. The lifecycle process then performs, in order:
- collection and persistence of trusted cumulative public-interface traffic;
- settlement of pending usage into the shared pool;
- activation, expiry and closing of entitlements;
- ordinary runtime and per-instance lifecycle reconciliation.
Shadow mode creates the real pool, entitlement, cursor and immutable ledger
records, but it does not add or remove the traffic_pool_exhausted network
restriction. Once a service period or pool is durably linked to the group,
shadow mode cannot be returned to disabled; failures must be repaired forward.
Keep the group in shadow for a representative usage and renewal window. At a minimum, observe several collection intervals, a multi-instance allocation, an entitlement boundary and repeated lifecycle ticks before enforcement.
3.3 Readiness: every blocker must be zero
Do not enable enforcement until the administrator response for the same group shows all three conditions:
ready = true
can_enforce = true
blockers = []
The readiness gate covers:
no_network_lines;line_frozen_identity_mismatch;pool_attentionandprojection_invalid;missing_entitlementsandmissing_pools;period_identity_mismatch;instance_not_readyandlegacy_network_unsettled;usage_unsettled;cursor_missingandcursor_attention;traffic_sample_stale(the normal freshness limit is 300 seconds);traffic_scan_failures;policy_unsettled;traffic_restriction_present;network_job_unsettled.
Fix the underlying sampling, identity, accounting or network-policy condition.
Normal lifecycle/recovery processes may then converge ordinary unsettled
records. pool_attention and cursor_attention are different: they are not
cleared automatically. They can be inspected and resolved only through the
local, audited break-glass CLI described in section 5.3; there is no HTTP or UI
resolve endpoint. Keep the group in its current safe mode (do not leave
shadow) until the immutable ledger, current counters and exact pool version
pass full verification. Never clear a blocker by deleting evidence or changing
a projection total.
3.4 Enforced: irreversible network policy
The administrator must review the empty blocker list, use the latest group
revision and type the complete group name in the confirmation dialog. Only
then advance the group to enforced.
Enforced mode is irreversible. It cannot be changed back to shadow or disabled, and its bound lines cannot be changed. This is intentional: after multiple instances have consumed shared entitlements, a per-instance balance cannot be reconstructed safely.
4. Acceptance checklist
Complete these checks in shadow where possible, then repeat the network-policy checks in enforced mode with controlled test instances:
- multiple finite instances owned by one user and group share one FEFO balance;
- instances in different regions but the same commercial line group share that one FEFO balance;
- different groups and different users never share quota;
- equal line names or codes in different groups remain isolated;
- an unlimited period neither contributes to nor consumes a finite pool;
- deleting one instance does not delete its unexpired paid entitlement;
- period expiry forfeits only that entitlement's unused balance;
- sampled excess is retained as
overage_bytes, and later quota settles the debt before exposing usable balance; - exhausting a pool adds the traffic restriction to every active finite instance in that user/group;
- new usable quota removes only
traffic_pool_exhausted, neverservice_expiredoradmin_hold; - the user page exposes only enforced groups and never presents shadow data as an active shared allowance;
- pool or cursor attention prevents an untrusted automatic network restore;
- repeated lifecycle ticks neither double-charge usage nor create duplicate network mutations;
- an Agent outage or unknown sample fails open for traffic accounting while ordinary service expiry remains enforceable.
Static and local checks do not replace a controlled Agent/network acceptance test. Record the release commit, group revision, involved instance IDs, sample timestamps, task IDs and operation IDs.
5. Incident response and forward recovery
5.1 Shadow incident
Keep the group in shadow and stop opening new sales on the affected lines through the existing commercial/scheduling controls. Continue the lifecycle and recovery processes so expiry and unrelated business state still advance. Repair the real cause, then wait for the readiness gate to become empty.
Returning shadow to disabled is allowed only before any service period or pool durably references the group. Do not remove those references to force a rollback.
5.2 Enforced incident
Pause new network mutation dispatch while preserving accounting and recovery:
stop virt-cloud-worker.service and every installed
virt-cloud-policy-worker@N.service. Stopping only policy workers is
insufficient because the ordinary worker is their fail-safe consumer. Keep
virt-cloud-lifecycle.service and virt-cloud-recovery-worker.service
running; lifecycle may persist target-state work, while recovery continues to
inspect already-dispatched operations without replaying them. This emergency
pause also stops unrelated ordinary jobs, so record its start and restore the
workers only after the original effects, accounting projection and queued
target state have been reviewed.
Do not stop lifecycle as a way to pause enforcement, and do not attempt a per-instance rollback. Preserve the immutable ledger and repair or rebuild its derived projections forward under a separately reviewed procedure.
Never:
- delete or rewrite pool, entitlement, ledger, cursor, restriction or policy rows to hide an inconsistency;
- reset cumulative instance counters;
- replay
network_suspendornetwork_resumeafter a timeout, disconnect, malformed receipt or empty response; - clear a policy blocker, release a reservation or issue a refund without the operation-bound evidence required by the original task.
For an unknown remote effect, retain the original operation_id and task.
Let virt-cloud-recovery-worker.service inspect and converge that exact
operation. A later explicit operation is allowed only after the original
effect has been resolved and its safety fence has been released by the normal
finalizer.
5.3 Attention inspection and audited resolution
The CLI is local to the panel host and does not contact an Agent, create a CloudJob or directly edit a network restriction. Shell access is the break-glass authorization boundary. Record the external incident/change ID, executing host, SSH/sudo operator, reason, pool ID, pool version and UUIDv4 in the release log.
Always use this sequence:
-
Run the read-only inspection:
cd /home/virt-cloud/web php xcat cloudTrafficPoolAttention inspect --pool-id=N -
Repair the real counter or projection cause. Do not edit the pool state, ledger, cursor totals or restriction rows.
-
Run
inspectagain. Use only this fresh response'spool_id,version,projection_valid, attention counts and categories. If the pool changes during inspection, the command fails and must be run again. -
Generate a new canonical lowercase UUIDv4 and bind it to the external incident/change record. Build the confirmation from that exact pool ID, fresh version and UUID.
-
Execute one explicit resolution attempt:
php xcat cloudTrafficPoolAttention resolve \ --pool-id=N \ --expected-version=V \ --resolution-key=UUID \ --apply \ --confirm=RESOLVE_TRAFFIC_POOL:N:V:UUID
The command still fails closed if the version changed, the projection is
invalid, a counter has not caught up, the UUID was already used, or the real
attention cause remains. A successful transaction appends one immutable
attention_resolved ledger event and recomputes the projection. It does not
perform a remote network mutation.
If stdout, SSH or the command response is lost, do not repeat resolve.
First query the immutable UUID:
php xcat cloudTrafficPoolAttention inspect-resolution \
--resolution-key=UUID
committed=true proves which pool and expected version were resolved; do not
submit it again. committed=false proves that UUID has no committed resolution,
but it is not permission to retry blindly: run a fresh pool inspect and
repeat the exact command with the same UUID only if the pool is still in
attention at the same expected version. Never use one UUID for a different
pool or incident.
For an enforced pool, resolving attention may make the projection trustworthy and allow the next lifecycle pass to request network restoration. Keep the ordinary and policy workers paused as described in section 5.2 until the resolved projection, active restrictions and queued target state have been reviewed; then restore the workers and observe the original task flow.
5.4 Rollback matrix
| State | Operational action |
|---|---|
disabled, no durable reference |
Edit line mapping or abandon the group. |
shadow, no durable reference |
Return to disabled if required. |
shadow, durable period/pool reference |
Keep shadow and repair forward. |
enforced |
No rollback; preserve the ledger and repair forward. |
| Network operation has unknown effect | Reconcile the original task and operation_id; never replay it. |
6. Release evidence
Before marking rollout complete, retain:
- the exact Git commit and schema marker/fingerprint used by the fresh installation or separately approved one-time migration;
- service status and recent logs for lifecycle, ordinary, policy and recovery workers;
- the commercial line group, coverage regions, exact bound-line mapping and revision;
- the complete readiness response immediately before enforcement;
- acceptance task IDs and operation IDs;
- any skipped live-host or failure-injection test.
If live Agent/network acceptance was not completed, report only that static and local tests passed; do not claim the shared traffic pool is production ready.