You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
358 lines
16 KiB
Markdown
358 lines
16 KiB
Markdown
# Commercial line traffic-pool rollout and recovery runbook
|
|
|
|
This runbook is the operational boundary for enabling the shared traffic pool.
|
|
The accounting contract and FEFO rules remain defined in
|
|
[`TRAFFIC_POOL.md`](TRAFFIC_POOL.md); this document covers deployment,
|
|
acceptance and recovery only.
|
|
|
|
## 1. Database and process prerequisites
|
|
|
|
`php xcat cloudInit` is a fresh-schema installer and validator. It is **not**
|
|
an incremental database upgrade command. A fresh deployment may run:
|
|
|
|
```bash
|
|
cd /home/virt-cloud/web
|
|
php xcat cloudInit
|
|
```
|
|
|
|
only after importing `web/sql/ss.sql` into a new business database and
|
|
configuring a new log database, `enable_log=true` and an empty `db_prefix`.
|
|
An installed database must already have the exact schema marker and
|
|
fingerprint required by the same source revision.
|
|
|
|
Do not run `cloudInit` to upgrade an existing installation. An existing
|
|
database needs a separately reviewed, one-time migration for the exact source
|
|
and database versions. Never make such a database appear current by editing
|
|
`cloud_schema_version` or `cloud_schema_fingerprint`.
|
|
|
|
The deployed panel source, schema and long-lived PHP processes must come from
|
|
one release. The following units are required for accounting, network policy
|
|
and operation-bound recovery:
|
|
|
|
- `virt-cloud-lifecycle.service`;
|
|
- `virt-cloud-worker.service`;
|
|
- at least `virt-cloud-policy-worker@1.service` and
|
|
`virt-cloud-policy-worker@2.service`;
|
|
- `virt-cloud-recovery-worker.service`.
|
|
|
|
The source commands behind those units are:
|
|
|
|
```bash
|
|
php xcat cloudLifecycle
|
|
php xcat cloudWorker
|
|
php xcat cloudWorker --policy-only
|
|
php xcat cloudWorker --recovery-only
|
|
```
|
|
|
|
`php xcat cloudLifecycle --once` is available for a bounded non-production
|
|
acceptance pass, but it is not a read-only health check: it also advances
|
|
billing, backup, traffic and instance lifecycle state.
|
|
|
|
## 2. Clean identity boundary; no automatic backfill
|
|
|
|
`traffic_pool_group_id` is the global commercial line identity. A group may
|
|
bind region-local lines from Hong Kong, Singapore or any other configured
|
|
region, and all of those regions consume one user/group FEFO balance.
|
|
Region is coverage metadata only; matching line names or codes do not create
|
|
identity.
|
|
|
|
The line-oriented user and administrator pages require no schema migration and
|
|
must not trigger an accounting backfill. Existing data is handled as follows:
|
|
|
|
| Existing state | Required action |
|
|
| --- | --- |
|
|
| `disabled`, no durable reference | Bind all intended region-local lines through the revision-checked administrator operation. |
|
|
| `shadow` or `enforced` | Preserve the current group and line membership; do not merge groups or rebind lines with SQL. |
|
|
| Legacy period with no group identity | Keep legacy per-instance accounting until the period finishes. |
|
|
| New purchase or renewal already carrying a group | Keep the frozen identity; a region or host move must not rewrite it. |
|
|
|
|
The line-to-pool mapping is frozen into a new order and service period. A
|
|
renewal inherits the preceding period's identity; it does not read the current
|
|
line mapping again. Therefore:
|
|
|
|
1. create the traffic-pool group and bind all intended network lines before
|
|
accepting new orders on those lines;
|
|
2. leave already-created old service periods on legacy accounting;
|
|
3. do not accept payment or attempt delivery for an old unfinished order that
|
|
lacks the required frozen `traffic_pool_group_id` and
|
|
`traffic_quota_mode`; let an unpaid invoice expire, or resolve a paid/failed
|
|
order through its original task and the existing order/refund workflow
|
|
before placing a new order from the current release;
|
|
4. do not update an active or scheduled period's `traffic_pool_group_id`;
|
|
5. do not edit an order's frozen `spec_json` or manufacture a pool,
|
|
entitlement, cursor or ledger event to make an old period appear migrated;
|
|
6. do not use renewal as a migration mechanism: a legacy `NULL` identity is
|
|
inherited as `NULL`.
|
|
|
|
If a mapped line still has a pending order or a finite
|
|
`scheduled`/`active`/`exhausted` period with a missing or different frozen
|
|
identity, readiness reports `line_frozen_identity_mismatch` and enforcement is
|
|
blocked. In the test environment, let such periods finish without renewal or
|
|
start again with a clean test database. Keep a legacy instance under lifecycle
|
|
management until every old scheduled/active/exhausted period is `finished`;
|
|
only then may the instance be removed and recreated through the normal product
|
|
workflow. Deleting it first is not a shortcut because deleted instances are no
|
|
longer advanced by lifecycle while their unfinished periods still block
|
|
readiness. An old unfinished order without the complete snapshot must be
|
|
expired or safely resolved; provisioning rejects it before Agent dispatch, so
|
|
it is not a deliverable legacy order. A previously deleted instance with such
|
|
an unfinished period needs a separately reviewed one-time data repair. Do not
|
|
repair either condition by an unreviewed SQL backfill.
|
|
|
|
## 3. Rollout state machine
|
|
|
|
Every group must advance in order:
|
|
|
|
```text
|
|
disabled -> shadow -> enforced
|
|
```
|
|
|
|
### 3.1 Disabled: configure the commercial identity
|
|
|
|
Open `/admin/resources?view=traffic`, create a line pool and bind every
|
|
region-local network line that represents that commercial line while it is
|
|
`disabled`.
|
|
|
|
- A group needs at least one line before entering shadow mode.
|
|
- One line can belong to only one group.
|
|
- One group may cover multiple regions; verify both the coverage-region count
|
|
and the exact bound-line list before creating orders.
|
|
- Line membership can be edited only while the group is disabled.
|
|
- Record the group name, ID, revision and bound line IDs in the release log.
|
|
- Finish mapping before creating any order, instance or service-period
|
|
reference for those lines.
|
|
|
|
### 3.2 Shadow: build and verify the real ledger
|
|
|
|
Advance the group to `shadow`. The lifecycle process then performs, in order:
|
|
|
|
1. collection and persistence of trusted cumulative public-interface traffic;
|
|
2. settlement of pending usage into the shared pool;
|
|
3. activation, expiry and closing of entitlements;
|
|
4. ordinary runtime and per-instance lifecycle reconciliation.
|
|
|
|
Shadow mode creates the real pool, entitlement, cursor and immutable ledger
|
|
records, but it does not add or remove the `traffic_pool_exhausted` network
|
|
restriction. Once a service period or pool is durably linked to the group,
|
|
shadow mode cannot be returned to disabled; failures must be repaired forward.
|
|
|
|
Keep the group in shadow for a representative usage and renewal window. At a
|
|
minimum, observe several collection intervals, a multi-instance allocation,
|
|
an entitlement boundary and repeated lifecycle ticks before enforcement.
|
|
|
|
### 3.3 Readiness: every blocker must be zero
|
|
|
|
Do not enable enforcement until the administrator response for the same group
|
|
shows all three conditions:
|
|
|
|
```text
|
|
ready = true
|
|
can_enforce = true
|
|
blockers = []
|
|
```
|
|
|
|
The readiness gate covers:
|
|
|
|
- `no_network_lines`;
|
|
- `line_frozen_identity_mismatch`;
|
|
- `pool_attention` and `projection_invalid`;
|
|
- `missing_entitlements` and `missing_pools`;
|
|
- `period_identity_mismatch`;
|
|
- `instance_not_ready` and `legacy_network_unsettled`;
|
|
- `usage_unsettled`;
|
|
- `cursor_missing` and `cursor_attention`;
|
|
- `traffic_sample_stale` (the normal freshness limit is 300 seconds);
|
|
- `traffic_scan_failures`;
|
|
- `policy_unsettled`;
|
|
- `traffic_restriction_present`;
|
|
- `network_job_unsettled`.
|
|
|
|
Fix the underlying sampling, identity, accounting or network-policy condition.
|
|
Normal lifecycle/recovery processes may then converge ordinary unsettled
|
|
records. `pool_attention` and `cursor_attention` are different: they are not
|
|
cleared automatically. They can be inspected and resolved only through the
|
|
local, audited break-glass CLI described in section 5.3; there is no HTTP or UI
|
|
resolve endpoint. Keep the group in its current safe mode (do not leave
|
|
shadow) until the immutable ledger, current counters and exact pool version
|
|
pass full verification. Never clear a blocker by deleting evidence or changing
|
|
a projection total.
|
|
|
|
### 3.4 Enforced: irreversible network policy
|
|
|
|
The administrator must review the empty blocker list, use the latest group
|
|
revision and type the complete group name in the confirmation dialog. Only
|
|
then advance the group to `enforced`.
|
|
|
|
Enforced mode is irreversible. It cannot be changed back to shadow or
|
|
disabled, and its bound lines cannot be changed. This is intentional: after
|
|
multiple instances have consumed shared entitlements, a per-instance balance
|
|
cannot be reconstructed safely.
|
|
|
|
## 4. Acceptance checklist
|
|
|
|
Complete these checks in shadow where possible, then repeat the network-policy
|
|
checks in enforced mode with controlled test instances:
|
|
|
|
- multiple finite instances owned by one user and group share one FEFO balance;
|
|
- instances in different regions but the same commercial line group share that
|
|
one FEFO balance;
|
|
- different groups and different users never share quota;
|
|
- equal line names or codes in different groups remain isolated;
|
|
- an unlimited period neither contributes to nor consumes a finite pool;
|
|
- deleting one instance does not delete its unexpired paid entitlement;
|
|
- period expiry forfeits only that entitlement's unused balance;
|
|
- sampled excess is retained as `overage_bytes`, and later quota settles the
|
|
debt before exposing usable balance;
|
|
- exhausting a pool adds the traffic restriction to every active finite
|
|
instance in that user/group;
|
|
- new usable quota removes only `traffic_pool_exhausted`, never
|
|
`service_expired` or `admin_hold`;
|
|
- the user page exposes only enforced groups and never presents shadow data as
|
|
an active shared allowance;
|
|
- pool or cursor attention prevents an untrusted automatic network restore;
|
|
- repeated lifecycle ticks neither double-charge usage nor create duplicate
|
|
network mutations;
|
|
- an Agent outage or unknown sample fails open for traffic accounting while
|
|
ordinary service expiry remains enforceable.
|
|
|
|
Static and local checks do not replace a controlled Agent/network acceptance
|
|
test. Record the release commit, group revision, involved instance IDs, sample
|
|
timestamps, task IDs and operation IDs.
|
|
|
|
## 5. Incident response and forward recovery
|
|
|
|
### 5.1 Shadow incident
|
|
|
|
Keep the group in shadow and stop opening new sales on the affected lines
|
|
through the existing commercial/scheduling controls. Continue the lifecycle
|
|
and recovery processes so expiry and unrelated business state still advance.
|
|
Repair the real cause, then wait for the readiness gate to become empty.
|
|
|
|
Returning shadow to disabled is allowed only before any service period or pool
|
|
durably references the group. Do not remove those references to force a
|
|
rollback.
|
|
|
|
### 5.2 Enforced incident
|
|
|
|
Pause new network mutation dispatch while preserving accounting and recovery:
|
|
stop `virt-cloud-worker.service` and every installed
|
|
`virt-cloud-policy-worker@N.service`. Stopping only policy workers is
|
|
insufficient because the ordinary worker is their fail-safe consumer. Keep
|
|
`virt-cloud-lifecycle.service` and `virt-cloud-recovery-worker.service`
|
|
running; lifecycle may persist target-state work, while recovery continues to
|
|
inspect already-dispatched operations without replaying them. This emergency
|
|
pause also stops unrelated ordinary jobs, so record its start and restore the
|
|
workers only after the original effects, accounting projection and queued
|
|
target state have been reviewed.
|
|
|
|
Do not stop lifecycle as a way to pause enforcement, and do not attempt a
|
|
per-instance rollback. Preserve the immutable ledger and repair or rebuild its
|
|
derived projections forward under a separately reviewed procedure.
|
|
|
|
Never:
|
|
|
|
- delete or rewrite pool, entitlement, ledger, cursor, restriction or policy
|
|
rows to hide an inconsistency;
|
|
- reset cumulative instance counters;
|
|
- replay `network_suspend` or `network_resume` after a timeout, disconnect,
|
|
malformed receipt or empty response;
|
|
- clear a policy blocker, release a reservation or issue a refund without the
|
|
operation-bound evidence required by the original task.
|
|
|
|
For an unknown remote effect, retain the original `operation_id` and task.
|
|
Let `virt-cloud-recovery-worker.service` inspect and converge that exact
|
|
operation. A later explicit operation is allowed only after the original
|
|
effect has been resolved and its safety fence has been released by the normal
|
|
finalizer.
|
|
|
|
### 5.3 Attention inspection and audited resolution
|
|
|
|
The CLI is local to the panel host and does not contact an Agent, create a
|
|
CloudJob or directly edit a network restriction. Shell access is the
|
|
break-glass authorization boundary. Record the external incident/change ID,
|
|
executing host, SSH/sudo operator, reason, pool ID, pool version and UUIDv4 in
|
|
the release log.
|
|
|
|
Always use this sequence:
|
|
|
|
1. Run the read-only inspection:
|
|
|
|
```bash
|
|
cd /home/virt-cloud/web
|
|
php xcat cloudTrafficPoolAttention inspect --pool-id=N
|
|
```
|
|
|
|
2. Repair the real counter or projection cause. Do not edit the pool state,
|
|
ledger, cursor totals or restriction rows.
|
|
3. Run `inspect` again. Use only this fresh response's `pool_id`, `version`,
|
|
`projection_valid`, attention counts and categories. If the pool changes
|
|
during inspection, the command fails and must be run again.
|
|
4. Generate a new canonical lowercase UUIDv4 and bind it to the external
|
|
incident/change record. Build the confirmation from that exact pool ID,
|
|
fresh version and UUID.
|
|
5. Execute one explicit resolution attempt:
|
|
|
|
```bash
|
|
php xcat cloudTrafficPoolAttention resolve \
|
|
--pool-id=N \
|
|
--expected-version=V \
|
|
--resolution-key=UUID \
|
|
--apply \
|
|
--confirm=RESOLVE_TRAFFIC_POOL:N:V:UUID
|
|
```
|
|
|
|
The command still fails closed if the version changed, the projection is
|
|
invalid, a counter has not caught up, the UUID was already used, or the real
|
|
attention cause remains. A successful transaction appends one immutable
|
|
`attention_resolved` ledger event and recomputes the projection. It does not
|
|
perform a remote network mutation.
|
|
|
|
If stdout, SSH or the command response is lost, do not repeat `resolve`.
|
|
First query the immutable UUID:
|
|
|
|
```bash
|
|
php xcat cloudTrafficPoolAttention inspect-resolution \
|
|
--resolution-key=UUID
|
|
```
|
|
|
|
`committed=true` proves which pool and expected version were resolved; do not
|
|
submit it again. `committed=false` proves that UUID has no committed resolution,
|
|
but it is not permission to retry blindly: run a fresh pool `inspect` and
|
|
repeat the exact command with the same UUID only if the pool is still in
|
|
attention at the same expected version. Never use one UUID for a different
|
|
pool or incident.
|
|
|
|
For an enforced pool, resolving attention may make the projection trustworthy
|
|
and allow the next lifecycle pass to request network restoration. Keep the
|
|
ordinary and policy workers paused as described in section 5.2 until the
|
|
resolved projection, active restrictions and queued target state have been
|
|
reviewed; then restore the workers and observe the original task flow.
|
|
|
|
### 5.4 Rollback matrix
|
|
|
|
| State | Operational action |
|
|
| --- | --- |
|
|
| `disabled`, no durable reference | Edit line mapping or abandon the group. |
|
|
| `shadow`, no durable reference | Return to disabled if required. |
|
|
| `shadow`, durable period/pool reference | Keep shadow and repair forward. |
|
|
| `enforced` | No rollback; preserve the ledger and repair forward. |
|
|
| Network operation has unknown effect | Reconcile the original task and `operation_id`; never replay it. |
|
|
|
|
## 6. Release evidence
|
|
|
|
Before marking rollout complete, retain:
|
|
|
|
- the exact Git commit and schema marker/fingerprint used by the fresh
|
|
installation or separately approved one-time migration;
|
|
- service status and recent logs for lifecycle, ordinary, policy and recovery
|
|
workers;
|
|
- the commercial line group, coverage regions, exact bound-line mapping and
|
|
revision;
|
|
- the complete readiness response immediately before enforcement;
|
|
- acceptance task IDs and operation IDs;
|
|
- any skipped live-host or failure-injection test.
|
|
|
|
If live Agent/network acceptance was not completed, report only that static
|
|
and local tests passed; do not claim the shared traffic pool is production
|
|
ready.
|