You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
virt/docs/TRAFFIC_POOL_ROLLOUT.md

16 KiB

Commercial line traffic-pool rollout and recovery runbook

This runbook is the operational boundary for enabling the shared traffic pool. The accounting contract and FEFO rules remain defined in TRAFFIC_POOL.md; this document covers deployment, acceptance and recovery only.

1. Database and process prerequisites

php xcat cloudInit is a fresh-schema installer and validator. It is not an incremental database upgrade command. A fresh deployment may run:

cd /home/virt-cloud/web
php xcat cloudInit

only after importing web/sql/ss.sql into a new business database and configuring a new log database, enable_log=true and an empty db_prefix. An installed database must already have the exact schema marker and fingerprint required by the same source revision.

Do not run cloudInit to upgrade an existing installation. An existing database needs a separately reviewed, one-time migration for the exact source and database versions. Never make such a database appear current by editing cloud_schema_version or cloud_schema_fingerprint.

The deployed panel source, schema and long-lived PHP processes must come from one release. The following units are required for accounting, network policy and operation-bound recovery:

  • virt-cloud-lifecycle.service;
  • virt-cloud-worker.service;
  • at least virt-cloud-policy-worker@1.service and virt-cloud-policy-worker@2.service;
  • virt-cloud-recovery-worker.service.

The source commands behind those units are:

php xcat cloudLifecycle
php xcat cloudWorker
php xcat cloudWorker --policy-only
php xcat cloudWorker --recovery-only

php xcat cloudLifecycle --once is available for a bounded non-production acceptance pass, but it is not a read-only health check: it also advances billing, backup, traffic and instance lifecycle state.

2. Clean identity boundary; no automatic backfill

traffic_pool_group_id is the global commercial line identity. A group may bind region-local lines from Hong Kong, Singapore or any other configured region, and all of those regions consume one user/group FEFO balance. Region is coverage metadata only; matching line names or codes do not create identity.

The line-oriented user and administrator pages require no schema migration and must not trigger an accounting backfill. Existing data is handled as follows:

Existing state Required action
disabled, no durable reference Bind all intended region-local lines through the revision-checked administrator operation.
shadow or enforced Preserve the current group and line membership; do not merge groups or rebind lines with SQL.
Legacy period with no group identity Keep legacy per-instance accounting until the period finishes.
New purchase or renewal already carrying a group Keep the frozen identity; a region or host move must not rewrite it.

The line-to-pool mapping is frozen into a new order and service period. A renewal inherits the preceding period's identity; it does not read the current line mapping again. Therefore:

  1. create the traffic-pool group and bind all intended network lines before accepting new orders on those lines;
  2. leave already-created old service periods on legacy accounting;
  3. do not accept payment or attempt delivery for an old unfinished order that lacks the required frozen traffic_pool_group_id and traffic_quota_mode; let an unpaid invoice expire, or resolve a paid/failed order through its original task and the existing order/refund workflow before placing a new order from the current release;
  4. do not update an active or scheduled period's traffic_pool_group_id;
  5. do not edit an order's frozen spec_json or manufacture a pool, entitlement, cursor or ledger event to make an old period appear migrated;
  6. do not use renewal as a migration mechanism: a legacy NULL identity is inherited as NULL.

If a mapped line still has a pending order or a finite scheduled/active/exhausted period with a missing or different frozen identity, readiness reports line_frozen_identity_mismatch and enforcement is blocked. In the test environment, let such periods finish without renewal or start again with a clean test database. Keep a legacy instance under lifecycle management until every old scheduled/active/exhausted period is finished; only then may the instance be removed and recreated through the normal product workflow. Deleting it first is not a shortcut because deleted instances are no longer advanced by lifecycle while their unfinished periods still block readiness. An old unfinished order without the complete snapshot must be expired or safely resolved; provisioning rejects it before Agent dispatch, so it is not a deliverable legacy order. A previously deleted instance with such an unfinished period needs a separately reviewed one-time data repair. Do not repair either condition by an unreviewed SQL backfill.

3. Rollout state machine

Every group must advance in order:

disabled -> shadow -> enforced

3.1 Disabled: configure the commercial identity

Open /admin/resources?view=traffic, create a line pool and bind every region-local network line that represents that commercial line while it is disabled.

  • A group needs at least one line before entering shadow mode.
  • One line can belong to only one group.
  • One group may cover multiple regions; verify both the coverage-region count and the exact bound-line list before creating orders.
  • Line membership can be edited only while the group is disabled.
  • Record the group name, ID, revision and bound line IDs in the release log.
  • Finish mapping before creating any order, instance or service-period reference for those lines.

3.2 Shadow: build and verify the real ledger

Advance the group to shadow. The lifecycle process then performs, in order:

  1. collection and persistence of trusted cumulative public-interface traffic;
  2. settlement of pending usage into the shared pool;
  3. activation, expiry and closing of entitlements;
  4. ordinary runtime and per-instance lifecycle reconciliation.

Shadow mode creates the real pool, entitlement, cursor and immutable ledger records, but it does not add or remove the traffic_pool_exhausted network restriction. Once a service period or pool is durably linked to the group, shadow mode cannot be returned to disabled; failures must be repaired forward.

Keep the group in shadow for a representative usage and renewal window. At a minimum, observe several collection intervals, a multi-instance allocation, an entitlement boundary and repeated lifecycle ticks before enforcement.

3.3 Readiness: every blocker must be zero

Do not enable enforcement until the administrator response for the same group shows all three conditions:

ready = true
can_enforce = true
blockers = []

The readiness gate covers:

  • no_network_lines;
  • line_frozen_identity_mismatch;
  • pool_attention and projection_invalid;
  • missing_entitlements and missing_pools;
  • period_identity_mismatch;
  • instance_not_ready and legacy_network_unsettled;
  • usage_unsettled;
  • cursor_missing and cursor_attention;
  • traffic_sample_stale (the normal freshness limit is 300 seconds);
  • traffic_scan_failures;
  • policy_unsettled;
  • traffic_restriction_present;
  • network_job_unsettled.

Fix the underlying sampling, identity, accounting or network-policy condition. Normal lifecycle/recovery processes may then converge ordinary unsettled records. pool_attention and cursor_attention are different: they are not cleared automatically. They can be inspected and resolved only through the local, audited break-glass CLI described in section 5.3; there is no HTTP or UI resolve endpoint. Keep the group in its current safe mode (do not leave shadow) until the immutable ledger, current counters and exact pool version pass full verification. Never clear a blocker by deleting evidence or changing a projection total.

3.4 Enforced: irreversible network policy

The administrator must review the empty blocker list, use the latest group revision and type the complete group name in the confirmation dialog. Only then advance the group to enforced.

Enforced mode is irreversible. It cannot be changed back to shadow or disabled, and its bound lines cannot be changed. This is intentional: after multiple instances have consumed shared entitlements, a per-instance balance cannot be reconstructed safely.

4. Acceptance checklist

Complete these checks in shadow where possible, then repeat the network-policy checks in enforced mode with controlled test instances:

  • multiple finite instances owned by one user and group share one FEFO balance;
  • instances in different regions but the same commercial line group share that one FEFO balance;
  • different groups and different users never share quota;
  • equal line names or codes in different groups remain isolated;
  • an unlimited period neither contributes to nor consumes a finite pool;
  • deleting one instance does not delete its unexpired paid entitlement;
  • period expiry forfeits only that entitlement's unused balance;
  • sampled excess is retained as overage_bytes, and later quota settles the debt before exposing usable balance;
  • exhausting a pool adds the traffic restriction to every active finite instance in that user/group;
  • new usable quota removes only traffic_pool_exhausted, never service_expired or admin_hold;
  • the user page exposes only enforced groups and never presents shadow data as an active shared allowance;
  • pool or cursor attention prevents an untrusted automatic network restore;
  • repeated lifecycle ticks neither double-charge usage nor create duplicate network mutations;
  • an Agent outage or unknown sample fails open for traffic accounting while ordinary service expiry remains enforceable.

Static and local checks do not replace a controlled Agent/network acceptance test. Record the release commit, group revision, involved instance IDs, sample timestamps, task IDs and operation IDs.

5. Incident response and forward recovery

5.1 Shadow incident

Keep the group in shadow and stop opening new sales on the affected lines through the existing commercial/scheduling controls. Continue the lifecycle and recovery processes so expiry and unrelated business state still advance. Repair the real cause, then wait for the readiness gate to become empty.

Returning shadow to disabled is allowed only before any service period or pool durably references the group. Do not remove those references to force a rollback.

5.2 Enforced incident

Pause new network mutation dispatch while preserving accounting and recovery: stop virt-cloud-worker.service and every installed virt-cloud-policy-worker@N.service. Stopping only policy workers is insufficient because the ordinary worker is their fail-safe consumer. Keep virt-cloud-lifecycle.service and virt-cloud-recovery-worker.service running; lifecycle may persist target-state work, while recovery continues to inspect already-dispatched operations without replaying them. This emergency pause also stops unrelated ordinary jobs, so record its start and restore the workers only after the original effects, accounting projection and queued target state have been reviewed.

Do not stop lifecycle as a way to pause enforcement, and do not attempt a per-instance rollback. Preserve the immutable ledger and repair or rebuild its derived projections forward under a separately reviewed procedure.

Never:

  • delete or rewrite pool, entitlement, ledger, cursor, restriction or policy rows to hide an inconsistency;
  • reset cumulative instance counters;
  • replay network_suspend or network_resume after a timeout, disconnect, malformed receipt or empty response;
  • clear a policy blocker, release a reservation or issue a refund without the operation-bound evidence required by the original task.

For an unknown remote effect, retain the original operation_id and task. Let virt-cloud-recovery-worker.service inspect and converge that exact operation. A later explicit operation is allowed only after the original effect has been resolved and its safety fence has been released by the normal finalizer.

5.3 Attention inspection and audited resolution

The CLI is local to the panel host and does not contact an Agent, create a CloudJob or directly edit a network restriction. Shell access is the break-glass authorization boundary. Record the external incident/change ID, executing host, SSH/sudo operator, reason, pool ID, pool version and UUIDv4 in the release log.

Always use this sequence:

  1. Run the read-only inspection:

    cd /home/virt-cloud/web
    php xcat cloudTrafficPoolAttention inspect --pool-id=N
    
  2. Repair the real counter or projection cause. Do not edit the pool state, ledger, cursor totals or restriction rows.

  3. Run inspect again. Use only this fresh response's pool_id, version, projection_valid, attention counts and categories. If the pool changes during inspection, the command fails and must be run again.

  4. Generate a new canonical lowercase UUIDv4 and bind it to the external incident/change record. Build the confirmation from that exact pool ID, fresh version and UUID.

  5. Execute one explicit resolution attempt:

    php xcat cloudTrafficPoolAttention resolve \
      --pool-id=N \
      --expected-version=V \
      --resolution-key=UUID \
      --apply \
      --confirm=RESOLVE_TRAFFIC_POOL:N:V:UUID
    

The command still fails closed if the version changed, the projection is invalid, a counter has not caught up, the UUID was already used, or the real attention cause remains. A successful transaction appends one immutable attention_resolved ledger event and recomputes the projection. It does not perform a remote network mutation.

If stdout, SSH or the command response is lost, do not repeat resolve. First query the immutable UUID:

php xcat cloudTrafficPoolAttention inspect-resolution \
  --resolution-key=UUID

committed=true proves which pool and expected version were resolved; do not submit it again. committed=false proves that UUID has no committed resolution, but it is not permission to retry blindly: run a fresh pool inspect and repeat the exact command with the same UUID only if the pool is still in attention at the same expected version. Never use one UUID for a different pool or incident.

For an enforced pool, resolving attention may make the projection trustworthy and allow the next lifecycle pass to request network restoration. Keep the ordinary and policy workers paused as described in section 5.2 until the resolved projection, active restrictions and queued target state have been reviewed; then restore the workers and observe the original task flow.

5.4 Rollback matrix

State Operational action
disabled, no durable reference Edit line mapping or abandon the group.
shadow, no durable reference Return to disabled if required.
shadow, durable period/pool reference Keep shadow and repair forward.
enforced No rollback; preserve the ledger and repair forward.
Network operation has unknown effect Reconcile the original task and operation_id; never replay it.

6. Release evidence

Before marking rollout complete, retain:

  • the exact Git commit and schema marker/fingerprint used by the fresh installation or separately approved one-time migration;
  • service status and recent logs for lifecycle, ordinary, policy and recovery workers;
  • the commercial line group, coverage regions, exact bound-line mapping and revision;
  • the complete readiness response immediately before enforcement;
  • acceptance task IDs and operation IDs;
  • any skipped live-host or failure-injection test.

If live Agent/network acceptance was not completed, report only that static and local tests passed; do not claim the shared traffic pool is production ready.