You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
virt/docs/TRAFFIC_POOL_ROLLOUT.md

21 KiB

Commercial line traffic-pool rollout and recovery runbook

This runbook is the operational boundary for enabling the shared traffic pool. The accounting contract and FEFO rules remain defined in TRAFFIC_POOL.md; this document covers deployment, acceptance and recovery only.

1. Database and process prerequisites

php xcat cloudInit is a fresh-schema installer and validator. It is not an incremental database upgrade command. A fresh deployment may run:

cd /home/virt-cloud/web
php xcat cloudInit

only after importing web/sql/ss.sql into a new business database and configuring a new log database, enable_log=true and an empty db_prefix. An installed database must already have the exact schema marker and fingerprint required by the same source revision.

Do not run cloudInit to upgrade an existing installation. An existing database needs a separately reviewed, one-time migration for the exact source and database versions. Never make such a database appear current by editing cloud_schema_version or cloud_schema_fingerprint.

The deployed panel source, schema and long-lived PHP processes must come from one release. The following units are required for accounting, network policy and operation-bound recovery:

  • virt-cloud-lifecycle.service;
  • virt-cloud-worker.service;
  • at least virt-cloud-policy-worker@1.service and virt-cloud-policy-worker@2.service;
  • virt-cloud-recovery-worker.service.

The source commands behind those units are:

php xcat cloudLifecycle
php xcat cloudWorker
php xcat cloudWorker --policy-only
php xcat cloudWorker --recovery-only

php xcat cloudLifecycle --once is available for a bounded non-production acceptance pass, but it is not a read-only health check: it also advances billing, backup, traffic and instance lifecycle state.

2. Clean identity boundary; no automatic backfill

traffic_pool_group_id is the global commercial line identity. A group may bind region-local lines from Hong Kong, Singapore or any other configured region, and all of those regions consume one user/group FEFO balance. Region is coverage metadata only; matching line names or codes do not create identity.

The line-oriented user and administrator pages require no schema migration and must not trigger an accounting backfill. Existing data is handled as follows:

Existing state Required action
disabled, no durable reference Bind all intended region-local lines through the revision-checked administrator operation.
shadow or enforced Preserve the current group and line membership; do not merge groups or rebind lines with SQL.
Legacy period with no group identity Keep legacy per-instance accounting until the period finishes.
New purchase or renewal already carrying a group Keep the frozen identity; a region or host move must not rewrite it.

The line-to-pool mapping is frozen into a new order and service period. A renewal inherits the preceding period's identity; it does not read the current line mapping again. Therefore:

  1. create the traffic-pool group and bind all intended network lines before accepting new orders on those lines;
  2. leave already-created old service periods on legacy accounting;
  3. do not accept payment or attempt delivery for an old unfinished order that lacks the required frozen traffic_pool_group_id and traffic_quota_mode; let an unpaid invoice expire, or resolve a paid/failed order through its original task and the existing order/refund workflow before placing a new order from the current release;
  4. do not update an active or scheduled period's traffic_pool_group_id;
  5. do not edit an order's frozen spec_json or manufacture a pool, entitlement, cursor or ledger event to make an old period appear migrated;
  6. do not use renewal as a migration mechanism: a legacy NULL identity is inherited as NULL.

If a mapped line still has a pending order or a finite scheduled/active/exhausted period with a missing or different frozen identity, readiness reports line_frozen_identity_mismatch and enforcement is blocked. In the test environment, let such periods finish without renewal or start again with a clean test database. Keep a legacy instance under lifecycle management until every old scheduled/active/exhausted period is finished; only then may the instance be removed and recreated through the normal product workflow. Deleting it first is not a shortcut because deleted instances are no longer advanced by lifecycle while their unfinished periods still block readiness. An old unfinished order without the complete snapshot must be expired or safely resolved; provisioning rejects it before Agent dispatch, so it is not a deliverable legacy order. A previously deleted instance with such an unfinished period needs a separately reviewed one-time data repair. Do not repair either condition by an unreviewed SQL backfill.

3. Rollout state machine

Every group must advance in order:

disabled -> shadow -> enforced

3.1 Disabled: configure the commercial identity

Open /admin/resources?view=traffic, create a line pool and bind every region-local network line that represents that commercial line while it is disabled.

  • A group needs at least one line before entering shadow mode.
  • One line can belong to only one group.
  • One group may cover multiple regions; verify both the coverage-region count and the exact bound-line list before creating orders.
  • Line membership can be edited only while the group is disabled.
  • Record the group name, ID, revision and bound line IDs in the release log.
  • Finish mapping before creating any order, instance or service-period reference for those lines.

3.2 Shadow: build and verify the real ledger

Advance the group to shadow. The lifecycle process then performs, in order:

  1. collection and persistence of trusted cumulative public-interface traffic;
  2. settlement of pending usage into the shared pool;
  3. activation, expiry and closing of entitlements;
  4. ordinary runtime and per-instance lifecycle reconciliation.

Shadow mode creates the real pool, entitlement, cursor, durable UTC daily settlement buckets and immutable sealed ledger records, but it does not add or remove the traffic_pool_exhausted network restriction. Open buckets are part of projection verification until sealed; they are not permission to replay a counter observation. Once a service period or pool is durably linked to the group, shadow mode cannot be returned to disabled; failures must be repaired forward.

Keep the group in shadow for a representative usage and renewal window. At a minimum, observe several collection intervals, a multi-instance allocation, an entitlement boundary and repeated lifecycle ticks before enforcement.

3.3 Readiness: every blocker must be zero

Do not enable enforcement until the administrator response for the same group shows all three conditions:

ready = true
can_enforce = true
blockers = []

The readiness gate covers:

  • no_network_lines;
  • line_frozen_identity_mismatch;
  • pool_attention and projection_invalid;
  • missing_entitlements and missing_pools;
  • period_identity_mismatch;
  • instance_not_ready and legacy_network_unsettled;
  • usage_unsettled;
  • cursor_missing and cursor_attention;
  • traffic_sample_stale (the normal freshness limit is 300 seconds);
  • traffic_scan_failures;
  • policy_unsettled;
  • traffic_restriction_present;
  • network_job_unsettled.

Fix the underlying sampling, identity, accounting or network-policy condition. Normal lifecycle/recovery processes may then converge ordinary unsettled records. pool_attention and cursor_attention are different: they are not cleared automatically. They can be inspected and resolved only through the local, audited break-glass CLI described in section 5.3; there is no HTTP or UI resolve endpoint. Keep the group in its current safe mode (do not leave shadow) until the immutable ledger, current counters and exact pool version pass full verification. Never clear a blocker by deleting evidence or changing a projection total.

3.4 Enforced: irreversible network policy

The administrator must review the empty blocker list, use the latest group revision and type the complete group name in the confirmation dialog. Only then advance the group to enforced.

Enforced mode is irreversible. It cannot be changed back to shadow or disabled, and its bound lines cannot be changed. This is intentional: after multiple instances have consumed shared entitlements, a per-instance balance cannot be reconstructed safely.

4. Acceptance checklist

Complete these checks in shadow where possible, then repeat the network-policy checks in enforced mode with controlled test instances:

  • multiple finite instances owned by one user and group share one FEFO balance;
  • instances in different regions but the same commercial line group share that one FEFO balance;
  • different groups and different users never share quota;
  • equal line names or codes in different groups remain isolated;
  • an unlimited period neither contributes to nor consumes a finite pool;
  • deleting one instance does not delete its unexpired paid entitlement;
  • period expiry forfeits only that entitlement's unused balance;
  • sampled excess is retained as overage_bytes, and later quota settles the debt before exposing usable balance;
  • exhausting a pool adds the traffic restriction to every active finite instance in that user/group;
  • new usable quota removes only traffic_pool_exhausted, never service_expired or admin_hold;
  • the user page exposes only enforced groups and never presents shadow data as an active shared allowance;
  • delayed or attention accounting is visible as a last-confirmed balance with its confirmation time, while per-instance raw samples remain available;
  • pool or cursor attention prevents an untrusted automatic network restore;
  • repeated lifecycle ticks neither double-charge usage nor create duplicate network mutations;
  • an Agent outage or unknown sample fails open for traffic accounting while ordinary service expiry remains enforceable.

Static and local checks do not replace a controlled Agent/network acceptance test. Record the release commit, group revision, involved instance IDs, sample timestamps, task IDs and operation IDs.

5. Incident response and forward recovery

5.1 Shadow incident

Keep the group in shadow and stop opening new sales on the affected lines through the existing commercial/scheduling controls. Continue the lifecycle and recovery processes so expiry and unrelated business state still advance. Repair the real cause, then wait for the readiness gate to become empty.

Returning shadow to disabled is allowed only before any service period or pool durably references the group. Do not remove those references to force a rollback.

5.2 Enforced incident

Pause new network mutation dispatch while preserving accounting and recovery: stop virt-cloud-worker.service and every installed virt-cloud-policy-worker@N.service. Stopping only policy workers is insufficient because the ordinary worker is their fail-safe consumer. Keep virt-cloud-lifecycle.service and virt-cloud-recovery-worker.service running; lifecycle may persist target-state work, while recovery continues to inspect already-dispatched operations without replaying them. This emergency pause also stops unrelated ordinary jobs, so record its start and restore the workers only after the original effects, accounting projection and queued target state have been reviewed.

Do not stop lifecycle as a way to pause enforcement, and do not attempt a per-instance rollback. Preserve the immutable ledger and repair or rebuild its derived projections forward under a separately reviewed procedure.

Never:

  • delete or rewrite pool, entitlement, ledger, cursor, restriction or policy rows to hide an inconsistency;
  • reset cumulative instance counters;
  • replay network_suspend or network_resume after a timeout, disconnect, malformed receipt or empty response;
  • clear a policy blocker, release a reservation or issue a refund without the operation-bound evidence required by the original task.

For an unknown remote effect, retain the original operation_id and task. Let virt-cloud-recovery-worker.service inspect and converge that exact operation. A later explicit operation is allowed only after the original effect has been resolved and its safety fence has been released by the normal finalizer.

5.3 Attention inspection and audited resolution

The CLI is local to the panel host and does not contact an Agent, create a CloudJob or directly edit a network restriction. Shell access is the break-glass authorization boundary. Record the external incident/change ID, executing host, SSH/sudo operator, reason, pool ID, pool version and UUIDv4 in the release log.

Always use this sequence:

  1. Run the read-only inspection:

    cd /home/virt-cloud/web
    php xcat cloudTrafficPoolAttention inspect --pool-id=N
    
  2. Repair the real counter or projection cause. Do not edit the pool state, ledger, cursor totals or restriction rows.

  3. Run inspect again. Use only this fresh response's pool_id, version, projection_valid, attention counts and categories. If the pool changes during inspection, the command fails and must be run again.

  4. Generate a new canonical lowercase UUIDv4 and bind it to the external incident/change record. Build the confirmation from that exact pool ID, fresh version and UUID.

  5. Execute one explicit resolution attempt:

    php xcat cloudTrafficPoolAttention resolve \
      --pool-id=N \
      --expected-version=V \
      --resolution-key=UUID \
      --apply \
      --confirm=RESOLVE_TRAFFIC_POOL:N:V:UUID
    

The command still fails closed if the version changed, the projection is invalid, a counter has not caught up, the UUID was already used, or the real attention cause remains. A successful transaction appends one immutable attention_resolved ledger event and recomputes the projection. It does not perform a remote network mutation.

If stdout, SSH or the command response is lost, do not repeat resolve. First query the immutable UUID:

php xcat cloudTrafficPoolAttention inspect-resolution \
  --resolution-key=UUID

committed=true proves which pool and expected version were resolved; do not submit it again. committed=false proves that UUID has no committed resolution, but it is not permission to retry blindly: run a fresh pool inspect and repeat the exact command with the same UUID only if the pool is still in attention at the same expected version. Never use one UUID for a different pool or incident.

For an enforced pool, resolving attention may make the projection trustworthy and allow the next lifecycle pass to request network restoration. Keep the ordinary and policy workers paused as described in section 5.2 until the resolved projection, active restrictions and queued target state have been reviewed; then restore the workers and observe the original task flow.

5.4 Delayed settlement recovery

When storage health reports overdue open settlement buckets or persisted instance counters that have not crossed the accounting seam, first generate a local read-only plan:

php xcat cloudTrafficSettlementRecovery inspect

The output binds the exact bucket identities, period states, instance counters and cursor positions into plan_hash. Review every blocker and action. The command refuses to apply while an event-identity conflict or attention cursor exists. If the plan is clean, execute that exact generation once:

php xcat cloudTrafficSettlementRecovery apply --plan-hash=<hash> --apply

Any intervening state change invalidates the hash and requires a new inspect. Apply uses only the ordinary bucket sealing, period reconciliation and usage settlement interfaces. It cannot delete or rewrite an immutable ledger event, change an entitlement balance manually, clear attention, or dispatch an Agent operation. A process interruption may leave a safe partial application; run a new inspect and continue only from its new plan.

5.5 Rollback matrix

State Operational action
disabled, no durable reference Edit line mapping or abandon the group.
shadow, no durable reference Return to disabled if required.
shadow, durable period/pool reference Keep shadow and repair forward.
enforced No rollback; preserve the ledger and repair forward.
Network operation has unknown effect Reconcile the original task and operation_id; never replay it.

6. Release evidence

Before marking rollout complete, retain:

  • the exact Git commit and schema marker/fingerprint used by the fresh installation or separately approved one-time migration;
  • service status and recent logs for lifecycle, ordinary, policy and recovery workers;
  • the commercial line group, coverage regions, exact bound-line mapping and revision;
  • the complete readiness response immediately before enforcement;
  • acceptance task IDs and operation IDs;
  • any skipped live-host or failure-injection test.

If live Agent/network acceptance was not completed, report only that static and local tests passed; do not claim the shared traffic pool is production ready.

7. Traffic-storage compaction rollout

Traffic-storage compaction changes retention only. It does not change the minute-level cumulative counter, FEFO allocation, pool projection or network policy decision. Deploy it as a clean schema/code boundary:

  1. Stop lifecycle before changing either database. Keep ordinary and recovery workers running unless the same change also modifies their schema.
  2. Back up the business and log databases. Record row counts and oldest timestamps for settlement buckets, ledger events, hourly checkpoints, daily traffic rows and service periods.
  3. Apply the exact reviewed v55 migration. Create the compact settlement receipt table, settlement retention index, daily checkpoint table and service-period boundary-anchor table. Add instance_id and its index to hourly checkpoints, including the first_observed_at-first retention index used by bounded cleanup and read-only health counts.
  4. Backfill an hourly checkpoint's logical instance only from one exact, unambiguous current or retained placement identity. Set an unmappable old row to the reserved value instance_id = 0; never infer ownership from a user, name, address or nearby timestamp. Those rows are retained for manual review and are excluded from automatic deletion.
  5. For each mapped instance/day, a reviewed one-time migration may derive one daily row from the earliest and latest existing hourly cumulative rows. Do not synthesize service-period anchors. Existing periods with persisted base counters need no anchor; an open period with null bases deliberately blocks deletion until the normal lifecycle path freezes real evidence.
  6. Deploy the matching code and restart lifecycle. Open the administrator task page and run its read-only health check. The “流量结算与存储” row exposes overdue open buckets, longest accounting delay, event-identity conflicts, unsettled and stale-cursor instances, recent accounting failures, due compaction rows and cursor attention without exposing IDs or payloads.
  7. Keep the pool in its existing rollout state while observing at least one complete collection, accounting, settlement-compaction and checkpoint-compaction pass. A backlog less than one hour beyond its retention window remains informational. An open settlement delay over five minutes is a warning; a delay over one hour, event-identity conflict, scan truncation or attention cursor requires immediate investigation.

Acceptance requires all of the following:

  • a repeated cumulative observation has no second accounting effect;
  • a sealed daily bucket older than seven days is replaced by one matching compact receipt while its immutable ledger event remains unchanged;
  • a late observation starts the next bucket revision after detail compaction;
  • an hourly checkpoint older than 72 hours is removed only when its daily row contains the exact cumulative values and every relevant open period has persisted bases or a valid anchor;
  • missing daily evidence, a malformed anchor or an attention cursor retains the hourly row and reports a warning;
  • deleting an hourly row does not change the activated period baseline;
  • 36-day performance-hour rows are pruned while 34-day rows and the complete 30-day chart remain readable;
  • account deletion removes the user's totals, checkpoints, daily history, anchors and performance history.

Rollback is a database restore plus matching source rollback performed while lifecycle is stopped. Do not run old code against the v55 log schema, remove receipts to recreate full settlement buckets, delete blockers, manufacture anchors or mark an attention cursor ready. A retention incident is fail-safe: leave the extra rows in place, repair the evidence or schema, and rerun the same bounded lifecycle stage.