Upgrade Overview
An upgrade may involve the control plane, data plane, worker components, Console UI, and database schema. Before execution, confirm the component versions, service availability requirements, backup state, and rollback boundary.
Typical upgrade sequence:
- Read the release notes and confirm the source and target versions.
- Evaluate the impact scope and upgrade window.
- Complete backup and pre-upgrade checks.
- Upgrade the application package, image, or deployment manifest.
- Run database migration.
- Upgrade or refresh the Console UI.
- Perform functional, routing, audit, and metric verification.
- Keep observation for one service cycle before closing the upgrade record.
Upgrade Scope
| Scope | Description | Typical Impact |
|---|---|---|
| Control plane | Channel, model, policy, credential, organization, quota, and audit management APIs | Administrators may be unable to modify configuration during restart |
| Data plane | OpenAI-compatible and LobsterPool-compatible model invocation entry points | In-flight requests may be interrupted if the entry layer or application is restarted |
| Worker | Asynchronous tasks such as usage aggregation and health probing | Logs or statistics may be delayed during restart |
| Console UI | Administrator web console | Existing browser sessions may need to refresh the page |
| Database | PostgreSQL schema and metadata | Migration must succeed before new application versions can work normally |
Upgrade Window
Plan the upgrade during a low-traffic window. If the environment has only one gateway instance, the upgrade usually requires a short service interruption. If the environment uses multiple gateway instances behind Nginx, Ingress, or a load balancer, upgrade nodes in batches and keep at least one healthy node serving traffic.
Before the upgrade, notify business owners of:
- Planned start time and end time.
- Expected impact on new requests and in-flight requests.
- Contact person during the upgrade.
- Rollback trigger conditions.
Important Limitations
Read this section before any operation.
- Database migrations are forward-only and append-only. Do not modify an already applied migration file, including comments.
- If a migration has changed table structure or data semantics, application-layer rollback may not be sufficient. Database restore may be required.
- Application-layer rollback applies to failures such as image startup failure, configuration file error, health check failure, and UI loading failure when no destructive schema or data change has been applied.
- Database restore applies to failures after schema or data migration when the old application version cannot read the new schema safely.
- When upgrading a production environment, prefer rolling back the entry layer first, then the application layer, and restore the database only as the last step.
- Do not reuse the production database for migration rehearsal. Use a copied database or a test database.
- During a grayscale upgrade, old and new versions may run at the same time only if the release notes explicitly state compatibility.
Applicable Boundaries
| Scenario | Applicable Method |
|---|---|
| Standard version upgrade | Follow the backup, stop-service, migration, startup, and verification procedure in this guide. |
| Console UI upgrade | Replace static resources only, and restart the service that reads the static directory when required. |
| LobsterPool grayscale | Route part of /openai/v1/* traffic to the new version at the entry layer. |
| Existing database migration | Use the migration tool to perform pre-check, backup, rehearsal import, formal import, and data validation. |
| Emergency fix | Switch traffic back to the stable entry path first, then recover application processes. |