Production Grayscale Upgrade
Scenario
This section applies to production environments with multiple gateway instances or an entry layer that can split traffic.
Prerequisites
- At least one old-version instance can continue serving traffic.
- The entry layer can remove or restore a gateway instance quickly.
- Database backup and configuration backup are complete.
- Release notes confirm whether old and new versions can coexist during grayscale.
Procedure
Production environments should use rolling upgrade when multiple gateway instances are available.
- Remove one gateway instance from the entry layer.
- Upgrade the application package or image on that instance.
- Run the migration only once according to the release notes. If the release notes require migration before application startup, complete migration before replacing traffic.
- Start the upgraded instance and confirm the health check is normal.
- Send a small amount of traffic to the upgraded instance.
- Verify model invocation, Route Policy, quota, Usage Log, Audit Log, and metrics.
- Continue upgrading the remaining instances in batches.
- Keep old-version images and configuration backups until the observation window ends.
During rolling upgrade, do not send production traffic to a new instance whose health check, model invocation, or audit writing has not been verified.
Verification
Verify the upgraded instance with grayscale traffic first. Check model invocation success rate, latency, route decision, Usage Log, Audit Log, quota, and upstream channel health.
Notes
Do not expand grayscale traffic if error rate, latency, audit writing, or usage statistics are abnormal.