How To Rehearse The Production Rollback

This guide is for VyDex maintainers authorized to approve the protected production environment. Use it when the deployment or rollback mechanism changes or another production rehearsal is explicitly required. The protected workflow temporarily switches the live Pages deployment, verifies it, restores the intended deployment, and verifies production again.

Before You Start

Confirm that the intended release is committed and pushed to main.
Confirm that the intended Deploy application run succeeded and that the selected deployment is currently serving production.
Confirm that https://vydex.pages.dev currently serves the intended Release ID.
Confirm that the GitHub production environment has required reviewers and that an authorized reviewer is available.
Confirm that no deployment, rollback, restoration, or production incident is in progress.
Be ready to use How To Restore A Production Deployment if the rehearsal cannot restore the intended deployment.

The initial Stage 1 launch rehearsal has passed. A new rehearsal still changes the live production deployment twice, so run it only when an explicit later check is needed; it is not part of every routine content deployment.

Run The Protected Workflow

Open the repository on GitHub and select Actions.
Select the Rehearse production rollback workflow.
Select Run workflow and choose the main branch.
Enter REHEARSE_PRODUCTION_ROLLBACK in the confirmation field. Any other value stops the workflow before production mutation.
Select Run workflow to start release reproduction and artifact preflight.
Wait for the Roll back and restore production job to request approval from the protected production environment.
Before approving, confirm that the workflow run uses the intended main commit and that no other production operation is active.
Select Review deployments, select production, and approve the deployment.
Wait for the workflow to finish. After approval, deployment discovery, optional byte-identical redeployment, rollback, hosted verification, restoration, final verification, and evidence capture are automatic. A verification phase may run the complete suite more than once while Pages edges converge; do not cancel it during the bounded retry window.

Do not cancel the job after production mutation begins unless emergency recovery requires it. The workflow's unconditional cleanup is responsible for restoring the intended deployment after an intermediate failure.

Confirm It Worked

Confirm that the Roll back and restore production job succeeded.
Download the vydex-rollback-evidence-<run-id>-<attempt> artifact from the workflow run.
Open runtime/hosted-verification/rollback-rehearsal-evidence.json and confirm that earlier_deployment_id and intended_deployment_id are different.
Confirm the earlier and intended VyDex Release IDs, source commits, manifest hashes, Dataset hashes, and inventory hashes match their archived state. They may differ across a cross-release rehearsal.
Confirm that the rollback report passed against https://vydex.pages.dev while the earlier deployment was canonical.
Confirm that the restoration report passed and the intended deployment ID became canonical again.
Open https://vydex.pages.dev and confirm that its Release ID matches the intended Stage 1 descriptor.

The rehearsal is complete only when rollback verification, restoration, and final hosted verification all pass. It prefers the immediate previous archived release and falls back to two deployment IDs exposing one byte-identical VyDex release when no usable previous-release deployment exists.

If Something Goes Wrong

The workflow still attempts restoration when rollback polling or verification fails. Preserve the failed run and its uploaded reports, browser output, screenshots, traces, and logs.

If the logs contain CRITICAL: restore production to deployment, do not start another rehearsal. Copy the intended deployment ID, follow the emitted manual recovery procedure, and use How To Restore A Production Deployment. Never substitute a preview deployment.

If restoration succeeds but the rollback phase failed, the workflow remains failed by design. Review the failed phase report before deciding whether the mechanism is ready for another explicitly approved rehearsal.

Next Steps