Rollback procedures - Chargeworx

When a deployment fails or introduces critical issues, Chargeworx provides multiple rollback strategies to restore service quickly. This guide covers rollback procedures for different scenarios.

Rollback strategies

AWS CodeDeploy rollback

Automatic rollback to previous deployment

Manual rollback

Manual restoration of previous version

Database rollback

Reverting database migrations

Configuration rollback

Restoring previous configuration

AWS CodeDeploy automatic rollback

AWS CodeDeploy can automatically rollback failed deployments.

Enable automatic rollback

Configure in CodeDeploy deployment group:

{
  "autoRollbackConfiguration": {
    "enabled": true,
    "events": [
      "DEPLOYMENT_FAILURE",
      "DEPLOYMENT_STOP_ON_ALARM"
    ]
  }
}

Rollback triggers

Automatic rollback occurs when:

Deployment fails on any instance
Health check fails after deployment
CloudWatch alarm triggers
Manual stop of deployment

Rollback process

CodeDeploy detects failure condition
Stops current deployment
Retrieves previous successful deployment
Redeploys previous version to all instances
Runs deployment hooks
Verifies health checks

Monitor rollback

# View deployment status
aws deploy get-deployment --deployment-id d-XXXXXXXXX

# Check rollback progress
aws deploy list-deployment-instances \
  --deployment-id d-XXXXXXXXX

Manual rollback via CodeDeploy

If automatic rollback is not configured, manually trigger rollback:

Step 1: Identify previous deployment

# List recent deployments
aws deploy list-deployments \
  --application-name Chargeworx \
  --deployment-group-name Production \
  --max-items 10

# Get deployment details
aws deploy get-deployment \
  --deployment-id d-XXXXXXXXX

Step 2: Create rollback deployment

# Redeploy previous revision
aws deploy create-deployment \
  --application-name Chargeworx \
  --deployment-group-name Production \
  --s3-location bucket=chargeworx-deployments,key=previous-version.zip,bundleType=zip \
  --description "Rollback to previous version"

Step 3: Monitor rollback

# Watch deployment progress
aws deploy get-deployment \
  --deployment-id d-XXXXXXXXX \
  --query 'deploymentInfo.status'

Manual rollback on Windows Server

If CodeDeploy is unavailable, manually rollback on the server.

Step 1: Stop IIS

# Stop application pools
Stop-WebAppPool -Name "Chargeworx.Api"
Stop-WebAppPool -Name "Chargeworx.Admin"

# Stop IIS
iisreset /stop

Step 2: Restore previous version

# Backup current deployment
$timestamp = Get-Date -Format "yyyyMMdd-HHmmss"
Copy-Item -Path "D:\APP-CODE\Chargeworx.Api" `
  -Destination "D:\APP-CODE\Backups\Chargeworx.Api-$timestamp" `
  -Recurse

# Restore previous version
Copy-Item -Path "D:\APP-CODE\Backups\Chargeworx.Api-previous" `
  -Destination "D:\APP-CODE\Chargeworx.Api" `
  -Recurse -Force

Step 3: Start IIS

# Start IIS
iisreset /start

# Start application pools
Start-WebAppPool -Name "Chargeworx.Api"
Start-WebAppPool -Name "Chargeworx.Admin"

Step 4: Verify application

# Test health endpoint
Invoke-WebRequest -Uri "https://api.chargeworx.com/Health" -UseBasicParsing

# Check application logs
Get-Content "D:\APP-CODE\Chargeworx.Api\Logs\log-latest.txt" -Tail 50

Database rollback

Migration rollback

If database migrations were applied, rollback using Entity Framework:

# Connect to server
ssh [email protected]

# Navigate to API directory
cd D:\APP-CODE\Chargeworx.Api

# Rollback to previous migration
dotnet ef database update PreviousMigrationName --context ChargeworxDbContext

# Verify migration status
dotnet ef migrations list --context ChargeworxDbContext

Database restore from backup

For critical database issues, restore from backup:

-- Stop application connections
ALTER DATABASE ChargeworxDb SET SINGLE_USER WITH ROLLBACK IMMEDIATE;

-- Restore database
RESTORE DATABASE ChargeworxDb
FROM DISK = 'D:\Backups\ChargeworxDb_20250101_120000.bak'
WITH REPLACE, RECOVERY;

-- Set multi-user mode
ALTER DATABASE ChargeworxDb SET MULTI_USER;

Verify database state

-- Check migration history
SELECT * FROM __EFMigrationsHistory
ORDER BY MigrationId DESC;

-- Verify data integrity
DBCC CHECKDB (ChargeworxDb);

-- Check table counts
SELECT 
    t.name AS TableName,
    p.rows AS RowCount
FROM sys.tables t
INNER JOIN sys.partitions p ON t.object_id = p.object_id
WHERE p.index_id IN (0,1)
ORDER BY p.rows DESC;

Configuration rollback

Restore configuration files

# Backup current configuration
Copy-Item -Path "D:\APP-CODE\Chargeworx.Api\appsettings.Production.json" `
  -Destination "D:\APP-CODE\Backups\appsettings.Production-$timestamp.json"

# Restore previous configuration
Copy-Item -Path "D:\APP-CODE\Backups\appsettings.Production-previous.json" `
  -Destination "D:\APP-CODE\Chargeworx.Api\appsettings.Production.json" `
  -Force

# Restart application
Restart-WebAppPool -Name "Chargeworx.Api"

Restore secrets from AWS Secrets Manager

# Retrieve previous secret version
aws secretsmanager get-secret-value \
  --secret-id chargeworx/production/database \
  --version-id previous-version-id

# Update secret to previous value
aws secretsmanager put-secret-value \
  --secret-id chargeworx/production/database \
  --secret-string "previous-connection-string"

Rollback verification

After rollback, verify system health:

Health checks

API health
Admin health
Database connectivity
Payment processing

# Check API health endpoint
curl https://api.chargeworx.com/Health

# Expected response
{
  "status": "Healthy",
  "checks": {
    "database": "Healthy",
    "identityServer": "Healthy",
    "cache": "Healthy"
  }
}

Application logs

# Check for errors in logs
Get-Content "D:\APP-CODE\Chargeworx.Api\Logs\log-latest.txt" | 
  Select-String -Pattern "ERROR|FATAL"

# Monitor real-time logs
Get-Content "D:\APP-CODE\Chargeworx.Api\Logs\log-latest.txt" -Wait -Tail 20

Performance metrics

Monitor key metrics after rollback:

Response times
Error rates
Database query performance
Memory and CPU usage
Active connections

Rollback decision matrix

Scenario	Recommended Action	Urgency
Failed deployment	AWS CodeDeploy automatic rollback	Immediate
Application crashes	Manual rollback + investigation	High
Database errors	Database rollback + application rollback	High
Performance degradation	Monitor, then rollback if worsening	Medium
Configuration issues	Configuration rollback only	Medium
Minor bugs	Fix forward in hotfix	Low

Post-rollback actions

After successful rollback:

1. Incident documentation

Document the incident:

What failed and why
Rollback steps taken
Time to recovery
Root cause analysis
Prevention measures

2. Notify stakeholders

Subject: Production Rollback - [Date/Time]

A deployment rollback was performed on [date] at [time].

Reason: [Brief description]
Impact: [User impact description]
Resolution: Rolled back to previous stable version
Status: System fully operational

Next steps:
- Root cause analysis in progress
- Fix will be deployed after thorough testing

3. Root cause analysis

Investigate the failure:

Review deployment logs
Analyze application errors
Check configuration changes
Review code changes
Identify testing gaps

4. Prevention measures

Implement safeguards:

Add automated tests
Improve deployment validation
Update runbooks
Enhance monitoring
Review change management process

Rollback best practices

Test rollback procedures

Regularly test rollback in non-production environments

Maintain backups

Keep multiple backup versions of deployments and databases

Document procedures

Keep rollback procedures up-to-date and accessible

Monitor after rollback

Closely monitor system after rollback for stability

Emergency contacts

For critical production issues:

On-call engineer: Check PagerDuty rotation
DevOps team: [email protected]
Database admin: [email protected]
Security team: [email protected]

Overview

Architecture

Getting started

Security

API reference

Admin UI

Data

Observability

Deployment

CI/CD

Runbooks

Contributing

​Rollback strategies

AWS CodeDeploy rollback

Manual rollback

Database rollback

Configuration rollback

​AWS CodeDeploy automatic rollback

​Enable automatic rollback

​Rollback triggers

​Rollback process

​Monitor rollback

​Manual rollback via CodeDeploy

​Step 1: Identify previous deployment

​Step 2: Create rollback deployment

​Step 3: Monitor rollback

​Manual rollback on Windows Server

​Step 1: Stop IIS

​Step 2: Restore previous version

​Step 3: Start IIS

​Step 4: Verify application

​Database rollback

​Migration rollback

​Database restore from backup

​Verify database state

​Configuration rollback

​Restore configuration files

​Restore secrets from AWS Secrets Manager

​Rollback verification

​Health checks

​Application logs

​Performance metrics

​Rollback decision matrix

​Post-rollback actions

​1. Incident documentation

​2. Notify stakeholders

​3. Root cause analysis

​4. Prevention measures

​Rollback best practices

Test rollback procedures

Maintain backups

Document procedures

Monitor after rollback

​Emergency contacts

​Related documentation

Rollback strategies

AWS CodeDeploy automatic rollback

Enable automatic rollback

Rollback triggers

Rollback process

Monitor rollback

Manual rollback via CodeDeploy

Step 1: Identify previous deployment

Step 2: Create rollback deployment

Step 3: Monitor rollback

Manual rollback on Windows Server

Step 1: Stop IIS

Step 2: Restore previous version

Step 3: Start IIS

Step 4: Verify application

Database rollback

Migration rollback

Database restore from backup

Verify database state

Configuration rollback

Restore configuration files

Restore secrets from AWS Secrets Manager

Rollback verification

Health checks

Application logs

Performance metrics

Rollback decision matrix

Post-rollback actions

1. Incident documentation

2. Notify stakeholders

3. Root cause analysis

4. Prevention measures

Rollback best practices

Emergency contacts

Related documentation