Documentation Index Fetch the complete documentation index at: https://docs.chargeworx.com/llms.txt
Use this file to discover all available pages before exploring further.
When a deployment fails or introduces critical issues, Chargeworx provides multiple rollback strategies to restore service quickly. This guide covers rollback procedures for different scenarios.
Rollback strategies
AWS CodeDeploy rollback Automatic rollback to previous deployment
Manual rollback Manual restoration of previous version
Database rollback Reverting database migrations
Configuration rollback Restoring previous configuration
AWS CodeDeploy automatic rollback AWS CodeDeploy can automatically rollback failed deployments.
Enable automatic rollback Configure in CodeDeploy deployment group:
{ "autoRollbackConfiguration" : { "enabled" : true , "events" : [ "DEPLOYMENT_FAILURE" , "DEPLOYMENT_STOP_ON_ALARM" ] } }
Rollback triggers Automatic rollback occurs when:
Deployment fails on any instance
Health check fails after deployment
CloudWatch alarm triggers
Manual stop of deployment
Rollback process
CodeDeploy detects failure condition
Stops current deployment
Retrieves previous successful deployment
Redeploys previous version to all instances
Runs deployment hooks
Verifies health checks
Monitor rollback # View deployment status aws deploy get-deployment --deployment-id d-XXXXXXXXX # Check rollback progress aws deploy list-deployment-instances \ --deployment-id d-XXXXXXXXX
Manual rollback via CodeDeploy If automatic rollback is not configured, manually trigger rollback:
Step 1: Identify previous deployment # List recent deployments aws deploy list-deployments \ --application-name Chargeworx \ --deployment-group-name Production \ --max-items 10 # Get deployment details aws deploy get-deployment \ --deployment-id d-XXXXXXXXX
Step 2: Create rollback deployment # Redeploy previous revision aws deploy create-deployment \ --application-name Chargeworx \ --deployment-group-name Production \ --s3-location bucket=chargeworx-deployments,key=previous-version.zip,bundleType=zip \ --description "Rollback to previous version"
Step 3: Monitor rollback # Watch deployment progress aws deploy get-deployment \ --deployment-id d-XXXXXXXXX \ --query 'deploymentInfo.status'
Manual rollback on Windows Server If CodeDeploy is unavailable, manually rollback on the server.
Step 1: Stop IIS # Stop application pools Stop-WebAppPool - Name "Chargeworx.Api" Stop-WebAppPool - Name "Chargeworx.Admin" # Stop IIS iisreset / stop
Step 2: Restore previous version # Backup current deployment $timestamp = Get-Date - Format "yyyyMMdd-HHmmss" Copy-Item - Path "D:\APP-CODE\Chargeworx.Api" ` - Destination "D:\APP-CODE\Backups\Chargeworx.Api- $timestamp " ` - Recurse # Restore previous version Copy-Item - Path "D:\APP-CODE\Backups\Chargeworx.Api-previous" ` - Destination "D:\APP-CODE\Chargeworx.Api" ` - Recurse - Force
Step 3: Start IIS # Start IIS iisreset / start # Start application pools Start-WebAppPool - Name "Chargeworx.Api" Start-WebAppPool - Name "Chargeworx.Admin"
Step 4: Verify application # Test health endpoint Invoke-WebRequest - Uri "https://api.chargeworx.com/Health" - UseBasicParsing # Check application logs Get-Content "D:\APP-CODE\Chargeworx.Api\Logs\log-latest.txt" - Tail 50
Database rollback Migration rollback If database migrations were applied, rollback using Entity Framework:
# Connect to server ssh admin@server.chargeworx.com # Navigate to API directory cd D: \A PP-CODE \C hargeworx.Api # Rollback to previous migration dotnet ef database update PreviousMigrationName --context ChargeworxDbContext # Verify migration status dotnet ef migrations list --context ChargeworxDbContext
Database restore from backup For critical database issues, restore from backup:
-- Stop application connections ALTER DATABASE ChargeworxDb SET SINGLE_USER WITH ROLLBACK IMMEDIATE ; -- Restore database RESTORE DATABASE ChargeworxDb FROM DISK = 'D:\Backups\ChargeworxDb_20250101_120000.bak' WITH REPLACE , RECOVERY ; -- Set multi-user mode ALTER DATABASE ChargeworxDb SET MULTI_USER ;
Verify database state -- Check migration history SELECT * FROM __EFMigrationsHistory ORDER BY MigrationId DESC ; -- Verify data integrity DBCC CHECKDB (ChargeworxDb); -- Check table counts SELECT t . name AS TableName, p . rows AS RowCount FROM sys . tables t INNER JOIN sys . partitions p ON t . object_id = p . object_id WHERE p . index_id IN ( 0 , 1 ) ORDER BY p . rows DESC ;
Configuration rollback Restore configuration files # Backup current configuration Copy-Item - Path "D:\APP-CODE\Chargeworx.Api\appsettings.Production.json" ` - Destination "D:\APP-CODE\Backups\appsettings.Production- $timestamp .json" # Restore previous configuration Copy-Item - Path "D:\APP-CODE\Backups\appsettings.Production-previous.json" ` - Destination "D:\APP-CODE\Chargeworx.Api\appsettings.Production.json" ` - Force # Restart application Restart-WebAppPool - Name "Chargeworx.Api"
Restore secrets from AWS Secrets Manager # Retrieve previous secret version aws secretsmanager get-secret-value \ --secret-id chargeworx/production/database \ --version-id previous-version-id # Update secret to previous value aws secretsmanager put-secret-value \ --secret-id chargeworx/production/database \ --secret-string "previous-connection-string"
Rollback verification After rollback, verify system health:
Health checks API health
Admin health
Database connectivity
Payment processing
# Check API health endpoint curl https://api.chargeworx.com/Health # Expected response { "status" : "Healthy", "checks" : { "database" : "Healthy", "identityServer" : "Healthy", "cache" : "Healthy" } }
# Check Admin health endpoint curl https://admin.chargeworx.com/Health # Verify login page loads curl -I https://admin.chargeworx.com/Auth/Login
-- Test database connection SELECT @@ VERSION ; -- Verify key tables SELECT COUNT ( * ) FROM Companies; SELECT COUNT ( * ) FROM Transactions;
# Test payment endpoint curl -X POST https://api.chargeworx.com/api/pay/{projectId}/transactions \ -H "Authorization: Bearer {token}" \ -H "Content-Type: application/json" \ -d '{...}'
Application logs # Check for errors in logs Get-Content "D:\APP-CODE\Chargeworx.Api\Logs\log-latest.txt" | Select-String - Pattern "ERROR|FATAL" # Monitor real-time logs Get-Content "D:\APP-CODE\Chargeworx.Api\Logs\log-latest.txt" - Wait - Tail 20
Monitor key metrics after rollback:
Response times
Error rates
Database query performance
Memory and CPU usage
Active connections
Rollback decision matrix Scenario Recommended Action Urgency Failed deployment AWS CodeDeploy automatic rollback Immediate Application crashes Manual rollback + investigation High Database errors Database rollback + application rollback High Performance degradation Monitor, then rollback if worsening Medium Configuration issues Configuration rollback only Medium Minor bugs Fix forward in hotfix Low
Post-rollback actions After successful rollback:
1. Incident documentation Document the incident:
What failed and why
Rollback steps taken
Time to recovery
Root cause analysis
Prevention measures
2. Notify stakeholders Subject: Production Rollback - [Date/Time] A deployment rollback was performed on [date] at [time]. Reason: [Brief description] Impact: [User impact description] Resolution: Rolled back to previous stable version Status: System fully operational Next steps: - Root cause analysis in progress - Fix will be deployed after thorough testing
3. Root cause analysis Investigate the failure:
Review deployment logs
Analyze application errors
Check configuration changes
Review code changes
Identify testing gaps
4. Prevention measures Implement safeguards:
Add automated tests
Improve deployment validation
Update runbooks
Enhance monitoring
Review change management process
Rollback best practices
Test rollback procedures Regularly test rollback in non-production environments
Maintain backups Keep multiple backup versions of deployments and databases
Document procedures Keep rollback procedures up-to-date and accessible
Monitor after rollback Closely monitor system after rollback for stability
For critical production issues:
