feat: Add safe deployment scripts with automatic rollback

install/deploy/README.md

@@ -0,0 +1,144 @@
+# Deployment Scripts
+
+These scripts implement safe, automated deployment with automatic rollback on failure.
+
+## Overview
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    DEPLOYMENT FLOW                          │
+├─────────────────────────────────────────────────────────────┤
+│  pre_deploy.sh  →  deploy.sh  →  post_deploy.sh            │
+│       ↓               ↓              ↓                      │
+│  Validate env    Update code    Health check                │
+│  Save state      Run migrations  Verify app                 │
+│       ↓               ↓              ↓                      │
+│                                 ┌────┴────┐                 │
+│                                 ↓         ↓                 │
+│                              SUCCESS   FAILURE              │
+│                                 ↓         ↓                 │
+│                               Done    rollback.sh           │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## Scripts
+
+### pre_deploy.sh
+Validates the environment before deployment:
+- Checks for concurrent deployments (lock file)
+- Verifies install folder and config.py exist
+- Tests database connection
+- Saves current commit and migration version for rollback
+- Checks disk space
+
+### deploy.sh
+Performs the actual deployment:
+- Fetches and updates code from git
+- Installs/updates Python dependencies
+- Runs database migrations
+- Copies CI scripts to test data directory
+- Reloads the platform service
+
+### post_deploy.sh
+Verifies deployment was successful:
+- Waits for application to start
+- Checks `/health` endpoint (with fallback to `/`)
+- Retries up to 6 times with 5-second delay
+- Returns success (0) or failure (1)
+
+### rollback.sh
+Restores previous working state:
+- Restores previous git commit
+- Downgrades database migrations if needed
+- Reinstalls dependencies
+- Reloads service
+- Verifies rollback was successful
+
+## Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `INSTALL_FOLDER` | `/var/www/sample-platform` | Application directory |
+| `SAMPLE_REPOSITORY` | `/repository` | Test data repository |
+| `DEPLOY_BRANCH` | `master` | Git branch to deploy |
+| `HEALTH_URL` | `http://127.0.0.1/health` | Health check endpoint |
+| `FALLBACK_URL` | `http://127.0.0.1/` | Fallback check URL |
+| `MAX_RETRIES` | `6` | Health check retry count |
+| `RETRY_DELAY` | `5` | Seconds between retries |
+
+## Manual Usage
+
+```bash
+# Set environment
+export INSTALL_FOLDER="/var/www/sample-platform"
+export SAMPLE_REPOSITORY="/repository"
+
+# Run deployment
+cd $INSTALL_FOLDER
+sudo bash install/deploy/pre_deploy.sh && \
+sudo bash install/deploy/deploy.sh && \
+sudo bash install/deploy/post_deploy.sh || \
+sudo bash install/deploy/rollback.sh
+```
+
+## GitHub Actions Integration
+
+These scripts are designed to be called from the GitHub Actions workflow:
+
+```yaml
+- name: Pre-deployment checks
+  run: sudo bash install/deploy/pre_deploy.sh
+
+- name: Deploy
+  run: sudo bash install/deploy/deploy.sh
+
+- name: Verify deployment
+  run: sudo bash install/deploy/post_deploy.sh
+
+- name: Rollback on failure
+  if: failure()
+  run: sudo bash install/deploy/rollback.sh
+```
+
+## Files Created During Deployment
+
+| File | Purpose |
+|------|---------|
+| `/tmp/sp-deploy.lock` | Prevents concurrent deployments |
+| `/tmp/sp-deploy-backup-dir.txt` | Points to backup directory |
+| `/tmp/sp-deploy-YYYYMMDD-HHMMSS/` | Backup directory with rollback info |
+| `/tmp/health_response.json` | Last health check response |
+
+## Troubleshooting
+
+### Deployment stuck
+Check for stale lock file:
+```bash
+ls -la /tmp/sp-deploy.lock
+rm /tmp/sp-deploy.lock  # If stale (>10 minutes old)
+```
+
+### Health check failing
+```bash
+# Check application logs
+tail -f /var/www/sample-platform/logs/error.log
+
+# Check service status
+systemctl status platform
+
+# Test health endpoint manually
+curl -v http://127.0.0.1/health
+```
+
+### Rollback failed
+```bash
+# Check current state
+cd /var/www/sample-platform
+git log --oneline -5
+FLASK_APP=./run.py flask db current
+
+# Manual rollback
+git checkout <previous-commit>
+FLASK_APP=./run.py flask db downgrade <migration-id>
+systemctl restart platform
+```

install/deploy/deploy.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+# Main deployment script
+# Exit codes: 0 = success, 1 = failure (triggers rollback)
+#
+# This script performs the actual deployment steps:
+# 1. Pull latest code from git
+# 2. Install/update Python dependencies
+# 3. Run database migrations
+# 4. Copy CI scripts
+# 5. Reload the application service
+
+set -e
+
+INSTALL_FOLDER="${INSTALL_FOLDER:-/var/www/sample-platform}"
+SAMPLE_REPOSITORY="${SAMPLE_REPOSITORY:-/repository}"
+DEPLOY_BRANCH="${DEPLOY_BRANCH:-master}"
+
+echo "=== Starting deployment ==="
+echo "Timestamp: $(date -Iseconds)"
+echo "Branch: $DEPLOY_BRANCH"
+echo "Install folder: $INSTALL_FOLDER"
+
+cd "$INSTALL_FOLDER"
+
+# Step 1: Fetch latest code
+echo ""
+echo "--- Step 1: Fetching latest code ---"
+git fetch origin "$DEPLOY_BRANCH"
+echo "✓ Fetched latest from origin/$DEPLOY_BRANCH"
+
+# Step 2: Check for local changes and handle them
+if ! git diff --quiet; then
+    echo "WARNING: Local changes detected, stashing..."
+    git stash push -m "pre-deploy-$(date +%Y%m%d-%H%M%S)" || true
+fi
+
+# Step 3: Update to latest code
+echo ""
+echo "--- Step 2: Updating code ---"
+git checkout "$DEPLOY_BRANCH"
+git reset --hard "origin/$DEPLOY_BRANCH"
+NEW_COMMIT=$(git rev-parse HEAD)
+echo "✓ Updated to commit: $NEW_COMMIT"
+
+# Step 4: Update dependencies
+echo ""
+echo "--- Step 3: Updating dependencies ---"
+python3 -m pip install -r requirements.txt --quiet --disable-pip-version-check
+echo "✓ Dependencies updated"
+
+# Step 5: Run database migrations
+echo ""
+echo "--- Step 4: Running database migrations ---"
+FLASK_APP=./run.py python3 -m flask db upgrade
+echo "✓ Database migrations complete"
+
+# Step 6: Copy CI scripts (if directories exist)
+echo ""
+echo "--- Step 5: Updating CI scripts ---"
+if [ -d "${SAMPLE_REPOSITORY}/TestData" ]; then
+    if [ -f "install/ci-vm/ci-linux/ci/bootstrap" ]; then
+        mkdir -p "${SAMPLE_REPOSITORY}/TestData/ci-linux"
+        cp "install/ci-vm/ci-linux/ci/bootstrap" "${SAMPLE_REPOSITORY}/TestData/ci-linux/bootstrap"
+        echo "✓ Copied ci-linux/bootstrap"
+    fi
+    if [ -f "install/ci-vm/ci-linux/ci/runCI" ]; then
+        mkdir -p "${SAMPLE_REPOSITORY}/TestData/ci-linux"
+        cp "install/ci-vm/ci-linux/ci/runCI" "${SAMPLE_REPOSITORY}/TestData/ci-linux/runCI"
+        echo "✓ Copied ci-linux/runCI"
+    fi
+    if [ -f "install/ci-vm/ci-windows/ci/runCI.bat" ]; then
+        mkdir -p "${SAMPLE_REPOSITORY}/TestData/ci-windows"
+        cp "install/ci-vm/ci-windows/ci/runCI.bat" "${SAMPLE_REPOSITORY}/TestData/ci-windows/runCI.bat"
+        echo "✓ Copied ci-windows/runCI.bat"
+    fi
+else
+    echo "⚠ TestData directory not found, skipping CI script copy"
+fi
+
+# Step 7: Reload application
+echo ""
+echo "--- Step 6: Reloading application ---"
+if systemctl is-active --quiet platform; then
+    systemctl reload platform
+    echo "✓ Platform service reloaded"
+else
+    echo "⚠ Platform service not running, attempting start..."
+    systemctl start platform
+    echo "✓ Platform service started"
+fi
+
+echo ""
+echo "=== Deployment complete ==="
+echo "Deployed commit: $NEW_COMMIT"
+echo "Waiting for application to start..."
+exit 0

install/deploy/post_deploy.sh

@@ -0,0 +1,75 @@
+#!/bin/bash
+# Post-deployment health check
+# Exit codes: 0 = healthy, 1 = unhealthy (triggers rollback)
+#
+# This script verifies the deployment was successful by checking:
+# 1. The /health endpoint (if available)
+# 2. Fallback: the root endpoint responds with 2xx/3xx
+
+set -e
+
+# Configuration
+HEALTH_URL="${HEALTH_URL:-http://127.0.0.1/health}"
+FALLBACK_URL="${FALLBACK_URL:-http://127.0.0.1/}"
+MAX_RETRIES="${MAX_RETRIES:-6}"
+RETRY_DELAY="${RETRY_DELAY:-5}"
+
+echo "=== Post-deployment health check ==="
+echo "Timestamp: $(date -Iseconds)"
+echo "Health URL: $HEALTH_URL"
+echo "Max retries: $MAX_RETRIES"
+echo "Retry delay: ${RETRY_DELAY}s"
+
+# Wait for application to start
+echo ""
+echo "Waiting for application to initialize..."
+sleep 3
+
+# Check health endpoint with retries
+echo ""
+echo "--- Health check ---"
+for i in $(seq 1 $MAX_RETRIES); do
+    echo "Attempt $i/$MAX_RETRIES..."
+
+    # Try the /health endpoint first
+    HTTP_CODE=$(curl -s -o /tmp/health_response.json -w "%{http_code}" "$HEALTH_URL" 2>/dev/null || echo "000")
+
+    if [ "$HTTP_CODE" = "200" ]; then
+        echo "✓ Health check passed (HTTP $HTTP_CODE)"
+        echo ""
+        echo "Response:"
+        cat /tmp/health_response.json 2>/dev/null | python3 -m json.tool 2>/dev/null || cat /tmp/health_response.json
+        echo ""
+        echo "=== Deployment verified successfully ==="
+        exit 0
+    elif [ "$HTTP_CODE" = "404" ]; then
+        # Health endpoint doesn't exist, try fallback
+        echo "Health endpoint not found, trying fallback URL..."
+        HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$FALLBACK_URL" 2>/dev/null || echo "000")
+        if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 400 ]; then
+            echo "✓ Fallback check passed (HTTP $HTTP_CODE)"
+            echo ""
+            echo "=== Deployment verified successfully (using fallback) ==="
+            exit 0
+        fi
+    elif [ "$HTTP_CODE" = "503" ]; then
+        echo "Health check returned unhealthy (HTTP 503)"
+        echo "Response:"
+        cat /tmp/health_response.json 2>/dev/null || echo "(no response body)"
+    fi
+
+    if [ "$i" -lt "$MAX_RETRIES" ]; then
+        echo "Health check returned HTTP $HTTP_CODE, retrying in ${RETRY_DELAY}s..."
+        sleep "$RETRY_DELAY"
+    fi
+done
+
+echo ""
+echo "=== HEALTH CHECK FAILED ==="
+echo "Health check failed after $MAX_RETRIES attempts"
+echo "Last HTTP code: $HTTP_CODE"
+echo "Last response:"
+cat /tmp/health_response.json 2>/dev/null || echo "(no response)"
+echo ""
+echo "Deployment verification FAILED - rollback recommended"
+exit 1