diff --git a/documentation/deployment-guides/gcp-managed/00-prerequisites-managed.md b/documentation/deployment-guides/gcp-managed/00-prerequisites-managed.md new file mode 100644 index 0000000..6cd5798 --- /dev/null +++ b/documentation/deployment-guides/gcp-managed/00-prerequisites-managed.md @@ -0,0 +1,438 @@ +# Google Cloud Prerequisites for ERPNext with Managed Services + +## Overview + +This guide covers the prerequisites and initial setup required for deploying ERPNext on Google Cloud Platform (GCP) using managed database services: Cloud SQL for MySQL and Memorystore for Redis. + +## 🔧 Required Tools + +### 1. Google Cloud SDK +```bash +# Install gcloud CLI +curl https://sdk.cloud.google.com | bash +exec -l $SHELL + +# Initialize gcloud +gcloud init +gcloud auth login +gcloud config set project YOUR_PROJECT_ID +``` + +### 2. kubectl (Kubernetes CLI) - For GKE Option +```bash +# Install kubectl +gcloud components install kubectl + +# Verify installation +kubectl version --client +``` + +### 3. Docker (for local testing and Cloud Run) +```bash +# Install Docker +curl -fsSL https://get.docker.com -o get-docker.sh && sh get-docker.sh + +# Enable Docker BuildKit +export DOCKER_BUILDKIT=1 +``` + +### 4. Helm (for GKE Kubernetes package management) +```bash +# Install Helm +curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash + +# Verify installation +helm version +``` + +## 🏗️ Google Cloud Project Setup + +### 1. Create or Select Project +```bash +# Create new project +gcloud projects create erpnext-production --name="ERPNext Production" + +# Set as current project +gcloud config set project erpnext-production + +# Enable billing (required for most services) +# This must be done via the Console: https://console.cloud.google.com/billing +``` + +### 2. Enable Required APIs +```bash +# Enable essential APIs for managed services deployment +gcloud services enable \ + container.googleapis.com \ + compute.googleapis.com \ + sqladmin.googleapis.com \ + redis.googleapis.com \ + secretmanager.googleapis.com \ + cloudbuild.googleapis.com \ + cloudresourcemanager.googleapis.com \ + monitoring.googleapis.com \ + logging.googleapis.com \ + run.googleapis.com \ + vpcaccess.googleapis.com \ + servicenetworking.googleapis.com +``` + +### 3. Set Default Region/Zone +```bash +# Set default compute region and zone +gcloud config set compute/region us-central1 +gcloud config set compute/zone us-central1-a + +# Verify configuration +gcloud config list +``` + +## 🔐 Security Setup + +### 1. Service Account Creation +```bash +# Create service account for ERPNext +gcloud iam service-accounts create erpnext-managed \ + --display-name="ERPNext Managed Services Account" \ + --description="Service account for ERPNext with managed database services" + +# Grant necessary roles for managed services +gcloud projects add-iam-policy-binding erpnext-production \ + --member="serviceAccount:erpnext-managed@erpnext-production.iam.gserviceaccount.com" \ + --role="roles/cloudsql.client" + +gcloud projects add-iam-policy-binding erpnext-production \ + --member="serviceAccount:erpnext-managed@erpnext-production.iam.gserviceaccount.com" \ + --role="roles/redis.editor" + +gcloud projects add-iam-policy-binding erpnext-production \ + --member="serviceAccount:erpnext-managed@erpnext-production.iam.gserviceaccount.com" \ + --role="roles/secretmanager.secretAccessor" + +# For GKE deployment +gcloud projects add-iam-policy-binding erpnext-production \ + --member="serviceAccount:erpnext-managed@erpnext-production.iam.gserviceaccount.com" \ + --role="roles/container.developer" + +# For Cloud Run deployment +gcloud projects add-iam-policy-binding erpnext-production \ + --member="serviceAccount:erpnext-managed@erpnext-production.iam.gserviceaccount.com" \ + --role="roles/run.developer" +``` + +### 2. Create Service Account Key (for local development) +```bash +# Generate service account key +gcloud iam service-accounts keys create ~/erpnext-managed-key.json \ + --iam-account=erpnext-managed@erpnext-production.iam.gserviceaccount.com + +# Set environment variable +export GOOGLE_APPLICATION_CREDENTIALS=~/erpnext-managed-key.json +``` + +### 3. Secret Manager Setup +```bash +# Create secrets for ERPNext +gcloud secrets create erpnext-admin-password \ + --data-file=<(echo -n "YourSecurePassword123!") + +gcloud secrets create erpnext-db-password \ + --data-file=<(echo -n "YourDBPassword123!") + +gcloud secrets create erpnext-api-key \ + --data-file=<(echo -n "your-api-key-here") + +gcloud secrets create erpnext-api-secret \ + --data-file=<(echo -n "your-api-secret-here") + +# Additional secret for database connection +gcloud secrets create erpnext-db-connection-name \ + --data-file=<(echo -n "erpnext-production:us-central1:erpnext-db") +``` + +## 🌐 Networking Setup + +### 1. VPC Network for Private Services +```bash +# Create custom VPC network for managed services +gcloud compute networks create erpnext-vpc \ + --subnet-mode=custom + +# Create subnet for compute resources +gcloud compute networks subnets create erpnext-subnet \ + --network=erpnext-vpc \ + --range=10.0.0.0/24 \ + --region=us-central1 + +# Create subnet for private services (Cloud SQL, Memorystore) +gcloud compute networks subnets create erpnext-private-subnet \ + --network=erpnext-vpc \ + --range=10.1.0.0/24 \ + --region=us-central1 + +# Allocate IP range for private services +gcloud compute addresses create erpnext-private-ip-range \ + --global \ + --purpose=VPC_PEERING \ + --prefix-length=16 \ + --network=erpnext-vpc + +# Create private connection for managed services +gcloud services vpc-peerings connect \ + --service=servicenetworking.googleapis.com \ + --ranges=erpnext-private-ip-range \ + --network=erpnext-vpc +``` + +### 2. VPC Access Connector (for Cloud Run) +```bash +# Create VPC access connector for Cloud Run to access private services +gcloud compute networks vpc-access connectors create erpnext-connector \ + --network=erpnext-vpc \ + --region=us-central1 \ + --range=10.2.0.0/28 \ + --min-instances=2 \ + --max-instances=10 +``` + +### 3. Firewall Rules +```bash +# Create firewall rules +gcloud compute firewall-rules create erpnext-allow-internal \ + --network=erpnext-vpc \ + --allow=tcp,udp,icmp \ + --source-ranges=10.0.0.0/16 + +gcloud compute firewall-rules create erpnext-allow-http \ + --network=erpnext-vpc \ + --allow=tcp:80,tcp:443,tcp:8080 \ + --source-ranges=0.0.0.0/0 + +# Allow Cloud SQL connections +gcloud compute firewall-rules create erpnext-allow-mysql \ + --network=erpnext-vpc \ + --allow=tcp:3306 \ + --source-ranges=10.0.0.0/16,10.1.0.0/16,10.2.0.0/28 + +# Allow Redis connections +gcloud compute firewall-rules create erpnext-allow-redis \ + --network=erpnext-vpc \ + --allow=tcp:6379 \ + --source-ranges=10.0.0.0/16,10.1.0.0/16,10.2.0.0/28 +``` + +## 💾 Managed Database Services Setup + +### 1. Cloud SQL (MySQL) Instance +```bash +# Create Cloud SQL instance +gcloud sql instances create erpnext-db \ + --database-version=MYSQL_8_0 \ + --tier=db-n1-standard-2 \ + --region=us-central1 \ + --network=erpnext-vpc \ + --no-assign-ip \ + --storage-size=100GB \ + --storage-type=SSD \ + --storage-auto-increase \ + --backup \ + --backup-start-time=02:00 \ + --maintenance-window-day=SUN \ + --maintenance-window-hour=3 \ + --maintenance-release-channel=production \ + --deletion-protection + +# Create database +gcloud sql databases create erpnext --instance=erpnext-db + +# Create database user +gcloud sql users create erpnext \ + --instance=erpnext-db \ + --password=YourDBPassword123! + +# Get connection name for applications +gcloud sql instances describe erpnext-db --format="value(connectionName)" +``` + +### 2. Memorystore (Redis) Instance +```bash +# Create Memorystore Redis instance +gcloud redis instances create erpnext-redis \ + --size=1 \ + --region=us-central1 \ + --network=erpnext-vpc \ + --redis-version=redis_6_x \ + --maintenance-window-day=sunday \ + --maintenance-window-hour=3 \ + --redis-config maxmemory-policy=allkeys-lru + +# Get Redis host IP for applications +gcloud redis instances describe erpnext-redis --region=us-central1 --format="value(host)" +``` + +### 3. Database Initialization +```bash +# Create initialization script +cat > /tmp/init_erpnext_db.sql < Dockerfile < startup.sh <<'EOF' +#!/bin/bash +set -e + +# Start Cloud SQL Proxy in background +if [ -n "$DB_CONNECTION_NAME" ]; then + echo "Starting Cloud SQL Proxy..." + /usr/local/bin/cloud_sql_proxy -instances=$DB_CONNECTION_NAME=tcp:3306 & + + # Wait for proxy to be ready + until nc -z localhost 3306; do + echo "Waiting for database connection..." + sleep 2 + done + echo "Database connection established" +fi + +# Set database connection +export DB_HOST="127.0.0.1" +export DB_PORT="3306" + +# Initialize site if needed +if [ ! -d "/home/frappe/frappe-bench/sites/frontend" ]; then + echo "Initializing ERPNext site..." + cd /home/frappe/frappe-bench + bench new-site frontend \ + --admin-password "$ADMIN_PASSWORD" \ + --mariadb-root-password "$DB_PASSWORD" \ + --install-app erpnext \ + --set-default +fi + +# Start ERPNext +echo "Starting ERPNext backend..." +cd /home/frappe/frappe-bench +exec gunicorn -b 0.0.0.0:$PORT -w $WORKERS -t $TIMEOUT frappe.app:application +EOF + +# Build and push image +gcloud builds submit --tag gcr.io/$PROJECT_ID/erpnext-backend:latest +``` + +### 2. Create Dockerfile for Frontend Service + +```bash +# Create directory for frontend build +mkdir -p ../frontend +cd ../frontend + +cat > Dockerfile < nginx.conf <<'EOF' +user nginx; +worker_processes auto; +error_log /var/log/nginx/error.log notice; +pid /var/run/nginx.pid; + +events { + worker_connections 1024; +} + +http { + include /etc/nginx/mime.types; + default_type application/octet-stream; + + log_format main '$remote_addr - $remote_user [$time_local] "$request" ' + '$status $body_bytes_sent "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for"'; + + access_log /var/log/nginx/access.log main; + + sendfile on; + tcp_nopush on; + tcp_nodelay on; + keepalive_timeout 65; + types_hash_max_size 2048; + client_max_body_size 50m; + + gzip on; + gzip_vary on; + gzip_proxied any; + gzip_comp_level 6; + gzip_types + text/plain + text/css + text/xml + text/javascript + application/json + application/javascript + application/xml+rss + application/atom+xml + image/svg+xml; + + # Upstream backend API + upstream backend { + server api.yourdomain.com:443; + keepalive 32; + } + + server { + listen 8080; + server_name _; + + # Security headers + add_header X-Frame-Options SAMEORIGIN; + add_header X-Content-Type-Options nosniff; + add_header X-XSS-Protection "1; mode=block"; + + # Serve static assets from Cloud Storage + location /assets { + proxy_pass https://storage.googleapis.com/erpnext-files-PROJECT_ID/assets; + proxy_set_header Host storage.googleapis.com; + expires 1y; + add_header Cache-Control "public, immutable"; + } + + # Proxy API requests to backend service + location /api { + proxy_pass https://backend; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } + + # Serve app from backend + location / { + proxy_pass https://backend; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } + + # Health check + location /health { + access_log off; + return 200 "healthy\n"; + add_header Content-Type text/plain; + } + } +} +EOF + +# Create startup script +cat > startup.sh <<'EOF' +#!/bin/bash +# Replace PROJECT_ID placeholder in nginx config +sed -i "s/PROJECT_ID/$PROJECT_ID/g" /etc/nginx/nginx.conf + +# Update backend upstream with actual API domain +sed -i "s/api.yourdomain.com/$API_DOMAIN/g" /etc/nginx/nginx.conf + +# Start nginx +exec nginx -g 'daemon off;' +EOF + +# Build and push frontend image +gcloud builds submit --tag gcr.io/$PROJECT_ID/erpnext-frontend:latest +``` + +### 3. Background Tasks Service + +```bash +# Create directory for background tasks +mkdir -p ../background +cd ../background + +cat > Dockerfile < task_processor.py <<'EOF' +import os +import json +import logging +from flask import Flask, request, jsonify +import frappe +from frappe.utils.background_jobs import execute_job + +app = Flask(__name__) + +@app.route('/process-task', methods=['POST']) +def process_task(): + """Process background task from Cloud Tasks""" + try: + # Get task data + task_data = request.get_json() + + # Set site context + frappe.init(site='frontend') + frappe.connect() + + # Execute the job + job_name = task_data.get('job_name') + kwargs = task_data.get('kwargs', {}) + + result = execute_job(job_name, **kwargs) + + frappe.db.commit() + frappe.destroy() + + return jsonify({'status': 'success', 'result': result}) + + except Exception as e: + logging.error(f"Task processing failed: {str(e)}") + return jsonify({'status': 'error', 'message': str(e)}), 500 + +@app.route('/health', methods=['GET']) +def health_check(): + return jsonify({'status': 'healthy'}) + +if __name__ == '__main__': + app.run(host='0.0.0.0', port=int(os.environ.get('PORT', 8080))) +EOF + +# Create startup script +cat > startup.sh <<'EOF' +#!/bin/bash +set -e + +# Start Cloud SQL Proxy +if [ -n "$DB_CONNECTION_NAME" ]; then + /usr/local/bin/cloud_sql_proxy -instances=$DB_CONNECTION_NAME=tcp:3306 & + + until nc -z localhost 3306; do + echo "Waiting for database connection..." + sleep 2 + done +fi + +export DB_HOST="127.0.0.1" +export DB_PORT="3306" + +# Start task processor +cd /home/frappe/frappe-bench +exec python /task_processor.py +EOF + +# Build and push background service image +gcloud builds submit --tag gcr.io/$PROJECT_ID/erpnext-background:latest +``` + +## 🚀 Deploy Cloud Run Services + +### 1. Deploy Backend API Service + +```bash +# Deploy backend service +gcloud run deploy erpnext-backend \ + --image gcr.io/$PROJECT_ID/erpnext-backend:latest \ + --platform managed \ + --region $REGION \ + --allow-unauthenticated \ + --vpc-connector $VPC_CONNECTOR \ + --set-env-vars="DB_CONNECTION_NAME=$DB_CONNECTION_NAME" \ + --set-env-vars="REDIS_HOST=$REDIS_HOST" \ + --set-env-vars="REDIS_PORT=6379" \ + --set-env-vars="PROJECT_ID=$PROJECT_ID" \ + --set-secrets="ADMIN_PASSWORD=erpnext-admin-password:latest" \ + --set-secrets="DB_PASSWORD=erpnext-db-password:latest" \ + --set-secrets="API_KEY=erpnext-api-key:latest" \ + --set-secrets="API_SECRET=erpnext-api-secret:latest" \ + --service-account erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com \ + --memory 2Gi \ + --cpu 2 \ + --concurrency 80 \ + --timeout 300 \ + --min-instances 1 \ + --max-instances 100 + +# Get backend service URL +export BACKEND_URL=$(gcloud run services describe erpnext-backend --region=$REGION --format="value(status.url)") +echo "Backend URL: $BACKEND_URL" +``` + +### 2. Deploy Frontend Service + +```bash +# Update API_DOMAIN environment variable +export API_DOMAIN=$(echo $BACKEND_URL | sed 's|https://||') + +# Deploy frontend service +gcloud run deploy erpnext-frontend \ + --image gcr.io/$PROJECT_ID/erpnext-frontend:latest \ + --platform managed \ + --region $REGION \ + --allow-unauthenticated \ + --set-env-vars="PROJECT_ID=$PROJECT_ID" \ + --set-env-vars="API_DOMAIN=$API_DOMAIN" \ + --memory 512Mi \ + --cpu 1 \ + --concurrency 1000 \ + --timeout 60 \ + --min-instances 0 \ + --max-instances 10 + +# Get frontend service URL +export FRONTEND_URL=$(gcloud run services describe erpnext-frontend --region=$REGION --format="value(status.url)") +echo "Frontend URL: $FRONTEND_URL" +``` + +### 3. Deploy Background Tasks Service + +```bash +# Deploy background service +gcloud run deploy erpnext-background \ + --image gcr.io/$PROJECT_ID/erpnext-background:latest \ + --platform managed \ + --region $REGION \ + --no-allow-unauthenticated \ + --vpc-connector $VPC_CONNECTOR \ + --set-env-vars="DB_CONNECTION_NAME=$DB_CONNECTION_NAME" \ + --set-env-vars="REDIS_HOST=$REDIS_HOST" \ + --set-env-vars="PROJECT_ID=$PROJECT_ID" \ + --set-secrets="DB_PASSWORD=erpnext-db-password:latest" \ + --service-account erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com \ + --memory 1Gi \ + --cpu 1 \ + --concurrency 10 \ + --timeout 900 \ + --min-instances 0 \ + --max-instances 10 + +# Get background service URL +export BACKGROUND_URL=$(gcloud run services describe erpnext-background --region=$REGION --format="value(status.url)") +echo "Background URL: $BACKGROUND_URL" +``` + +## 📅 Setup Cloud Scheduler for Scheduled Tasks + +### 1. Create Cloud Tasks Queue + +```bash +# Create task queue for background jobs +gcloud tasks queues create erpnext-tasks \ + --location=$REGION \ + --max-dispatches-per-second=100 \ + --max-concurrent-dispatches=1000 \ + --max-attempts=3 +``` + +### 2. Create Scheduled Jobs + +```bash +# Daily backup job +gcloud scheduler jobs create http erpnext-daily-backup \ + --location=$REGION \ + --schedule="0 2 * * *" \ + --uri="$BACKGROUND_URL/process-task" \ + --http-method=POST \ + --headers="Content-Type=application/json" \ + --message-body='{"job_name": "frappe.utils.backup.daily_backup", "kwargs": {}}' \ + --oidc-service-account-email=erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com + +# ERPNext scheduler (every 5 minutes) +gcloud scheduler jobs create http erpnext-scheduler \ + --location=$REGION \ + --schedule="*/5 * * * *" \ + --uri="$BACKGROUND_URL/process-task" \ + --http-method=POST \ + --headers="Content-Type=application/json" \ + --message-body='{"job_name": "frappe.utils.scheduler.execute_all", "kwargs": {}}' \ + --oidc-service-account-email=erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com + +# Email queue processor (every minute) +gcloud scheduler jobs create http erpnext-email-queue \ + --location=$REGION \ + --schedule="* * * * *" \ + --uri="$BACKGROUND_URL/process-task" \ + --http-method=POST \ + --headers="Content-Type=application/json" \ + --message-body='{"job_name": "frappe.email.queue.flush", "kwargs": {}}' \ + --oidc-service-account-email=erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com +``` + +## 🌐 Custom Domain and Load Balancer Setup + +### 1. Create Global Load Balancer + +```bash +# Create HTTP health check +gcloud compute health-checks create http erpnext-health-check \ + --request-path="/health" \ + --port=8080 + +# Create backend service for frontend +gcloud compute backend-services create erpnext-frontend-backend \ + --protocol=HTTP \ + --health-checks=erpnext-health-check \ + --global + +# Add Cloud Run NEG for frontend +gcloud compute network-endpoint-groups create erpnext-frontend-neg \ + --region=$REGION \ + --network-endpoint-type=serverless \ + --cloud-run-service=erpnext-frontend + +gcloud compute backend-services add-backend erpnext-frontend-backend \ + --global \ + --network-endpoint-group=erpnext-frontend-neg \ + --network-endpoint-group-region=$REGION + +# Create backend service for API +gcloud compute backend-services create erpnext-api-backend \ + --protocol=HTTP \ + --health-checks=erpnext-health-check \ + --global + +# Add Cloud Run NEG for backend API +gcloud compute network-endpoint-groups create erpnext-backend-neg \ + --region=$REGION \ + --network-endpoint-type=serverless \ + --cloud-run-service=erpnext-backend + +gcloud compute backend-services add-backend erpnext-api-backend \ + --global \ + --network-endpoint-group=erpnext-backend-neg \ + --network-endpoint-group-region=$REGION +``` + +### 2. Create URL Map and SSL Certificate + +```bash +# Create URL map +gcloud compute url-maps create erpnext-url-map \ + --default-service=erpnext-frontend-backend + +# Add path matcher for API routes +gcloud compute url-maps add-path-matcher erpnext-url-map \ + --path-matcher-name=api-matcher \ + --default-service=erpnext-frontend-backend \ + --path-rules="/api/*=erpnext-api-backend,/method/*=erpnext-api-backend" + +# Create managed SSL certificate +gcloud compute ssl-certificates create erpnext-ssl-cert \ + --domains=$DOMAIN + +# Create HTTPS proxy +gcloud compute target-https-proxies create erpnext-https-proxy \ + --ssl-certificates=erpnext-ssl-cert \ + --url-map=erpnext-url-map + +# Create global forwarding rule +gcloud compute forwarding-rules create erpnext-https-rule \ + --global \ + --target-https-proxy=erpnext-https-proxy \ + --ports=443 + +# Get global IP address +export GLOBAL_IP=$(gcloud compute forwarding-rules describe erpnext-https-rule --global --format="value(IPAddress)") +echo "Global IP: $GLOBAL_IP" +echo "Point your domain $DOMAIN to this IP address" +``` + +## 🔍 Initialize ERPNext Site + +### 1. Run Site Creation + +```bash +# Trigger site creation via background service +curl -X POST "$BACKGROUND_URL/process-task" \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $(gcloud auth print-identity-token)" \ + -d '{ + "job_name": "frappe.installer.install_app", + "kwargs": { + "site": "frontend", + "app": "erpnext" + } + }' +``` + +### 2. Alternative: Manual Site Creation + +```bash +# Create a temporary Cloud Run job for site creation +gcloud run jobs create erpnext-init-site \ + --image gcr.io/$PROJECT_ID/erpnext-backend:latest \ + --region $REGION \ + --vpc-connector $VPC_CONNECTOR \ + --set-env-vars="DB_CONNECTION_NAME=$DB_CONNECTION_NAME" \ + --set-secrets="ADMIN_PASSWORD=erpnext-admin-password:latest" \ + --set-secrets="DB_PASSWORD=erpnext-db-password:latest" \ + --service-account erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com \ + --memory 2Gi \ + --cpu 2 \ + --max-retries 3 \ + --parallelism 1 \ + --task-count 1 \ + --args="bench,new-site,frontend,--admin-password,\$ADMIN_PASSWORD,--mariadb-root-password,\$DB_PASSWORD,--install-app,erpnext,--set-default" + +# Execute the job +gcloud run jobs execute erpnext-init-site --region $REGION --wait +``` + +## 📊 Monitoring and Observability + +### 1. Cloud Run Metrics + +```bash +# Enable detailed monitoring +gcloud services enable cloudmonitoring.googleapis.com + +# Create custom dashboard +cat > monitoring-dashboard.json < alert-policy.json < disaster_recovery.md < /dev/null & +done +wait + +echo "Load test completed. Check Cloud Run metrics in console." +``` + +## 🚨 Troubleshooting + +### 1. Common Issues + +```bash +# Check service logs +gcloud logging read "resource.type=\"cloud_run_revision\" AND resource.labels.service_name=\"erpnext-backend\"" --limit=50 + +# Check database connectivity +gcloud run services logs read erpnext-backend --region=$REGION --limit=20 + +# Test VPC connector +gcloud compute networks vpc-access connectors describe $VPC_CONNECTOR --region=$REGION +``` + +### 2. Performance Issues + +```bash +# Check instance allocation +gcloud run services describe erpnext-backend --region=$REGION --format="value(status.traffic[].allocation)" + +# Monitor cold starts +gcloud logging read "resource.type=\"cloud_run_revision\" AND textPayload:\"Container called exit\"" --limit=10 +``` + +## 📈 Performance Benefits + +### 1. Scalability +- **Auto-scaling**: 0 to 1000+ instances automatically +- **Global distribution**: Multi-region deployment capability +- **Pay-per-use**: No idle costs + +### 2. Reliability +- **Managed infrastructure**: Google handles all infrastructure +- **Automatic deployments**: Zero-downtime deployments +- **Health checks**: Automatic instance replacement + +### 3. Cost Efficiency +- **No minimum costs**: Pay only for actual usage +- **Automatic optimization**: CPU and memory auto-scaling +- **Reduced operational overhead**: No server management + +## 📚 Additional Resources + +- [Cloud Run Documentation](https://cloud.google.com/run/docs) +- [Cloud Tasks Documentation](https://cloud.google.com/tasks/docs) +- [Cloud Scheduler Documentation](https://cloud.google.com/scheduler/docs) +- [Cloud Run Best Practices](https://cloud.google.com/run/docs/best-practices) + +## ➡️ Next Steps + +1. **Production Hardening**: Follow `03-production-managed-setup.md` +2. **Monitoring Setup**: Configure detailed monitoring and alerting +3. **Performance Tuning**: Optimize based on usage patterns +4. **Custom Development**: Add custom ERPNext apps and configurations + +--- + +**⚠️ Important Notes**: +- Cloud Run bills per 100ms of CPU time and memory allocation +- Cold starts may affect initial response times +- Database connections should be managed carefully due to Cloud SQL limits +- Consider using Cloud CDN for better global performance \ No newline at end of file diff --git a/documentation/deployment-guides/gcp-managed/03-production-managed-setup.md b/documentation/deployment-guides/gcp-managed/03-production-managed-setup.md new file mode 100644 index 0000000..7aa40d3 --- /dev/null +++ b/documentation/deployment-guides/gcp-managed/03-production-managed-setup.md @@ -0,0 +1,1292 @@ +# ERPNext Production Setup with Managed Services + +## Overview + +This guide covers production-ready configurations, security hardening, monitoring, backup strategies, and operational best practices for ERPNext using Google Cloud managed services (Cloud SQL, Memorystore, Cloud Run, and GKE). + +## 🔐 Enhanced Security Configuration + +### 1. Private Service Connect for Managed Services + +```bash +# Create Private Service Connect endpoint for Cloud SQL +gcloud compute addresses create cloudsql-psc-ip \ + --global \ + --purpose=PRIVATE_SERVICE_CONNECT \ + --network=erpnext-vpc + +# Create Private Service Connect endpoint +gcloud compute forwarding-rules create cloudsql-psc-endpoint \ + --global \ + --network=erpnext-vpc \ + --address=cloudsql-psc-ip \ + --target-service-attachment=projects/PROJECT_ID/regions/us-central1/serviceAttachments/cloudsql-psc + +# Update firewall rules for PSC +gcloud compute firewall-rules create allow-psc-cloudsql \ + --network=erpnext-vpc \ + --allow=tcp:3306 \ + --source-ranges=10.0.0.0/16 \ + --target-tags=cloudsql-psc +``` + +### 2. Advanced IAM and Service Account Security + +```bash +# Create least-privilege service accounts for different components +gcloud iam service-accounts create erpnext-frontend \ + --display-name="ERPNext Frontend Service Account" + +gcloud iam service-accounts create erpnext-backend \ + --display-name="ERPNext Backend Service Account" + +gcloud iam service-accounts create erpnext-worker \ + --display-name="ERPNext Worker Service Account" + +# Grant minimal required permissions +# Backend service permissions +gcloud projects add-iam-policy-binding $PROJECT_ID \ + --member="serviceAccount:erpnext-backend@$PROJECT_ID.iam.gserviceaccount.com" \ + --role="roles/cloudsql.client" + +gcloud projects add-iam-policy-binding $PROJECT_ID \ + --member="serviceAccount:erpnext-backend@$PROJECT_ID.iam.gserviceaccount.com" \ + --role="roles/redis.editor" + +gcloud projects add-iam-policy-binding $PROJECT_ID \ + --member="serviceAccount:erpnext-backend@$PROJECT_ID.iam.gserviceaccount.com" \ + --role="roles/secretmanager.secretAccessor" + +# Worker service permissions (more restricted) +gcloud projects add-iam-policy-binding $PROJECT_ID \ + --member="serviceAccount:erpnext-worker@$PROJECT_ID.iam.gserviceaccount.com" \ + --role="roles/cloudsql.client" + +gcloud projects add-iam-policy-binding $PROJECT_ID \ + --member="serviceAccount:erpnext-worker@$PROJECT_ID.iam.gserviceaccount.com" \ + --role="roles/secretmanager.secretAccessor" + +# Frontend service permissions (most restricted) +gcloud projects add-iam-policy-binding $PROJECT_ID \ + --member="serviceAccount:erpnext-frontend@$PROJECT_ID.iam.gserviceaccount.com" \ + --role="roles/storage.objectViewer" +``` + +### 3. Cloud SQL Security Hardening + +```bash +# Enable SSL enforcement +gcloud sql instances patch erpnext-db \ + --require-ssl + +# Create SSL certificates for secure connections +gcloud sql ssl-certs create erpnext-client-cert \ + --instance=erpnext-db + +# Download client certificates +gcloud sql ssl-certs describe erpnext-client-cert \ + --instance=erpnext-db \ + --format="value(cert)" > client-cert.pem + +gcloud sql ssl-certs describe erpnext-client-cert \ + --instance=erpnext-db \ + --format="value(private_key)" > client-key.pem + +gcloud sql instances describe erpnext-db \ + --format="value(serverCaCert.cert)" > server-ca.pem + +# Store SSL certificates in Secret Manager +gcloud secrets create cloudsql-client-cert --data-file=client-cert.pem +gcloud secrets create cloudsql-client-key --data-file=client-key.pem +gcloud secrets create cloudsql-server-ca --data-file=server-ca.pem + +# Enable database flags for security +gcloud sql instances patch erpnext-db \ + --database-flags=slow_query_log=on,log_queries_not_using_indexes=on,general_log=off + +# Enable audit logging +gcloud sql instances patch erpnext-db \ + --database-flags=log_output=FILE +``` + +### 4. Memorystore Security Configuration + +```bash +# Enable AUTH for Redis +gcloud redis instances update erpnext-redis \ + --region=us-central1 \ + --auth-enabled + +# Get AUTH string and store in Secret Manager +AUTH_STRING=$(gcloud redis instances describe erpnext-redis \ + --region=us-central1 \ + --format="value(authString)") + +gcloud secrets create redis-auth-string \ + --data-file=<(echo -n "$AUTH_STRING") + +# Enable transit encryption +gcloud redis instances update erpnext-redis \ + --region=us-central1 \ + --transit-encryption-mode=SERVER_AUTH +``` + +### 5. Binary Authorization for Container Security + +```bash +# Enable Binary Authorization +gcloud container binauthz policy import policy.yaml + +# Create policy.yaml +cat > policy.yaml < custom-metrics.yaml < + resource.type="cloud_run_revision" + resource.labels.service_name="erpnext-backend" + httpRequest.latency + labelExtractors: + status: EXTRACT(httpRequest.status) + method: EXTRACT(httpRequest.requestMethod) + valueExtractor: EXTRACT(httpRequest.latency) + metricDescriptor: + metricKind: GAUGE + valueType: DOUBLE + displayName: ERPNext Response Time + +- name: erpnext-error-rate + type: logging.v2.LogMetric + properties: + name: erpnext_error_rate + description: ERPNext error rate + filter: > + resource.type="cloud_run_revision" + resource.labels.service_name="erpnext-backend" + severity>=ERROR + metricDescriptor: + metricKind: GAUGE + valueType: INT64 + displayName: ERPNext Error Rate +EOF + +gcloud deployment-manager deployments create erpnext-metrics \ + --config custom-metrics.yaml +``` + +### 2. SLO Configuration + +```bash +# Create SLO for ERPNext availability +cat > slo-config.json < alerting-policy.json < transfer-job.json < dr-test.sh <<'EOF' +#!/bin/bash +set -e + +PROJECT_ID_DR="erpnext-dr-test" +REGION_DR="us-east1" + +echo "Starting DR test..." + +# Create test Cloud SQL instance from backup +LATEST_BACKUP=$(gcloud sql backups list --instance=erpnext-db --limit=1 --format="value(id)") +gcloud sql instances create erpnext-db-dr-test \ + --project=$PROJECT_ID_DR \ + --backup=$LATEST_BACKUP \ + --backup-project=$PROJECT_ID \ + --region=$REGION_DR + +# Deploy test Cloud Run service +gcloud run deploy erpnext-backend-dr-test \ + --project=$PROJECT_ID_DR \ + --image gcr.io/$PROJECT_ID/erpnext-backend:latest \ + --region $REGION_DR \ + --set-env-vars="DB_CONNECTION_NAME=$PROJECT_ID_DR:$REGION_DR:erpnext-db-dr-test" + +# Test application functionality +BACKEND_URL=$(gcloud run services describe erpnext-backend-dr-test \ + --project=$PROJECT_ID_DR \ + --region=$REGION_DR \ + --format="value(status.url)") + +# Run health check +if curl -f "$BACKEND_URL/api/method/ping"; then + echo "DR test successful" +else + echo "DR test failed" + exit 1 +fi + +# Cleanup +gcloud sql instances delete erpnext-db-dr-test --project=$PROJECT_ID_DR --quiet +gcloud run services delete erpnext-backend-dr-test --project=$PROJECT_ID_DR --region=$REGION_DR --quiet + +echo "DR test completed and cleaned up" +EOF + +chmod +x dr-test.sh + +# Schedule monthly DR tests +gcloud scheduler jobs create http dr-test-monthly \ + --location=$REGION \ + --schedule="0 2 1 * *" \ + --uri="https://us-central1-$PROJECT_ID.cloudfunctions.net/dr-test-function" \ + --http-method=POST +``` + +### 3. Point-in-Time Recovery Procedures + +```bash +# Create PITR script +cat > pitr-recovery.sh <<'EOF' +#!/bin/bash +# Point-in-Time Recovery Script + +RECOVERY_TIME="$1" +NEW_INSTANCE_NAME="erpnext-db-pitr-$(date +%Y%m%d-%H%M%S)" + +if [ -z "$RECOVERY_TIME" ]; then + echo "Usage: $0 " + echo "Example: $0 '2024-01-15T14:30:00Z'" + exit 1 +fi + +echo "Creating PITR instance for time: $RECOVERY_TIME" + +# Create instance from PITR +gcloud sql instances clone erpnext-db $NEW_INSTANCE_NAME \ + --point-in-time="$RECOVERY_TIME" + +echo "PITR instance created: $NEW_INSTANCE_NAME" +echo "Connection name: $PROJECT_ID:$REGION:$NEW_INSTANCE_NAME" + +# Update application configuration to use new instance +echo "Update your application configuration to use the new instance" +echo "Test the recovery, then promote if successful" +EOF + +chmod +x pitr-recovery.sh +``` + +## 🚀 Performance Optimization + +### 1. Database Performance Tuning + +```bash +# Optimize Cloud SQL for ERPNext workload +gcloud sql instances patch erpnext-db \ + --database-flags=innodb_buffer_pool_size=75%,innodb_log_file_size=256M,max_connections=200,query_cache_size=64M + +# Enable Query Insights for performance monitoring +gcloud sql instances patch erpnext-db \ + --insights-config-query-insights-enabled \ + --insights-config-record-application-tags \ + --insights-config-record-client-address + +# Create read replica for read-heavy workloads +gcloud sql instances create erpnext-db-read-replica \ + --master-instance-name=erpnext-db \ + --region=us-central1 \ + --tier=db-n1-standard-1 +``` + +### 2. Redis Performance Optimization + +```bash +# Scale Redis instance based on memory usage +gcloud redis instances update erpnext-redis \ + --region=us-central1 \ + --size=5 + +# Configure maxmemory policies for different use cases +gcloud redis instances update erpnext-redis \ + --region=us-central1 \ + --redis-config maxmemory-policy=allkeys-lru,timeout=300 +``` + +### 3. Application Performance Optimization + +For Cloud Run: +```bash +# Optimize Cloud Run for performance +gcloud run services update erpnext-backend \ + --region=$REGION \ + --memory=4Gi \ + --cpu=2 \ + --concurrency=80 \ + --min-instances=2 \ + --max-instances=100 \ + --execution-environment=gen2 + +# Enable CPU boost +gcloud run services update erpnext-backend \ + --region=$REGION \ + --cpu-boost +``` + +For GKE: +```bash +# Update GKE deployments with performance optimizations +kubectl apply -f - < blue-green-deploy.sh <<'EOF' +#!/bin/bash +set -e + +SERVICE_NAME="erpnext-backend" +NEW_IMAGE="$1" +REGION="us-central1" + +if [ -z "$NEW_IMAGE" ]; then + echo "Usage: $0 " + exit 1 +fi + +echo "Starting blue-green deployment for $SERVICE_NAME" + +# Deploy new version with 0% traffic +gcloud run deploy $SERVICE_NAME-green \ + --image=$NEW_IMAGE \ + --region=$REGION \ + --no-traffic + +# Run health checks on green version +GREEN_URL=$(gcloud run services describe $SERVICE_NAME-green \ + --region=$REGION --format="value(status.url)") + +if ! curl -f "$GREEN_URL/api/method/ping"; then + echo "Health check failed on green version" + exit 1 +fi + +# Gradually shift traffic +echo "Shifting 10% traffic to green version" +gcloud run services update-traffic $SERVICE_NAME \ + --region=$REGION \ + --to-revisions=$SERVICE_NAME-green=10 + +sleep 30 + +# Monitor metrics for 5 minutes +echo "Monitoring for 5 minutes..." +sleep 300 + +# Full cutover +echo "Completing traffic shift to green version" +gcloud run services update-traffic $SERVICE_NAME \ + --region=$REGION \ + --to-revisions=$SERVICE_NAME-green=100 + +# Cleanup old version after successful deployment +gcloud run revisions delete $(gcloud run revisions list \ + --service=$SERVICE_NAME \ + --region=$REGION \ + --filter="traffic.percent=0" \ + --format="value(metadata.name)" \ + --limit=1) \ + --region=$REGION --quiet + +echo "Blue-green deployment completed successfully" +EOF + +chmod +x blue-green-deploy.sh +``` + +### 2. Canary Deployment for GKE + +```bash +# Create canary deployment using Argo Rollouts +kubectl apply -f - < custom-autoscaler.yaml < dlp-config.json < audit-policy.yaml < security-policy.yaml < business-metrics-dashboard.json < security-events.yaml < + resource.type="cloud_run_revision" + resource.labels.service_name="erpnext-backend" + jsonPayload.event="login_failed" + jsonPayload.ip_address!=NULL + labelExtractors: + ip_address: EXTRACT(jsonPayload.ip_address) + user: EXTRACT(jsonPayload.user) + metricDescriptor: + metricKind: COUNTER + valueType: INT64 + +- name: data-export-events + type: logging.v2.LogMetric + properties: + name: data_export_events + description: Track data export events + filter: > + resource.type="cloud_run_revision" + jsonPayload.event="data_export" + labelExtractors: + user: EXTRACT(jsonPayload.user) + data_type: EXTRACT(jsonPayload.data_type) + metricDescriptor: + metricKind: COUNTER + valueType: INT64 +EOF + +gcloud deployment-manager deployments create security-metrics \ + --config security-events.yaml +``` + +### 2. Automated Incident Response + +```bash +# Create Cloud Function for automated incident response +cat > incident-response.py <<'EOF' +import json +import logging +from google.cloud import run_v2 +from google.cloud import sql_v1 + +def incident_response(request): + """Automated incident response function""" + try: + alert_data = request.get_json() + + # Parse alert + alert_type = alert_data.get('incident', {}).get('condition_name', '') + + if 'High Error Rate' in alert_type: + # Scale up Cloud Run service + client = run_v2.ServicesClient() + service_path = client.service_path( + project='PROJECT_ID', + location='us-central1', + service='erpnext-backend' + ) + + # Increase max instances + service = client.get_service(name=service_path) + service.spec.template.scaling.max_instance_count = 20 + + operation = client.update_service(service=service) + logging.info(f"Scaled up service: {operation.name}") + + elif 'Database Connection' in alert_type: + # Restart Cloud SQL instance if needed + sql_client = sql_v1.SqlInstancesServiceClient() + project = 'PROJECT_ID' + instance = 'erpnext-db' + + # Check instance status and restart if needed + instance_info = sql_client.get(project=project, instance=instance) + if instance_info.state != 'RUNNABLE': + sql_client.restart(project=project, instance=instance) + logging.info(f"Restarted Cloud SQL instance: {instance}") + + return {'status': 'success'} + + except Exception as e: + logging.error(f"Incident response failed: {str(e)}") + return {'status': 'error', 'message': str(e)}, 500 +EOF + +# Deploy incident response function +gcloud functions deploy incident-response \ + --runtime=python39 \ + --trigger-http \ + --source=. \ + --entry-point=incident_response \ + --service-account=erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com +``` + +## 🧹 Maintenance and Lifecycle Management + +### 1. Automated Updates and Patching + +```bash +# Create automated update pipeline +cat > update-pipeline.yaml < cleanup-resources.sh <<'EOF' +#!/bin/bash +set -e + +echo "Starting resource cleanup..." + +# Clean up old Cloud Run revisions +gcloud run revisions list --service=erpnext-backend \ + --region=us-central1 \ + --filter="traffic.percent=0" \ + --format="value(metadata.name)" | \ + head -n -5 | \ + xargs -I {} gcloud run revisions delete {} --region=us-central1 --quiet + +# Clean up old container images +gcloud container images list-tags gcr.io/$PROJECT_ID/erpnext-backend \ + --filter="NOT tags:latest" \ + --format="value(digest)" | \ + head -n -10 | \ + xargs -I {} gcloud container images delete gcr.io/$PROJECT_ID/erpnext-backend@{} --quiet + +# Clean up old Cloud SQL backups (keep last 30) +gcloud sql backups list --instance=erpnext-db \ + --format="value(id)" | \ + tail -n +31 | \ + xargs -I {} gcloud sql backups delete {} --instance=erpnext-db --quiet + +# Clean up old logs (keep last 30 days) +gcloud logging sinks create old-logs-cleanup \ + storage.googleapis.com/cleanup-logs-bucket \ + --log-filter="timestamp<\"$(date -d '30 days ago' -Iseconds)\"" + +echo "Resource cleanup completed" +EOF + +chmod +x cleanup-resources.sh + +# Schedule monthly cleanup +gcloud scheduler jobs create http monthly-cleanup \ + --location=us-central1 \ + --schedule="0 3 1 * *" \ + --uri="https://us-central1-$PROJECT_ID.cloudfunctions.net/cleanup-function" \ + --http-method=POST +``` + +## 📚 Documentation and Knowledge Management + +### 1. Automated Documentation Generation + +```bash +# Create automated documentation pipeline +cat > generate-docs.sh <<'EOF' +#!/bin/bash +set -e + +# Generate infrastructure documentation +echo "# ERPNext Infrastructure Documentation" > infrastructure.md +echo "" >> infrastructure.md +echo "## Cloud SQL Instances" >> infrastructure.md +gcloud sql instances list --format="table(name,region,tier,status)" >> infrastructure.md + +echo "" >> infrastructure.md +echo "## Cloud Run Services" >> infrastructure.md +gcloud run services list --format="table(metadata.name,status.url,status.conditions[0].status)" >> infrastructure.md + +echo "" >> infrastructure.md +echo "## Memorystore Instances" >> infrastructure.md +gcloud redis instances list --format="table(name,region,tier,state)" >> infrastructure.md + +# Generate monitoring documentation +echo "# Monitoring Configuration" > monitoring.md +gcloud alpha monitoring policies list --format="table(displayName,enabled,conditions[0].displayName)" >> monitoring.md + +# Upload to Cloud Storage for team access +gsutil cp *.md gs://erpnext-docs-$PROJECT_ID/ +EOF + +chmod +x generate-docs.sh +``` + +## 📊 Cost Optimization and FinOps + +### 1. Advanced Cost Monitoring + +```bash +# Create cost anomaly detection +cat > cost-anomaly-detection.json < rightsizing-analysis.sh <<'EOF' +#!/bin/bash +set -e + +echo "ERPNext Resource Right-sizing Analysis" +echo "=====================================" + +# Analyze Cloud SQL utilization +echo "Cloud SQL CPU Utilization (Last 7 days):" +gcloud logging read "resource.type=\"cloudsql_database\" AND metric.type=\"cloudsql.googleapis.com/database/cpu/utilization\"" \ + --freshness=7d \ + --format="table(timestamp,jsonPayload.value)" + +# Analyze Cloud Run utilization +echo "Cloud Run Memory Utilization (Last 7 days):" +gcloud logging read "resource.type=\"cloud_run_revision\" AND metric.type=\"run.googleapis.com/container/memory/utilizations\"" \ + --freshness=7d \ + --format="table(timestamp,jsonPayload.value)" + +# Analyze Redis utilization +echo "Redis Memory Utilization (Last 7 days):" +gcloud logging read "resource.type=\"redis_instance\" AND metric.type=\"redis.googleapis.com/stats/memory/usage_ratio\"" \ + --freshness=7d \ + --format="table(timestamp,jsonPayload.value)" + +# Generate recommendations +echo "" +echo "Recommendations:" +echo "1. If Cloud SQL CPU < 50%, consider downsizing" +echo "2. If Cloud Run memory < 60%, reduce memory allocation" +echo "3. If Redis memory < 70%, consider smaller instance" +EOF + +chmod +x rightsizing-analysis.sh +``` + +This production setup guide provides comprehensive coverage of security, monitoring, backup, performance optimization, and operational procedures for ERPNext running on Google Cloud managed services. The configuration supports both Cloud Run and GKE deployments with enterprise-grade reliability and security. + +## ➡️ Next Steps + +1. **Security Audit**: Conduct regular security assessments +2. **Performance Baseline**: Establish performance benchmarks +3. **Disaster Recovery Testing**: Regular DR drills +4. **Cost Optimization**: Monthly cost reviews and optimization +5. **Documentation Updates**: Keep documentation current with changes + +--- + +**⚠️ Important**: Regular reviews and updates of these configurations are essential for maintaining security and performance in production environments. \ No newline at end of file diff --git a/documentation/deployment-guides/gcp-managed/README.md b/documentation/deployment-guides/gcp-managed/README.md new file mode 100644 index 0000000..7c0fc3c --- /dev/null +++ b/documentation/deployment-guides/gcp-managed/README.md @@ -0,0 +1,334 @@ +# ERPNext Google Cloud Deployment with Managed Services + +## Overview + +This directory contains comprehensive guides and resources for deploying ERPNext on Google Cloud Platform (GCP) using **managed database services**: Cloud SQL for MySQL and Memorystore for Redis. This approach provides better reliability, security, and operational efficiency compared to self-hosted databases. + +## 🏗️ Architecture Overview + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Google Cloud Platform │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌─────────────────┐ ┌─────────────────┐ │ +│ │ Cloud Run │ │ GKE │ │ +│ │ (Serverless) │ │ (Kubernetes) │ │ +│ │ │ │ │ │ +│ │ ┌─────────────┐ │ │ ┌─────────────┐ │ │ +│ │ │ Frontend │ │ │ │ Pods │ │ │ +│ │ │ Backend │ │ │ │ - Frontend │ │ │ +│ │ │ Workers │ │ │ │ - Backend │ │ │ +│ │ └─────────────┘ │ │ │ - Workers │ │ │ +│ └─────────────────┘ │ └─────────────┘ │ │ +│ └─────────────────┘ │ +│ │ │ +│ ┌─────────────────────────────┼─────────────────────────────┐ │ +│ │ Managed Services │ │ │ +│ │ │ │ │ +│ │ ┌──────────────┐ ┌─────────────┐ ┌──────────────┐ │ │ +│ │ │ Cloud SQL │ │ Memorystore │ │Cloud Storage │ │ │ +│ │ │ (MySQL) │ │ (Redis) │ │ (Files) │ │ │ +│ │ └──────────────┘ └─────────────┘ └──────────────┘ │ │ +│ └─────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +## 📁 Directory Structure + +``` +gcp-managed/ +├── README.md # This file +├── 00-prerequisites-managed.md # Prerequisites for managed services +├── 01-gke-managed-deployment.md # GKE with managed databases +├── 02-cloud-run-deployment.md # Cloud Run serverless deployment +├── 03-production-managed-setup.md # Production hardening +├── kubernetes-manifests/ # K8s manifests for managed services +│ ├── namespace.yaml # Namespace with reduced quotas +│ ├── storage.yaml # Only application file storage +│ ├── configmap.yaml # Config for managed services +│ ├── secrets.yaml # External Secrets integration +│ ├── erpnext-backend.yaml # Backend with Cloud SQL proxy +│ ├── erpnext-frontend.yaml # Optimized frontend +│ ├── erpnext-workers.yaml # Workers with managed DB +│ ├── ingress.yaml # Enhanced ingress config +│ └── jobs.yaml # Site creation and backup jobs +└── scripts/ # Automation scripts + ├── deploy-managed.sh # GKE deployment script + └── cloud-run-deploy.sh # Cloud Run deployment script +``` + +## 🚀 Quick Start + +### Option 1: GKE with Managed Services (Recommended for Production) + +```bash +# 1. Complete prerequisites +cd gcp-managed/ +# Follow 00-prerequisites-managed.md + +# 2. Deploy to GKE +cd scripts/ +export PROJECT_ID="your-gcp-project" +export DOMAIN="erpnext.yourdomain.com" +export EMAIL="admin@yourdomain.com" +./deploy-managed.sh deploy +``` + +### Option 2: Cloud Run Serverless Deployment + +```bash +# 1. Complete prerequisites +cd gcp-managed/ +# Follow 00-prerequisites-managed.md + +# 2. Deploy to Cloud Run +cd scripts/ +export PROJECT_ID="your-gcp-project" +export DOMAIN="erpnext.yourdomain.com" +./cloud-run-deploy.sh deploy +``` + +## 🎯 Key Benefits of Managed Services + +### 🛡️ Enhanced Reliability +- **99.95% SLA** for Cloud SQL and Memorystore +- **Automatic failover** and disaster recovery +- **Point-in-time recovery** for databases +- **Automated backups** with cross-region replication + +### 🔧 Operational Efficiency +- **Zero database administration** overhead +- **Automatic security patches** and updates +- **Performance insights** and optimization recommendations +- **Built-in monitoring** and alerting + +### 🔒 Enterprise Security +- **Private IP connectivity** within VPC +- **Encryption at rest and in transit** by default +- **IAM integration** for access control +- **Audit logging** for compliance + +### 💰 Cost Optimization +- **Pay-as-you-scale** pricing model +- **Automatic storage scaling** without downtime +- **Right-sizing recommendations** based on usage +- **No over-provisioning** of database resources + +## 📊 Deployment Options Comparison + +| Feature | GKE + Managed DB | Cloud Run + Managed DB | Self-Hosted DB | +|---------|------------------|------------------------|-----------------| +| **Scalability** | Manual/Auto HPA | Automatic (0-1000+) | Manual | +| **Operational Overhead** | Medium | Very Low | High | +| **Database Reliability** | 99.95% SLA | 99.95% SLA | Depends on setup | +| **Cost (Small)** | ~$450/month | ~$200/month | ~$300/month | +| **Cost (Large)** | ~$800/month | ~$400/month | ~$600/month | +| **Cold Start** | None | 1-3 seconds | None | +| **Customization** | High | Medium | Very High | +| **Multi-tenancy** | Supported | Limited | Supported | + +## 🛠️ Managed Services Configuration + +### Cloud SQL (MySQL) +- **Instance Types**: db-n1-standard-2 to db-n1-standard-96 +- **Storage**: 10GB to 64TB, automatic scaling +- **Backup**: Automated daily backups with 7-day retention +- **High Availability**: Regional persistent disks with automatic failover +- **Security**: Private IP, SSL/TLS encryption, IAM database authentication + +### Memorystore (Redis) +- **Tiers**: Basic (1-5GB) or Standard (1-300GB) with HA +- **Features**: Persistence, AUTH, in-transit encryption +- **Performance**: Up to 12 Gbps network throughput +- **Monitoring**: Built-in metrics and alerting + +### Additional Services +- **Cloud Storage**: File uploads and static assets +- **Secret Manager**: Secure credential management +- **VPC Access Connector**: Secure serverless-to-VPC communication +- **Cloud Tasks**: Background job processing (Cloud Run) +- **Cloud Scheduler**: Cron jobs and scheduled tasks + +## 🔧 Advanced Features + +### Auto-scaling Configuration +- **GKE**: Horizontal Pod Autoscaler based on CPU/memory +- **Cloud Run**: Automatic scaling from 0 to 1000+ instances +- **Database**: Automatic storage scaling, manual compute scaling +- **Redis**: Manual scaling with zero-downtime + +### Security Hardening +- **Network isolation** with private subnets +- **Workload Identity** for secure GCP API access +- **External Secrets Operator** for credential management +- **Network policies** for pod-to-pod communication +- **Binary Authorization** for container security + +### Monitoring & Observability +- **Stackdriver integration** for logs and metrics +- **Custom dashboards** for ERPNext-specific metrics +- **SLO/SLI monitoring** with alerting +- **Distributed tracing** with Cloud Trace +- **Error reporting** with automatic grouping + +### Backup & Disaster Recovery +- **Cloud SQL**: Automated backups with point-in-time recovery +- **Application files**: Automated backup to Cloud Storage +- **Cross-region replication** for disaster recovery +- **Automated DR testing** with validation + +## 💰 Cost Estimation & Optimization + +### Typical Monthly Costs (US-Central1) + +#### Small Deployment (< 50 users) +``` +Cloud SQL (db-n1-standard-1): $50 +Memorystore Redis (1GB): $37 +Cloud Run (avg 2 instances): $60 +Cloud Storage (50GB): $1 +Load Balancer: $18 +Total: ~$166/month +``` + +#### Medium Deployment (50-200 users) +``` +Cloud SQL (db-n1-standard-2): $278 +Memorystore Redis (5GB): $185 +GKE (3 e2-standard-4 nodes): $420 +Cloud Storage (200GB): $4 +Load Balancer: $18 +Total: ~$905/month +``` + +#### Large Deployment (200+ users) +``` +Cloud SQL (db-n1-standard-4): $556 +Memorystore Redis (10GB): $370 +GKE (6 e2-standard-4 nodes): $840 +Cloud Storage (500GB): $10 +Load Balancer: $18 +Total: ~$1,794/month +``` + +### Cost Optimization Strategies +1. **Use committed use discounts** (up to 57% savings) +2. **Right-size instances** based on monitoring data +3. **Use preemptible nodes** for non-critical workloads +4. **Implement storage lifecycle policies** for Cloud Storage +5. **Scale down during off-hours** with automation + +## 🚨 Migration Path from Self-Hosted + +### Phase 1: Assessment (Week 1) +- [ ] Audit current database size and performance +- [ ] Identify custom configurations and extensions +- [ ] Plan migration windows and rollback procedures +- [ ] Set up managed services in parallel + +### Phase 2: Data Migration (Week 2) +- [ ] Export data from existing MySQL/Redis +- [ ] Import to Cloud SQL/Memorystore +- [ ] Validate data integrity and performance +- [ ] Update connection strings and test + +### Phase 3: Application Migration (Week 3) +- [ ] Deploy ERPNext with managed services +- [ ] Migrate file storage to Cloud Storage +- [ ] Update backup procedures +- [ ] Conduct full testing + +### Phase 4: Cutover and Optimization (Week 4) +- [ ] DNS cutover to new deployment +- [ ] Monitor performance and costs +- [ ] Optimize resource allocation +- [ ] Decommission old infrastructure + +## 🔍 Troubleshooting Common Issues + +### Cloud SQL Connection Issues +```bash +# Test connectivity from GKE +kubectl run mysql-test --rm -i --tty --image=mysql:8.0 -- mysql -h PRIVATE_IP -u erpnext -p + +# Check Cloud SQL Proxy logs +kubectl logs deployment/erpnext-backend -c cloud-sql-proxy +``` + +### Redis Connection Issues +```bash +# Test Redis connectivity +kubectl run redis-test --rm -i --tty --image=redis:alpine -- redis-cli -h REDIS_IP ping + +# Check AUTH configuration +gcloud redis instances describe erpnext-redis --region=us-central1 +``` + +### Performance Issues +```bash +# Check database performance +gcloud sql operations list --instance=erpnext-db + +# Monitor Redis memory usage +gcloud redis instances describe erpnext-redis --region=us-central1 --format="value(memorySizeGb,redisMemoryUsage)" +``` + +## 📚 Additional Resources + +### Google Cloud Documentation +- [Cloud SQL Best Practices](https://cloud.google.com/sql/docs/mysql/best-practices) +- [Memorystore Best Practices](https://cloud.google.com/memorystore/docs/redis/memory-management-best-practices) +- [Cloud Run Best Practices](https://cloud.google.com/run/docs/best-practices) +- [GKE Networking Best Practices](https://cloud.google.com/kubernetes-engine/docs/best-practices/networking) + +### ERPNext Specific +- [ERPNext Database Configuration](https://docs.erpnext.com/docs/user/manual/en/setting-up/database-setup) +- [Performance Optimization](https://docs.erpnext.com/docs/user/manual/en/setting-up/performance) +- [Backup Strategies](https://docs.erpnext.com/docs/user/manual/en/setting-up/backup) + +### Monitoring & Operations +- [SRE Best Practices](https://sre.google/books/) +- [Prometheus Monitoring](https://prometheus.io/docs/practices/naming/) +- [Grafana Dashboards](https://grafana.com/grafana/dashboards/) + +## 🎯 Decision Matrix + +### Choose GKE + Managed Services if: +- ✅ Need full control over application deployment +- ✅ Require complex networking or multi-tenancy +- ✅ Have existing Kubernetes expertise +- ✅ Need consistent performance with no cold starts +- ✅ Plan to run multiple applications in same cluster + +### Choose Cloud Run + Managed Services if: +- ✅ Want minimal operational overhead +- ✅ Have variable or unpredictable traffic +- ✅ Need rapid scaling capabilities +- ✅ Want to minimize costs for smaller deployments +- ✅ Prefer serverless architecture + +## 📞 Support & Contributing + +### Getting Help +- **Documentation Issues**: Create issues in the repository +- **Deployment Support**: Follow troubleshooting guides +- **Performance Issues**: Check monitoring dashboards +- **Cost Optimization**: Use GCP billing reports and recommendations + +### Contributing +- **Documentation improvements**: Submit pull requests +- **Script enhancements**: Share automation improvements +- **Best practices**: Contribute lessons learned +- **Cost optimizations**: Share optimization strategies + +--- + +**⚠️ Important Notes**: +- Managed services incur continuous costs even when not in use +- Always test deployments in staging before production +- Monitor costs and usage regularly +- Keep credentials secure and rotate regularly +- Follow GCP security best practices + +**🎯 Recommendation**: For most production deployments, GKE with managed services provides the best balance of control, reliability, and operational efficiency. \ No newline at end of file diff --git a/documentation/deployment-guides/gcp-managed/kubernetes-manifests/configmap.yaml b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/configmap.yaml new file mode 100644 index 0000000..8db8084 --- /dev/null +++ b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/configmap.yaml @@ -0,0 +1,279 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: erpnext-managed-config + namespace: erpnext + labels: + app: erpnext + component: config + deployment-type: managed-services +data: + # ERPNext Application Configuration + APP_VERSION: "v14" + APP_URL: "erpnext.yourdomain.com" + APP_USER: "Administrator" + APP_DB_PARAM: "db" + DEVELOPER_MODE: "0" + ENABLE_SCHEDULER: "1" + SOCKETIO_PORT: "9000" + + # Cloud SQL Configuration (via Cloud SQL Proxy) + DB_HOST: "127.0.0.1" + DB_PORT: "3306" + DB_NAME: "erpnext" + DB_USER: "erpnext" + DB_TIMEOUT: "60" + DB_CHARSET: "utf8mb4" + + # Memorystore Redis Configuration + # Note: REDIS_HOST will be set by deployment script + REDIS_PORT: "6379" + REDIS_CACHE_URL: "redis://REDIS_HOST:6379/0" + REDIS_QUEUE_URL: "redis://REDIS_HOST:6379/1" + REDIS_SOCKETIO_URL: "redis://REDIS_HOST:6379/2" + REDIS_TIMEOUT: "5" + REDIS_MAX_CONNECTIONS: "20" + + # Performance settings optimized for managed services + WORKER_PROCESSES: "4" + WORKER_TIMEOUT: "120" + WORKER_MAX_REQUESTS: "1000" + WORKER_MAX_REQUESTS_JITTER: "50" + + # Logging settings for Cloud Logging + LOG_LEVEL: "INFO" + LOG_FORMAT: "json" + ENABLE_CLOUD_LOGGING: "1" + + # Cloud SQL Proxy settings + CLOUDSQL_CONNECTION_NAME: "PROJECT_ID:REGION:erpnext-db" + ENABLE_CLOUDSQL_PROXY: "1" + + # Cloud Storage settings for file uploads + USE_CLOUD_STORAGE: "1" + CLOUD_STORAGE_BUCKET: "erpnext-files-PROJECT_ID" + + # Security settings + SECURE_HEADERS: "1" + CSRF_PROTECTION: "1" + SESSION_SECURE: "1" +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: nginx-managed-config + namespace: erpnext + labels: + app: erpnext + component: nginx + deployment-type: managed-services +data: + nginx.conf: | + user nginx; + worker_processes auto; + error_log /var/log/nginx/error.log notice; + pid /var/run/nginx.pid; + + events { + worker_connections 1024; + use epoll; + multi_accept on; + } + + http { + include /etc/nginx/mime.types; + default_type application/octet-stream; + + # Enhanced logging for managed services + log_format main '$remote_addr - $remote_user [$time_local] "$request" ' + '$status $body_bytes_sent "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for" ' + 'rt=$request_time uct="$upstream_connect_time" ' + 'uht="$upstream_header_time" urt="$upstream_response_time"'; + + access_log /var/log/nginx/access.log main; + + sendfile on; + tcp_nopush on; + tcp_nodelay on; + keepalive_timeout 65; + types_hash_max_size 2048; + client_max_body_size 50m; + + # Enhanced gzip configuration + gzip on; + gzip_vary on; + gzip_proxied any; + gzip_comp_level 6; + gzip_types + text/plain + text/css + text/xml + text/javascript + application/json + application/javascript + application/xml+rss + application/atom+xml + image/svg+xml; + + # Connection pooling for backend + upstream backend { + server erpnext-backend:8000; + keepalive 32; + keepalive_requests 100; + keepalive_timeout 60s; + } + + upstream socketio { + server erpnext-backend:9000; + keepalive 16; + } + + # Rate limiting zones + limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s; + limit_req_zone $binary_remote_addr zone=login:10m rate=5r/m; + + server { + listen 8080; + server_name _; + root /home/frappe/frappe-bench/sites; + + # Enhanced security headers for managed services + add_header X-Frame-Options SAMEORIGIN always; + add_header X-Content-Type-Options nosniff always; + add_header X-XSS-Protection "1; mode=block" always; + add_header Referrer-Policy "strict-origin-when-cross-origin" always; + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' data:; connect-src 'self' wss:; frame-ancestors 'self';" always; + + # Static assets with enhanced caching + location /assets { + try_files $uri =404; + expires 1y; + add_header Cache-Control "public, immutable"; + add_header Vary "Accept-Encoding"; + + # Serve from Cloud Storage for managed deployments + location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$ { + proxy_pass https://storage.googleapis.com/erpnext-files-PROJECT_ID/assets$uri; + proxy_set_header Host storage.googleapis.com; + proxy_cache_valid 200 1y; + expires 1y; + add_header Cache-Control "public, immutable"; + } + } + + # Protected files + location ~ ^/protected/(.*) { + internal; + try_files /frontend/$1 =404; + } + + # WebSocket connections with enhanced handling + location /socket.io/ { + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header X-Frappe-Site-Name frontend; + proxy_set_header Origin $scheme://$http_host; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + # Enhanced timeouts for WebSocket + proxy_read_timeout 300s; + proxy_send_timeout 300s; + proxy_connect_timeout 10s; + + proxy_pass http://socketio; + } + + # API endpoints with rate limiting + location /api/ { + limit_req zone=api burst=20 nodelay; + + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Frappe-Site-Name frontend; + proxy_set_header Host $host; + proxy_set_header X-Use-X-Accel-Redirect True; + proxy_set_header X-Real-IP $remote_addr; + + # Enhanced timeouts for API calls + proxy_read_timeout 120s; + proxy_send_timeout 120s; + proxy_connect_timeout 10s; + + proxy_redirect off; + proxy_buffering off; + + proxy_pass http://backend; + } + + # Login endpoint with stricter rate limiting + location /api/method/login { + limit_req zone=login burst=3 nodelay; + + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Frappe-Site-Name frontend; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + + proxy_pass http://backend; + } + + # Main application routes + location / { + try_files /frontend/public/$uri @webserver; + } + + location @webserver { + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Frappe-Site-Name frontend; + proxy_set_header Host $host; + proxy_set_header X-Use-X-Accel-Redirect True; + proxy_set_header X-Real-IP $remote_addr; + + proxy_read_timeout 120s; + proxy_send_timeout 120s; + proxy_connect_timeout 10s; + + proxy_redirect off; + proxy_buffering on; + proxy_buffer_size 4k; + proxy_buffers 8 4k; + + proxy_pass http://backend; + } + + # Health check endpoint + location /health { + access_log off; + return 200 "healthy\n"; + add_header Content-Type text/plain; + } + + # Metrics endpoint for monitoring + location /metrics { + access_log off; + proxy_pass http://backend/api/method/frappe.utils.response.get_response_length; + allow 10.0.0.0/8; # VPC internal only + deny all; + } + + # Block access to sensitive files + location ~* \.(conf|htaccess|htpasswd|ini|log|sh|sql|tar|gz|bak|backup)$ { + deny all; + return 404; + } + + # Block access to hidden files + location ~ /\. { + deny all; + return 404; + } + } + } \ No newline at end of file diff --git a/documentation/deployment-guides/gcp-managed/kubernetes-manifests/erpnext-backend.yaml b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/erpnext-backend.yaml new file mode 100644 index 0000000..992b958 --- /dev/null +++ b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/erpnext-backend.yaml @@ -0,0 +1,374 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: erpnext-backend + namespace: erpnext + labels: + app: erpnext-backend + component: backend + environment: production + version: v14 + deployment-type: managed-services +spec: + replicas: 3 + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 1 + selector: + matchLabels: + app: erpnext-backend + template: + metadata: + labels: + app: erpnext-backend + component: backend + environment: production + version: v14 + deployment-type: managed-services + annotations: + prometheus.io/scrape: "true" + prometheus.io/port: "8000" + prometheus.io/path: "/api/method/frappe.utils.response.get_response_length" + spec: + serviceAccountName: erpnext-ksa + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault + affinity: + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - erpnext-backend + topologyKey: kubernetes.io/hostname + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: cloud.google.com/gke-preemptible + operator: DoesNotExist + initContainers: + - name: wait-for-cloudsql + image: gcr.io/cloudsql-docker/gce-proxy:1.35.4-alpine + command: + - sh + - -c + - | + echo 'Starting Cloud SQL Proxy for health check...' + /cloud_sql_proxy -instances=$DB_CONNECTION_NAME=tcp:3306 & + PROXY_PID=$! + + # Wait for proxy to be ready + until nc -z localhost 3306; do + echo 'Waiting for Cloud SQL connection...' + sleep 5 + done + + # Test database connection + echo 'Testing database connection...' + timeout 10 sh -c 'until nc -z localhost 3306; do sleep 1; done' + + echo 'Cloud SQL connection verified' + kill $PROXY_PID + env: + - name: DB_CONNECTION_NAME + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: db-connection-name + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 2 + capabilities: + drop: + - ALL + resources: + requests: + memory: "64Mi" + cpu: "50m" + limits: + memory: "128Mi" + cpu: "100m" + containers: + - name: erpnext-backend + image: frappe/erpnext-worker:v14 + envFrom: + - configMapRef: + name: erpnext-managed-config + env: + - name: DB_PASSWORD + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: db-password + - name: REDIS_AUTH + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: redis-auth + - name: GOOGLE_APPLICATION_CREDENTIALS + value: /var/secrets/google/key.json + ports: + - containerPort: 8000 + name: http + - containerPort: 9000 + name: socketio + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + capabilities: + drop: + - ALL + volumeMounts: + - name: sites-data + mountPath: /home/frappe/frappe-bench/sites + - name: assets-data + mountPath: /home/frappe/frappe-bench/sites/assets + - name: tmp + mountPath: /tmp + - name: logs + mountPath: /home/frappe/frappe-bench/logs + - name: google-cloud-key + mountPath: /var/secrets/google + readOnly: true + resources: + requests: + memory: "1.5Gi" + cpu: "750m" + limits: + memory: "3Gi" + cpu: "1500m" + livenessProbe: + httpGet: + path: /api/method/ping + port: 8000 + initialDelaySeconds: 90 + periodSeconds: 30 + timeoutSeconds: 10 + failureThreshold: 3 + readinessProbe: + httpGet: + path: /api/method/ping + port: 8000 + initialDelaySeconds: 30 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 3 + startupProbe: + httpGet: + path: /api/method/ping + port: 8000 + initialDelaySeconds: 60 + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: 30 + - name: cloud-sql-proxy + image: gcr.io/cloudsql-docker/gce-proxy:1.35.4-alpine + command: + - /cloud_sql_proxy + - -instances=$(DB_CONNECTION_NAME)=tcp:127.0.0.1:3306 + - -credential_file=/var/secrets/google/key.json + - -enable_iam_login + env: + - name: DB_CONNECTION_NAME + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: db-connection-name + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 2 + capabilities: + drop: + - ALL + volumeMounts: + - name: google-cloud-key + mountPath: /var/secrets/google + readOnly: true + resources: + requests: + memory: "128Mi" + cpu: "100m" + limits: + memory: "256Mi" + cpu: "200m" + livenessProbe: + tcpSocket: + port: 3306 + initialDelaySeconds: 30 + periodSeconds: 10 + readinessProbe: + tcpSocket: + port: 3306 + initialDelaySeconds: 5 + periodSeconds: 5 + volumes: + - name: sites-data + persistentVolumeClaim: + claimName: erpnext-sites-pvc + - name: assets-data + persistentVolumeClaim: + claimName: erpnext-assets-pvc + - name: tmp + emptyDir: {} + - name: logs + emptyDir: {} + - name: google-cloud-key + secret: + secretName: gcp-service-account-key +--- +apiVersion: v1 +kind: Service +metadata: + name: erpnext-backend + namespace: erpnext + labels: + app: erpnext-backend + component: backend + deployment-type: managed-services + annotations: + cloud.google.com/neg: '{"ingress": true}' + prometheus.io/scrape: "true" + prometheus.io/port: "8000" +spec: + selector: + app: erpnext-backend + ports: + - name: http + port: 8000 + targetPort: 8000 + protocol: TCP + - name: socketio + port: 9000 + targetPort: 9000 + protocol: TCP + type: ClusterIP +--- +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: erpnext-backend-hpa + namespace: erpnext + labels: + app: erpnext-backend + component: backend + deployment-type: managed-services +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: erpnext-backend + minReplicas: 3 + maxReplicas: 15 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 70 + - type: Resource + resource: + name: memory + target: + type: Utilization + averageUtilization: 80 + behavior: + scaleDown: + stabilizationWindowSeconds: 300 + policies: + - type: Percent + value: 50 + periodSeconds: 60 + scaleUp: + stabilizationWindowSeconds: 60 + policies: + - type: Percent + value: 100 + periodSeconds: 15 + - type: Pods + value: 2 + periodSeconds: 60 +--- +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: erpnext-backend-pdb + namespace: erpnext + labels: + app: erpnext-backend + component: backend + deployment-type: managed-services +spec: + minAvailable: 2 + selector: + matchLabels: + app: erpnext-backend +--- +# Network Policy for backend service +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: erpnext-backend-netpol + namespace: erpnext +spec: + podSelector: + matchLabels: + app: erpnext-backend + policyTypes: + - Ingress + - Egress + ingress: + - from: + - podSelector: + matchLabels: + app: erpnext-frontend + - namespaceSelector: + matchLabels: + name: ingress-nginx + ports: + - protocol: TCP + port: 8000 + - protocol: TCP + port: 9000 + egress: + # Allow DNS resolution + - to: [] + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 + # Allow access to managed services (Cloud SQL, Redis) + - to: [] + ports: + - protocol: TCP + port: 3306 # Cloud SQL + - protocol: TCP + port: 6379 # Redis + - protocol: TCP + port: 443 # HTTPS for GCP APIs + # Allow internal cluster communication + - to: + - namespaceSelector: {} + ports: + - protocol: TCP + port: 8080 \ No newline at end of file diff --git a/documentation/deployment-guides/gcp-managed/kubernetes-manifests/erpnext-frontend.yaml b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/erpnext-frontend.yaml new file mode 100644 index 0000000..29efc17 --- /dev/null +++ b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/erpnext-frontend.yaml @@ -0,0 +1,338 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: erpnext-frontend + namespace: erpnext + labels: + app: erpnext-frontend + component: frontend + environment: production + version: v14 + deployment-type: managed-services +spec: + replicas: 2 + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 0 + selector: + matchLabels: + app: erpnext-frontend + template: + metadata: + labels: + app: erpnext-frontend + component: frontend + environment: production + version: v14 + deployment-type: managed-services + annotations: + prometheus.io/scrape: "true" + prometheus.io/port: "8080" + prometheus.io/path: "/health" + spec: + securityContext: + runAsNonRoot: true + runAsUser: 101 + runAsGroup: 101 + fsGroup: 101 + seccompProfile: + type: RuntimeDefault + affinity: + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - erpnext-frontend + topologyKey: kubernetes.io/hostname + initContainers: + - name: setup-nginx-config + image: busybox:1.35 + command: + - sh + - -c + - | + # Copy base nginx config + cp /config/nginx.conf /shared/nginx.conf + + # Replace placeholders with actual values + sed -i "s/PROJECT_ID/$PROJECT_ID/g" /shared/nginx.conf + + echo "Nginx configuration prepared" + env: + - name: PROJECT_ID + valueFrom: + configMapRef: + name: erpnext-managed-config + key: PROJECT_ID + optional: true + volumeMounts: + - name: nginx-config + mountPath: /config + - name: nginx-config-processed + mountPath: /shared + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + capabilities: + drop: + - ALL + containers: + - name: erpnext-frontend + image: frappe/erpnext-nginx:v14 + ports: + - containerPort: 8080 + name: http + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 101 + runAsGroup: 101 + capabilities: + drop: + - ALL + add: + - NET_BIND_SERVICE + volumeMounts: + - name: sites-data + mountPath: /home/frappe/frappe-bench/sites + readOnly: true + - name: assets-data + mountPath: /home/frappe/frappe-bench/sites/assets + readOnly: true + - name: nginx-config-processed + mountPath: /etc/nginx/nginx.conf + subPath: nginx.conf + readOnly: true + - name: tmp + mountPath: /tmp + - name: var-cache + mountPath: /var/cache/nginx + - name: var-run + mountPath: /var/run + - name: var-log + mountPath: /var/log/nginx + resources: + requests: + memory: "256Mi" + cpu: "100m" + limits: + memory: "512Mi" + cpu: "500m" + livenessProbe: + httpGet: + path: /health + port: 8080 + initialDelaySeconds: 30 + periodSeconds: 30 + timeoutSeconds: 5 + failureThreshold: 3 + readinessProbe: + httpGet: + path: /health + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 10 + timeoutSeconds: 3 + failureThreshold: 3 + startupProbe: + httpGet: + path: /health + port: 8080 + initialDelaySeconds: 5 + periodSeconds: 5 + timeoutSeconds: 3 + failureThreshold: 12 + env: + - name: NGINX_WORKER_PROCESSES + value: "auto" + - name: NGINX_WORKER_CONNECTIONS + value: "1024" + volumes: + - name: sites-data + persistentVolumeClaim: + claimName: erpnext-sites-pvc + - name: assets-data + persistentVolumeClaim: + claimName: erpnext-assets-pvc + - name: nginx-config + configMap: + name: nginx-managed-config + - name: nginx-config-processed + emptyDir: {} + - name: tmp + emptyDir: {} + - name: var-cache + emptyDir: {} + - name: var-run + emptyDir: {} + - name: var-log + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: + name: erpnext-frontend + namespace: erpnext + labels: + app: erpnext-frontend + component: frontend + deployment-type: managed-services + annotations: + cloud.google.com/neg: '{"ingress": true}' + prometheus.io/scrape: "true" + prometheus.io/port: "8080" +spec: + selector: + app: erpnext-frontend + ports: + - port: 8080 + targetPort: 8080 + name: http + protocol: TCP + type: ClusterIP +--- +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: erpnext-frontend-hpa + namespace: erpnext + labels: + app: erpnext-frontend + component: frontend + deployment-type: managed-services +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: erpnext-frontend + minReplicas: 2 + maxReplicas: 8 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 70 + - type: Resource + resource: + name: memory + target: + type: Utilization + averageUtilization: 80 + behavior: + scaleDown: + stabilizationWindowSeconds: 300 + policies: + - type: Percent + value: 50 + periodSeconds: 60 + scaleUp: + stabilizationWindowSeconds: 60 + policies: + - type: Percent + value: 100 + periodSeconds: 15 + - type: Pods + value: 1 + periodSeconds: 60 +--- +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: erpnext-frontend-pdb + namespace: erpnext + labels: + app: erpnext-frontend + component: frontend + deployment-type: managed-services +spec: + minAvailable: 1 + selector: + matchLabels: + app: erpnext-frontend +--- +# Network Policy for frontend service +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: erpnext-frontend-netpol + namespace: erpnext +spec: + podSelector: + matchLabels: + app: erpnext-frontend + policyTypes: + - Ingress + - Egress + ingress: + # Allow traffic from ingress controller + - from: + - namespaceSelector: + matchLabels: + name: ingress-nginx + ports: + - protocol: TCP + port: 8080 + # Allow traffic from monitoring + - from: + - namespaceSelector: + matchLabels: + name: monitoring + ports: + - protocol: TCP + port: 8080 + egress: + # Allow DNS resolution + - to: [] + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 + # Allow communication with backend + - to: + - podSelector: + matchLabels: + app: erpnext-backend + ports: + - protocol: TCP + port: 8000 + - protocol: TCP + port: 9000 + # Allow access to Cloud Storage for assets + - to: [] + ports: + - protocol: TCP + port: 443 # HTTPS for Cloud Storage +--- +# Service Monitor for Prometheus (if using Prometheus Operator) +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: erpnext-frontend-monitor + namespace: erpnext + labels: + app: erpnext-frontend + component: frontend + deployment-type: managed-services +spec: + selector: + matchLabels: + app: erpnext-frontend + endpoints: + - port: http + path: /health + interval: 30s + scrapeTimeout: 10s \ No newline at end of file diff --git a/documentation/deployment-guides/gcp-managed/kubernetes-manifests/erpnext-workers.yaml b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/erpnext-workers.yaml new file mode 100644 index 0000000..5ff988d --- /dev/null +++ b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/erpnext-workers.yaml @@ -0,0 +1,696 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: erpnext-queue-default + namespace: erpnext + labels: + app: erpnext-queue-default + component: worker + queue: default + environment: production + version: v14 + deployment-type: managed-services +spec: + replicas: 2 + selector: + matchLabels: + app: erpnext-queue-default + template: + metadata: + labels: + app: erpnext-queue-default + component: worker + queue: default + environment: production + version: v14 + deployment-type: managed-services + spec: + serviceAccountName: erpnext-ksa + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: queue-worker + image: frappe/erpnext-worker:v14 + command: + - bench + - worker + - --queue + - default + envFrom: + - configMapRef: + name: erpnext-managed-config + env: + - name: DB_PASSWORD + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: db-password + - name: REDIS_AUTH + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: redis-auth + - name: GOOGLE_APPLICATION_CREDENTIALS + value: /var/secrets/google/key.json + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + capabilities: + drop: + - ALL + volumeMounts: + - name: sites-data + mountPath: /home/frappe/frappe-bench/sites + - name: tmp + mountPath: /tmp + - name: logs + mountPath: /home/frappe/frappe-bench/logs + - name: google-cloud-key + mountPath: /var/secrets/google + readOnly: true + resources: + requests: + memory: "512Mi" + cpu: "250m" + limits: + memory: "1Gi" + cpu: "500m" + livenessProbe: + exec: + command: + - pgrep + - -f + - "bench worker" + initialDelaySeconds: 30 + periodSeconds: 30 + timeoutSeconds: 5 + failureThreshold: 3 + - name: cloud-sql-proxy + image: gcr.io/cloudsql-docker/gce-proxy:1.35.4-alpine + command: + - /cloud_sql_proxy + - -instances=$(DB_CONNECTION_NAME)=tcp:127.0.0.1:3306 + - -credential_file=/var/secrets/google/key.json + env: + - name: DB_CONNECTION_NAME + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: db-connection-name + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 2 + capabilities: + drop: + - ALL + volumeMounts: + - name: google-cloud-key + mountPath: /var/secrets/google + readOnly: true + resources: + requests: + memory: "64Mi" + cpu: "50m" + limits: + memory: "128Mi" + cpu: "100m" + volumes: + - name: sites-data + persistentVolumeClaim: + claimName: erpnext-sites-pvc + - name: tmp + emptyDir: {} + - name: logs + emptyDir: {} + - name: google-cloud-key + secret: + secretName: gcp-service-account-key +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: erpnext-queue-long + namespace: erpnext + labels: + app: erpnext-queue-long + component: worker + queue: long + environment: production + version: v14 + deployment-type: managed-services +spec: + replicas: 1 + selector: + matchLabels: + app: erpnext-queue-long + template: + metadata: + labels: + app: erpnext-queue-long + component: worker + queue: long + environment: production + version: v14 + deployment-type: managed-services + spec: + serviceAccountName: erpnext-ksa + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: queue-worker + image: frappe/erpnext-worker:v14 + command: + - bench + - worker + - --queue + - long + envFrom: + - configMapRef: + name: erpnext-managed-config + env: + - name: DB_PASSWORD + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: db-password + - name: REDIS_AUTH + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: redis-auth + - name: GOOGLE_APPLICATION_CREDENTIALS + value: /var/secrets/google/key.json + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + capabilities: + drop: + - ALL + volumeMounts: + - name: sites-data + mountPath: /home/frappe/frappe-bench/sites + - name: tmp + mountPath: /tmp + - name: logs + mountPath: /home/frappe/frappe-bench/logs + - name: google-cloud-key + mountPath: /var/secrets/google + readOnly: true + resources: + requests: + memory: "512Mi" + cpu: "250m" + limits: + memory: "1Gi" + cpu: "500m" + livenessProbe: + exec: + command: + - pgrep + - -f + - "bench worker" + initialDelaySeconds: 30 + periodSeconds: 30 + timeoutSeconds: 5 + failureThreshold: 3 + - name: cloud-sql-proxy + image: gcr.io/cloudsql-docker/gce-proxy:1.35.4-alpine + command: + - /cloud_sql_proxy + - -instances=$(DB_CONNECTION_NAME)=tcp:127.0.0.1:3306 + - -credential_file=/var/secrets/google/key.json + env: + - name: DB_CONNECTION_NAME + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: db-connection-name + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 2 + capabilities: + drop: + - ALL + volumeMounts: + - name: google-cloud-key + mountPath: /var/secrets/google + readOnly: true + resources: + requests: + memory: "64Mi" + cpu: "50m" + limits: + memory: "128Mi" + cpu: "100m" + volumes: + - name: sites-data + persistentVolumeClaim: + claimName: erpnext-sites-pvc + - name: tmp + emptyDir: {} + - name: logs + emptyDir: {} + - name: google-cloud-key + secret: + secretName: gcp-service-account-key +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: erpnext-queue-short + namespace: erpnext + labels: + app: erpnext-queue-short + component: worker + queue: short + environment: production + version: v14 + deployment-type: managed-services +spec: + replicas: 2 + selector: + matchLabels: + app: erpnext-queue-short + template: + metadata: + labels: + app: erpnext-queue-short + component: worker + queue: short + environment: production + version: v14 + deployment-type: managed-services + spec: + serviceAccountName: erpnext-ksa + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: queue-worker + image: frappe/erpnext-worker:v14 + command: + - bench + - worker + - --queue + - short + envFrom: + - configMapRef: + name: erpnext-managed-config + env: + - name: DB_PASSWORD + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: db-password + - name: REDIS_AUTH + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: redis-auth + - name: GOOGLE_APPLICATION_CREDENTIALS + value: /var/secrets/google/key.json + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + capabilities: + drop: + - ALL + volumeMounts: + - name: sites-data + mountPath: /home/frappe/frappe-bench/sites + - name: tmp + mountPath: /tmp + - name: logs + mountPath: /home/frappe/frappe-bench/logs + - name: google-cloud-key + mountPath: /var/secrets/google + readOnly: true + resources: + requests: + memory: "512Mi" + cpu: "250m" + limits: + memory: "1Gi" + cpu: "500m" + livenessProbe: + exec: + command: + - pgrep + - -f + - "bench worker" + initialDelaySeconds: 30 + periodSeconds: 30 + timeoutSeconds: 5 + failureThreshold: 3 + - name: cloud-sql-proxy + image: gcr.io/cloudsql-docker/gce-proxy:1.35.4-alpine + command: + - /cloud_sql_proxy + - -instances=$(DB_CONNECTION_NAME)=tcp:127.0.0.1:3306 + - -credential_file=/var/secrets/google/key.json + env: + - name: DB_CONNECTION_NAME + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: db-connection-name + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 2 + capabilities: + drop: + - ALL + volumeMounts: + - name: google-cloud-key + mountPath: /var/secrets/google + readOnly: true + resources: + requests: + memory: "64Mi" + cpu: "50m" + limits: + memory: "128Mi" + cpu: "100m" + volumes: + - name: sites-data + persistentVolumeClaim: + claimName: erpnext-sites-pvc + - name: tmp + emptyDir: {} + - name: logs + emptyDir: {} + - name: google-cloud-key + secret: + secretName: gcp-service-account-key +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: erpnext-scheduler + namespace: erpnext + labels: + app: erpnext-scheduler + component: scheduler + environment: production + version: v14 + deployment-type: managed-services +spec: + replicas: 1 + selector: + matchLabels: + app: erpnext-scheduler + template: + metadata: + labels: + app: erpnext-scheduler + component: scheduler + environment: production + version: v14 + deployment-type: managed-services + spec: + serviceAccountName: erpnext-ksa + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: scheduler + image: frappe/erpnext-worker:v14 + command: + - bench + - schedule + envFrom: + - configMapRef: + name: erpnext-managed-config + env: + - name: DB_PASSWORD + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: db-password + - name: REDIS_AUTH + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: redis-auth + - name: GOOGLE_APPLICATION_CREDENTIALS + value: /var/secrets/google/key.json + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + capabilities: + drop: + - ALL + volumeMounts: + - name: sites-data + mountPath: /home/frappe/frappe-bench/sites + - name: tmp + mountPath: /tmp + - name: logs + mountPath: /home/frappe/frappe-bench/logs + - name: google-cloud-key + mountPath: /var/secrets/google + readOnly: true + resources: + requests: + memory: "256Mi" + cpu: "100m" + limits: + memory: "512Mi" + cpu: "250m" + livenessProbe: + exec: + command: + - pgrep + - -f + - "bench schedule" + initialDelaySeconds: 30 + periodSeconds: 30 + timeoutSeconds: 5 + failureThreshold: 3 + - name: cloud-sql-proxy + image: gcr.io/cloudsql-docker/gce-proxy:1.35.4-alpine + command: + - /cloud_sql_proxy + - -instances=$(DB_CONNECTION_NAME)=tcp:127.0.0.1:3306 + - -credential_file=/var/secrets/google/key.json + env: + - name: DB_CONNECTION_NAME + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: db-connection-name + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 2 + capabilities: + drop: + - ALL + volumeMounts: + - name: google-cloud-key + mountPath: /var/secrets/google + readOnly: true + resources: + requests: + memory: "64Mi" + cpu: "50m" + limits: + memory: "128Mi" + cpu: "100m" + volumes: + - name: sites-data + persistentVolumeClaim: + claimName: erpnext-sites-pvc + - name: tmp + emptyDir: {} + - name: logs + emptyDir: {} + - name: google-cloud-key + secret: + secretName: gcp-service-account-key + nodeSelector: + cloud.google.com/gke-preemptible: "false" +--- +# HPA for queue workers based on queue depth +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: erpnext-queue-default-hpa + namespace: erpnext + labels: + app: erpnext-queue-default + component: worker + deployment-type: managed-services +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: erpnext-queue-default + minReplicas: 2 + maxReplicas: 8 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 70 + - type: Resource + resource: + name: memory + target: + type: Utilization + averageUtilization: 80 + behavior: + scaleUp: + stabilizationWindowSeconds: 60 + policies: + - type: Pods + value: 2 + periodSeconds: 60 + scaleDown: + stabilizationWindowSeconds: 300 + policies: + - type: Pods + value: 1 + periodSeconds: 60 +--- +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: erpnext-queue-short-hpa + namespace: erpnext + labels: + app: erpnext-queue-short + component: worker + deployment-type: managed-services +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: erpnext-queue-short + minReplicas: 2 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 70 + - type: Resource + resource: + name: memory + target: + type: Utilization + averageUtilization: 80 + behavior: + scaleUp: + stabilizationWindowSeconds: 30 + policies: + - type: Pods + value: 2 + periodSeconds: 30 + scaleDown: + stabilizationWindowSeconds: 180 + policies: + - type: Pods + value: 1 + periodSeconds: 60 +--- +# Network Policy for worker services +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: erpnext-workers-netpol + namespace: erpnext +spec: + podSelector: + matchLabels: + component: worker + policyTypes: + - Egress + egress: + # Allow DNS resolution + - to: [] + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 + # Allow access to managed services + - to: [] + ports: + - protocol: TCP + port: 3306 # Cloud SQL + - protocol: TCP + port: 6379 # Redis + - protocol: TCP + port: 443 # HTTPS for GCP APIs +--- +# Network Policy for scheduler +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: erpnext-scheduler-netpol + namespace: erpnext +spec: + podSelector: + matchLabels: + app: erpnext-scheduler + policyTypes: + - Egress + egress: + # Allow DNS resolution + - to: [] + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 + # Allow access to managed services + - to: [] + ports: + - protocol: TCP + port: 3306 # Cloud SQL + - protocol: TCP + port: 6379 # Redis + - protocol: TCP + port: 443 # HTTPS for GCP APIs \ No newline at end of file diff --git a/documentation/deployment-guides/gcp-managed/kubernetes-manifests/ingress.yaml b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/ingress.yaml new file mode 100644 index 0000000..fa1eab4 --- /dev/null +++ b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/ingress.yaml @@ -0,0 +1,319 @@ +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: erpnext-ingress + namespace: erpnext + labels: + app: erpnext + component: ingress + environment: production + deployment-type: managed-services + annotations: + kubernetes.io/ingress.class: nginx + nginx.ingress.kubernetes.io/proxy-body-size: 50m + nginx.ingress.kubernetes.io/proxy-read-timeout: "600" + nginx.ingress.kubernetes.io/proxy-send-timeout: "600" + nginx.ingress.kubernetes.io/proxy-connect-timeout: "60" + nginx.ingress.kubernetes.io/ssl-redirect: "true" + nginx.ingress.kubernetes.io/force-ssl-redirect: "true" + nginx.ingress.kubernetes.io/use-regex: "true" + + # CORS configuration for managed services + nginx.ingress.kubernetes.io/cors-allow-origin: "*" + nginx.ingress.kubernetes.io/cors-allow-methods: "GET, POST, PUT, DELETE, OPTIONS" + nginx.ingress.kubernetes.io/cors-allow-headers: "DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization" + nginx.ingress.kubernetes.io/cors-expose-headers: "Content-Length,Content-Range" + nginx.ingress.kubernetes.io/enable-cors: "true" + + # SSL configuration + cert-manager.io/cluster-issuer: letsencrypt-prod + + # Rate limiting for API protection + nginx.ingress.kubernetes.io/rate-limit: "100" + nginx.ingress.kubernetes.io/rate-limit-window: "1m" + + # Enhanced security headers for managed services + nginx.ingress.kubernetes.io/configuration-snippet: | + add_header X-Frame-Options SAMEORIGIN always; + add_header X-Content-Type-Options nosniff always; + add_header X-XSS-Protection "1; mode=block" always; + add_header Referrer-Policy "strict-origin-when-cross-origin" always; + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval' https://storage.googleapis.com; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; img-src 'self' data: https: https://storage.googleapis.com; font-src 'self' data: https://fonts.gstatic.com; connect-src 'self' wss: https://storage.googleapis.com; frame-ancestors 'self';" always; + + # Custom headers for managed services + add_header X-Deployment-Type "managed-services" always; + add_header X-Backend-Type "cloud-sql-redis" always; + + # Performance optimizations + add_header Cache-Control "no-cache, no-store, must-revalidate" always; + add_header Pragma "no-cache" always; + add_header Expires "0" always; + + # Security for API endpoints + if ($request_uri ~* "^/api/") { + add_header X-API-Rate-Limit "100/min" always; + } + + # Server-side configuration snippet for managed services + nginx.ingress.kubernetes.io/server-snippet: | + # Rate limiting zones + limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s; + limit_req_zone $binary_remote_addr zone=login:10m rate=5r/m; + limit_req_zone $binary_remote_addr zone=upload:10m rate=2r/s; + + # Connection limiting + limit_conn_zone $binary_remote_addr zone=conn_limit_per_ip:10m; + limit_conn conn_limit_per_ip 20; + + # Security configurations + server_tokens off; + client_header_timeout 10s; + client_body_timeout 10s; + + # Logging for managed services + access_log /var/log/nginx/erpnext-managed.access.log combined; + error_log /var/log/nginx/erpnext-managed.error.log warn; + +spec: + tls: + - hosts: + - erpnext.yourdomain.com + - api.yourdomain.com + secretName: erpnext-tls + rules: + - host: erpnext.yourdomain.com + http: + paths: + # Static assets - serve from Cloud Storage or CDN + - path: /assets + pathType: Prefix + backend: + service: + name: erpnext-frontend + port: + number: 8080 + + # File uploads and downloads + - path: /files + pathType: Prefix + backend: + service: + name: erpnext-frontend + port: + number: 8080 + + # Protected files + - path: /protected + pathType: Prefix + backend: + service: + name: erpnext-frontend + port: + number: 8080 + + # WebSocket connections for real-time features + - path: /socket.io + pathType: Prefix + backend: + service: + name: erpnext-backend + port: + number: 9000 + + # API endpoints with rate limiting + - path: /api + pathType: Prefix + backend: + service: + name: erpnext-frontend + port: + number: 8080 + + # Method endpoints (Frappe framework) + - path: /method + pathType: Prefix + backend: + service: + name: erpnext-frontend + port: + number: 8080 + + # Main application + - path: / + pathType: Prefix + backend: + service: + name: erpnext-frontend + port: + number: 8080 + + # API subdomain for cleaner separation + - host: api.yourdomain.com + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: erpnext-backend + port: + number: 8000 +--- +# Additional ingress for API-only access with stricter security +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: erpnext-api-ingress + namespace: erpnext + labels: + app: erpnext + component: api-ingress + environment: production + deployment-type: managed-services + annotations: + kubernetes.io/ingress.class: nginx + nginx.ingress.kubernetes.io/proxy-body-size: 10m + nginx.ingress.kubernetes.io/proxy-read-timeout: "300" + nginx.ingress.kubernetes.io/proxy-send-timeout: "300" + nginx.ingress.kubernetes.io/ssl-redirect: "true" + cert-manager.io/cluster-issuer: letsencrypt-prod + + # Stricter rate limiting for API + nginx.ingress.kubernetes.io/rate-limit: "50" + nginx.ingress.kubernetes.io/rate-limit-window: "1m" + + # API-specific security + nginx.ingress.kubernetes.io/configuration-snippet: | + # Stricter security for API access + add_header X-API-Version "v14" always; + add_header X-Rate-Limit "50/min" always; + + # Block non-API requests + if ($request_uri !~* "^/api/") { + return 404; + } + + # Additional API security headers + add_header X-API-Security "enhanced" always; + +spec: + tls: + - hosts: + - api-secure.yourdomain.com + secretName: erpnext-api-tls + rules: + - host: api-secure.yourdomain.com + http: + paths: + - path: /api + pathType: Prefix + backend: + service: + name: erpnext-backend + port: + number: 8000 +--- +# ClusterIssuer for Let's Encrypt SSL certificates +apiVersion: cert-manager.io/v1 +kind: ClusterIssuer +metadata: + name: letsencrypt-prod +spec: + acme: + server: https://acme-v02.api.letsencrypt.org/directory + email: admin@yourdomain.com + privateKeySecretRef: + name: letsencrypt-prod + solvers: + - http01: + ingress: + class: nginx + podTemplate: + spec: + nodeSelector: + cloud.google.com/gke-preemptible: "false" +--- +# Staging issuer for testing +apiVersion: cert-manager.io/v1 +kind: ClusterIssuer +metadata: + name: letsencrypt-staging +spec: + acme: + server: https://acme-staging-v02.api.letsencrypt.org/directory + email: admin@yourdomain.com + privateKeySecretRef: + name: letsencrypt-staging + solvers: + - http01: + ingress: + class: nginx +--- +# Certificate for main domain +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: erpnext-cert + namespace: erpnext +spec: + secretName: erpnext-tls + issuerRef: + name: letsencrypt-prod + kind: ClusterIssuer + dnsNames: + - erpnext.yourdomain.com + - api.yourdomain.com +--- +# Certificate for API domain +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: erpnext-api-cert + namespace: erpnext +spec: + secretName: erpnext-api-tls + issuerRef: + name: letsencrypt-prod + kind: ClusterIssuer + dnsNames: + - api-secure.yourdomain.com +--- +# Nginx configuration for managed services optimization +apiVersion: v1 +kind: ConfigMap +metadata: + name: nginx-ingress-config + namespace: ingress-nginx +data: + # Global nginx configurations optimized for managed services + proxy-connect-timeout: "10" + proxy-send-timeout: "120" + proxy-read-timeout: "120" + proxy-body-size: "50m" + proxy-buffer-size: "4k" + proxy-buffers-number: "8" + + # Performance optimizations + worker-processes: "auto" + worker-connections: "16384" + worker-rlimit-nofile: "65536" + + # SSL optimizations + ssl-protocols: "TLSv1.2 TLSv1.3" + ssl-ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384" + ssl-prefer-server-ciphers: "true" + ssl-session-cache: "shared:SSL:10m" + ssl-session-timeout: "10m" + + # Connection keep-alive + keep-alive: "75" + keep-alive-requests: "100" + + # Rate limiting for managed services + limit-req-status-code: "429" + limit-conn-status-code: "429" + + # Logging optimizations + log-format-escape-json: "true" + log-format-upstream: '{"time": "$time_iso8601", "remote_addr": "$proxy_protocol_addr", "x_forwarded_for": "$proxy_add_x_forwarded_for", "request_id": "$req_id", "remote_user": "$remote_user", "bytes_sent": $bytes_sent, "request_time": $request_time, "status": $status, "vhost": "$host", "request_proto": "$server_protocol", "path": "$uri", "request_query": "$args", "request_length": $request_length, "duration": $request_time,"method": "$request_method", "http_referrer": "$http_referer", "http_user_agent": "$http_user_agent", "upstream_addr": "$upstream_addr", "upstream_response_time": $upstream_response_time, "upstream_response_length": $upstream_response_length, "upstream_status": $upstream_status }' \ No newline at end of file diff --git a/documentation/deployment-guides/gcp-managed/kubernetes-manifests/jobs.yaml b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/jobs.yaml new file mode 100644 index 0000000..34fb901 --- /dev/null +++ b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/jobs.yaml @@ -0,0 +1,606 @@ +apiVersion: batch/v1 +kind: Job +metadata: + name: erpnext-create-site + namespace: erpnext + labels: + app: erpnext + component: setup + job-type: create-site + deployment-type: managed-services +spec: + backoffLimit: 3 + activeDeadlineSeconds: 1800 # 30 minutes timeout + template: + metadata: + labels: + app: erpnext + component: setup + job-type: create-site + deployment-type: managed-services + spec: + serviceAccountName: erpnext-ksa + restartPolicy: Never + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault + initContainers: + - name: wait-for-managed-services + image: busybox:1.35 + command: + - sh + - -c + - | + echo 'Waiting for Cloud SQL and Redis managed services...' + + # Wait for Cloud SQL Proxy to establish connection + until nc -z 127.0.0.1 3306; do + echo 'Waiting for Cloud SQL connection...' + sleep 10 + done + + # Test Redis connection + echo 'Testing Redis connection...' + timeout 30 sh -c 'until nc -z $REDIS_HOST 6379; do sleep 5; done' + + echo 'All managed services are ready!' + + # Additional wait for services to be fully ready + sleep 30 + env: + - name: REDIS_HOST + value: "REDIS_HOST_PLACEHOLDER" # Will be replaced by deployment script + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + capabilities: + drop: + - ALL + containers: + - name: create-site + image: frappe/erpnext-worker:v14 + command: + - bash + - -c + - | + set -e + echo "Starting ERPNext site creation with managed services..." + + # Set up environment for managed services + export DB_HOST="127.0.0.1" + export DB_PORT="3306" + + # Check if site already exists + if [ -d "/home/frappe/frappe-bench/sites/frontend" ]; then + echo "Site 'frontend' already exists. Checking configuration..." + + # Update site configuration for managed services + cd /home/frappe/frappe-bench + bench --site frontend set-config db_host "$DB_HOST" + bench --site frontend set-config db_port "$DB_PORT" + bench --site frontend set-config redis_cache "redis://$REDIS_HOST:6379/0" + bench --site frontend set-config redis_queue "redis://$REDIS_HOST:6379/1" + bench --site frontend set-config redis_socketio "redis://$REDIS_HOST:6379/2" + + echo "Site configuration updated for managed services." + exit 0 + fi + + echo "Creating new ERPNext site with managed services..." + cd /home/frappe/frappe-bench + + # Create the site with managed database + bench new-site frontend \ + --admin-password "$ADMIN_PASSWORD" \ + --mariadb-root-password "$DB_PASSWORD" \ + --db-host "$DB_HOST" \ + --db-port "$DB_PORT" \ + --install-app erpnext \ + --set-default + + # Configure Redis for managed Memorystore + echo "Configuring Redis for Memorystore..." + bench --site frontend set-config redis_cache "redis://$REDIS_HOST:6379/0" + bench --site frontend set-config redis_queue "redis://$REDIS_HOST:6379/1" + bench --site frontend set-config redis_socketio "redis://$REDIS_HOST:6379/2" + + # Set additional configurations for managed services + bench --site frontend set-config developer_mode 0 + bench --site frontend set-config server_script_enabled 1 + bench --site frontend set-config allow_tests 0 + bench --site frontend set-config auto_update 0 + bench --site frontend set-config enable_scheduler 1 + + # Configure for Cloud Storage + bench --site frontend set-config use_google_cloud_storage 1 + bench --site frontend set-config google_cloud_storage_bucket "$CLOUD_STORAGE_BUCKET" + + # Set up email configuration (optional) + bench --site frontend set-config mail_server "smtp.gmail.com" + bench --site frontend set-config mail_port 587 + bench --site frontend set-config use_tls 1 + + # Install additional apps if needed + # bench --site frontend install-app custom_app + + # Run post-installation setup + bench --site frontend migrate + bench --site frontend clear-cache + + echo "Site creation completed successfully with managed services!" + + # Verify the setup + echo "Verifying installation..." + bench --site frontend list-apps + + # Create initial data if needed + # bench --site frontend execute frappe.utils.install.make_demo_data + + echo "ERPNext installation verification completed." + envFrom: + - configMapRef: + name: erpnext-managed-config + env: + - name: ADMIN_PASSWORD + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: admin-password + - name: DB_PASSWORD + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: db-password + - name: REDIS_HOST + value: "REDIS_HOST_PLACEHOLDER" # Will be replaced by deployment script + - name: CLOUD_STORAGE_BUCKET + value: "erpnext-files-PROJECT_ID" + - name: GOOGLE_APPLICATION_CREDENTIALS + value: /var/secrets/google/key.json + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + capabilities: + drop: + - ALL + volumeMounts: + - name: sites-data + mountPath: /home/frappe/frappe-bench/sites + - name: tmp + mountPath: /tmp + - name: google-cloud-key + mountPath: /var/secrets/google + readOnly: true + resources: + requests: + memory: "1Gi" + cpu: "500m" + limits: + memory: "2Gi" + cpu: "1000m" + - name: cloud-sql-proxy + image: gcr.io/cloudsql-docker/gce-proxy:1.35.4-alpine + command: + - /cloud_sql_proxy + - -instances=$(DB_CONNECTION_NAME)=tcp:127.0.0.1:3306 + - -credential_file=/var/secrets/google/key.json + - -enable_iam_login + env: + - name: DB_CONNECTION_NAME + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: db-connection-name + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 2 + capabilities: + drop: + - ALL + volumeMounts: + - name: google-cloud-key + mountPath: /var/secrets/google + readOnly: true + resources: + requests: + memory: "128Mi" + cpu: "100m" + limits: + memory: "256Mi" + cpu: "200m" + volumes: + - name: sites-data + persistentVolumeClaim: + claimName: erpnext-sites-pvc + - name: tmp + emptyDir: {} + - name: google-cloud-key + secret: + secretName: gcp-service-account-key +--- +# CronJob for backing up site files (database is handled by Cloud SQL) +apiVersion: batch/v1 +kind: CronJob +metadata: + name: erpnext-files-backup + namespace: erpnext + labels: + app: erpnext + component: backup + backup-type: files + deployment-type: managed-services +spec: + schedule: "0 3 * * *" # Daily at 3 AM + concurrencyPolicy: Forbid + successfulJobsHistoryLimit: 7 + failedJobsHistoryLimit: 3 + jobTemplate: + spec: + backoffLimit: 2 + activeDeadlineSeconds: 3600 # 1 hour timeout + template: + metadata: + labels: + app: erpnext + component: backup + backup-type: files + deployment-type: managed-services + spec: + serviceAccountName: erpnext-ksa + restartPolicy: OnFailure + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + containers: + - name: files-backup + image: google/cloud-sdk:alpine + command: + - /bin/bash + - -c + - | + set -e + + BACKUP_DATE=$(date +%Y%m%d_%H%M%S) + BACKUP_FILE="erpnext_files_backup_${BACKUP_DATE}.tar.gz" + + echo "Starting ERPNext files backup: $BACKUP_FILE" + + # Create compressed backup of sites directory + echo "Creating compressed archive..." + tar -czf /tmp/$BACKUP_FILE -C /sites \ + --exclude='*.log' \ + --exclude='*.tmp' \ + --exclude='__pycache__' \ + . + + # Get file size + BACKUP_SIZE=$(stat -f%z /tmp/$BACKUP_FILE 2>/dev/null || stat -c%s /tmp/$BACKUP_FILE 2>/dev/null || echo "unknown") + echo "Backup file size: $BACKUP_SIZE bytes" + + # Upload to Cloud Storage with metadata + echo "Uploading to Cloud Storage..." + gsutil -h "x-goog-meta-backup-date:$BACKUP_DATE" \ + -h "x-goog-meta-backup-type:files" \ + -h "x-goog-meta-deployment-type:managed-services" \ + -h "x-goog-meta-source:erpnext-gke" \ + cp /tmp/$BACKUP_FILE gs://erpnext-backups-$PROJECT_ID/files/ + + echo "Files backup uploaded to GCS: gs://erpnext-backups-$PROJECT_ID/files/$BACKUP_FILE" + + # Verify upload + if gsutil ls gs://erpnext-backups-$PROJECT_ID/files/$BACKUP_FILE > /dev/null 2>&1; then + echo "Backup verification successful" + else + echo "Backup verification failed!" + exit 1 + fi + + # Clean up local file + rm /tmp/$BACKUP_FILE + + # Clean up old backups (keep last 30 days) + echo "Cleaning up old backups..." + gsutil -m rm -r gs://erpnext-backups-$PROJECT_ID/files/erpnext_files_backup_$(date -d '30 days ago' +%Y%m%d)_* 2>/dev/null || true + + echo "Files backup completed successfully!" + env: + - name: PROJECT_ID + value: "PROJECT_ID_PLACEHOLDER" # Will be replaced by deployment script + - name: GOOGLE_APPLICATION_CREDENTIALS + value: /var/secrets/google/key.json + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + capabilities: + drop: + - ALL + volumeMounts: + - name: sites-data + mountPath: /sites + readOnly: true + - name: tmp + mountPath: /tmp + - name: google-cloud-key + mountPath: /var/secrets/google + readOnly: true + resources: + requests: + memory: "256Mi" + cpu: "100m" + limits: + memory: "512Mi" + cpu: "250m" + volumes: + - name: sites-data + persistentVolumeClaim: + claimName: erpnext-sites-pvc + - name: tmp + emptyDir: {} + - name: google-cloud-key + secret: + secretName: gcp-service-account-key +--- +# Job for database migration (for upgrades) +apiVersion: batch/v1 +kind: Job +metadata: + name: erpnext-migrate + namespace: erpnext + labels: + app: erpnext + component: maintenance + job-type: migrate + deployment-type: managed-services +spec: + backoffLimit: 3 + activeDeadlineSeconds: 3600 # 1 hour timeout + template: + metadata: + labels: + app: erpnext + component: maintenance + job-type: migrate + deployment-type: managed-services + spec: + serviceAccountName: erpnext-ksa + restartPolicy: Never + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault + initContainers: + - name: wait-for-services + image: busybox:1.35 + command: + - sh + - -c + - | + echo 'Waiting for managed services...' + until nc -z 127.0.0.1 3306; do + echo 'Waiting for Cloud SQL...' + sleep 5 + done + echo 'Services ready for migration' + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + capabilities: + drop: + - ALL + containers: + - name: migrate + image: frappe/erpnext-worker:v14 + command: + - bash + - -c + - | + set -e + echo "Starting ERPNext migration with managed services..." + + cd /home/frappe/frappe-bench + + # Create backup before migration + echo "Creating pre-migration backup..." + bench --site all backup --with-files + + # Run database migrations + echo "Running database migrations..." + bench --site all migrate + + # Clear cache + echo "Clearing cache..." + bench --site all clear-cache + + # Clear website cache + echo "Clearing website cache..." + bench --site all clear-website-cache + + # Rebuild search index if needed + echo "Rebuilding search index..." + bench --site all build-search-index || true + + # Update translations + echo "Updating translations..." + bench --site all build-translations || true + + # Verify migration + echo "Verifying migration..." + bench --site all list-apps + + echo "Migration completed successfully with managed services!" + envFrom: + - configMapRef: + name: erpnext-managed-config + env: + - name: DB_PASSWORD + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: db-password + - name: GOOGLE_APPLICATION_CREDENTIALS + value: /var/secrets/google/key.json + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + capabilities: + drop: + - ALL + volumeMounts: + - name: sites-data + mountPath: /home/frappe/frappe-bench/sites + - name: tmp + mountPath: /tmp + - name: google-cloud-key + mountPath: /var/secrets/google + readOnly: true + resources: + requests: + memory: "1Gi" + cpu: "500m" + limits: + memory: "2Gi" + cpu: "1000m" + - name: cloud-sql-proxy + image: gcr.io/cloudsql-docker/gce-proxy:1.35.4-alpine + command: + - /cloud_sql_proxy + - -instances=$(DB_CONNECTION_NAME)=tcp:127.0.0.1:3306 + - -credential_file=/var/secrets/google/key.json + env: + - name: DB_CONNECTION_NAME + valueFrom: + secretKeyRef: + name: erpnext-managed-secrets + key: db-connection-name + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 2 + capabilities: + drop: + - ALL + volumeMounts: + - name: google-cloud-key + mountPath: /var/secrets/google + readOnly: true + resources: + requests: + memory: "128Mi" + cpu: "100m" + limits: + memory: "256Mi" + cpu: "200m" + volumes: + - name: sites-data + persistentVolumeClaim: + claimName: erpnext-sites-pvc + - name: tmp + emptyDir: {} + - name: google-cloud-key + secret: + secretName: gcp-service-account-key +--- +# CronJob for health checks and maintenance +apiVersion: batch/v1 +kind: CronJob +metadata: + name: erpnext-health-check + namespace: erpnext + labels: + app: erpnext + component: maintenance + job-type: health-check + deployment-type: managed-services +spec: + schedule: "*/15 * * * *" # Every 15 minutes + concurrencyPolicy: Forbid + successfulJobsHistoryLimit: 3 + failedJobsHistoryLimit: 3 + jobTemplate: + spec: + backoffLimit: 1 + activeDeadlineSeconds: 300 # 5 minutes timeout + template: + metadata: + labels: + app: erpnext + component: maintenance + job-type: health-check + deployment-type: managed-services + spec: + serviceAccountName: erpnext-ksa + restartPolicy: Never + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + containers: + - name: health-check + image: curlimages/curl:latest + command: + - sh + - -c + - | + set -e + echo "Running ERPNext health checks..." + + # Check frontend service + echo "Checking frontend service..." + curl -f http://erpnext-frontend:8080/health || exit 1 + + # Check backend service + echo "Checking backend service..." + curl -f http://erpnext-backend:8000/api/method/ping || exit 1 + + # Check Redis connectivity (if accessible from cluster) + echo "Checking Redis connectivity..." + # This would require a Redis client in the image + # nc -z $REDIS_HOST 6379 || exit 1 + + echo "All health checks passed!" + env: + - name: REDIS_HOST + value: "REDIS_HOST_PLACEHOLDER" + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 65534 # nobody user in curl image + capabilities: + drop: + - ALL + resources: + requests: + memory: "64Mi" + cpu: "50m" + limits: + memory: "128Mi" + cpu: "100m" \ No newline at end of file diff --git a/documentation/deployment-guides/gcp-managed/kubernetes-manifests/namespace.yaml b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/namespace.yaml new file mode 100644 index 0000000..b680657 --- /dev/null +++ b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/namespace.yaml @@ -0,0 +1,43 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: erpnext + labels: + name: erpnext + environment: production + deployment-type: managed-services + pod-security.kubernetes.io/enforce: restricted + pod-security.kubernetes.io/audit: restricted + pod-security.kubernetes.io/warn: restricted +--- +apiVersion: v1 +kind: ResourceQuota +metadata: + name: erpnext-quota + namespace: erpnext +spec: + hard: + requests.cpu: "8" + requests.memory: 16Gi + limits.cpu: "16" + limits.memory: 32Gi + persistentvolumeclaims: "5" # Reduced since no DB storage needed + pods: "15" # Reduced since no DB/Redis pods + services: "8" + secrets: "10" + configmaps: "10" +--- +apiVersion: v1 +kind: LimitRange +metadata: + name: erpnext-limits + namespace: erpnext +spec: + limits: + - default: + cpu: "500m" + memory: "1Gi" + defaultRequest: + cpu: "100m" + memory: "256Mi" + type: Container \ No newline at end of file diff --git a/documentation/deployment-guides/gcp-managed/kubernetes-manifests/secrets.yaml b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/secrets.yaml new file mode 100644 index 0000000..b3ab098 --- /dev/null +++ b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/secrets.yaml @@ -0,0 +1,181 @@ +# Note: This file shows the structure of secrets that need to be created +# In practice, these should be created using kubectl commands or External Secrets Operator +# DO NOT commit actual secret values to version control + +apiVersion: v1 +kind: Secret +metadata: + name: erpnext-db-credentials + namespace: erpnext + labels: + app: erpnext + component: database + deployment-type: managed-services +type: Opaque +stringData: + # Cloud SQL database password + password: "PLACEHOLDER_DB_PASSWORD" + # Cloud SQL connection name + connection-name: "PROJECT_ID:REGION:erpnext-db" +--- +apiVersion: v1 +kind: Secret +metadata: + name: erpnext-admin-credentials + namespace: erpnext + labels: + app: erpnext + component: admin +type: Opaque +stringData: + # ERPNext administrator password + password: "PLACEHOLDER_ADMIN_PASSWORD" +--- +apiVersion: v1 +kind: Secret +metadata: + name: erpnext-api-credentials + namespace: erpnext + labels: + app: erpnext + component: api +type: Opaque +stringData: + # ERPNext API credentials + api-key: "PLACEHOLDER_API_KEY" + api-secret: "PLACEHOLDER_API_SECRET" +--- +apiVersion: v1 +kind: Secret +metadata: + name: redis-auth + namespace: erpnext + labels: + app: erpnext + component: redis + deployment-type: managed-services +type: Opaque +stringData: + # Memorystore Redis AUTH string + auth-string: "PLACEHOLDER_REDIS_AUTH" +--- +apiVersion: v1 +kind: Secret +metadata: + name: cloudsql-ssl-certs + namespace: erpnext + labels: + app: erpnext + component: database + security: ssl +type: Opaque +stringData: + # Cloud SQL SSL certificates + client-cert.pem: "PLACEHOLDER_CLIENT_CERT" + client-key.pem: "PLACEHOLDER_CLIENT_KEY" + server-ca.pem: "PLACEHOLDER_SERVER_CA" +--- +apiVersion: v1 +kind: Secret +metadata: + name: gcp-service-account-key + namespace: erpnext + labels: + app: erpnext + component: auth + deployment-type: managed-services +type: Opaque +stringData: + # Service account key for accessing managed services + key.json: "PLACEHOLDER_SERVICE_ACCOUNT_KEY" +--- +# External Secrets Operator configuration (recommended approach) +apiVersion: external-secrets.io/v1beta1 +kind: SecretStore +metadata: + name: gcpsm-secret-store + namespace: erpnext +spec: + provider: + gcpsm: + projectId: "PROJECT_ID" + auth: + workloadIdentity: + clusterLocation: us-central1-a + clusterName: erpnext-managed-cluster + serviceAccountRef: + name: erpnext-ksa +--- +apiVersion: external-secrets.io/v1beta1 +kind: ExternalSecret +metadata: + name: erpnext-managed-secrets + namespace: erpnext +spec: + refreshInterval: 15m + secretStoreRef: + name: gcpsm-secret-store + kind: SecretStore + target: + name: erpnext-managed-secrets + creationPolicy: Owner + data: + - secretKey: admin-password + remoteRef: + key: erpnext-admin-password + - secretKey: db-password + remoteRef: + key: erpnext-db-password + - secretKey: api-key + remoteRef: + key: erpnext-api-key + - secretKey: api-secret + remoteRef: + key: erpnext-api-secret + - secretKey: redis-auth + remoteRef: + key: redis-auth-string + - secretKey: db-connection-name + remoteRef: + key: erpnext-db-connection-name +--- +# Service Account for Workload Identity +apiVersion: v1 +kind: ServiceAccount +metadata: + name: erpnext-ksa + namespace: erpnext + annotations: + iam.gke.io/gcp-service-account: erpnext-managed@PROJECT_ID.iam.gserviceaccount.com + labels: + app: erpnext + component: auth + deployment-type: managed-services +--- +# ClusterRole for accessing secrets +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + namespace: erpnext + name: erpnext-secrets-reader +rules: +- apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "list"] +- apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "list"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: erpnext-secrets-binding + namespace: erpnext +subjects: +- kind: ServiceAccount + name: erpnext-ksa + namespace: erpnext +roleRef: + kind: Role + name: erpnext-secrets-reader + apiGroup: rbac.authorization.k8s.io \ No newline at end of file diff --git a/documentation/deployment-guides/gcp-managed/kubernetes-manifests/storage.yaml b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/storage.yaml new file mode 100644 index 0000000..9bb1497 --- /dev/null +++ b/documentation/deployment-guides/gcp-managed/kubernetes-manifests/storage.yaml @@ -0,0 +1,75 @@ +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: ssd-retain +provisioner: kubernetes.io/gce-pd +parameters: + type: pd-ssd +reclaimPolicy: Retain +allowVolumeExpansion: true +volumeBindingMode: WaitForFirstConsumer +--- +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: standard-retain +provisioner: kubernetes.io/gce-pd +parameters: + type: pd-standard +reclaimPolicy: Retain +allowVolumeExpansion: true +volumeBindingMode: WaitForFirstConsumer +--- +# ERPNext sites storage (smaller since database is external) +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: erpnext-sites-pvc + namespace: erpnext + labels: + app: erpnext + component: sites + storage-type: application-files +spec: + storageClassName: ssd-retain + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 30Gi # Reduced since no database files +--- +# ERPNext assets storage (for static files and uploads) +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: erpnext-assets-pvc + namespace: erpnext + labels: + app: erpnext + component: assets + storage-type: static-files +spec: + storageClassName: ssd-retain + accessModes: + - ReadWriteMany + resources: + requests: + storage: 15Gi # Reduced for managed services +--- +# Backup storage for application files only +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: backup-pvc + namespace: erpnext + labels: + app: erpnext + component: backup + storage-type: backup +spec: + storageClassName: standard-retain + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 50Gi # Smaller since database backups are handled by Cloud SQL \ No newline at end of file diff --git a/documentation/deployment-guides/gcp-managed/scripts/cloud-run-deploy.sh b/documentation/deployment-guides/gcp-managed/scripts/cloud-run-deploy.sh new file mode 100755 index 0000000..c56248a --- /dev/null +++ b/documentation/deployment-guides/gcp-managed/scripts/cloud-run-deploy.sh @@ -0,0 +1,980 @@ +#!/bin/bash + +# ERPNext Cloud Run Deployment Script with Managed Services +# This script automates the deployment of ERPNext on Cloud Run using Cloud SQL and Memorystore + +set -e + +# Color codes for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Configuration +PROJECT_ID=${PROJECT_ID:-""} +REGION=${REGION:-"us-central1"} +DOMAIN=${DOMAIN:-"erpnext.yourdomain.com"} +API_DOMAIN=${API_DOMAIN:-"api.yourdomain.com"} + +# Managed services configuration +DB_INSTANCE_NAME=${DB_INSTANCE_NAME:-"erpnext-db"} +REDIS_INSTANCE_NAME=${REDIS_INSTANCE_NAME:-"erpnext-redis"} +VPC_NAME=${VPC_NAME:-"erpnext-vpc"} +VPC_CONNECTOR=${VPC_CONNECTOR:-"erpnext-connector"} + +# Cloud Run service names +BACKEND_SERVICE=${BACKEND_SERVICE:-"erpnext-backend"} +FRONTEND_SERVICE=${FRONTEND_SERVICE:-"erpnext-frontend"} +BACKGROUND_SERVICE=${BACKGROUND_SERVICE:-"erpnext-background"} + +# Function to print colored output +print_status() { + echo -e "${BLUE}[INFO]${NC} $1" +} + +print_success() { + echo -e "${GREEN}[SUCCESS]${NC} $1" +} + +print_warning() { + echo -e "${YELLOW}[WARNING]${NC} $1" +} + +print_error() { + echo -e "${RED}[ERROR]${NC} $1" +} + +# Function to check prerequisites +check_prerequisites() { + print_status "Checking prerequisites for Cloud Run deployment..." + + # Check if required tools are installed + local required_tools=("gcloud" "docker") + for tool in "${required_tools[@]}"; do + if ! command -v "$tool" &> /dev/null; then + print_error "$tool is not installed. Please install it first." + exit 1 + fi + done + + # Check if user is authenticated + if ! gcloud auth list --filter=status:ACTIVE --format="value(account)" | head -n 1 &> /dev/null; then + print_error "Not authenticated with gcloud. Please run 'gcloud auth login'" + exit 1 + fi + + # Check if project ID is set + if [[ -z "$PROJECT_ID" ]]; then + PROJECT_ID=$(gcloud config get-value project) + if [[ -z "$PROJECT_ID" ]]; then + print_error "PROJECT_ID not set. Please set it or configure gcloud project." + exit 1 + fi + fi + + # Check if Docker is configured for gcloud + if ! gcloud auth configure-docker --quiet; then + print_error "Failed to configure Docker for gcloud" + exit 1 + fi + + print_success "Prerequisites check passed" +} + +# Function to check managed services +check_managed_services() { + print_status "Checking managed services status..." + + # Check Cloud SQL instance + if ! gcloud sql instances describe "$DB_INSTANCE_NAME" &> /dev/null; then + print_error "Cloud SQL instance '$DB_INSTANCE_NAME' not found. Please create it first." + exit 1 + fi + + # Check Memorystore Redis instance + if ! gcloud redis instances describe "$REDIS_INSTANCE_NAME" --region="$REGION" &> /dev/null; then + print_error "Memorystore Redis instance '$REDIS_INSTANCE_NAME' not found. Please create it first." + exit 1 + fi + + # Check VPC connector + if ! gcloud compute networks vpc-access connectors describe "$VPC_CONNECTOR" --region="$REGION" &> /dev/null; then + print_error "VPC connector '$VPC_CONNECTOR' not found. Please create it first." + exit 1 + fi + + print_success "All managed services are available" +} + +# Function to get managed services information +get_managed_services_info() { + print_status "Gathering managed services information..." + + # Get Cloud SQL connection name + DB_CONNECTION_NAME=$(gcloud sql instances describe "$DB_INSTANCE_NAME" --format="value(connectionName)") + print_status "Cloud SQL connection name: $DB_CONNECTION_NAME" + + # Get Redis host IP + REDIS_HOST=$(gcloud redis instances describe "$REDIS_INSTANCE_NAME" --region="$REGION" --format="value(host)") + print_status "Redis host IP: $REDIS_HOST" + + # Get Redis AUTH if enabled + REDIS_AUTH="" + if gcloud redis instances describe "$REDIS_INSTANCE_NAME" --region="$REGION" --format="value(authEnabled)" | grep -q "True"; then + REDIS_AUTH=$(gcloud redis instances describe "$REDIS_INSTANCE_NAME" --region="$REGION" --format="value(authString)") + print_status "Redis AUTH enabled" + fi + + print_success "Managed services information gathered" +} + +# Function to setup Cloud Storage for files +setup_cloud_storage() { + print_status "Setting up Cloud Storage for ERPNext files..." + + local bucket_name="erpnext-files-$PROJECT_ID" + + # Create bucket if it doesn't exist + if ! gsutil ls -b gs://"$bucket_name" &> /dev/null; then + gsutil mb gs://"$bucket_name" + print_success "Created Cloud Storage bucket: $bucket_name" + else + print_warning "Bucket $bucket_name already exists" + fi + + # Set lifecycle policy + gsutil lifecycle set - gs://"$bucket_name" < /dev/null; then + local admin_password=${ADMIN_PASSWORD:-$(openssl rand -base64 32)} + gcloud secrets create erpnext-admin-password --data-file=<(echo -n "$admin_password") + print_warning "Admin password: $admin_password" + print_warning "Please save this password securely!" + fi + + if ! gcloud secrets describe erpnext-db-password &> /dev/null; then + local db_password=${DB_PASSWORD:-$(openssl rand -base64 32)} + gcloud secrets create erpnext-db-password --data-file=<(echo -n "$db_password") + print_warning "Database password: $db_password" + print_warning "Please save this password securely!" + fi + + if ! gcloud secrets describe erpnext-api-key &> /dev/null; then + local api_key=${API_KEY:-$(openssl rand -hex 32)} + gcloud secrets create erpnext-api-key --data-file=<(echo -n "$api_key") + fi + + if ! gcloud secrets describe erpnext-api-secret &> /dev/null; then + local api_secret=${API_SECRET:-$(openssl rand -hex 32)} + gcloud secrets create erpnext-api-secret --data-file=<(echo -n "$api_secret") + fi + + # Store connection information + gcloud secrets create erpnext-db-connection-name --data-file=<(echo -n "$DB_CONNECTION_NAME") --quiet || \ + gcloud secrets versions add erpnext-db-connection-name --data-file=<(echo -n "$DB_CONNECTION_NAME") + + if [[ -n "$REDIS_AUTH" ]]; then + gcloud secrets create redis-auth-string --data-file=<(echo -n "$REDIS_AUTH") --quiet || \ + gcloud secrets versions add redis-auth-string --data-file=<(echo -n "$REDIS_AUTH") + fi + + print_success "Secrets created in Secret Manager" +} + +# Function to build and push container images +build_and_push_images() { + print_status "Building and pushing container images..." + + # Create temporary directory for builds + local build_dir="/tmp/erpnext-cloudrun-builds" + mkdir -p "$build_dir" + + # Build backend image + print_status "Building ERPNext backend image..." + cat > "$build_dir/Dockerfile.backend" <<'EOF' +FROM frappe/erpnext-worker:v14 + +# Install Cloud SQL Proxy +RUN apt-get update && apt-get install -y wget netcat-openbsd && \ + wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy && \ + chmod +x cloud_sql_proxy && \ + mv cloud_sql_proxy /usr/local/bin/ && \ + apt-get clean && rm -rf /var/lib/apt/lists/* + +# Install additional Python packages +RUN pip install --no-cache-dir google-cloud-storage google-cloud-secret-manager + +# Create startup script +COPY startup-backend.sh /startup.sh +RUN chmod +x /startup.sh + +# Set environment for Cloud Run +ENV PORT=8080 +ENV WORKERS=4 +ENV TIMEOUT=300 +ENV MAX_REQUESTS=1000 + +EXPOSE 8080 + +CMD ["/startup.sh"] +EOF + + # Create backend startup script + cat > "$build_dir/startup-backend.sh" <<'EOF' +#!/bin/bash +set -e + +echo "Starting ERPNext backend for Cloud Run..." + +# Start Cloud SQL Proxy in background +if [ -n "$DB_CONNECTION_NAME" ]; then + echo "Starting Cloud SQL Proxy..." + /usr/local/bin/cloud_sql_proxy -instances=$DB_CONNECTION_NAME=tcp:3306 & + + # Wait for proxy to be ready + echo "Waiting for database connection..." + until nc -z localhost 3306; do + sleep 2 + done + echo "Database connection established" +fi + +# Set database connection environment +export DB_HOST="127.0.0.1" +export DB_PORT="3306" + +# Set Redis connection +export REDIS_CACHE_URL="redis://$REDIS_HOST:6379/0" +export REDIS_QUEUE_URL="redis://$REDIS_HOST:6379/1" +export REDIS_SOCKETIO_URL="redis://$REDIS_HOST:6379/2" + +# Add Redis AUTH if available +if [ -n "$REDIS_AUTH" ]; then + export REDIS_CACHE_URL="redis://:$REDIS_AUTH@$REDIS_HOST:6379/0" + export REDIS_QUEUE_URL="redis://:$REDIS_AUTH@$REDIS_HOST:6379/1" + export REDIS_SOCKETIO_URL="redis://:$REDIS_AUTH@$REDIS_HOST:6379/2" +fi + +# Initialize site if it doesn't exist (for first run) +cd /home/frappe/frappe-bench +if [ ! -d "sites/frontend" ] && [ "$INIT_SITE" = "true" ]; then + echo "Initializing ERPNext site..." + bench new-site frontend \ + --admin-password "$ADMIN_PASSWORD" \ + --mariadb-root-password "$DB_PASSWORD" \ + --install-app erpnext \ + --set-default +fi + +# Start ERPNext with Gunicorn optimized for Cloud Run +echo "Starting ERPNext backend..." +exec gunicorn -b 0.0.0.0:$PORT \ + -w $WORKERS \ + -t $TIMEOUT \ + --max-requests $MAX_REQUESTS \ + --preload \ + --access-logfile - \ + --error-logfile - \ + frappe.app:application +EOF + + # Build and push backend image + cd "$build_dir" + gcloud builds submit --tag gcr.io/$PROJECT_ID/erpnext-backend-cloudrun:latest \ + --dockerfile=Dockerfile.backend . + + # Build frontend image + print_status "Building ERPNext frontend image..." + cat > "$build_dir/Dockerfile.frontend" <<'EOF' +FROM nginx:alpine + +# Install envsubst for template processing +RUN apk add --no-cache gettext + +# Copy nginx configuration template +COPY nginx.conf.template /etc/nginx/nginx.conf.template +COPY startup-frontend.sh /startup.sh +RUN chmod +x /startup.sh + +# Set Cloud Run port +ENV PORT=8080 + +EXPOSE 8080 + +CMD ["/startup.sh"] +EOF + + # Create nginx configuration template + cat > "$build_dir/nginx.conf.template" <<'EOF' +user nginx; +worker_processes auto; +error_log /var/log/nginx/error.log notice; +pid /var/run/nginx.pid; + +events { + worker_connections 1024; +} + +http { + include /etc/nginx/mime.types; + default_type application/octet-stream; + + log_format main '$remote_addr - $remote_user [$time_local] "$request" ' + '$status $body_bytes_sent "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for"'; + + access_log /var/log/nginx/access.log main; + + sendfile on; + tcp_nopush on; + tcp_nodelay on; + keepalive_timeout 65; + types_hash_max_size 2048; + client_max_body_size 50m; + + gzip on; + gzip_vary on; + gzip_proxied any; + gzip_comp_level 6; + gzip_types + text/plain + text/css + text/xml + text/javascript + application/json + application/javascript + application/xml+rss + application/atom+xml + image/svg+xml; + + upstream backend { + server ${BACKEND_URL}; + keepalive 32; + } + + server { + listen ${PORT}; + server_name _; + + # Security headers + add_header X-Frame-Options SAMEORIGIN; + add_header X-Content-Type-Options nosniff; + add_header X-XSS-Protection "1; mode=block"; + add_header Referrer-Policy "strict-origin-when-cross-origin"; + + # Serve static assets from Cloud Storage + location /assets/ { + proxy_pass https://storage.googleapis.com/${STORAGE_BUCKET}/assets/; + proxy_set_header Host storage.googleapis.com; + expires 1y; + add_header Cache-Control "public, immutable"; + } + + # Proxy API requests to backend service + location /api/ { + proxy_pass http://backend; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_read_timeout 300s; + } + + # WebSocket support + location /socket.io/ { + proxy_pass http://backend; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } + + # Main application + location / { + proxy_pass http://backend; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_read_timeout 300s; + } + + # Health check + location /health { + access_log off; + return 200 "healthy\n"; + add_header Content-Type text/plain; + } + } +} +EOF + + # Create frontend startup script + cat > "$build_dir/startup-frontend.sh" <<'EOF' +#!/bin/sh +set -e + +echo "Starting ERPNext frontend for Cloud Run..." + +# Process nginx configuration template +envsubst '${PORT} ${BACKEND_URL} ${STORAGE_BUCKET}' < /etc/nginx/nginx.conf.template > /etc/nginx/nginx.conf + +echo "Nginx configuration processed" +cat /etc/nginx/nginx.conf + +# Start nginx +exec nginx -g 'daemon off;' +EOF + + # Build and push frontend image + gcloud builds submit --tag gcr.io/$PROJECT_ID/erpnext-frontend-cloudrun:latest \ + --dockerfile=Dockerfile.frontend . + + # Build background worker image + print_status "Building ERPNext background worker image..." + cat > "$build_dir/Dockerfile.background" <<'EOF' +FROM frappe/erpnext-worker:v14 + +# Install Cloud SQL Proxy and additional tools +RUN apt-get update && apt-get install -y wget netcat-openbsd && \ + wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy && \ + chmod +x cloud_sql_proxy && \ + mv cloud_sql_proxy /usr/local/bin/ && \ + apt-get clean && rm -rf /var/lib/apt/lists/* + +# Install additional Python packages +RUN pip install --no-cache-dir google-cloud-tasks google-cloud-storage flask + +# Copy task processor +COPY background_processor.py /background_processor.py +COPY startup-background.sh /startup.sh +RUN chmod +x /startup.sh + +ENV PORT=8080 + +EXPOSE 8080 + +CMD ["/startup.sh"] +EOF + + # Create background processor + cat > "$build_dir/background_processor.py" <<'EOF' +import os +import json +import logging +from flask import Flask, request, jsonify +import frappe +from frappe.utils.background_jobs import execute_job + +app = Flask(__name__) +logging.basicConfig(level=logging.INFO) + +@app.route('/process-task', methods=['POST']) +def process_task(): + """Process background task from Cloud Tasks or Cloud Scheduler""" + try: + # Get task data + task_data = request.get_json() + if not task_data: + return jsonify({'status': 'error', 'message': 'No task data provided'}), 400 + + # Set site context + frappe.init(site='frontend') + frappe.connect() + + # Execute the job + job_name = task_data.get('job_name') + kwargs = task_data.get('kwargs', {}) + + if not job_name: + return jsonify({'status': 'error', 'message': 'No job_name provided'}), 400 + + logging.info(f"Executing job: {job_name}") + result = execute_job(job_name, **kwargs) + + frappe.db.commit() + frappe.destroy() + + return jsonify({'status': 'success', 'result': str(result)}) + + except Exception as e: + logging.error(f"Task processing failed: {str(e)}") + return jsonify({'status': 'error', 'message': str(e)}), 500 + +@app.route('/health', methods=['GET']) +def health_check(): + return jsonify({'status': 'healthy'}) + +@app.route('/', methods=['GET']) +def root(): + return jsonify({'service': 'erpnext-background', 'status': 'running'}) + +if __name__ == '__main__': + port = int(os.environ.get('PORT', 8080)) + app.run(host='0.0.0.0', port=port) +EOF + + # Create background startup script + cat > "$build_dir/startup-background.sh" <<'EOF' +#!/bin/bash +set -e + +echo "Starting ERPNext background processor for Cloud Run..." + +# Start Cloud SQL Proxy in background +if [ -n "$DB_CONNECTION_NAME" ]; then + echo "Starting Cloud SQL Proxy..." + /usr/local/bin/cloud_sql_proxy -instances=$DB_CONNECTION_NAME=tcp:3306 & + + # Wait for proxy to be ready + until nc -z localhost 3306; do + echo "Waiting for database connection..." + sleep 2 + done + echo "Database connection established" +fi + +export DB_HOST="127.0.0.1" +export DB_PORT="3306" + +# Start background processor +cd /home/frappe/frappe-bench +exec python /background_processor.py +EOF + + # Build and push background image + gcloud builds submit --tag gcr.io/$PROJECT_ID/erpnext-background-cloudrun:latest \ + --dockerfile=Dockerfile.background . + + # Cleanup + rm -rf "$build_dir" + + print_success "Container images built and pushed" +} + +# Function to deploy Cloud Run services +deploy_cloud_run_services() { + print_status "Deploying Cloud Run services..." + + # Deploy backend service + print_status "Deploying backend service..." + gcloud run deploy "$BACKEND_SERVICE" \ + --image gcr.io/$PROJECT_ID/erpnext-backend-cloudrun:latest \ + --platform managed \ + --region "$REGION" \ + --allow-unauthenticated \ + --vpc-connector "$VPC_CONNECTOR" \ + --set-env-vars="DB_CONNECTION_NAME=$DB_CONNECTION_NAME,REDIS_HOST=$REDIS_HOST,PROJECT_ID=$PROJECT_ID" \ + --set-secrets="ADMIN_PASSWORD=erpnext-admin-password:latest,DB_PASSWORD=erpnext-db-password:latest,API_KEY=erpnext-api-key:latest,API_SECRET=erpnext-api-secret:latest" \ + --service-account "erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com" \ + --memory 2Gi \ + --cpu 2 \ + --concurrency 80 \ + --timeout 300 \ + --min-instances 1 \ + --max-instances 100 + + if [[ -n "$REDIS_AUTH" ]]; then + gcloud run services update "$BACKEND_SERVICE" \ + --region "$REGION" \ + --update-secrets="REDIS_AUTH=redis-auth-string:latest" + fi + + # Get backend service URL + BACKEND_URL=$(gcloud run services describe "$BACKEND_SERVICE" --region="$REGION" --format="value(status.url)") + BACKEND_HOSTNAME=$(echo "$BACKEND_URL" | sed 's|https://||') + + print_success "Backend service deployed: $BACKEND_URL" + + # Deploy frontend service + print_status "Deploying frontend service..." + gcloud run deploy "$FRONTEND_SERVICE" \ + --image gcr.io/$PROJECT_ID/erpnext-frontend-cloudrun:latest \ + --platform managed \ + --region "$REGION" \ + --allow-unauthenticated \ + --set-env-vars="BACKEND_URL=$BACKEND_HOSTNAME,STORAGE_BUCKET=erpnext-files-$PROJECT_ID" \ + --memory 512Mi \ + --cpu 1 \ + --concurrency 1000 \ + --timeout 60 \ + --min-instances 0 \ + --max-instances 10 + + # Get frontend service URL + FRONTEND_URL=$(gcloud run services describe "$FRONTEND_SERVICE" --region="$REGION" --format="value(status.url)") + + print_success "Frontend service deployed: $FRONTEND_URL" + + # Deploy background service + print_status "Deploying background service..." + gcloud run deploy "$BACKGROUND_SERVICE" \ + --image gcr.io/$PROJECT_ID/erpnext-background-cloudrun:latest \ + --platform managed \ + --region "$REGION" \ + --no-allow-unauthenticated \ + --vpc-connector "$VPC_CONNECTOR" \ + --set-env-vars="DB_CONNECTION_NAME=$DB_CONNECTION_NAME,REDIS_HOST=$REDIS_HOST,PROJECT_ID=$PROJECT_ID" \ + --set-secrets="DB_PASSWORD=erpnext-db-password:latest" \ + --service-account "erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com" \ + --memory 1Gi \ + --cpu 1 \ + --concurrency 10 \ + --timeout 900 \ + --min-instances 0 \ + --max-instances 10 + + if [[ -n "$REDIS_AUTH" ]]; then + gcloud run services update "$BACKGROUND_SERVICE" \ + --region "$REGION" \ + --update-secrets="REDIS_AUTH=redis-auth-string:latest" + fi + + # Get background service URL + BACKGROUND_URL=$(gcloud run services describe "$BACKGROUND_SERVICE" --region="$REGION" --format="value(status.url)") + + print_success "Background service deployed: $BACKGROUND_URL" +} + +# Function to initialize ERPNext site +initialize_site() { + print_status "Initializing ERPNext site..." + + # Create a temporary Cloud Run job to initialize the site + gcloud run jobs create erpnext-init \ + --image gcr.io/$PROJECT_ID/erpnext-backend-cloudrun:latest \ + --region "$REGION" \ + --vpc-connector "$VPC_CONNECTOR" \ + --set-env-vars="DB_CONNECTION_NAME=$DB_CONNECTION_NAME,REDIS_HOST=$REDIS_HOST,INIT_SITE=true" \ + --set-secrets="ADMIN_PASSWORD=erpnext-admin-password:latest,DB_PASSWORD=erpnext-db-password:latest" \ + --service-account "erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com" \ + --memory 2Gi \ + --cpu 2 \ + --max-retries 3 \ + --parallelism 1 \ + --task-count 1 \ + --task-timeout 1800 + + if [[ -n "$REDIS_AUTH" ]]; then + gcloud run jobs update erpnext-init \ + --region "$REGION" \ + --update-secrets="REDIS_AUTH=redis-auth-string:latest" + fi + + # Execute the job + print_status "Executing site initialization job..." + gcloud run jobs execute erpnext-init --region "$REGION" --wait + + # Check if job succeeded + local job_status=$(gcloud run jobs describe erpnext-init --region="$REGION" --format="value(status.conditions[0].type)") + if [[ "$job_status" == "Succeeded" ]]; then + print_success "ERPNext site initialized successfully" + else + print_error "Site initialization failed. Check job logs:" + gcloud logging read "resource.type=\"cloud_run_job\" AND resource.labels.job_name=\"erpnext-init\"" --limit=50 + exit 1 + fi + + # Clean up the job + gcloud run jobs delete erpnext-init --region="$REGION" --quiet +} + +# Function to setup Cloud Tasks and Cloud Scheduler +setup_background_processing() { + print_status "Setting up Cloud Tasks and Cloud Scheduler..." + + # Create task queue + gcloud tasks queues create erpnext-tasks \ + --location="$REGION" \ + --max-dispatches-per-second=100 \ + --max-concurrent-dispatches=1000 \ + --max-attempts=3 || true + + # Create scheduled jobs + print_status "Creating scheduled jobs..." + + # ERPNext scheduler (every 5 minutes) + gcloud scheduler jobs create http erpnext-scheduler \ + --location="$REGION" \ + --schedule="*/5 * * * *" \ + --uri="$BACKGROUND_URL/process-task" \ + --http-method=POST \ + --headers="Content-Type=application/json" \ + --message-body='{"job_name": "frappe.utils.scheduler.execute_all", "kwargs": {}}' \ + --oidc-service-account-email="erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com" \ + --quiet || true + + # Email queue processor (every minute) + gcloud scheduler jobs create http erpnext-email-queue \ + --location="$REGION" \ + --schedule="* * * * *" \ + --uri="$BACKGROUND_URL/process-task" \ + --http-method=POST \ + --headers="Content-Type=application/json" \ + --message-body='{"job_name": "frappe.email.queue.flush", "kwargs": {}}' \ + --oidc-service-account-email="erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com" \ + --quiet || true + + # Daily backup job + gcloud scheduler jobs create http erpnext-daily-backup \ + --location="$REGION" \ + --schedule="0 2 * * *" \ + --uri="$BACKGROUND_URL/process-task" \ + --http-method=POST \ + --headers="Content-Type=application/json" \ + --message-body='{"job_name": "frappe.utils.backup.backup_to_cloud", "kwargs": {}}' \ + --oidc-service-account-email="erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com" \ + --quiet || true + + print_success "Background processing setup completed" +} + +# Function to setup custom domain and load balancer +setup_custom_domain() { + print_status "Setting up custom domain and load balancer..." + + # Create health check + gcloud compute health-checks create http erpnext-health-check \ + --request-path="/health" \ + --port=8080 \ + --quiet || true + + # Create backend service for frontend + gcloud compute backend-services create erpnext-frontend-backend \ + --protocol=HTTP \ + --health-checks=erpnext-health-check \ + --global \ + --quiet || true + + # Create NEG for frontend service + gcloud compute network-endpoint-groups create erpnext-frontend-neg \ + --region="$REGION" \ + --network-endpoint-type=serverless \ + --cloud-run-service="$FRONTEND_SERVICE" \ + --quiet || true + + gcloud compute backend-services add-backend erpnext-frontend-backend \ + --global \ + --network-endpoint-group=erpnext-frontend-neg \ + --network-endpoint-group-region="$REGION" \ + --quiet || true + + # Create URL map + gcloud compute url-maps create erpnext-url-map \ + --default-service=erpnext-frontend-backend \ + --quiet || true + + # Create managed SSL certificate + gcloud compute ssl-certificates create erpnext-ssl-cert \ + --domains="$DOMAIN" \ + --quiet || true + + # Create HTTPS proxy + gcloud compute target-https-proxies create erpnext-https-proxy \ + --ssl-certificates=erpnext-ssl-cert \ + --url-map=erpnext-url-map \ + --quiet || true + + # Create global forwarding rule + gcloud compute forwarding-rules create erpnext-https-rule \ + --global \ + --target-https-proxy=erpnext-https-proxy \ + --ports=443 \ + --quiet || true + + # Get global IP address + GLOBAL_IP=$(gcloud compute forwarding-rules describe erpnext-https-rule --global --format="value(IPAddress)") + + print_success "Custom domain setup completed" + print_warning "Point your domain $DOMAIN to IP address: $GLOBAL_IP" +} + +# Function to show deployment status +show_status() { + print_status "Cloud Run deployment status:" + + echo "" + echo "=== Cloud Run Services ===" + gcloud run services list --region="$REGION" --filter="metadata.name:erpnext" + + echo "" + echo "=== Managed Services ===" + echo "Cloud SQL:" + gcloud sql instances describe "$DB_INSTANCE_NAME" --format="table(name,state,region)" + echo "" + echo "Redis:" + gcloud redis instances describe "$REDIS_INSTANCE_NAME" --region="$REGION" --format="table(name,state,host)" + + echo "" + echo "=== Cloud Scheduler Jobs ===" + gcloud scheduler jobs list --location="$REGION" --filter="name:erpnext" + + echo "" + echo "=== Load Balancer ===" + gcloud compute forwarding-rules list --global --filter="name:erpnext" + + echo "" + echo "=== Service URLs ===" + echo "Frontend: $(gcloud run services describe "$FRONTEND_SERVICE" --region="$REGION" --format="value(status.url)")" + echo "Backend API: $(gcloud run services describe "$BACKEND_SERVICE" --region="$REGION" --format="value(status.url)")" + echo "Custom Domain: https://$DOMAIN (if DNS is configured)" +} + +# Function to cleanup deployment +cleanup() { + print_warning "This will delete the Cloud Run deployment but preserve managed services. Are you sure? (y/N)" + read -r response + if [[ "$response" =~ ^([yY][eE][sS]|[yY])$ ]]; then + print_status "Cleaning up Cloud Run deployment..." + + # Delete Cloud Run services + gcloud run services delete "$FRONTEND_SERVICE" --region="$REGION" --quiet || true + gcloud run services delete "$BACKEND_SERVICE" --region="$REGION" --quiet || true + gcloud run services delete "$BACKGROUND_SERVICE" --region="$REGION" --quiet || true + + # Delete Cloud Scheduler jobs + gcloud scheduler jobs delete erpnext-scheduler --location="$REGION" --quiet || true + gcloud scheduler jobs delete erpnext-email-queue --location="$REGION" --quiet || true + gcloud scheduler jobs delete erpnext-daily-backup --location="$REGION" --quiet || true + + # Delete Cloud Tasks queue + gcloud tasks queues delete erpnext-tasks --location="$REGION" --quiet || true + + # Delete load balancer components + gcloud compute forwarding-rules delete erpnext-https-rule --global --quiet || true + gcloud compute target-https-proxies delete erpnext-https-proxy --quiet || true + gcloud compute ssl-certificates delete erpnext-ssl-cert --quiet || true + gcloud compute url-maps delete erpnext-url-map --quiet || true + gcloud compute backend-services delete erpnext-frontend-backend --global --quiet || true + gcloud compute network-endpoint-groups delete erpnext-frontend-neg --region="$REGION" --quiet || true + gcloud compute health-checks delete erpnext-health-check --quiet || true + + print_success "Cloud Run deployment cleaned up" + print_warning "Managed services (Cloud SQL, Redis) and container images are preserved" + else + print_status "Cleanup cancelled" + fi +} + +# Function to show help +show_help() { + echo "ERPNext Cloud Run Deployment Script with Managed Services" + echo "" + echo "Usage: $0 [COMMAND]" + echo "" + echo "Commands:" + echo " deploy - Full Cloud Run deployment (default)" + echo " status - Show deployment status" + echo " cleanup - Delete deployment (preserves managed services)" + echo " help - Show this help" + echo "" + echo "Environment Variables:" + echo " PROJECT_ID - GCP Project ID" + echo " REGION - GCP region (default: us-central1)" + echo " DOMAIN - Domain name (default: erpnext.yourdomain.com)" + echo " API_DOMAIN - API domain (default: api.yourdomain.com)" + echo " DB_INSTANCE_NAME - Cloud SQL instance (default: erpnext-db)" + echo " REDIS_INSTANCE_NAME - Memorystore instance (default: erpnext-redis)" + echo " VPC_NAME - VPC network (default: erpnext-vpc)" + echo " VPC_CONNECTOR - VPC connector (default: erpnext-connector)" + echo "" + echo "Prerequisites:" + echo " - Complete setup in 00-prerequisites-managed.md" + echo " - Cloud SQL and Memorystore instances must exist" + echo " - VPC network with VPC Access Connector" + echo "" + echo "Example:" + echo " PROJECT_ID=my-project DOMAIN=erp.mycompany.com $0 deploy" +} + +# Main deployment function +main_deploy() { + print_status "Starting ERPNext Cloud Run deployment with managed services..." + + check_prerequisites + check_managed_services + get_managed_services_info + setup_cloud_storage + setup_secrets + build_and_push_images + deploy_cloud_run_services + initialize_site + setup_background_processing + setup_custom_domain + + print_success "Cloud Run deployment completed successfully!" + echo "" + print_status "Service URLs:" + print_status " Frontend: $FRONTEND_URL" + print_status " Backend API: $BACKEND_URL" + print_status " Custom Domain: https://$DOMAIN (after DNS configuration)" + echo "" + print_status "Managed Services:" + print_status " Cloud SQL: $DB_CONNECTION_NAME" + print_status " Redis: $REDIS_HOST:6379" + print_status " Storage: gs://erpnext-files-$PROJECT_ID" + echo "" + print_warning "Configure DNS to point $DOMAIN to $GLOBAL_IP" + print_warning "Retrieve admin password: gcloud secrets versions access latest --secret=erpnext-admin-password" +} + +# Main script logic +case "${1:-deploy}" in + "deploy") + main_deploy + ;; + "status") + show_status + ;; + "cleanup") + cleanup + ;; + "help"|"-h"|"--help") + show_help + ;; + *) + print_error "Unknown command: $1" + show_help + exit 1 + ;; +esac \ No newline at end of file diff --git a/documentation/deployment-guides/gcp-managed/scripts/deploy-managed.sh b/documentation/deployment-guides/gcp-managed/scripts/deploy-managed.sh new file mode 100755 index 0000000..17de1c5 --- /dev/null +++ b/documentation/deployment-guides/gcp-managed/scripts/deploy-managed.sh @@ -0,0 +1,683 @@ +#!/bin/bash + +# ERPNext GKE Deployment Script with Managed Services +# This script automates the deployment of ERPNext on GKE using Cloud SQL and Memorystore + +set -e + +# Color codes for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Configuration +CLUSTER_NAME=${CLUSTER_NAME:-"erpnext-managed-cluster"} +ZONE=${ZONE:-"us-central1-a"} +REGION=${REGION:-"us-central1"} +PROJECT_ID=${PROJECT_ID:-""} +DOMAIN=${DOMAIN:-"erpnext.yourdomain.com"} +EMAIL=${EMAIL:-"admin@yourdomain.com"} +NAMESPACE=${NAMESPACE:-"erpnext"} + +# Managed services configuration +DB_INSTANCE_NAME=${DB_INSTANCE_NAME:-"erpnext-db"} +REDIS_INSTANCE_NAME=${REDIS_INSTANCE_NAME:-"erpnext-redis"} +VPC_NAME=${VPC_NAME:-"erpnext-vpc"} +VPC_CONNECTOR=${VPC_CONNECTOR:-"erpnext-connector"} + +# Function to print colored output +print_status() { + echo -e "${BLUE}[INFO]${NC} $1" +} + +print_success() { + echo -e "${GREEN}[SUCCESS]${NC} $1" +} + +print_warning() { + echo -e "${YELLOW}[WARNING]${NC} $1" +} + +print_error() { + echo -e "${RED}[ERROR]${NC} $1" +} + +# Function to check prerequisites +check_prerequisites() { + print_status "Checking prerequisites for managed services deployment..." + + # Check if required tools are installed + local required_tools=("gcloud" "kubectl" "helm") + for tool in "${required_tools[@]}"; do + if ! command -v "$tool" &> /dev/null; then + print_error "$tool is not installed. Please install it first." + exit 1 + fi + done + + # Check if user is authenticated + if ! gcloud auth list --filter=status:ACTIVE --format="value(account)" | head -n 1 &> /dev/null; then + print_error "Not authenticated with gcloud. Please run 'gcloud auth login'" + exit 1 + fi + + # Check if project ID is set + if [[ -z "$PROJECT_ID" ]]; then + PROJECT_ID=$(gcloud config get-value project) + if [[ -z "$PROJECT_ID" ]]; then + print_error "PROJECT_ID not set. Please set it or configure gcloud project." + exit 1 + fi + fi + + print_success "Prerequisites check passed" +} + +# Function to check managed services +check_managed_services() { + print_status "Checking managed services status..." + + # Check Cloud SQL instance + if ! gcloud sql instances describe "$DB_INSTANCE_NAME" &> /dev/null; then + print_error "Cloud SQL instance '$DB_INSTANCE_NAME' not found. Please create it first." + exit 1 + fi + + # Check Memorystore Redis instance + if ! gcloud redis instances describe "$REDIS_INSTANCE_NAME" --region="$REGION" &> /dev/null; then + print_error "Memorystore Redis instance '$REDIS_INSTANCE_NAME' not found. Please create it first." + exit 1 + fi + + # Check VPC network + if ! gcloud compute networks describe "$VPC_NAME" &> /dev/null; then + print_error "VPC network '$VPC_NAME' not found. Please create it first." + exit 1 + fi + + print_success "All managed services are available" +} + +# Function to get managed services information +get_managed_services_info() { + print_status "Gathering managed services information..." + + # Get Cloud SQL connection name + DB_CONNECTION_NAME=$(gcloud sql instances describe "$DB_INSTANCE_NAME" --format="value(connectionName)") + print_status "Cloud SQL connection name: $DB_CONNECTION_NAME" + + # Get Redis host IP + REDIS_HOST=$(gcloud redis instances describe "$REDIS_INSTANCE_NAME" --region="$REGION" --format="value(host)") + print_status "Redis host IP: $REDIS_HOST" + + # Get VPC connector + if ! gcloud compute networks vpc-access connectors describe "$VPC_CONNECTOR" --region="$REGION" &> /dev/null; then + print_warning "VPC connector '$VPC_CONNECTOR' not found. This is needed for Cloud Run deployment." + fi + + print_success "Managed services information gathered" +} + +# Function to create GKE cluster for managed services +create_cluster() { + print_status "Creating GKE cluster: $CLUSTER_NAME" + + # Check if cluster already exists + if gcloud container clusters describe "$CLUSTER_NAME" --zone="$ZONE" &> /dev/null; then + print_warning "Cluster $CLUSTER_NAME already exists" + return 0 + fi + + gcloud container clusters create "$CLUSTER_NAME" \ + --zone="$ZONE" \ + --num-nodes=3 \ + --node-locations="$ZONE" \ + --machine-type=e2-standard-4 \ + --disk-type=pd-ssd \ + --disk-size=50GB \ + --enable-autoscaling \ + --min-nodes=2 \ + --max-nodes=15 \ + --enable-autorepair \ + --enable-autoupgrade \ + --enable-network-policy \ + --enable-ip-alias \ + --network="$VPC_NAME" \ + --subnetwork=erpnext-subnet \ + --enable-private-nodes \ + --master-ipv4-cidr-block=172.16.0.0/28 \ + --enable-cloud-logging \ + --enable-cloud-monitoring \ + --workload-pool="$PROJECT_ID.svc.id.goog" \ + --enable-shielded-nodes \ + --enable-image-streaming \ + --logging=SYSTEM,WORKLOAD,API_SERVER \ + --monitoring=SYSTEM,WORKLOAD,STORAGE,POD,DEPLOYMENT,STATEFULSET,DAEMONSET,HPA,CADVISOR,KUBELET + + print_success "Cluster created successfully" +} + +# Function to configure kubectl +configure_kubectl() { + print_status "Configuring kubectl..." + + gcloud container clusters get-credentials "$CLUSTER_NAME" --zone="$ZONE" + + # Verify connection + if kubectl cluster-info &> /dev/null; then + print_success "kubectl configured successfully" + else + print_error "Failed to configure kubectl" + exit 1 + fi +} + +# Function to install required operators and controllers +install_operators() { + print_status "Installing required operators and controllers..." + + # Install External Secrets Operator + helm repo add external-secrets https://charts.external-secrets.io + helm repo update + helm install external-secrets external-secrets/external-secrets \ + -n external-secrets-system \ + --create-namespace \ + --wait + + # Install nginx ingress controller + kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.2/deploy/static/provider/cloud/deploy.yaml + + # Wait for ingress controller to be ready + kubectl wait --namespace ingress-nginx \ + --for=condition=ready pod \ + --selector=app.kubernetes.io/component=controller \ + --timeout=300s + + # Install cert-manager + kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml + + # Wait for cert-manager to be ready + kubectl wait --for=condition=available --timeout=300s deployment/cert-manager -n cert-manager + kubectl wait --for=condition=available --timeout=300s deployment/cert-manager-webhook -n cert-manager + + print_success "Operators and controllers installed" +} + +# Function to create namespace and basic resources +create_namespace() { + print_status "Creating namespace and basic resources..." + + # Update namespace.yaml with correct labels + cp ../kubernetes-manifests/namespace.yaml /tmp/namespace-updated.yaml + + kubectl apply -f /tmp/namespace-updated.yaml + rm /tmp/namespace-updated.yaml + + print_success "Namespace created" +} + +# Function to create service account and workload identity +setup_workload_identity() { + print_status "Setting up Workload Identity..." + + # Create Kubernetes service account + kubectl create serviceaccount erpnext-ksa --namespace="$NAMESPACE" --dry-run=client -o yaml | kubectl apply -f - + + # Create Google Cloud service account if it doesn't exist + if ! gcloud iam service-accounts describe "erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com" &> /dev/null; then + gcloud iam service-accounts create erpnext-managed \ + --display-name="ERPNext Managed Services Account" + fi + + # Grant necessary permissions + gcloud projects add-iam-policy-binding "$PROJECT_ID" \ + --member="serviceAccount:erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com" \ + --role="roles/cloudsql.client" + + gcloud projects add-iam-policy-binding "$PROJECT_ID" \ + --member="serviceAccount:erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com" \ + --role="roles/redis.editor" + + gcloud projects add-iam-policy-binding "$PROJECT_ID" \ + --member="serviceAccount:erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com" \ + --role="roles/secretmanager.secretAccessor" + + gcloud projects add-iam-policy-binding "$PROJECT_ID" \ + --member="serviceAccount:erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com" \ + --role="roles/storage.admin" + + # Bind service accounts + gcloud iam service-accounts add-iam-policy-binding \ + --role roles/iam.workloadIdentityUser \ + --member "serviceAccount:$PROJECT_ID.svc.id.goog[$NAMESPACE/erpnext-ksa]" \ + "erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com" + + # Annotate Kubernetes service account + kubectl annotate serviceaccount erpnext-ksa \ + --namespace="$NAMESPACE" \ + "iam.gke.io/gcp-service-account=erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com" \ + --overwrite + + print_success "Workload Identity configured" +} + +# Function to create secrets using External Secrets Operator +create_secrets() { + print_status "Creating secrets with External Secrets Operator..." + + # Generate random passwords if secrets don't exist in Secret Manager + if ! gcloud secrets describe erpnext-admin-password &> /dev/null; then + local admin_password=${ADMIN_PASSWORD:-$(openssl rand -base64 32)} + gcloud secrets create erpnext-admin-password --data-file=<(echo -n "$admin_password") + print_warning "Admin password: $admin_password" + fi + + if ! gcloud secrets describe erpnext-db-password &> /dev/null; then + local db_password=${DB_PASSWORD:-$(openssl rand -base64 32)} + gcloud secrets create erpnext-db-password --data-file=<(echo -n "$db_password") + print_warning "Database password: $db_password" + fi + + if ! gcloud secrets describe erpnext-api-key &> /dev/null; then + local api_key=${API_KEY:-$(openssl rand -hex 32)} + gcloud secrets create erpnext-api-key --data-file=<(echo -n "$api_key") + fi + + if ! gcloud secrets describe erpnext-api-secret &> /dev/null; then + local api_secret=${API_SECRET:-$(openssl rand -hex 32)} + gcloud secrets create erpnext-api-secret --data-file=<(echo -n "$api_secret") + fi + + # Create connection name secret + gcloud secrets create erpnext-db-connection-name --data-file=<(echo -n "$DB_CONNECTION_NAME") --quiet || \ + gcloud secrets versions add erpnext-db-connection-name --data-file=<(echo -n "$DB_CONNECTION_NAME") + + # Get Redis AUTH string if enabled + local redis_auth="" + if gcloud redis instances describe "$REDIS_INSTANCE_NAME" --region="$REGION" --format="value(authEnabled)" | grep -q "True"; then + redis_auth=$(gcloud redis instances describe "$REDIS_INSTANCE_NAME" --region="$REGION" --format="value(authString)") + gcloud secrets create redis-auth-string --data-file=<(echo -n "$redis_auth") --quiet || \ + gcloud secrets versions add redis-auth-string --data-file=<(echo -n "$redis_auth") + fi + + # Apply External Secrets configuration + cp ../kubernetes-manifests/secrets.yaml /tmp/secrets-updated.yaml + sed -i "s/PROJECT_ID/$PROJECT_ID/g" /tmp/secrets-updated.yaml + sed -i "s/REGION/$REGION/g" /tmp/secrets-updated.yaml + sed -i "s/erpnext-managed-cluster/$CLUSTER_NAME/g" /tmp/secrets-updated.yaml + + kubectl apply -f /tmp/secrets-updated.yaml + rm /tmp/secrets-updated.yaml + + # Wait for External Secrets to sync + print_status "Waiting for secrets to be synced..." + sleep 30 + + # Create service account key for Cloud SQL Proxy + if ! kubectl get secret gcp-service-account-key -n "$NAMESPACE" &> /dev/null; then + local key_file="/tmp/erpnext-managed-key.json" + gcloud iam service-accounts keys create "$key_file" \ + --iam-account="erpnext-managed@$PROJECT_ID.iam.gserviceaccount.com" + + kubectl create secret generic gcp-service-account-key \ + --namespace="$NAMESPACE" \ + --from-file=key.json="$key_file" + + rm "$key_file" + fi + + print_success "Secrets created" + print_warning "Please save the generated credentials securely!" +} + +# Function to update ConfigMap with managed services configuration +update_configmap() { + print_status "Updating ConfigMap with managed services configuration..." + + # Copy and update configmap + cp ../kubernetes-manifests/configmap.yaml /tmp/configmap-updated.yaml + sed -i "s/erpnext.yourdomain.com/$DOMAIN/g" /tmp/configmap-updated.yaml + sed -i "s/PROJECT_ID/$PROJECT_ID/g" /tmp/configmap-updated.yaml + sed -i "s/REGION/$REGION/g" /tmp/configmap-updated.yaml + sed -i "s/REDIS_HOST/$REDIS_HOST/g" /tmp/configmap-updated.yaml + + kubectl apply -f /tmp/configmap-updated.yaml + rm /tmp/configmap-updated.yaml + + print_success "ConfigMap updated" +} + +# Function to deploy storage +deploy_storage() { + print_status "Deploying storage resources..." + + kubectl apply -f ../kubernetes-manifests/storage.yaml + + print_success "Storage resources deployed" +} + +# Function to deploy ERPNext application with managed services +deploy_application() { + print_status "Deploying ERPNext application with managed services..." + + # Update manifests with managed services configuration + local manifests=("erpnext-backend.yaml" "erpnext-frontend.yaml" "erpnext-workers.yaml") + + for manifest in "${manifests[@]}"; do + cp "../kubernetes-manifests/$manifest" "/tmp/${manifest%.yaml}-updated.yaml" + sed -i "s/PROJECT_ID/$PROJECT_ID/g" "/tmp/${manifest%.yaml}-updated.yaml" + sed -i "s/REDIS_HOST/$REDIS_HOST/g" "/tmp/${manifest%.yaml}-updated.yaml" + + kubectl apply -f "/tmp/${manifest%.yaml}-updated.yaml" + rm "/tmp/${manifest%.yaml}-updated.yaml" + done + + # Wait for backend to be ready + print_status "Waiting for ERPNext backend to be ready..." + kubectl wait --for=condition=available deployment/erpnext-backend -n "$NAMESPACE" --timeout=600s + + print_success "ERPNext application deployed" +} + +# Function to create ERPNext site +create_site() { + print_status "Creating ERPNext site with managed services..." + + # Update jobs manifest + cp ../kubernetes-manifests/jobs.yaml /tmp/jobs-updated.yaml + sed -i "s/PROJECT_ID_PLACEHOLDER/$PROJECT_ID/g" /tmp/jobs-updated.yaml + sed -i "s/REDIS_HOST_PLACEHOLDER/$REDIS_HOST/g" /tmp/jobs-updated.yaml + + # Apply site creation job + kubectl apply -f /tmp/jobs-updated.yaml + rm /tmp/jobs-updated.yaml + + # Wait for job to complete + print_status "Waiting for site creation to complete..." + kubectl wait --for=condition=complete job/erpnext-create-site -n "$NAMESPACE" --timeout=1200s + + if kubectl get job erpnext-create-site -n "$NAMESPACE" -o jsonpath='{.status.conditions[?(@.type=="Complete")].status}' | grep -q "True"; then + print_success "ERPNext site created successfully with managed services" + else + print_error "Site creation failed. Check job logs:" + kubectl logs job/erpnext-create-site -n "$NAMESPACE" + exit 1 + fi +} + +# Function to deploy ingress with managed services optimization +deploy_ingress() { + print_status "Deploying ingress with managed services optimization..." + + # Update ingress with correct domains and managed services configuration + cp ../kubernetes-manifests/ingress.yaml /tmp/ingress-updated.yaml + sed -i "s/erpnext.yourdomain.com/$DOMAIN/g" /tmp/ingress-updated.yaml + sed -i "s/api.yourdomain.com/api.$DOMAIN/g" /tmp/ingress-updated.yaml + sed -i "s/admin@yourdomain.com/$EMAIL/g" /tmp/ingress-updated.yaml + sed -i "s/PROJECT_ID/$PROJECT_ID/g" /tmp/ingress-updated.yaml + + kubectl apply -f /tmp/ingress-updated.yaml + rm /tmp/ingress-updated.yaml + + print_success "Ingress deployed with managed services optimization" +} + +# Function to setup monitoring for managed services +setup_monitoring() { + print_status "Setting up monitoring for managed services..." + + # Install Prometheus stack with managed services configuration + helm repo add prometheus-community https://prometheus-community.github.io/helm-charts + helm repo update + + # Create values file for managed services monitoring + cat > /tmp/prometheus-values.yaml </dev/null || echo "No certificates found" + + echo "" + echo "=== External IP ===" + kubectl get service -n ingress-nginx ingress-nginx-controller + + echo "" + echo "=== Secrets Status ===" + kubectl get externalsecret -n "$NAMESPACE" 2>/dev/null || echo "External Secrets not found" +} + +# Function to cleanup deployment +cleanup() { + print_warning "This will delete the entire ERPNext deployment but preserve managed services. Are you sure? (y/N)" + read -r response + if [[ "$response" =~ ^([yY][eE][sS]|[yY])$ ]]; then + print_status "Cleaning up deployment..." + + kubectl delete namespace "$NAMESPACE" --ignore-not-found=true + + print_status "Deleting cluster..." + gcloud container clusters delete "$CLUSTER_NAME" --zone="$ZONE" --quiet + + print_warning "Managed services (Cloud SQL, Redis) are preserved." + print_warning "To delete them manually:" + print_warning " gcloud sql instances delete $DB_INSTANCE_NAME" + print_warning " gcloud redis instances delete $REDIS_INSTANCE_NAME --region=$REGION" + + print_success "Cleanup completed" + else + print_status "Cleanup cancelled" + fi +} + +# Function to show help +show_help() { + echo "ERPNext GKE Deployment Script with Managed Services" + echo "" + echo "Usage: $0 [COMMAND]" + echo "" + echo "Commands:" + echo " deploy - Full deployment with managed services (default)" + echo " status - Show deployment status" + echo " cleanup - Delete deployment (preserves managed services)" + echo " help - Show this help" + echo "" + echo "Environment Variables:" + echo " PROJECT_ID - GCP Project ID" + echo " CLUSTER_NAME - GKE cluster name (default: erpnext-managed-cluster)" + echo " ZONE - GCP zone (default: us-central1-a)" + echo " REGION - GCP region (default: us-central1)" + echo " DOMAIN - Domain name (default: erpnext.yourdomain.com)" + echo " EMAIL - Email for Let's Encrypt (default: admin@yourdomain.com)" + echo " NAMESPACE - Kubernetes namespace (default: erpnext)" + echo " DB_INSTANCE_NAME - Cloud SQL instance name (default: erpnext-db)" + echo " REDIS_INSTANCE_NAME - Memorystore instance name (default: erpnext-redis)" + echo " VPC_NAME - VPC network name (default: erpnext-vpc)" + echo "" + echo "Prerequisites:" + echo " - Complete setup in 00-prerequisites-managed.md" + echo " - Cloud SQL and Memorystore instances must exist" + echo " - VPC network with proper configuration" + echo "" + echo "Example:" + echo " PROJECT_ID=my-project DOMAIN=erp.mycompany.com $0 deploy" +} + +# Main deployment function +main_deploy() { + print_status "Starting ERPNext GKE deployment with managed services..." + + check_prerequisites + check_managed_services + get_managed_services_info + create_cluster + configure_kubectl + install_operators + create_namespace + setup_workload_identity + create_secrets + update_configmap + deploy_storage + deploy_application + create_site + deploy_ingress + setup_monitoring + setup_backup + + print_success "Deployment completed successfully with managed services!" + echo "" + print_status "Access your ERPNext instance at: https://$DOMAIN" + print_status "API endpoint: https://api.$DOMAIN" + echo "" + print_status "Managed Services:" + print_status " Cloud SQL: $DB_CONNECTION_NAME" + print_status " Redis: $REDIS_HOST:6379" + echo "" + print_warning "It may take a few minutes for the SSL certificate to be issued." + print_warning "Monitor certificate status with: kubectl get certificate -n $NAMESPACE" + echo "" + print_status "Default credentials are stored in Secret Manager." + print_status "Retrieve admin password with: gcloud secrets versions access latest --secret=erpnext-admin-password" +} + +# Main script logic +case "${1:-deploy}" in + "deploy") + main_deploy + ;; + "status") + get_status + ;; + "cleanup") + cleanup + ;; + "help"|"-h"|"--help") + show_help + ;; + *) + print_error "Unknown command: $1" + show_help + exit 1 + ;; +esac \ No newline at end of file