Jenkins CI/CD 파이프라인 구축 완료

- Kustomize 기반 환경별 배포 구조 구축 • Base 매니페스트: deployment/cicd/kustomize/base/ • 환경별 오버레이: overlays/{dev,staging,prod} • 기존 k8s 매니페스트를 Kustomize 구조로 마이그레이션 - Jenkins 파이프라인 설정 • Jenkinsfile: Pod Template, SonarQube, 배포 자동화 • 환경별 설정 파일: config/deploy_env_vars_{env} • 수동 배포 스크립트: scripts/deploy.sh - Azure 연동 설정 • ACR (acrdigitalgarage01) 및 AKS (aks-digitalgarage-01) • 환경별 리소스 분리 및 보안 설정 - 완전한 구축 가이드 문서 • deployment/cicd/jenkins-pipeline-guide.md • Jenkins 플러그인, RBAC, 트러블슈팅 가이드 포함 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-06 08:06:24 +00:00 · 2025-09-12 10:46:05 +09:00 · 2025-09-12 10:46:05 +09:00 · 892f30ba44
commit 892f30ba44
parent 0f1e22c5dc
67 changed files with 2430 additions and 10 deletions
--- a/.claude/commands/deploy-jenkins-cicd-guide-back.md
+++ b/.claude/commands/deploy-jenkins-cicd-guide-back.md
@ -0,0 +1,15 @@
+---
+command: "/deploy-jenkins-cicd-guide-back"
+category: "배포"
+purpose: "백엔드 Jenkins CI/CD 가이드 작성"
+---
+
+@cicd 
+'백엔드Jenkins파이프라인작성가이드'에 따라 Jenkins를 이용한 CI/CD 가이드를 작성해 주세요. 
+프롬프트에 '[실행정보]'항목이 없으면 수행을 중단하고 안내 메시지를 표시해 주세요. 
+{안내메시지}
+'[실행정보]'섹션 하위에 아래 예와 같이 필요한 정보를 제시해 주세요.   
+[실행정보]
+- ACR명: acrdigitalgarage01
+- RESOURCE_GROUP: rg-digitalgarage-01
+- AKS_CLUSTER: aks-digitalgarage-01 
--- a/.claude/settings.local.json
+++ b/.claude/settings.local.json
@ -0,0 +1,12 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(mkdir:*)",
+      "Bash(chmod:*)",
+      "Bash(sed:*)",
+      "Bash(find:*)"
+    ],
+    "deny": [],
+    "ask": []
+  }
+}
--- a/CLAUDE.md
+++ b/CLAUDE.md
@ -97,9 +97,9 @@ QA Engineer

 [URL링크 참조]
 - URL링크는 WebFetch가 아닌 'curl {URL} > claude/{filename}'명령으로 저장
- 'claude'디렉토리가 없으면 생성하고 다운로드   
+- 동일한 파일이 있으면 덮어 씀
+- 'claude'디렉토리가 없으면 생성하고 다운로드
 - 저장된 파일을 읽어 사용함
- 작업을 완료한 후 다운로드한 파일은 삭제함 

 ---

@ -306,7 +306,11 @@ QA Engineer
  - 설명: 프론트엔드 서비스를 쿠버네티스 클러스터에 배포하는 가이드  
  - URL: https://raw.githubusercontent.com/cna-bootcamp/clauding-guide/refs/heads/main/guides/deploy/deploy-k8s-front.md
  - 파일명: deploy-k8s-front.md  
-
+- 백엔드Jenkins파이프라인작성가이드
+  - 설명: 백엔드 서비스를 Jenkins를 이용하여 CI/CD하는 배포 가이드  
+  - URL: https://raw.githubusercontent.com/cna-bootcamp/clauding-guide/refs/heads/main/guides/deploy/deploy-jenkins-cicd-back.md
+  - 파일명: deploy-jenkins-cicd-back.md  
+  
 ## 참조 문서
 - 프로젝트지침템플릿
  - 설명: 프로젝트 지침인 CLAUDE.md 파일 템플릿 
--- a/deployment/cicd/Jenkinsfile
+++ b/deployment/cicd/Jenkinsfile
@ -0,0 +1,426 @@
+#!/usr/bin/env groovy
+
+/**
+ * Jenkins Pipeline for phonebill Microservices
+ * Supports multi-environment deployment (dev, staging, prod)
+ * Services: api-gateway, user-service, bill-service, product-service, kos-mock
+ */
+
+pipeline {
+    agent {
+        kubernetes {
+            yaml '''
+apiVersion: v1
+kind: Pod
+metadata:
+  labels:
+    jenkins: agent
+spec:
+  serviceAccountName: jenkins-agent
+  containers:
+  - name: gradle
+    image: gradle:8.5-jdk17
+    command:
+    - cat
+    tty: true
+    resources:
+      requests:
+        memory: "1Gi"
+        cpu: "500m"
+      limits:
+        memory: "2Gi"
+        cpu: "1"
+    volumeMounts:
+    - name: gradle-cache
+      mountPath: /home/gradle/.gradle
+  - name: podman
+    image: quay.io/podman/stable:latest
+    command:
+    - cat
+    tty: true
+    securityContext:
+      privileged: true
+    resources:
+      requests:
+        memory: "1Gi"
+        cpu: "500m"
+      limits:
+        memory: "2Gi"
+        cpu: "1"
+  - name: azure-cli
+    image: mcr.microsoft.com/azure-cli:latest
+    command:
+    - cat
+    tty: true
+    resources:
+      requests:
+        memory: "512Mi"
+        cpu: "250m"
+      limits:
+        memory: "1Gi"
+        cpu: "500m"
+  - name: kubectl
+    image: bitnami/kubectl:latest
+    command:
+    - cat
+    tty: true
+    resources:
+      requests:
+        memory: "256Mi"
+        cpu: "100m"
+      limits:
+        memory: "512Mi"
+        cpu: "250m"
+  volumes:
+  - name: gradle-cache
+    persistentVolumeClaim:
+      claimName: jenkins-gradle-cache
+            '''
+        }
+    }
+    
+    parameters {
+        choice(
+            name: 'ENVIRONMENT',
+            choices: ['dev', 'staging', 'prod'],
+            description: 'Target deployment environment'
+        )
+        choice(
+            name: 'SERVICES_TO_BUILD',
+            choices: ['all', 'api-gateway', 'user-service', 'bill-service', 'product-service', 'kos-mock'],
+            description: 'Services to build and deploy'
+        )
+        booleanParam(
+            name: 'SKIP_TESTS',
+            defaultValue: false,
+            description: 'Skip unit tests during build'
+        )
+        booleanParam(
+            name: 'SKIP_SONAR',
+            defaultValue: false,
+            description: 'Skip SonarQube analysis'
+        )
+        booleanParam(
+            name: 'FORCE_DEPLOY',
+            defaultValue: false,
+            description: 'Force deployment even if no changes detected'
+        )
+    }
+    
+    environment {
+        // Load environment-specific variables
+        CONFIG_FILE = "deployment/cicd/config/deploy_env_vars_${params.ENVIRONMENT}"
+        
+        // Build configuration
+        GRADLE_USER_HOME = '/home/gradle/.gradle'
+        GRADLE_OPTS = '-Xmx2048m -XX:MaxPermSize=512m'
+        
+        // Azure credentials
+        AZURE_CREDENTIALS = credentials('azure-service-principal')
+        ACR_CREDENTIALS = credentials('acr-credentials')
+        
+        // SonarQube
+        SONAR_TOKEN = credentials('sonarqube-token')
+        
+        // Slack notification
+        SLACK_TOKEN = credentials('slack-token')
+    }
+    
+    stages {
+        stage('Initialize') {
+            steps {
+                script {
+                    echo "🚀 Starting phonebill pipeline for ${params.ENVIRONMENT} environment"
+                    
+                    // Load environment-specific configuration
+                    if (fileExists(env.CONFIG_FILE)) {
+                        def props = readProperties file: env.CONFIG_FILE
+                        props.each { key, value ->
+                            env[key] = value
+                        }
+                        echo "✅ Loaded configuration from ${env.CONFIG_FILE}"
+                    } else {
+                        error "❌ Configuration file not found: ${env.CONFIG_FILE}"
+                    }
+                    
+                    // Set services to build
+                    if (params.SERVICES_TO_BUILD == 'all') {
+                        env.SERVICES_LIST = env.SERVICES
+                    } else {
+                        env.SERVICES_LIST = params.SERVICES_TO_BUILD
+                    }
+                    
+                    echo "🎯 Services to build: ${env.SERVICES_LIST}"
+                    echo "📦 Target namespace: ${env.AKS_NAMESPACE}"
+                }
+            }
+        }
+        
+        stage('Checkout & Prepare') {
+            steps {
+                checkout scm
+                script {
+                    env.BUILD_TIMESTAMP = sh(
+                        script: 'date +%Y%m%d-%H%M%S',
+                        returnStdout: true
+                    ).trim()
+                    
+                    env.IMAGE_TAG = "${env.BUILD_NUMBER}-${params.ENVIRONMENT}-${env.BUILD_TIMESTAMP}"
+                    echo "🏷️ Image tag: ${env.IMAGE_TAG}"
+                }
+            }
+        }
+        
+        stage('Build & Test') {
+            parallel {
+                stage('Gradle Build') {
+                    steps {
+                        container('gradle') {
+                            script {
+                                def services = env.SERVICES_LIST.split(',')
+                                for (service in services) {
+                                    echo "🔨 Building ${service}..."
+                                    
+                                    if (!params.SKIP_TESTS) {
+                                        sh """
+                                            ./gradlew ${service}:clean ${service}:test ${service}:build \
+                                                --no-daemon \
+                                                --parallel \
+                                                --build-cache
+                                        """
+                                    } else {
+                                        sh """
+                                            ./gradlew ${service}:clean ${service}:build -x test \
+                                                --no-daemon \
+                                                --parallel \
+                                                --build-cache
+                                        """
+                                    }
+                                }
+                            }
+                        }
+                    }
+                    post {
+                        always {
+                            // Publish test results
+                            script {
+                                def services = env.SERVICES_LIST.split(',')
+                                for (service in services) {
+                                    if (fileExists("${service}/build/test-results/test/*.xml")) {
+                                        publishTestResults(
+                                            testResultsPattern: "${service}/build/test-results/test/*.xml",
+                                            allowEmptyResults: true
+                                        )
+                                    }
+                                }
+                            }
+                        }
+                    }
+                }
+            }
+        }
+        
+        stage('SonarQube Analysis') {
+            when {
+                not { params.SKIP_SONAR }
+            }
+            steps {
+                container('gradle') {
+                    withSonarQubeEnv('SonarQube') {
+                        sh '''
+                            ./gradlew sonarqube \
+                                -Dsonar.projectKey=${SONAR_PROJECT_KEY} \
+                                -Dsonar.sources=${SONAR_SOURCES} \
+                                -Dsonar.exclusions="${SONAR_EXCLUSIONS}" \
+                                --no-daemon
+                        '''
+                    }
+                }
+            }
+        }
+        
+        stage('Quality Gate') {
+            when {
+                not { params.SKIP_SONAR }
+            }
+            steps {
+                timeout(time: 5, unit: 'MINUTES') {
+                    waitForQualityGate abortPipeline: true
+                }
+            }
+        }
+        
+        stage('Container Build & Push') {
+            parallel {
+                stage('Build Images') {
+                    steps {
+                        container('podman') {
+                            script {
+                                // Login to ACR
+                                sh """
+                                    echo "${ACR_CREDENTIALS_PSW}" | podman login \
+                                        --username "${ACR_CREDENTIALS_USR}" \
+                                        --password-stdin \
+                                        ${REGISTRY_URL}
+                                """
+                                
+                                def services = env.SERVICES_LIST.split(',')
+                                for (service in services) {
+                                    echo "🐳 Building container image for ${service}..."
+                                    
+                                    sh """
+                                        cd ${service}
+                                        podman build \
+                                            --tag ${REGISTRY_URL}/phonebill/${service}:${IMAGE_TAG} \
+                                            --tag ${REGISTRY_URL}/phonebill/${service}:latest-${params.ENVIRONMENT} \
+                                            --file Dockerfile \
+                                            .
+                                        
+                                        podman push ${REGISTRY_URL}/phonebill/${service}:${IMAGE_TAG}
+                                        podman push ${REGISTRY_URL}/phonebill/${service}:latest-${params.ENVIRONMENT}
+                                    """
+                                }
+                            }
+                        }
+                    }
+                }
+            }
+        }
+        
+        stage('Deploy to Kubernetes') {
+            steps {
+                container('kubectl') {
+                    script {
+                        // Login to Azure and get AKS credentials
+                        sh """
+                            az login --service-principal \
+                                --username "${AZURE_CREDENTIALS_USR}" \
+                                --password "${AZURE_CREDENTIALS_PSW}" \
+                                --tenant "${AZURE_CREDENTIALS_TENANT_ID}"
+                            
+                            az aks get-credentials \
+                                --resource-group ${AZURE_RESOURCE_GROUP} \
+                                --name ${AKS_CLUSTER_NAME} \
+                                --overwrite-existing
+                        """
+                        
+                        // Deploy services using kustomize
+                        def services = env.SERVICES_LIST.split(',')
+                        for (service in services) {
+                            echo "🚀 Deploying ${service} to ${params.ENVIRONMENT}..."
+                            
+                            sh """
+                                cd ${KUSTOMIZE_BASE}/${service}
+                                
+                                # Update image tag in kustomization.yaml
+                                sed -i 's|newTag:.*|newTag: ${IMAGE_TAG}|' ${KUSTOMIZE_OVERLAY}/kustomization.yaml
+                                
+                                # Apply deployment
+                                kubectl apply -k ${KUSTOMIZE_OVERLAY} -n ${AKS_NAMESPACE}
+                                
+                                # Wait for rollout
+                                kubectl rollout status deployment/${service} -n ${AKS_NAMESPACE} --timeout=300s
+                            """
+                        }
+                    }
+                }
+            }
+        }
+        
+        stage('Health Check') {
+            steps {
+                container('kubectl') {
+                    script {
+                        def services = env.SERVICES_LIST.split(',')
+                        for (service in services) {
+                            echo "🏥 Health checking ${service}..."
+                            
+                            retry(count: env.HEALTH_CHECK_RETRY as Integer) {
+                                sh """
+                                    kubectl get deployment ${service} -n ${AKS_NAMESPACE} -o json | \
+                                        jq -e '.status.readyReplicas == .status.replicas and .status.replicas > 0'
+                                """
+                                
+                                sleep time: 30, unit: 'SECONDS'
+                            }
+                            
+                            echo "✅ ${service} is healthy"
+                        }
+                    }
+                }
+            }
+        }
+    }
+    
+    post {
+        always {
+            script {
+                // Archive build artifacts
+                archiveArtifacts(
+                    artifacts: '**/build/libs/*.jar',
+                    allowEmptyArchive: true,
+                    fingerprint: true
+                )
+                
+                // Clean workspace
+                cleanWs()
+            }
+        }
+        
+        success {
+            script {
+                def message = """
+                    ✅ *phonebill Deployment Successful*
+                    • Environment: `${params.ENVIRONMENT}`
+                    • Services: `${env.SERVICES_LIST}`
+                    • Image Tag: `${env.IMAGE_TAG}`
+                    • Build: `${env.BUILD_NUMBER}`
+                    • Duration: `${currentBuild.durationString}`
+                """.stripIndent()
+                
+                // Send Slack notification
+                slackSend(
+                    channel: env.SLACK_CHANNEL,
+                    color: 'good',
+                    message: message,
+                    token: env.SLACK_TOKEN
+                )
+                
+                // Send email notification
+                emailext(
+                    subject: "✅ phonebill Deployment Success - ${params.ENVIRONMENT}",
+                    body: message,
+                    to: env.EMAIL_RECIPIENTS
+                )
+            }
+        }
+        
+        failure {
+            script {
+                def message = """
+                    ❌ *phonebill Deployment Failed*
+                    • Environment: `${params.ENVIRONMENT}`
+                    • Services: `${env.SERVICES_LIST}`
+                    • Build: `${env.BUILD_NUMBER}`
+                    • Error: `${currentBuild.result}`
+                    • Console: ${env.BUILD_URL}console
+                """.stripIndent()
+                
+                // Send Slack notification
+                slackSend(
+                    channel: env.SLACK_CHANNEL,
+                    color: 'danger',
+                    message: message,
+                    token: env.SLACK_TOKEN
+                )
+                
+                // Send email notification
+                emailext(
+                    subject: "❌ phonebill Deployment Failed - ${params.ENVIRONMENT}",
+                    body: message,
+                    to: env.EMAIL_RECIPIENTS
+                )
+            }
+        }
+    }
+}
--- a/deployment/cicd/README.md
+++ b/deployment/cicd/README.md
@ -0,0 +1,77 @@
+# Phonebill CI/CD 디렉토리 구조
+
+## 개요
+이 디렉토리는 Phonebill 마이크로서비스의 CI/CD 파이프라인을 위한 Kustomize 기반 구조를 포함합니다.
+
+## 디렉토리 구조
+
+```
+deployment/cicd/
+├── kustomize/
+│   ├── base/                     # Base 매니페스트 (환경 독립적)
+│   │   ├── common/              # 공통 리소스 (Ingress, Secret 등)
+│   │   ├── api-gateway/         # API Gateway 서비스
+│   │   ├── user-service/        # 사용자 서비스
+│   │   ├── bill-service/        # 요금 조회 서비스
+│   │   ├── product-service/     # 상품 변경 서비스
+│   │   ├── kos-mock/           # KOS Mock 서비스
+│   │   └── kustomization.yaml   # Base 통합 설정
+│   └── overlays/               # 환경별 오버레이
+│       ├── dev/                # 개발 환경
+│       ├── staging/            # 스테이징 환경
+│       └── prod/               # 프로덕션 환경
+├── config/                     # CI/CD 설정 파일
+└── scripts/                    # 배포 스크립트
+```
+
+## 주요 특징
+
+### 1. 네임스페이스 분리
+- **개발**: `phonebill-dev`
+- **스테이징**: `phonebill-staging`
+- **프로덕션**: `phonebill-prod`
+
+### 2. 환경별 리소스 설정
+- **개발**: 최소 리소스 (CPU: 250m, Memory: 512Mi)
+- **스테이징**: 중간 리소스, 복제본 2개
+- **프로덕션**: 최대 리소스, 복제본 3개
+
+### 3. 서비스 구성
+- **api-gateway**: API 게이트웨이
+- **user-service**: 사용자 인증/관리
+- **bill-service**: 요금 조회
+- **product-service**: 상품 변경
+- **kos-mock**: KOS 시스템 모킹
+
+## 사용 방법
+
+### 개발 환경 배포
+```bash
+kubectl apply -k deployment/cicd/kustomize/overlays/dev
+```
+
+### 스테이징 환경 배포
+```bash
+kubectl apply -k deployment/cicd/kustomize/overlays/staging
+```
+
+### 프로덕션 환경 배포
+```bash
+kubectl apply -k deployment/cicd/kustomize/overlays/prod
+```
+
+### 매니페스트 미리보기
+```bash
+kubectl kustomize deployment/cicd/kustomize/overlays/dev
+```
+
+## 주요 변경사항
+1. 기존 `deployment/k8s/` 매니페스트를 `base/`로 복사
+2. 하드코딩된 네임스페이스 제거 (`phonebill-dev`)
+3. 환경별 오버레이 구조 적용
+4. 리소스 제한 및 복제본 수 환경별 차별화
+
+## Azure 연동 정보
+- **ACR**: acrdigitalgarage01
+- **리소스그룹**: rg-digitalgarage-01
+- **AKS클러스터**: aks-digitalgarage-01
--- a/deployment/cicd/config/deploy_env_vars_dev
+++ b/deployment/cicd/config/deploy_env_vars_dev
@ -0,0 +1,34 @@
+# Development Environment Configuration for phonebill
+# Jenkins Pipeline Environment Variables
+
+# Azure Configuration
+AZURE_RESOURCE_GROUP=rg-digitalgarage-01
+ACR_NAME=acrdigitalgarage01
+AKS_CLUSTER_NAME=aks-digitalgarage-01
+AKS_NAMESPACE=phonebill-dev
+
+# Service Names
+SERVICES=api-gateway,user-service,bill-service,product-service,kos-mock
+
+# Build Configuration
+GRADLE_OPTS="-Xmx2048m -XX:MaxPermSize=512m"
+JAVA_OPTS="-Xmx1024m -Xms512m"
+
+# Docker Configuration
+REGISTRY_URL=${ACR_NAME}.azurecr.io
+IMAGE_TAG_PATTERN=${BUILD_NUMBER}-dev
+
+# Deployment Configuration
+KUSTOMIZE_BASE=deployment/k8s
+KUSTOMIZE_OVERLAY=overlays/dev
+HEALTH_CHECK_TIMEOUT=300
+HEALTH_CHECK_RETRY=10
+
+# SonarQube Configuration
+SONAR_PROJECT_KEY=phonebill-dev
+SONAR_SOURCES=.
+SONAR_EXCLUSIONS=**/target/**,**/build/**,**/*.generated.java
+
+# Notification Configuration
+SLACK_CHANNEL=#phonebill-dev
+EMAIL_RECIPIENTS=dev-team@company.com
--- a/deployment/cicd/config/deploy_env_vars_prod
+++ b/deployment/cicd/config/deploy_env_vars_prod
@ -0,0 +1,34 @@
+# Production Environment Configuration for phonebill
+# Jenkins Pipeline Environment Variables
+
+# Azure Configuration
+AZURE_RESOURCE_GROUP=rg-digitalgarage-01
+ACR_NAME=acrdigitalgarage01
+AKS_CLUSTER_NAME=aks-digitalgarage-01
+AKS_NAMESPACE=phonebill-prod
+
+# Service Names
+SERVICES=api-gateway,user-service,bill-service,product-service,kos-mock
+
+# Build Configuration
+GRADLE_OPTS="-Xmx3072m -XX:MaxPermSize=1024m"
+JAVA_OPTS="-Xmx2048m -Xms1024m"
+
+# Docker Configuration
+REGISTRY_URL=${ACR_NAME}.azurecr.io
+IMAGE_TAG_PATTERN=${BUILD_NUMBER}-prod
+
+# Deployment Configuration
+KUSTOMIZE_BASE=deployment/k8s
+KUSTOMIZE_OVERLAY=overlays/prod
+HEALTH_CHECK_TIMEOUT=600
+HEALTH_CHECK_RETRY=15
+
+# SonarQube Configuration
+SONAR_PROJECT_KEY=phonebill-prod
+SONAR_SOURCES=.
+SONAR_EXCLUSIONS=**/target/**,**/build/**,**/*.generated.java
+
+# Notification Configuration
+SLACK_CHANNEL=#phonebill-prod
+EMAIL_RECIPIENTS=prod-team@company.com,ops-team@company.com
--- a/deployment/cicd/config/deploy_env_vars_staging
+++ b/deployment/cicd/config/deploy_env_vars_staging
@ -0,0 +1,34 @@
+# Staging Environment Configuration for phonebill
+# Jenkins Pipeline Environment Variables
+
+# Azure Configuration
+AZURE_RESOURCE_GROUP=rg-digitalgarage-01
+ACR_NAME=acrdigitalgarage01
+AKS_CLUSTER_NAME=aks-digitalgarage-01
+AKS_NAMESPACE=phonebill-staging
+
+# Service Names
+SERVICES=api-gateway,user-service,bill-service,product-service,kos-mock
+
+# Build Configuration
+GRADLE_OPTS="-Xmx2048m -XX:MaxPermSize=512m"
+JAVA_OPTS="-Xmx1024m -Xms512m"
+
+# Docker Configuration
+REGISTRY_URL=${ACR_NAME}.azurecr.io
+IMAGE_TAG_PATTERN=${BUILD_NUMBER}-staging
+
+# Deployment Configuration
+KUSTOMIZE_BASE=deployment/k8s
+KUSTOMIZE_OVERLAY=overlays/staging
+HEALTH_CHECK_TIMEOUT=300
+HEALTH_CHECK_RETRY=10
+
+# SonarQube Configuration
+SONAR_PROJECT_KEY=phonebill-staging
+SONAR_SOURCES=.
+SONAR_EXCLUSIONS=**/target/**,**/build/**,**/*.generated.java
+
+# Notification Configuration
+SLACK_CHANNEL=#phonebill-staging
+EMAIL_RECIPIENTS=staging-team@company.com
--- a/deployment/cicd/jenkins-pipeline-guide.md
+++ b/deployment/cicd/jenkins-pipeline-guide.md
@ -0,0 +1,423 @@
+# Jenkins CI/CD Pipeline 구축 가이드
+
+## 📋 개요
+
+phonebill 마이크로서비스를 위한 Jenkins CI/CD 파이프라인 구축 가이드입니다.
+Azure Kubernetes Service(AKS)와 Azure Container Registry(ACR)를 활용한 자동화된 배포 시스템을 구성합니다.
+
+## 🏗️ 아키텍처
+
+### 시스템 구성요소
+- **Jenkins**: CI/CD 오케스트레이션
+- **Azure Container Registry (ACR)**: 컨테이너 이미지 저장소
+- **Azure Kubernetes Service (AKS)**: 컨테이너 오케스트레이션
+- **SonarQube**: 코드 품질 분석
+- **Gradle**: 빌드 도구
+- **Kustomize**: Kubernetes 매니페스트 관리
+
+### 배포 환경
+- **Development** (`phonebill-dev`)
+- **Staging** (`phonebill-staging`)
+- **Production** (`phonebill-prod`)
+
+## 🚀 파이프라인 워크플로우
+
+```mermaid
+graph LR
+    A[Code Commit] --> B[Jenkins Trigger]
+    B --> C[Build & Test]
+    C --> D[SonarQube Analysis]
+    D --> E[Quality Gate]
+    E --> F[Container Build]
+    F --> G[Push to ACR]
+    G --> H[Deploy to AKS]
+    H --> I[Health Check]
+    I --> J[Notification]
+```
+
+### 주요 단계
+
+1. **Initialize**: 환경별 설정 로드 및 파라미터 설정
+2. **Checkout & Prepare**: 소스 코드 체크아웃 및 빌드 태그 생성
+3. **Build & Test**: Gradle을 이용한 빌드 및 단위 테스트
+4. **SonarQube Analysis**: 코드 품질 분석
+5. **Quality Gate**: 품질 기준 검증
+6. **Container Build & Push**: 컨테이너 이미지 빌드 및 ACR 푸시
+7. **Deploy to Kubernetes**: AKS 클러스터에 배포
+8. **Health Check**: 배포된 서비스 상태 확인
+
+## 📂 파일 구조
+
+```
+deployment/cicd/
+├── Jenkinsfile                           # Jenkins 파이프라인 정의
+├── config/
+│   ├── deploy_env_vars_dev              # 개발 환경 설정
+│   ├── deploy_env_vars_staging          # 스테이징 환경 설정
+│   └── deploy_env_vars_prod             # 운영 환경 설정
+├── scripts/
+│   └── deploy.sh                        # 수동 배포 스크립트
+└── jenkins-pipeline-guide.md            # 이 가이드 문서
+```
+
+## 🔧 Jenkins 구성
+
+### 1. Jenkins 플러그인 설치
+
+필수 플러그인 목록:
+```bash
+# Kubernetes 관련
+Kubernetes Plugin
+Pipeline: Kubernetes Steps
+
+# Azure 관련
+Azure CLI Plugin
+Azure Container Registry Plugin
+
+# 빌드 도구
+Gradle Plugin
+Pipeline: Gradle Plugin
+
+# 코드 품질
+SonarQube Scanner Plugin
+Pipeline: SonarQube Plugin
+
+# 알림
+Slack Notification Plugin
+Email Extension Plugin
+
+# Git 관련
+Git Plugin
+GitHub Plugin
+Pipeline: GitHub Plugin
+
+# 기타
+Pipeline Plugin
+Pipeline: Stage View Plugin
+Blue Ocean Plugin
+```
+
+### 2. 글로벌 설정
+
+#### Azure Service Principal 설정
+```bash
+# Jenkins 관리 > 시스템 설정 > Global properties
+# Environment variables 추가
+AZURE_TENANT_ID=<your-tenant-id>
+AZURE_SUBSCRIPTION_ID=<your-subscription-id>
+```
+
+#### Credentials 설정
+Jenkins 관리 > Manage Credentials에서 다음 설정:
+
+1. **azure-service-principal** (Azure Service Principal)
+   - ID: `azure-service-principal`
+   - Type: Microsoft Azure Service Principal
+   - Tenant ID: Azure 테넌트 ID
+   - Client ID: 서비스 프린시pal 클라이언트 ID
+   - Client Secret: 서비스 프린시pal 시크릿
+
+2. **acr-credentials** (ACR 인증 정보)
+   - ID: `acr-credentials`
+   - Type: Username with password
+   - Username: ACR 사용자명
+   - Password: ACR 패스워드
+
+3. **sonarqube-token** (SonarQube 토큰)
+   - ID: `sonarqube-token`
+   - Type: Secret text
+   - Secret: SonarQube 액세스 토큰
+
+4. **slack-token** (Slack 토큰)
+   - ID: `slack-token`
+   - Type: Secret text
+   - Secret: Slack Bot 토큰
+
+### 3. Kubernetes Agent 설정
+
+Jenkins가 Kubernetes 클러스터에서 빌드 에이전트를 실행할 수 있도록 설정:
+
+#### ServiceAccount 및 RBAC 생성
+```yaml
+# jenkins-rbac.yaml
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: jenkins-agent
+  namespace: jenkins
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: jenkins-agent
+rules:
+- apiGroups: [\"\"]
+  resources: [\"pods\", \"pods/exec\", \"pods/log\", \"persistentvolumeclaims\"]
+  verbs: [\"*\"]
+- apiGroups: [\"apps\"]
+  resources: [\"deployments\", \"replicasets\"]
+  verbs: [\"*\"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: jenkins-agent
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: jenkins-agent
+subjects:
+- kind: ServiceAccount
+  name: jenkins-agent
+  namespace: jenkins
+```
+
+#### Gradle Cache PVC 생성
+```yaml
+# gradle-cache-pvc.yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: jenkins-gradle-cache
+  namespace: jenkins
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 10Gi
+  storageClassName: managed-premium
+```
+
+적용:
+```bash
+kubectl apply -f jenkins-rbac.yaml
+kubectl apply -f gradle-cache-pvc.yaml
+```
+
+### 4. SonarQube 설정
+
+#### SonarQube Server 구성
+Jenkins 관리 > Configure System > SonarQube servers:
+- Name: `SonarQube`
+- Server URL: `http://sonarqube.example.com`
+- Server authentication token: `sonarqube-token` credential 선택
+
+## 🔨 파이프라인 생성
+
+### 1. 새 Pipeline Job 생성
+1. Jenkins 대시보드에서 \"New Item\" 클릭
+2. Job 이름: `phonebill-pipeline`
+3. Type: \"Pipeline\" 선택
+4. OK 클릭
+
+### 2. Pipeline 설정
+1. **General** 탭:
+   - Description: \"phonebill microservices CI/CD pipeline\"
+   - \"GitHub project\" 체크하고 프로젝트 URL 입력
+
+2. **Build Triggers** 탭:
+   - \"GitHub hook trigger for GITScm polling\" 체크 (GitHub Webhook 사용 시)
+   - \"Poll SCM\" 설정: `H/5 * * * *` (5분마다 폴링)
+
+3. **Pipeline** 탭:
+   - Definition: \"Pipeline script from SCM\"
+   - SCM: Git
+   - Repository URL: GitHub 저장소 URL
+   - Credentials: GitHub 인증 정보
+   - Branch: `*/main`
+   - Script Path: `deployment/cicd/Jenkinsfile`
+
+### 3. 환경별 파이프라인 생성
+동일한 방식으로 환경별 파이프라인 생성:
+- `phonebill-dev-pipeline`
+- `phonebill-staging-pipeline`
+- `phonebill-prod-pipeline`
+
+## 🧪 파이프라인 실행
+
+### 1. 수동 실행
+1. Jenkins에서 파이프라인 Job 선택
+2. \"Build with Parameters\" 클릭
+3. 파라미터 설정:
+   - **ENVIRONMENT**: `dev` / `staging` / `prod`
+   - **SERVICES_TO_BUILD**: `all` 또는 특정 서비스
+   - **SKIP_TESTS**: 테스트 스킵 여부
+   - **SKIP_SONAR**: SonarQube 분석 스킵 여부
+   - **FORCE_DEPLOY**: 강제 배포 여부
+4. \"Build\" 클릭
+
+### 2. 자동 실행 (Webhook)
+GitHub에서 코드 푸시 시 자동으로 파이프라인이 트리거됩니다.
+
+#### GitHub Webhook 설정
+1. GitHub 저장소 > Settings > Webhooks
+2. Add webhook:
+   - Payload URL: `http://jenkins.example.com/github-webhook/`
+   - Content type: `application/json`
+   - Secret: 설정한 시크릿
+   - Events: \"Just the push event\"
+
+## 📊 모니터링 및 알림
+
+### 1. 빌드 상태 모니터링
+- Jenkins Blue Ocean 인터페이스 활용
+- 파이프라인 실행 상태 실시간 확인
+- 로그 및 아티팩트 확인
+
+### 2. 알림 설정
+파이프라인 실행 결과를 다음 채널로 알림:
+- **Slack**: 지정된 채널에 빌드 상태 알림
+- **Email**: 담당자에게 결과 메일 발송
+
+### 3. 메트릭 수집
+- 빌드 시간 추적
+- 성공/실패율 모니터링
+- 배포 빈도 측정
+
+## 🔍 트러블슈팅
+
+### 일반적인 문제 해결
+
+#### 1. Azure 인증 실패
+```bash
+# 서비스 프린시pal 권한 확인
+az role assignment list --assignee <service-principal-client-id>
+
+# 필요한 권한 할당
+az role assignment create \
+  --assignee <service-principal-client-id> \
+  --role \"AKS Cluster Admin\" \
+  --scope /subscriptions/<subscription-id>/resourceGroups/<rg-name>
+```
+
+#### 2. ACR 푸시 실패
+```bash
+# ACR 로그인 확인
+az acr login --name <acr-name>
+
+# ACR 권한 확인
+az acr show --name <acr-name> --resource-group <rg-name>
+```
+
+#### 3. Kubernetes 배포 실패
+```bash
+# kubectl 컨텍스트 확인
+kubectl config current-context
+
+# 네임스페이스 확인
+kubectl get namespaces
+
+# 리소스 상태 확인
+kubectl get all -n <namespace>
+```
+
+#### 4. 파드 시작 실패
+```bash
+# 파드 로그 확인
+kubectl logs <pod-name> -n <namespace>
+
+# 파드 상세 정보 확인
+kubectl describe pod <pod-name> -n <namespace>
+
+# 이벤트 확인
+kubectl get events -n <namespace> --sort-by='.lastTimestamp'
+```
+
+### 로그 위치
+- **Jenkins 로그**: `/var/log/jenkins/jenkins.log`
+- **파이프라인 로그**: Jenkins UI에서 Build History > Console Output
+- **Kubernetes 로그**: `kubectl logs` 명령어 사용
+
+## 🚀 수동 배포 스크립트 사용법
+
+Jenkins 파이프라인 외에도 수동 배포 스크립트를 제공합니다.
+
+### 기본 사용법
+```bash
+# 모든 서비스를 dev 환경에 배포
+./deployment/cicd/scripts/deploy.sh dev
+
+# 특정 서비스만 staging 환경에 배포
+./deployment/cicd/scripts/deploy.sh staging user-service
+
+# 여러 서비스를 prod 환경에 배포
+./deployment/cicd/scripts/deploy.sh prod api-gateway,user-service,bill-service
+
+# 옵션 사용 예시
+./deployment/cicd/scripts/deploy.sh dev all --skip-build --skip-test
+```
+
+### 주요 옵션
+- `--skip-build`: Gradle 빌드 스킵
+- `--skip-test`: 단위 테스트 스킵
+- `--skip-push`: 컨테이너 이미지 푸시 스킵
+- `--force`: 변경사항이 없어도 강제 배포
+- `--dry-run`: 실제 배포 없이 미리보기
+
+## 📈 성능 최적화
+
+### 1. 빌드 성능 개선
+- **Gradle Daemon** 활용: `--daemon` 옵션
+- **병렬 빌드**: `--parallel` 옵션
+- **Build Cache** 활용: `--build-cache` 옵션
+- **Incremental Build** 활용
+
+### 2. 컨테이너 이미지 최적화
+- **Multi-stage Build** 사용
+- **Layer Caching** 최적화
+- **Base Image** 최적화
+- **.dockerignore** 활용
+
+### 3. Kubernetes 배포 최적화
+- **Rolling Update** 전략 사용
+- **Resource Limits** 설정
+- **Readiness/Liveness Probe** 설정
+- **Pod Disruption Budget** 설정
+
+## 🔒 보안 고려사항
+
+### 1. 인증 및 권한 관리
+- Azure Service Principal 최소 권한 원칙
+- Jenkins Credentials 암호화 저장
+- Kubernetes RBAC 적절한 권한 할당
+- 시크릿 정보 환경 변수로 분리
+
+### 2. 컨테이너 보안
+- 취약점 스캐닝 도구 통합
+- 비특권 사용자로 컨테이너 실행
+- 읽기 전용 루트 파일시스템
+- 보안 컨텍스트 설정
+
+### 3. 네트워크 보안
+- Private Registry 사용
+- Network Policy 적용
+- Service Mesh 보안 정책
+- TLS/SSL 암호화
+
+## 📚 참고 자료
+
+### 공식 문서
+- [Jenkins Pipeline](https://jenkins.io/doc/book/pipeline/)
+- [Azure Kubernetes Service](https://docs.microsoft.com/en-us/azure/aks/)
+- [Azure Container Registry](https://docs.microsoft.com/en-us/azure/container-registry/)
+- [Kubernetes](https://kubernetes.io/docs/)
+- [Gradle](https://docs.gradle.org/)
+
+### 모범 사례
+- [Jenkins Best Practices](https://wiki.jenkins.io/display/JENKINS/Jenkins+Best+Practices)
+- [Kubernetes Best Practices](https://kubernetes.io/docs/concepts/configuration/overview/)
+- [Docker Best Practices](https://docs.docker.com/develop/dev-best-practices/)
+
+## 📞 지원
+
+문제가 발생하거나 추가 지원이 필요한 경우:
+
+1. **로그 확인**: Jenkins 콘솔 출력 및 Kubernetes 로그 검토
+2. **문서 검토**: 이 가이드 및 공식 문서 참조
+3. **커뮤니티**: Stack Overflow, Jenkins 커뮤니티 포럼 활용
+4. **팀 지원**: DevOps 팀 또는 플랫폼 팀에 문의
+
+---
+
+*이 가이드는 phonebill 프로젝트의 Jenkins CI/CD 파이프라인 구축을 위한 완전한 가이드입니다. 프로젝트 요구사항에 따라 설정을 조정하여 사용하시기 바랍니다.*
--- a/deployment/cicd/kustomize/base/api-gateway/cm-api-gateway.yaml
+++ b/deployment/cicd/kustomize/base/api-gateway/cm-api-gateway.yaml
@ -0,0 +1,10 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cm-api-gateway
+data:
+  SERVER_PORT: "8080"
+  BILL_SERVICE_URL: "http://bill-service"
+  PRODUCT_SERVICE_URL: "http://product-service"
+  USER_SERVICE_URL: "http://user-service"
+  KOS_MOCK_URL: "http://kos-mock"
--- a/deployment/cicd/kustomize/base/api-gateway/deployment.yaml
+++ b/deployment/cicd/kustomize/base/api-gateway/deployment.yaml
@ -0,0 +1,57 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: api-gateway
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: api-gateway
+  template:
+    metadata:
+      labels:
+        app: api-gateway
+    spec:
+      imagePullSecrets:
+      - name: phonebill
+      containers:
+      - name: api-gateway
+        image: acrdigitalgarage01.azurecr.io/phonebill/api-gateway:latest
+        imagePullPolicy: Always
+        ports:
+        - containerPort: 8080
+        envFrom:
+        - configMapRef:
+            name: cm-common
+        - configMapRef:
+            name: cm-api-gateway
+        - secretRef:
+            name: secret-common
+        resources:
+          requests:
+            cpu: 256m
+            memory: 256Mi
+          limits:
+            cpu: 1024m
+            memory: 1024Mi
+        startupProbe:
+          httpGet:
+            path: /health
+            port: 8080
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          failureThreshold: 6
+        readinessProbe:
+          httpGet:
+            path: /health
+            port: 8080
+          initialDelaySeconds: 10
+          periodSeconds: 5
+          failureThreshold: 3
+        livenessProbe:
+          httpGet:
+            path: /health
+            port: 8080
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          failureThreshold: 3
--- a/deployment/cicd/kustomize/base/api-gateway/kustomization.yaml
+++ b/deployment/cicd/kustomize/base/api-gateway/kustomization.yaml
@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+- cm-api-gateway.yaml
+- deployment.yaml
+- service.yaml
--- a/deployment/cicd/kustomize/base/api-gateway/service.yaml
+++ b/deployment/cicd/kustomize/base/api-gateway/service.yaml
@ -0,0 +1,11 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: api-gateway
+spec:
+  selector:
+    app: api-gateway
+  ports:
+  - port: 80
+    targetPort: 8080
+  type: ClusterIP
--- a/deployment/cicd/kustomize/base/bill-service/cm-bill-service.yaml
+++ b/deployment/cicd/kustomize/base/bill-service/cm-bill-service.yaml
@ -0,0 +1,21 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cm-bill-service
+data:
+  SERVER_PORT: "8082"
+  DB_KIND: "postgresql"
+  DB_PORT: "5432"
+  DB_CONNECTION_TIMEOUT: "30000"
+  DB_IDLE_TIMEOUT: "600000"
+  DB_LEAK_DETECTION: "60000"
+  DB_MAX_LIFETIME: "1800000"
+  DB_MAX_POOL: "20"
+  DB_MIN_IDLE: "5"
+  KOS_BASE_URL: "http://kos-mock"
+  REDIS_DATABASE: "1"
+  REDIS_MAX_ACTIVE: "8"
+  REDIS_MAX_IDLE: "8"
+  REDIS_MAX_WAIT: "-1"
+  REDIS_MIN_IDLE: "0"
+  REDIS_TIMEOUT: "2000"
--- a/deployment/cicd/kustomize/base/bill-service/deployment.yaml
+++ b/deployment/cicd/kustomize/base/bill-service/deployment.yaml
@ -0,0 +1,59 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: bill-service
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: bill-service
+  template:
+    metadata:
+      labels:
+        app: bill-service
+    spec:
+      imagePullSecrets:
+      - name: phonebill
+      containers:
+      - name: bill-service
+        image: acrdigitalgarage01.azurecr.io/phonebill/bill-service:latest
+        imagePullPolicy: Always
+        ports:
+        - containerPort: 8082
+        envFrom:
+        - configMapRef:
+            name: cm-common
+        - configMapRef:
+            name: cm-bill-service
+        - secretRef:
+            name: secret-common
+        - secretRef:
+            name: secret-bill-service
+        resources:
+          requests:
+            cpu: 256m
+            memory: 256Mi
+          limits:
+            cpu: 1024m
+            memory: 1024Mi
+        startupProbe:
+          httpGet:
+            path: /actuator/health
+            port: 8082
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          failureThreshold: 6
+        readinessProbe:
+          httpGet:
+            path: /actuator/health/readiness
+            port: 8082
+          initialDelaySeconds: 10
+          periodSeconds: 5
+          failureThreshold: 3
+        livenessProbe:
+          httpGet:
+            path: /actuator/health/liveness
+            port: 8082
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          failureThreshold: 3
--- a/deployment/cicd/kustomize/base/bill-service/kustomization.yaml
+++ b/deployment/cicd/kustomize/base/bill-service/kustomization.yaml
@ -0,0 +1,8 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+- cm-bill-service.yaml
+- deployment.yaml
+- secret-bill-service.yaml
+- service.yaml
--- a/deployment/cicd/kustomize/base/bill-service/secret-bill-service.yaml
+++ b/deployment/cicd/kustomize/base/bill-service/secret-bill-service.yaml
@ -0,0 +1,10 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-bill-service
+type: Opaque
+stringData:
+  DB_HOST: "bill-inquiry-postgres-dev-postgresql"
+  DB_NAME: "bill_inquiry_db"
+  DB_USERNAME: "bill_inquiry_user"
+  DB_PASSWORD: "BillUser2025!"
--- a/deployment/cicd/kustomize/base/bill-service/service.yaml
+++ b/deployment/cicd/kustomize/base/bill-service/service.yaml
@ -0,0 +1,11 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: bill-service
+spec:
+  selector:
+    app: bill-service
+  ports:
+  - port: 80
+    targetPort: 8082
+  type: ClusterIP
--- a/deployment/cicd/kustomize/base/common/cm-common.yaml
+++ b/deployment/cicd/kustomize/base/common/cm-common.yaml
@ -0,0 +1,11 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cm-common
+data:
+  CORS_ALLOWED_ORIGINS: "http://localhost:8081,http://localhost:8082,http://localhost:8083,http://localhost:8084,http://phonebill.20.214.196.128.nip.io"
+  JWT_ACCESS_TOKEN_VALIDITY: "18000000"
+  JWT_REFRESH_TOKEN_VALIDITY: "86400000"
+  REDIS_PORT: "6379"
+  SPRING_PROFILES_ACTIVE: "dev"
+  DDL_AUTO: "update"
--- a/deployment/cicd/kustomize/base/common/ingress.yaml
+++ b/deployment/cicd/kustomize/base/common/ingress.yaml
@ -0,0 +1,48 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: phonebill
+  annotations:
+    kubernetes.io/ingress.class: nginx
+    nginx.ingress.kubernetes.io/ssl-redirect: "false"
+spec:
+  ingressClassName: nginx
+  rules:
+  - host: phonebill-api.20.214.196.128.nip.io
+    http:
+      paths:
+      - path: /api/v1/auth
+        pathType: Prefix
+        backend:
+          service:
+            name: user-service
+            port:
+              number: 80
+      - path: /api/v1/users
+        pathType: Prefix
+        backend:
+          service:
+            name: user-service
+            port:
+              number: 80
+      - path: /api/v1/bills
+        pathType: Prefix
+        backend:
+          service:
+            name: bill-service
+            port:
+              number: 80
+      - path: /api/v1/products
+        pathType: Prefix
+        backend:
+          service:
+            name: product-service
+            port:
+              number: 80
+      - path: /api/v1/kos
+        pathType: Prefix
+        backend:
+          service:
+            name: kos-mock
+            port:
+              number: 80
--- a/deployment/cicd/kustomize/base/common/kustomization.yaml
+++ b/deployment/cicd/kustomize/base/common/kustomization.yaml
@ -0,0 +1,8 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+- cm-common.yaml
+- ingress.yaml
+- secret-common.yaml
+- secret-imagepull.yaml
--- a/deployment/cicd/kustomize/base/common/secret-common.yaml
+++ b/deployment/cicd/kustomize/base/common/secret-common.yaml
@ -0,0 +1,9 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-common
+type: Opaque
+stringData:
+  JWT_SECRET: "nwe5Yo9qaJ6FBD/Thl2/j6/SFAfNwUorAY1ZcWO2KI7uA4bmVLOCPxE9hYuUpRCOkgV2UF2DdHXtqHi3+BU/ecbz2zpHyf/720h48UbA3XOMYOX1sdM+dQ=="
+  REDIS_HOST: "redis-cache-dev-master"
+  REDIS_PASSWORD: "Redis2025Dev!"
--- a/deployment/cicd/kustomize/base/common/secret-imagepull.yaml
+++ b/deployment/cicd/kustomize/base/common/secret-imagepull.yaml
@ -0,0 +1,16 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: phonebill
+type: kubernetes.io/dockerconfigjson
+stringData:
+  .dockerconfigjson: |
+    {
+      "auths": {
+        "acrdigitalgarage01.azurecr.io": {
+          "username": "acrdigitalgarage01",
+          "password": "+OY+rmOagorjWvQe/tTk6oqvnZI8SmNbY/Y2o5EDcY+ACRDCDbYk",
+          "auth": "YWNyZGlnaXRhbGdhcmFnZTAxOitPWStybU9hZ29yald2UWUvdFRrNm9xdm5aSThTbU5iWS9ZMm81RURjWStBQ1JEQ0RiWWs="
+        }
+      }
+    }
--- a/deployment/cicd/kustomize/base/kos-mock/cm-kos-mock.yaml
+++ b/deployment/cicd/kustomize/base/kos-mock/cm-kos-mock.yaml
@ -0,0 +1,6 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cm-kos-mock
+data:
+  SERVER_PORT: "8084"
--- a/deployment/cicd/kustomize/base/kos-mock/deployment.yaml
+++ b/deployment/cicd/kustomize/base/kos-mock/deployment.yaml
@ -0,0 +1,57 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: kos-mock
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: kos-mock
+  template:
+    metadata:
+      labels:
+        app: kos-mock
+    spec:
+      imagePullSecrets:
+      - name: phonebill
+      containers:
+      - name: kos-mock
+        image: acrdigitalgarage01.azurecr.io/phonebill/kos-mock:latest
+        imagePullPolicy: Always
+        ports:
+        - containerPort: 8084
+        envFrom:
+        - configMapRef:
+            name: cm-common
+        - configMapRef:
+            name: cm-kos-mock
+        - secretRef:
+            name: secret-common
+        resources:
+          requests:
+            cpu: 256m
+            memory: 256Mi
+          limits:
+            cpu: 1024m
+            memory: 1024Mi
+        startupProbe:
+          httpGet:
+            path: /actuator/health
+            port: 8084
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          failureThreshold: 6
+        readinessProbe:
+          httpGet:
+            path: /actuator/health/readiness
+            port: 8084
+          initialDelaySeconds: 10
+          periodSeconds: 5
+          failureThreshold: 3
+        livenessProbe:
+          httpGet:
+            path: /actuator/health/liveness
+            port: 8084
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          failureThreshold: 3
--- a/deployment/cicd/kustomize/base/kos-mock/kustomization.yaml
+++ b/deployment/cicd/kustomize/base/kos-mock/kustomization.yaml
@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+- cm-kos-mock.yaml
+- deployment.yaml
+- service.yaml
--- a/deployment/cicd/kustomize/base/kos-mock/service.yaml
+++ b/deployment/cicd/kustomize/base/kos-mock/service.yaml
@ -0,0 +1,11 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: kos-mock
+spec:
+  selector:
+    app: kos-mock
+  ports:
+  - port: 80
+    targetPort: 8084
+  type: ClusterIP
--- a/deployment/cicd/kustomize/base/kustomization.yaml
+++ b/deployment/cicd/kustomize/base/kustomization.yaml
@ -0,0 +1,10 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+- common
+- api-gateway
+- user-service
+- bill-service
+- product-service
+- kos-mock
--- a/deployment/cicd/kustomize/base/product-service/cm-product-service.yaml
+++ b/deployment/cicd/kustomize/base/product-service/cm-product-service.yaml
@ -0,0 +1,10 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cm-product-service
+data:
+  SERVER_PORT: "8083"
+  DB_KIND: "postgresql"
+  DB_PORT: "5432"
+  KOS_BASE_URL: "http://kos-mock"
+  REDIS_DATABASE: "2"
--- a/deployment/cicd/kustomize/base/product-service/deployment.yaml
+++ b/deployment/cicd/kustomize/base/product-service/deployment.yaml
@ -0,0 +1,59 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: product-service
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: product-service
+  template:
+    metadata:
+      labels:
+        app: product-service
+    spec:
+      imagePullSecrets:
+      - name: phonebill
+      containers:
+      - name: product-service
+        image: acrdigitalgarage01.azurecr.io/phonebill/product-service:latest
+        imagePullPolicy: Always
+        ports:
+        - containerPort: 8083
+        envFrom:
+        - configMapRef:
+            name: cm-common
+        - configMapRef:
+            name: cm-product-service
+        - secretRef:
+            name: secret-common
+        - secretRef:
+            name: secret-product-service
+        resources:
+          requests:
+            cpu: 256m
+            memory: 256Mi
+          limits:
+            cpu: 1024m
+            memory: 1024Mi
+        startupProbe:
+          httpGet:
+            path: /actuator/health
+            port: 8083
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          failureThreshold: 6
+        readinessProbe:
+          httpGet:
+            path: /actuator/health/readiness
+            port: 8083
+          initialDelaySeconds: 10
+          periodSeconds: 5
+          failureThreshold: 3
+        livenessProbe:
+          httpGet:
+            path: /actuator/health/liveness
+            port: 8083
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          failureThreshold: 3
--- a/deployment/cicd/kustomize/base/product-service/kustomization.yaml
+++ b/deployment/cicd/kustomize/base/product-service/kustomization.yaml
@ -0,0 +1,8 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+- cm-product-service.yaml
+- deployment.yaml
+- secret-product-service.yaml
+- service.yaml
--- a/deployment/cicd/kustomize/base/product-service/secret-product-service.yaml
+++ b/deployment/cicd/kustomize/base/product-service/secret-product-service.yaml
@ -0,0 +1,10 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-product-service
+type: Opaque
+stringData:
+  DB_HOST: "product-change-postgres-dev-postgresql"
+  DB_NAME: "product_change_db"
+  DB_USERNAME: "product_change_user"
+  DB_PASSWORD: "ProductUser2025!"
--- a/deployment/cicd/kustomize/base/product-service/service.yaml
+++ b/deployment/cicd/kustomize/base/product-service/service.yaml
@ -0,0 +1,11 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: product-service
+spec:
+  selector:
+    app: product-service
+  ports:
+  - port: 80
+    targetPort: 8083
+  type: ClusterIP
--- a/deployment/cicd/kustomize/base/user-service/cm-user-service.yaml
+++ b/deployment/cicd/kustomize/base/user-service/cm-user-service.yaml
@ -0,0 +1,11 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cm-user-service
+data:
+  SERVER_PORT: "8081"
+  DB_KIND: "postgresql"
+  DB_PORT: "5432"
+  DDL_AUTO: "update"
+  REDIS_DATABASE: "0"
+  SHOW_SQL: "true"
--- a/deployment/cicd/kustomize/base/user-service/deployment.yaml
+++ b/deployment/cicd/kustomize/base/user-service/deployment.yaml
@ -0,0 +1,59 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: user-service
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: user-service
+  template:
+    metadata:
+      labels:
+        app: user-service
+    spec:
+      imagePullSecrets:
+      - name: phonebill
+      containers:
+      - name: user-service
+        image: acrdigitalgarage01.azurecr.io/phonebill/user-service:latest
+        imagePullPolicy: Always
+        ports:
+        - containerPort: 8081
+        envFrom:
+        - configMapRef:
+            name: cm-common
+        - configMapRef:
+            name: cm-user-service
+        - secretRef:
+            name: secret-common
+        - secretRef:
+            name: secret-user-service
+        resources:
+          requests:
+            cpu: 256m
+            memory: 256Mi
+          limits:
+            cpu: 1024m
+            memory: 1024Mi
+        startupProbe:
+          httpGet:
+            path: /actuator/health
+            port: 8081
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          failureThreshold: 6
+        readinessProbe:
+          httpGet:
+            path: /actuator/health/readiness
+            port: 8081
+          initialDelaySeconds: 10
+          periodSeconds: 5
+          failureThreshold: 3
+        livenessProbe:
+          httpGet:
+            path: /actuator/health/liveness
+            port: 8081
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          failureThreshold: 3
--- a/deployment/cicd/kustomize/base/user-service/kustomization.yaml
+++ b/deployment/cicd/kustomize/base/user-service/kustomization.yaml
@ -0,0 +1,8 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+- cm-user-service.yaml
+- deployment.yaml
+- secret-user-service.yaml
+- service.yaml
--- a/deployment/cicd/kustomize/base/user-service/secret-user-service.yaml
+++ b/deployment/cicd/kustomize/base/user-service/secret-user-service.yaml
@ -0,0 +1,10 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-user-service
+type: Opaque
+stringData:
+  DB_HOST: "auth-postgres-dev-postgresql"
+  DB_NAME: "phonebill_auth"
+  DB_USERNAME: "auth_user"
+  DB_PASSWORD: "AuthUser2025!"
--- a/deployment/cicd/kustomize/base/user-service/service.yaml
+++ b/deployment/cicd/kustomize/base/user-service/service.yaml
@ -0,0 +1,11 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: user-service
+spec:
+  selector:
+    app: user-service
+  ports:
+  - port: 80
+    targetPort: 8081
+  type: ClusterIP
--- a/deployment/cicd/kustomize/overlays/dev/configmap-common-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/dev/configmap-common-patch.yaml
@ -0,0 +1,8 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cm-common
+data:
+  CORS_ALLOWED_ORIGINS: "http://localhost:8081,http://localhost:8082,http://localhost:8083,http://localhost:8084,http://phonebill.20.214.196.128.nip.io"
+  SPRING_PROFILES_ACTIVE: "dev"
+  DDL_AUTO: "update"
--- a/deployment/cicd/kustomize/overlays/dev/ingress-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/dev/ingress-patch.yaml
@ -0,0 +1,10 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: phonebill
+  annotations:
+    kubernetes.io/ingress.class: nginx
+    nginx.ingress.kubernetes.io/ssl-redirect: "false"
+spec:
+  rules:
+  - host: phonebill-api.20.214.196.128.nip.io
--- a/deployment/cicd/kustomize/overlays/dev/kustomization.yaml
+++ b/deployment/cicd/kustomize/overlays/dev/kustomization.yaml
@ -0,0 +1,19 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: phonebill-dev
+
+resources:
+- ../../base
+
+commonLabels:
+  env: dev
+  
+patchesStrategicMerge:
+- configmap-common-patch.yaml
+- secret-common-patch.yaml
+- ingress-patch.yaml
+- replica-patch.yaml
+- secret-user-service-patch.yaml
+- secret-bill-service-patch.yaml
+- secret-product-service-patch.yaml
--- a/deployment/cicd/kustomize/overlays/dev/replica-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/dev/replica-patch.yaml
@ -0,0 +1,35 @@
+# Replica count patches for dev environment
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: api-gateway
+spec:
+  replicas: 1
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: user-service
+spec:
+  replicas: 1
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: bill-service
+spec:
+  replicas: 1
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: product-service
+spec:
+  replicas: 1
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: kos-mock
+spec:
+  replicas: 1
--- a/deployment/cicd/kustomize/overlays/dev/secret-bill-service-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/dev/secret-bill-service-patch.yaml
@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-bill-service
+type: Opaque
+stringData:
+  DB_HOST: "bill-inquiry-postgres-dev-postgresql"
+  DB_PASSWORD: "BillUser2025!"
--- a/deployment/cicd/kustomize/overlays/dev/secret-common-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/dev/secret-common-patch.yaml
@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-common
+type: Opaque
+stringData:
+  REDIS_HOST: "redis-cache-dev-master"
+  REDIS_PASSWORD: "Redis2025Dev!"
--- a/deployment/cicd/kustomize/overlays/dev/secret-product-service-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/dev/secret-product-service-patch.yaml
@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-product-service
+type: Opaque
+stringData:
+  DB_HOST: "product-change-postgres-dev-postgresql"
+  DB_PASSWORD: "ProductUser2025!"
--- a/deployment/cicd/kustomize/overlays/dev/secret-user-service-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/dev/secret-user-service-patch.yaml
@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-user-service
+type: Opaque
+stringData:
+  DB_HOST: "auth-postgres-dev-postgresql"
+  DB_PASSWORD: "AuthUser2025!"
--- a/deployment/cicd/kustomize/overlays/prod/configmap-common-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/prod/configmap-common-patch.yaml
@ -0,0 +1,8 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cm-common
+data:
+  CORS_ALLOWED_ORIGINS: "https://phonebill.example.com,https://phonebill-app.example.com"
+  SPRING_PROFILES_ACTIVE: "prod"
+  DDL_AUTO: "validate"
--- a/deployment/cicd/kustomize/overlays/prod/ingress-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/prod/ingress-patch.yaml
@ -0,0 +1,16 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: phonebill
+  annotations:
+    kubernetes.io/ingress.class: nginx
+    nginx.ingress.kubernetes.io/ssl-redirect: "true"
+    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+spec:
+  tls:
+  - hosts:
+    - phonebill-api.example.com
+    secretName: phonebill-prod-tls
+  rules:
+  - host: phonebill-api.example.com
--- a/deployment/cicd/kustomize/overlays/prod/kustomization.yaml
+++ b/deployment/cicd/kustomize/overlays/prod/kustomization.yaml
@ -0,0 +1,13 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: phonebill-prod
+
+resources:
+- ../../base
+
+commonLabels:
+  env: prod
+  
+patchesStrategicMerge:
+- env-patches.yaml
--- a/deployment/cicd/kustomize/overlays/prod/replica-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/prod/replica-patch.yaml
@ -0,0 +1,35 @@
+# Replica count patches for production environment
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: api-gateway
+spec:
+  replicas: 3
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: user-service
+spec:
+  replicas: 3
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: bill-service
+spec:
+  replicas: 3
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: product-service
+spec:
+  replicas: 3
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: kos-mock
+spec:
+  replicas: 2
--- a/deployment/cicd/kustomize/overlays/prod/secret-bill-service-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/prod/secret-bill-service-patch.yaml
@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-bill-service
+type: Opaque
+stringData:
+  DB_HOST: "bill-inquiry-postgres-prod-postgresql"
+  DB_PASSWORD: "BillUserProd$ecure2025!"
--- a/deployment/cicd/kustomize/overlays/prod/secret-common-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/prod/secret-common-patch.yaml
@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-common
+type: Opaque
+stringData:
+  REDIS_HOST: "redis-cache-prod-master"
+  REDIS_PASSWORD: "Redis2025Prod$ecure!"
--- a/deployment/cicd/kustomize/overlays/prod/secret-product-service-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/prod/secret-product-service-patch.yaml
@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-product-service
+type: Opaque
+stringData:
+  DB_HOST: "product-change-postgres-prod-postgresql"
+  DB_PASSWORD: "ProductUserProd$ecure2025!"
--- a/deployment/cicd/kustomize/overlays/prod/secret-user-service-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/prod/secret-user-service-patch.yaml
@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-user-service
+type: Opaque
+stringData:
+  DB_HOST: "auth-postgres-prod-postgresql"
+  DB_PASSWORD: "AuthUserProd$ecure2025!"
--- a/deployment/cicd/kustomize/overlays/staging/configmap-common-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/staging/configmap-common-patch.yaml
@ -0,0 +1,8 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cm-common
+data:
+  CORS_ALLOWED_ORIGINS: "https://phonebill-staging.example.com,https://phonebill.staging.example.com"
+  SPRING_PROFILES_ACTIVE: "staging"
+  DDL_AUTO: "validate"
--- a/deployment/cicd/kustomize/overlays/staging/ingress-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/staging/ingress-patch.yaml
@ -0,0 +1,15 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: phonebill
+  annotations:
+    kubernetes.io/ingress.class: nginx
+    nginx.ingress.kubernetes.io/ssl-redirect: "true"
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+spec:
+  tls:
+  - hosts:
+    - phonebill-api-staging.example.com
+    secretName: phonebill-staging-tls
+  rules:
+  - host: phonebill-api-staging.example.com
--- a/deployment/cicd/kustomize/overlays/staging/kustomization.yaml
+++ b/deployment/cicd/kustomize/overlays/staging/kustomization.yaml
@ -0,0 +1,19 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: phonebill-staging
+
+resources:
+- ../../base
+
+commonLabels:
+  env: staging
+  
+patchesStrategicMerge:
+- configmap-common-patch.yaml
+- secret-common-patch.yaml
+- ingress-patch.yaml
+- replica-patch.yaml
+- secret-user-service-patch.yaml
+- secret-bill-service-patch.yaml
+- secret-product-service-patch.yaml
--- a/deployment/cicd/kustomize/overlays/staging/replica-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/staging/replica-patch.yaml
@ -0,0 +1,35 @@
+# Replica count patches for staging environment
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: api-gateway
+spec:
+  replicas: 2
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: user-service
+spec:
+  replicas: 2
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: bill-service
+spec:
+  replicas: 2
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: product-service
+spec:
+  replicas: 2
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: kos-mock
+spec:
+  replicas: 1
--- a/deployment/cicd/kustomize/overlays/staging/secret-bill-service-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/staging/secret-bill-service-patch.yaml
@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-bill-service
+type: Opaque
+stringData:
+  DB_HOST: "bill-inquiry-postgres-staging-postgresql"
+  DB_PASSWORD: "BillUserStaging2025!"
--- a/deployment/cicd/kustomize/overlays/staging/secret-common-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/staging/secret-common-patch.yaml
@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-common
+type: Opaque
+stringData:
+  REDIS_HOST: "redis-cache-staging-master"
+  REDIS_PASSWORD: "Redis2025Staging!"
--- a/deployment/cicd/kustomize/overlays/staging/secret-product-service-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/staging/secret-product-service-patch.yaml
@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-product-service
+type: Opaque
+stringData:
+  DB_HOST: "product-change-postgres-staging-postgresql"
+  DB_PASSWORD: "ProductUserStaging2025!"
--- a/deployment/cicd/kustomize/overlays/staging/secret-user-service-patch.yaml
+++ b/deployment/cicd/kustomize/overlays/staging/secret-user-service-patch.yaml
@ -0,0 +1,8 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-user-service
+type: Opaque
+stringData:
+  DB_HOST: "auth-postgres-staging-postgresql"
+  DB_PASSWORD: "AuthUserStaging2025!"
--- a/deployment/cicd/scripts/deploy.sh
+++ b/deployment/cicd/scripts/deploy.sh
@ -0,0 +1,474 @@
+#!/bin/bash
+
+# Manual Deployment Script for phonebill Microservices
+# Usage: ./deploy.sh <environment> [service1,service2,...] [options]
+# Example: ./deploy.sh dev all --skip-build
+# Example: ./deploy.sh prod user-service,bill-service --force
+
+set -euo pipefail
+
+# Script configuration
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
+CICD_DIR="${PROJECT_ROOT}/deployment/cicd"
+K8S_DIR="${PROJECT_ROOT}/deployment/k8s"
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Logging functions
+log_info() {
+    echo -e "${BLUE}[INFO]${NC} $1"
+}
+
+log_warn() {
+    echo -e "${YELLOW}[WARN]${NC} $1"
+}
+
+log_error() {
+    echo -e "${RED}[ERROR]${NC} $1"
+}
+
+log_success() {
+    echo -e "${GREEN}[SUCCESS]${NC} $1"
+}
+
+# Usage information
+show_usage() {
+    cat << EOF
+Usage: $0 <environment> [services] [options]
+
+Arguments:
+  environment     Target environment (dev|staging|prod)
+  services        Services to deploy (default: all)
+                  Options: all, api-gateway, user-service, bill-service, product-service, kos-mock
+                  Multiple services: service1,service2,service3
+
+Options:
+  --skip-build    Skip Gradle build step
+  --skip-test     Skip unit tests during build
+  --skip-push     Skip container image push
+  --force         Force deployment even if no changes
+  --dry-run       Show what would be deployed without actually deploying
+  --help          Show this help message
+
+Examples:
+  $0 dev                              # Deploy all services to dev
+  $0 staging user-service             # Deploy user-service to staging
+  $0 prod api-gateway,bill-service    # Deploy specific services to prod
+  $0 dev all --skip-build             # Deploy without building
+  $0 staging all --dry-run            # Preview deployment
+
+Environment Files:
+  dev: ${CICD_DIR}/config/deploy_env_vars_dev
+  staging: ${CICD_DIR}/config/deploy_env_vars_staging
+  prod: ${CICD_DIR}/config/deploy_env_vars_prod
+EOF
+}
+
+# Parse command line arguments
+parse_arguments() {
+    if [[ $# -eq 0 ]] || [[ "$1" == "--help" ]]; then
+        show_usage
+        exit 0
+    fi
+
+    ENVIRONMENT="$1"
+    shift
+
+    # Validate environment
+    if [[ ! "$ENVIRONMENT" =~ ^(dev|staging|prod)$ ]]; then
+        log_error "Invalid environment: $ENVIRONMENT"
+        log_error "Valid environments: dev, staging, prod"
+        exit 1
+    fi
+
+    # Set services (default to 'all')
+    SERVICES_TO_DEPLOY="all"
+    if [[ $# -gt 0 ]] && [[ ! "$1" =~ ^-- ]]; then
+        SERVICES_TO_DEPLOY="$1"
+        shift
+    fi
+
+    # Parse options
+    SKIP_BUILD=false
+    SKIP_TEST=false
+    SKIP_PUSH=false
+    FORCE_DEPLOY=false
+    DRY_RUN=false
+
+    while [[ $# -gt 0 ]]; do
+        case $1 in
+            --skip-build)
+                SKIP_BUILD=true
+                shift
+                ;;
+            --skip-test)
+                SKIP_TEST=true
+                shift
+                ;;
+            --skip-push)
+                SKIP_PUSH=true
+                shift
+                ;;
+            --force)
+                FORCE_DEPLOY=true
+                shift
+                ;;
+            --dry-run)
+                DRY_RUN=true
+                shift
+                ;;
+            *)
+                log_error "Unknown option: $1"
+                show_usage
+                exit 1
+                ;;
+        esac
+    done
+}
+
+# Load environment configuration
+load_environment_config() {
+    local config_file="${CICD_DIR}/config/deploy_env_vars_${ENVIRONMENT}"
+    
+    if [[ ! -f "$config_file" ]]; then
+        log_error "Configuration file not found: $config_file"
+        exit 1
+    fi
+
+    # Source the configuration file
+    set -a  # automatically export all variables
+    source "$config_file"
+    set +a
+
+    log_info "Loaded configuration from $config_file"
+}
+
+# Validate prerequisites
+validate_prerequisites() {
+    local missing_tools=()
+
+    # Check required tools
+    command -v gradle >/dev/null 2>&1 || missing_tools+=("gradle")
+    command -v docker >/dev/null 2>&1 || missing_tools+=("docker")
+    command -v kubectl >/dev/null 2>&1 || missing_tools+=("kubectl")
+    command -v az >/dev/null 2>&1 || missing_tools+=("azure-cli")
+
+    if [[ ${#missing_tools[@]} -ne 0 ]]; then
+        log_error "Missing required tools: ${missing_tools[*]}"
+        log_error "Please install the missing tools and try again"
+        exit 1
+    fi
+
+    # Check if we're in the project root
+    if [[ ! -f "${PROJECT_ROOT}/settings.gradle" ]]; then
+        log_error "Not in phonebill project root directory"
+        exit 1
+    fi
+
+    # Check Azure login
+    if ! az account show >/dev/null 2>&1; then
+        log_error "Not logged into Azure CLI. Please run: az login"
+        exit 1
+    fi
+
+    # Check kubectl context
+    if ! kubectl config current-context >/dev/null 2>&1; then
+        log_warn "No kubectl context set. Will attempt to configure AKS credentials"
+    fi
+
+    log_success "Prerequisites validation passed"
+}
+
+# Resolve services list
+resolve_services() {
+    if [[ "$SERVICES_TO_DEPLOY" == "all" ]]; then
+        SERVICE_LIST=(${SERVICES//,/ })
+    else
+        IFS=',' read -ra SERVICE_LIST <<< "$SERVICES_TO_DEPLOY"
+    fi
+
+    log_info "Services to deploy: ${SERVICE_LIST[*]}"
+    
+    # Validate service names
+    local valid_services=(${SERVICES//,/ })
+    for service in "${SERVICE_LIST[@]}"; do
+        if [[ ! " ${valid_services[*]} " =~ " ${service} " ]]; then
+            log_error "Invalid service name: $service"
+            log_error "Valid services: ${valid_services[*]}"
+            exit 1
+        fi
+    done
+}
+
+# Build services
+build_services() {
+    if [[ "$SKIP_BUILD" == true ]]; then
+        log_info "Skipping build step"
+        return 0
+    fi
+
+    log_info "Building services with Gradle..."
+    cd "$PROJECT_ROOT"
+
+    for service in "${SERVICE_LIST[@]}"; do
+        log_info "Building $service..."
+        
+        local build_cmd="./gradlew ${service}:clean ${service}:build --no-daemon --parallel"
+        
+        if [[ "$SKIP_TEST" == true ]]; then
+            build_cmd="$build_cmd -x test"
+        fi
+
+        if [[ "$DRY_RUN" == true ]]; then
+            log_info "[DRY-RUN] Would execute: $build_cmd"
+        else
+            if ! $build_cmd; then
+                log_error "Build failed for $service"
+                exit 1
+            fi
+        fi
+    done
+
+    log_success "Build completed successfully"
+}
+
+# Build and push container images
+build_and_push_images() {
+    if [[ "$SKIP_PUSH" == true ]]; then
+        log_info "Skipping container image build and push"
+        return 0
+    fi
+
+    log_info "Building and pushing container images..."
+    
+    # Generate image tag
+    local timestamp=$(date +%Y%m%d-%H%M%S)
+    local build_number="${BUILD_NUMBER:-$(date +%s)}"
+    IMAGE_TAG="${build_number}-${ENVIRONMENT}-${timestamp}"
+    
+    log_info "Using image tag: $IMAGE_TAG"
+
+    # Login to ACR
+    if [[ "$DRY_RUN" == false ]]; then
+        log_info "Logging into Azure Container Registry..."
+        az acr login --name "$ACR_NAME"
+    fi
+
+    for service in "${SERVICE_LIST[@]}"; do
+        log_info "Building container image for $service..."
+        
+        local image_name="${REGISTRY_URL}/phonebill/${service}"
+        local service_dir="${PROJECT_ROOT}/${service}"
+        
+        if [[ ! -d "$service_dir" ]]; then
+            log_error "Service directory not found: $service_dir"
+            exit 1
+        fi
+
+        if [[ ! -f "${service_dir}/Dockerfile" ]]; then
+            log_error "Dockerfile not found: ${service_dir}/Dockerfile"
+            exit 1
+        fi
+
+        if [[ "$DRY_RUN" == true ]]; then
+            log_info "[DRY-RUN] Would build and push: ${image_name}:${IMAGE_TAG}"
+        else
+            # Build image
+            docker build \
+                -t "${image_name}:${IMAGE_TAG}" \
+                -t "${image_name}:latest-${ENVIRONMENT}" \
+                "$service_dir"
+
+            # Push image
+            docker push "${image_name}:${IMAGE_TAG}"
+            docker push "${image_name}:latest-${ENVIRONMENT}"
+        fi
+    done
+
+    log_success "Container images built and pushed successfully"
+}
+
+# Configure kubectl
+configure_kubectl() {
+    log_info "Configuring kubectl for AKS cluster..."
+    
+    if [[ "$DRY_RUN" == false ]]; then
+        az aks get-credentials \
+            --resource-group "$AZURE_RESOURCE_GROUP" \
+            --name "$AKS_CLUSTER_NAME" \
+            --overwrite-existing
+    fi
+
+    log_success "kubectl configured for $AKS_CLUSTER_NAME"
+}
+
+# Deploy to Kubernetes
+deploy_to_kubernetes() {
+    log_info "Deploying services to Kubernetes namespace: $AKS_NAMESPACE"
+    
+    # Ensure namespace exists
+    if [[ "$DRY_RUN" == false ]]; then
+        kubectl create namespace "$AKS_NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -
+    fi
+
+    for service in "${SERVICE_LIST[@]}"; do
+        log_info "Deploying $service..."
+        
+        local kustomize_path="${K8S_DIR}/${service}"
+        local overlay_path="${kustomize_path}/${KUSTOMIZE_OVERLAY}"
+        
+        if [[ ! -d "$overlay_path" ]]; then
+            log_error "Kustomize overlay not found: $overlay_path"
+            exit 1
+        fi
+
+        if [[ "$DRY_RUN" == true ]]; then
+            log_info "[DRY-RUN] Would deploy $service using: kubectl apply -k $overlay_path -n $AKS_NAMESPACE"
+        else
+            # Update image tag in kustomization.yaml if IMAGE_TAG is set
+            if [[ -n "${IMAGE_TAG:-}" ]]; then
+                local kustomization_file="${overlay_path}/kustomization.yaml"
+                if [[ -f "$kustomization_file" ]]; then
+                    # Backup original file
+                    cp "$kustomization_file" "${kustomization_file}.backup"
+                    
+                    # Update image tag
+                    sed -i "s|newTag:.*|newTag: ${IMAGE_TAG}|" "$kustomization_file"
+                fi
+            fi
+
+            # Deploy using kubectl + kustomize
+            kubectl apply -k "$overlay_path" -n "$AKS_NAMESPACE"
+            
+            # Wait for rollout to complete
+            kubectl rollout status deployment/"$service" -n "$AKS_NAMESPACE" --timeout=300s
+            
+            # Restore backup if exists
+            if [[ -f "${kustomization_file}.backup" ]]; then
+                mv "${kustomization_file}.backup" "$kustomization_file"
+            fi
+        fi
+    done
+
+    log_success "Deployment completed successfully"
+}
+
+# Perform health checks
+perform_health_checks() {
+    if [[ "$DRY_RUN" == true ]]; then
+        log_info "[DRY-RUN] Would perform health checks for deployed services"
+        return 0
+    fi
+
+    log_info "Performing health checks..."
+
+    for service in "${SERVICE_LIST[@]}"; do
+        log_info "Health checking $service..."
+        
+        local max_retries=${HEALTH_CHECK_RETRY:-10}
+        local retry_count=0
+        local is_healthy=false
+
+        while [[ $retry_count -lt $max_retries ]]; do
+            if kubectl get deployment "$service" -n "$AKS_NAMESPACE" -o json | \
+               jq -e '.status.readyReplicas == .status.replicas and .status.replicas > 0' >/dev/null 2>&1; then
+                is_healthy=true
+                break
+            fi
+
+            retry_count=$((retry_count + 1))
+            log_info "Health check attempt $retry_count/$max_retries for $service..."
+            sleep 30
+        done
+
+        if [[ "$is_healthy" == true ]]; then
+            log_success "$service is healthy"
+        else
+            log_error "$service failed health check"
+            kubectl describe deployment "$service" -n "$AKS_NAMESPACE"
+            exit 1
+        fi
+    done
+
+    log_success "All services passed health checks"
+}
+
+# Show deployment summary
+show_summary() {
+    log_info "========================================="
+    log_info "Deployment Summary"
+    log_info "========================================="
+    log_info "Environment: $ENVIRONMENT"
+    log_info "Namespace: $AKS_NAMESPACE"
+    log_info "Services: ${SERVICE_LIST[*]}"
+    
+    if [[ -n "${IMAGE_TAG:-}" ]]; then
+        log_info "Image Tag: $IMAGE_TAG"
+    fi
+    
+    log_info "Options:"
+    log_info "  Skip Build: $SKIP_BUILD"
+    log_info "  Skip Test: $SKIP_TEST"
+    log_info "  Skip Push: $SKIP_PUSH"
+    log_info "  Force Deploy: $FORCE_DEPLOY"
+    log_info "  Dry Run: $DRY_RUN"
+    log_info "========================================="
+}
+
+# Cleanup function
+cleanup() {
+    local exit_code=$?
+    if [[ $exit_code -ne 0 ]]; then
+        log_error "Deployment failed with exit code: $exit_code"
+        
+        # Show recent events for debugging
+        if [[ "$DRY_RUN" == false ]] && command -v kubectl >/dev/null 2>&1; then
+            log_info "Recent events in namespace $AKS_NAMESPACE:"
+            kubectl get events -n "$AKS_NAMESPACE" --sort-by='.lastTimestamp' --field-selector type=Warning | tail -10
+        fi
+    fi
+}
+
+# Main execution function
+main() {
+    # Set up error handling
+    trap cleanup EXIT
+
+    log_info "🚀 Starting phonebill deployment script"
+    
+    # Parse arguments and validate
+    parse_arguments "$@"
+    show_summary
+    
+    # Load configuration and validate prerequisites
+    load_environment_config
+    validate_prerequisites
+    resolve_services
+    
+    # Execute deployment steps
+    build_services
+    build_and_push_images
+    configure_kubectl
+    deploy_to_kubernetes
+    perform_health_checks
+    
+    log_success "🎉 Deployment completed successfully!"
+    
+    if [[ "$DRY_RUN" == false ]]; then
+        log_info "You can check the deployment status with:"
+        log_info "  kubectl get pods -n $AKS_NAMESPACE"
+        log_info "  kubectl get services -n $AKS_NAMESPACE"
+        log_info "  kubectl get ingress -n $AKS_NAMESPACE"
+    fi
+}
+
+# Execute main function if script is run directly
+if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
+    main "$@"
+fi
--- a/deployment/k8s/bill-service/cm-bill-service.yaml
+++ b/deployment/k8s/bill-service/cm-bill-service.yaml
@ -14,7 +14,6 @@ data:
  DB_MAX_POOL: "20"
  DB_MIN_IDLE: "5"
  KOS_BASE_URL: "http://kos-mock"
-  LOG_FILE_NAME: "logs/bill-service.log"
  REDIS_DATABASE: "1"
  REDIS_MAX_ACTIVE: "8"
  REDIS_MAX_IDLE: "8"
--- a/deployment/k8s/common/cm-common.yaml
+++ b/deployment/k8s/common/cm-common.yaml
@ -8,4 +8,5 @@ data:
  JWT_ACCESS_TOKEN_VALIDITY: "18000000"
  JWT_REFRESH_TOKEN_VALIDITY: "86400000"
  REDIS_PORT: "6379"
-  SPRING_PROFILES_ACTIVE: "dev"
+  SPRING_PROFILES_ACTIVE: "dev"
+  DDL_AUTO: "update"
--- a/deployment/k8s/product-service/cm-product-service.yaml
+++ b/deployment/k8s/product-service/cm-product-service.yaml
@ -7,8 +7,5 @@ data:
  SERVER_PORT: "8083"
  DB_KIND: "postgresql"
  DB_PORT: "5432"
-  DDL_AUTO: "update"
  KOS_BASE_URL: "http://kos-mock"
-  KOS_CLIENT_ID: "product-service-dev"
-  KOS_MOCK_ENABLED: "true"
  REDIS_DATABASE: "2"
--- a/deployment/k8s/product-service/secret-product-service.yaml
+++ b/deployment/k8s/product-service/secret-product-service.yaml
@ -8,5 +8,4 @@ stringData:
  DB_HOST: "product-change-postgres-dev-postgresql"
  DB_NAME: "product_change_db"
  DB_USERNAME: "product_change_user"
-  DB_PASSWORD: "ProductUser2025!"
-  KOS_API_KEY: "dev-api-key"
+  DB_PASSWORD: "ProductUser2025!"