Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions containers/devops-agent-operator/sample-app/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
FROM public.ecr.aws/docker/library/python:3.12-alpine
WORKDIR /app
COPY app.py .
CMD ["python", "app.py"]
141 changes: 141 additions & 0 deletions containers/devops-agent-operator/sample-app/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
# Sample App — CodeBuild 기반 데모

DevOps Agent Operator의 장애 감지 → 자동 트러블슈팅 데모를 위한 샘플 애플리케이션입니다.

CodeBuild로 이미지를 빌드하고 EKS에 배포하여, 버그 버전 배포 시 Operator가 자동으로 장애를 감지하고 데이터를 수집하는 전체 흐름을 시연합니다.

## 구조

```
sample-app/
├── app.py # 애플리케이션 코드 (데모 시 stable/buggy로 교체)
├── versions/
│ ├── stable.py # 정상 버전 (기본 config 사용)
│ └── buggy.py # 버그 버전 (APP_CONFIG 필수, 미설정 시 crash)
├── Dockerfile # 컨테이너 이미지 빌드
├── buildspec.yml # CodeBuild 빌드 스펙
├── demo-run.sh # 데모 실행 스크립트 (stable/buggy/status/cleanup)
├── deploy.sh # 로컬 Docker 빌드 & 배포
├── deploy-app.sh # ConfigMap 기반 배포 (이미지 빌드 없이)
├── build-in-cluster.sh # Kaniko 기반 클러스터 내 빌드
└── k8s-deployment.yaml.tpl # Kubernetes Deployment 템플릿
```

## 사전 요구사항

- EKS 클러스터 (Operator가 배포된 상태)
- AWS CLI 설정 완료
- CodeBuild 프로젝트 생성 (아래 설정 참조)
- ECR 리포지토리 생성
- AWS DevOps Agent에서 GitHub repository 연동 완료
- Agent가 장애 발생 시 관련 commit diff를 자동으로 추적하려면 소스 repository 연결이 필요합니다
- 설정 방법: [Connecting to GitHub](https://docs.aws.amazon.com/devopsagent/latest/userguide/connecting-to-cicd-pipelines-connecting-github.html)

## 환경 설정

`demo-run.sh` 상단의 변수를 자신의 환경에 맞게 수정합니다:

```bash
AWS_ACCOUNT_ID="<YOUR_AWS_ACCOUNT_ID>" # 예: 123456789012
AWS_REGION="<YOUR_AWS_REGION>" # 예: ap-northeast-2
ECR_REPO="sample-app" # ECR 리포지토리 이름
S3_BUCKET="<YOUR_CODEBUILD_SOURCE_BUCKET>" # CodeBuild 소스용 S3 버킷
CODEBUILD_PROJECT="<YOUR_CODEBUILD_PROJECT>" # CodeBuild 프로젝트 이름
```

## CodeBuild 프로젝트 설정

### 1. S3 소스 버킷 생성

```bash
aws s3 mb s3://<YOUR_CODEBUILD_SOURCE_BUCKET> --region <YOUR_AWS_REGION>
```

### 2. ECR 리포지토리 생성

```bash
aws ecr create-repository --repository-name sample-app --region <YOUR_AWS_REGION>
```

### 3. CodeBuild 프로젝트 생성

콘솔 또는 CLI로 다음 설정의 CodeBuild 프로젝트를 생성합니다:

| 항목 | 값 |
|------|-----|
| 소스 | S3 (`s3://<YOUR_CODEBUILD_SOURCE_BUCKET>/source.zip`) |
| 환경 이미지 | `aws/codebuild/amazonlinux2-x86_64-standard:5.0` |
| 권한 모드 | Privileged (Docker 빌드용) |
| 빌드 스펙 | `sample-app/buildspec.yml` |
| 환경 변수 | `AWS_DEFAULT_REGION` = `<YOUR_AWS_REGION>` |

CodeBuild IAM Role에 ECR push 권한과 S3 read 권한이 필요합니다.

## 사용 방법

### 데모 스크립트 (CodeBuild 사용)

```bash
cd sample-app/

# 정상 버전 빌드 & 배포
./demo-run.sh stable

# 버그 버전 배포 → CrashLoopBackOff 발생 → Operator 감지
./demo-run.sh buggy

# 상태 확인
./demo-run.sh status

# 정리
./demo-run.sh cleanup
```

### 로컬 Docker 빌드 (CodeBuild 없이)

```bash
# 환경 변수 설정
export AWS_ACCOUNT_ID="<YOUR_AWS_ACCOUNT_ID>"
export AWS_REGION="<YOUR_AWS_REGION>"

./deploy.sh
```

### ConfigMap 기반 배포 (이미지 빌드 없이)

```bash
./deploy-app.sh
```

## 데모 시나리오

### 정상 → 장애 발생 흐름

1. **stable 배포**: `app.py`는 기본 config(`{"app_name": "sample-app", "port": 8080}`)를 사용하므로 정상 동작
2. **buggy 배포**: `app.py`가 `APP_CONFIG` 환경변수를 필수로 요구하도록 변경. Deployment에 해당 환경변수가 없으므로 즉시 crash
3. **Operator 감지**: CrashLoopBackOff 상태를 감지하고, pod manifest/logs/events/node logs를 수집
4. **결과 확인**: S3에 인시던트 데이터가 저장되고, Webhook을 통해 DevOps Agent에 알림 전달

### 버그의 핵심

```python
# stable: 기본값 사용
config_str = os.environ.get("APP_CONFIG", '{"app_name": "sample-app", "port": 8080}')

# buggy: 환경변수 필수 → Deployment에 미설정 → crash
config_str = os.environ.get("APP_CONFIG")
if config_str is None:
sys.exit(1) # FATAL
```

실제 운영에서 발생할 수 있는 "리팩토링 시 기본값 제거" 패턴을 재현합니다.


## 트러블슈팅

| 증상 | 대응 |
|------|------|
| 빌드 실패 | `aws codebuild batch-get-builds --ids <BUILD_ID>` 로 에러 확인 |
| Pod가 계속 Running | stable 버전 배포됨. `./demo-run.sh buggy` 재실행 |
| Operator가 감지 안 함 | `kubectl rollout restart deploy/devops-agent-operator -n devops-agent-operator-system` |
| S3에 인시던트 없음 | Operator 환경변수에 `S3_BUCKET` 설정 확인 |
37 changes: 37 additions & 0 deletions containers/devops-agent-operator/sample-app/app.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
"""Sample application - stable version"""
import http.server
import json
import os
import sys


def load_config():
"""Load configuration from environment variable."""
config_str = os.environ.get("APP_CONFIG", '{"app_name": "sample-app", "port": 8080}')
config = json.loads(config_str)
return config


def main():
config = load_config()
port = config["port"]
app_name = config["app_name"]

class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
if self.path == "/health":
self.send_response(200)
self.end_headers()
self.wfile.write(b"ok")
else:
self.send_response(200)
self.end_headers()
self.wfile.write(f"Hello from {app_name}!\n".encode())

print(f"Starting {app_name} on port {port}")
server = http.server.HTTPServer(("", port), Handler)
server.serve_forever()


if __name__ == "__main__":
main()
104 changes: 104 additions & 0 deletions containers/devops-agent-operator/sample-app/build-in-cluster.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
#!/bin/bash
set -e

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"

# Configuration
AWS_ACCOUNT_ID="${AWS_ACCOUNT_ID:-<YOUR_AWS_ACCOUNT_ID>}"
AWS_REGION="${AWS_REGION:-<YOUR_AWS_REGION>}"
ECR_REPO="sample-app"
COMMIT_SHA=$(git -C "$REPO_ROOT" rev-parse HEAD)
IMAGE_TAG=$(git -C "$REPO_ROOT" rev-parse --short HEAD)
FULL_IMAGE="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${ECR_REPO}:${IMAGE_TAG}"

echo "=== Building sample-app in cluster (Kaniko) ==="
echo "Image: $FULL_IMAGE"
echo "Commit: $COMMIT_SHA"
echo ""

# 1. Create ECR repo if not exists
echo "[1/4] Ensuring ECR repository..."
aws ecr describe-repositories --repository-names "$ECR_REPO" --region "$AWS_REGION" 2>/dev/null || \
aws ecr create-repository --repository-name "$ECR_REPO" --region "$AWS_REGION" --output text >/dev/null

# 2. Create ConfigMap with build context (Dockerfile + source)
echo "[2/4] Creating build context..."
kubectl delete configmap sample-app-build-context --ignore-not-found -n default >/dev/null 2>&1
kubectl create configmap sample-app-build-context \
--from-file=Dockerfile="$SCRIPT_DIR/Dockerfile" \
--from-file=app.py="$SCRIPT_DIR/app.py" \
-n default

# 3. Run Kaniko build pod
echo "[3/4] Running Kaniko build..."
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: kaniko-build
namespace: default
spec:
restartPolicy: Never
serviceAccountName: default
initContainers:
- name: setup-context
image: busybox
command: ['sh', '-c', 'cp /build-context/* /workspace/']
volumeMounts:
- name: build-context
mountPath: /build-context
- name: workspace
mountPath: /workspace
containers:
- name: kaniko
image: gcr.io/kaniko-project/executor:latest
args:
- "--dockerfile=/workspace/Dockerfile"
- "--context=/workspace"
- "--destination=${FULL_IMAGE}"
volumeMounts:
- name: workspace
mountPath: /workspace
env:
- name: AWS_SDK_LOAD_CONFIG
value: "true"
- name: AWS_REGION
value: "${AWS_REGION}"
volumes:
- name: build-context
configMap:
name: sample-app-build-context
- name: workspace
emptyDir: {}
EOF

# Wait for build
echo " Waiting for build to complete..."
kubectl wait --for=condition=Ready pod/kaniko-build --timeout=10s 2>/dev/null || true
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/kaniko-build --timeout=300s 2>/dev/null || {
echo " Build logs:"
kubectl logs kaniko-build -c kaniko 2>/dev/null | tail -10
STATUS=$(kubectl get pod kaniko-build -o jsonpath='{.status.phase}')
if [ "$STATUS" != "Succeeded" ]; then
echo " ERROR: Build failed (status: $STATUS)"
kubectl delete pod kaniko-build --ignore-not-found >/dev/null 2>&1
exit 1
fi
}
echo " ✓ Image built: $FULL_IMAGE"
kubectl delete pod kaniko-build --ignore-not-found >/dev/null 2>&1

# 4. Deploy
echo "[4/4] Deploying..."
sed -e "s|__COMMIT_SHA__|${COMMIT_SHA}|g" \
-e "s|__IMAGE__|${FULL_IMAGE}|g" \
"$SCRIPT_DIR/k8s-deployment.yaml.tpl" > "$SCRIPT_DIR/k8s-deployment.yaml"
kubectl apply -f "$SCRIPT_DIR/k8s-deployment.yaml"
kubectl rollout status deployment/sample-app --timeout=120s

echo ""
echo "=== Done ==="
echo "Commit: $COMMIT_SHA"
echo "Image: $FULL_IMAGE"
kubectl get pods -l app=sample-app
25 changes: 25 additions & 0 deletions containers/devops-agent-operator/sample-app/buildspec.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
version: 0.2

env:
variables:
ECR_REPO: sample-app
AWS_ACCOUNT_ID: "<YOUR_AWS_ACCOUNT_ID>"

phases:
pre_build:
commands:
- echo Logging in to ECR...
- aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
- IMAGE_TAG=$(date +%Y%m%d%H%M%S)
- FULL_IMAGE=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$ECR_REPO:$IMAGE_TAG
- echo "Building image $FULL_IMAGE"
- ls -la && ls -la sample-app/ || true
build:
commands:
- echo Building Docker image...
- cd sample-app && docker build -t $FULL_IMAGE .
post_build:
commands:
- echo Pushing to ECR...
- docker push $FULL_IMAGE
- echo "Image pushed successfully - $FULL_IMAGE"
Loading