Data Engineering 1

QUICK_REFERENCE_LAB2

7 min de lecture

QUICK REFERENCE: Lab 2 Deployment & Command Guide

🚀 Fast Navigation

Documentation Files

Main Overview:
  FINAL_LAB2_COMPREHENSIVE_SUMMARY.md   ← Start here!

Section Summaries:
  SUMMARY_LAB2.md                       (Sections 1-5)
  SECTION_6_SUMMARY.md                  (Modular architecture)
  SECTION_7_SUMMARY.md                  (GitHub modules concepts)

Deployment Guides:
  SECTION_7_DEPLOYMENT_GUIDE.md         (GitHub modules practical)

Exercise Explanations:
  scripts/EXERCISE_9_AND_10.md      (Parameterized & scalable)
  scripts/EXERCISE_11_AND_12.md     (Git versioning & public modules)
  scripts/packer/EXERCISE_5_EXPLANATION.md
  scripts/packer/EXERCISE_6_EXPLANATION.md
  scripts/tofu/EXERCISE_7_AND_8.md

📍 Directory Paths

Bash (Section 1)

cd /home/sable/devops_base/scripts/bash
ls -la deploy-ec2-instance.sh user-data.sh

Ansible (Section 2)

cd /home/sable/devops_base/scripts/ansible
ls -la *.yml inventory.aws_ec2.yml roles/sample-app/

Packer (Section 3)

cd /home/sable/devops_base/scripts/packer
ls -la *.pkr.hcl *.json EXERCISE_*.md

OpenTofu Modules (Section 6)

cd /home/sable/devops_base/scripts/modules/ec2-instance
cat main.tf variables.tf outputs.tf

OpenTofu Live Configurations (Sections 4-7)

# Single instance
cd /home/sable/devops_base/scripts/tofu/ec2-instance
 
# Multiple instances
cd /home/sable/devops_base/scripts/tofu/ec2-multi
 
# Static modules
cd /home/sable/devops_base/scripts/live/sample-app
 
# Scalable modules
cd /home/sable/devops_base/scripts/live/sample-app-scalable
 
# GitHub modules (Section 7)
cd /home/sable/devops_base/scripts/live/github-modules

🔧 Common Commands

OpenTofu Commands

# Navigate to a configuration directory
cd /home/sable/devops_base/scripts/live/sample-app
 
# Initialize (download providers)
tofu init
 
# Format code
tofu fmt
 
# Validate syntax
tofu validate
 
# Preview changes
tofu plan
 
# Deploy infrastructure
tofu apply -auto-approve
 
# View outputs
tofu output
 
# Destroy infrastructure
tofu destroy -auto-approve
 
# View state
tofu state list
tofu state show aws_instance.sample_app

Packer Commands

cd /home/sable/devops_base/scripts/packer
 
# Initialize plugins
packer init sample-app.pkr.hcl
 
# Format code
packer fmt sample-app.pkr.hcl
 
# Validate template
packer validate sample-app.pkr.hcl
 
# Build AMI
AWS_PROFILE=labs-devops_diallo AWS_REGION=us-east-2 packer build sample-app.pkr.hcl

Ansible Commands

cd /home/sable/devops_base/scripts/ansible
 
# Test inventory (dynamic AWS)
ansible-inventory -i inventory.aws_ec2.yml --list | head -20
 
# Validate playbook syntax
ansible-playbook create_ec2_instance_playbook.yml --syntax-check
 
# Dry run (show what would happen)
ansible-playbook create_ec2_instance_playbook.yml --check
 
# Run playbook
ansible-playbook create_ec2_instance_playbook.yml

AWS CLI Commands

# List instances in us-east-2
aws ec2 describe-instances \
  --region us-east-2 \
  --profile labs-devops_diallo \
  --query 'Reservations[*].Instances[*].[InstanceId,State.Name,PublicIpAddress,Tags[?Key==`Name`].Value|[0]]' \
  --output table
 
# List AMIs you created
aws ec2 describe-images \
  --owners self \
  --region us-east-2 \
  --profile labs-devops_diallo \
  --query 'Images[*].[ImageId,Name,CreationDate]' \
  --output table
 
# Describe instance types
aws ec2 describe-instance-types \
  --instance-types t3.micro \
  --region us-east-2 \
  --profile labs-devops_diallo

Testing Deployed Instances

# Get public IP from tofu output
INSTANCE_IP=$(tofu output -raw instance_public_ip)
 
# Test application (wait 30-60 seconds after deploy)
curl http://$INSTANCE_IP:8080/
 
# SSH into instance (if you have the key)
ssh -i ansible-ch2.key ec2-user@$INSTANCE_IP
 
# Check cloud-init logs
ssh -i ansible-ch2.key ec2-user@$INSTANCE_IP cat /var/log/cloud-init-output.log
 
# Test from multiple instances
for ip in 18.222.76.46 3.137.158.194 18.116.203.64; do
  echo "Testing $ip:"
  curl http://$ip:8080/
done

📊 Deployment Reference

Current Status

  • Total Exercises: 12 (all completed)
  • Deployments Created: 20+ EC2 instances throughout lab
  • Current Running: 0 (all cleaned up)
  • AMI Created: ami-07eb809c44dd0fcab (Packer-built)
  • Cost: Currently $0/month (all resources destroyed)

If You Want to Deploy Again

Option 1: Deploy Section 6 (Modular Architecture)

cd /home/sable/devops_base/scripts/live/sample-app
tofu apply -auto-approve
# Creates: 2 EC2 instances, 2 security groups
# Estimated cost: $0 (Free Tier) or ~$3/month per instance

Option 2: Deploy Section 6 Scalable

cd /home/sable/devops_base/scripts/live/sample-app-scalable
tofu apply -auto-approve
# Creates: 3 EC2 instances
# Estimated cost: $0 (Free Tier) or ~$9/month total

Option 3: Custom Deployment Count

cd /home/sable/devops_base/scripts/live/sample-app-scalable
# Edit main.tf, modify var.instances map to have desired number
# Then:
tofu apply -auto-approve

🧪 Testing Checklist

After Each Deployment

# 1. Verify tofu output
tofu output
 
# 2. Get public IP
INSTANCE_IP=$(tofu output -raw public_ip)
echo $INSTANCE_IP
 
# 3. Wait for instance initialization
echo "Waiting 30 seconds for app to start..."
sleep 30
 
# 4. Test application
curl http://$INSTANCE_IP:8080/
# Expected: "Hello from [hostname]!"
 
# 5. Verify security group allows traffic
tofu output security_group_id
 
# 6. Verify instance type
aws ec2 describe-instances --instance-ids $(tofu output instance_id) \
  --query 'Reservations[0].Instances[0].InstanceType' \
  --profile labs-devops_diallo
# Expected: "t3.micro"

🧹 Cleanup Commands

Destroy Everything

# Sample-app deployment
cd /home/sable/devops_base/scripts/live/sample-app
tofu destroy -auto-approve
 
# Sample-app-scalable deployment
cd /home/sable/devops_base/scripts/live/sample-app-scalable
tofu destroy -auto-approve
 
# GitHub modules (Section 7)
cd /home/sable/devops_base/scripts/live/github-modules
tofu destroy -auto-approve
 
# Any other deployments
cd /home/sable/devops_base/scripts/tofu/ec2-instance
tofu destroy -auto-approve
 
cd /home/sable/devops_base/scripts/tofu/ec2-multi
tofu destroy -auto-approve

Remove Terraform State Files

# Warning: Only do this if you're sure you want to delete resources!
cd /home/sable/devops_base/scripts/live/sample-app
rm -rf .terraform* terraform.tfstate*
 
cd /home/sable/devops_base/scripts/live/sample-app-scalable
rm -rf .terraform* terraform.tfstate*

📝 Key File Locations

Reusable Module (Foundation)

/home/sable/devops_base/scripts/modules/ec2-instance/
├── main.tf              # EC2 + Security Group resources
├── variables.tf         # Input variables
├── outputs.tf           # Output values
└── user-data.sh         # Application startup script

Module Variables (Customization)

# Quick reference - variables.tf
variable "ami_id" {
  default = "ami-07eb809c44dd0fcab"  # Packer-built AMI
}
 
variable "instance_type" {
  default = "t3.micro"  # Free Tier eligible
}
 
variable "port" {
  default = 8080  # Application port
}
 
variable "name" {
  # Set by calling module
}

Application Code (user-data.sh)

#!/bin/bash
# Installs Node.js binary (v16.20.0)
# Creates simple HTTP server on port 8080
# Returns: "Hello from [hostname]!"

🔐 Security Notes

Credentials & Keys

# AWS Credentials (do NOT commit!)
~/.aws/credentials          # Profile: labs-devops_diallo
~/.aws/config              # Region: us-east-2
 
# SSH Key for Ansible
/home/sable/devops_base/scripts/ansible/ansible-ch2.key
# Keep secure! Use: chmod 600 ansible-ch2.key

Best Practices Applied

✅ No credentials in code files
✅ No credentials in Git history
✅ Use AWS profiles (not access key ID in code)
✅ Security groups restrict port 8080 to 0.0.0.0/0 (for demo)
✅ t3.micro instances used (minimal exposed surface)

For Production

❌ Don’t use 0.0.0.0/0 for open ports
❌ Use VPC security groups properly
❌ Implement secrets management (AWS Secrets Manager)
❌ Enable VPC Flow Logs
❌ Use Systems Manager Session Manager (no SSH keys)

💡 Troubleshooting Quick Fixes

”Error: terraform not found"

# Solution: Use 'tofu' instead
tofu --version
# or alias: alias terraform=tofu

"Error: Creating EC2 Instance: VcpuLimitExceeded"

# Solution 1: Destroy unused instances
tofu destroy -auto-approve
 
# Solution 2: Request limit increase
# Go to AWS Console → Service Quotas
# Search for "EC2 On-Demand t3 instances"
# Request limit increase
 
# Solution 3: Use smaller instance type (nano)
# Edit variables.tf: instance_type = "t3.nano"

"Error: User-Data Script Not Running"

# Check logs on instance:
ssh -i ansible-ch2.key ec2-user@<IP> \
  cat /var/log/cloud-init-output.log
 
# Solution: Add to OpenTofu EC2 resource:
user_data_replace_on_change = true
 
# Wait longer for app to start:
sleep 60 && curl http://IP:8080/

"Error: Instance not responding on port 8080”

# Check security group allows traffic:
tofu output security_group_id
aws ec2 describe-security-groups \
  --group-ids sg-xxxxx \
  --profile labs-devops_diallo
 
# Check instance is running:
aws ec2 describe-instances \
  --instance-ids i-xxxxx \
  --query 'Reservations[0].Instances[0].State.Name'
 
# Wait for cloud-init to complete:
sleep 60
curl http://<IP>:8080/

📈 Next Learning Steps

Immediate (This week)

  • Read FINAL_LAB2_COMPREHENSIVE_SUMMARY.md
  • Review each exercise explanation document
  • Study module structure in modules/ec2-instance/

Short-term (Next 2 weeks)

  • Create GitHub account (if needed)
  • Push modules to GitHub repository
  • Create semantic version tags
  • Deploy Section 7 with GitHub modules

Medium-term (Next month)

  • Explore Terraform Cloud
  • Implement CI/CD pipeline (GitHub Actions)
  • Add monitoring (CloudWatch)
  • Create module testing framework

Advanced (3+ months)

  • Publish modules to Terraform Registry
  • Multi-cloud deployments
  • Kubernetes infrastructure
  • Disaster recovery implementation

📚 Recommended Reading

Official Documentation

  1. OpenTofu Language Reference
  2. Terraform Registry
  3. AWS Provider Documentation

Best Practices

  1. Terraform Best Practices
  2. Semantic Versioning
  3. Infrastructure as Code Principles

Advanced Topics

  1. State Management
  2. Module Design Patterns
  3. Testing Infrastructure (terratest)
  4. Infrastructure Pipelines (GitOps)

🎯 Success Criteria Checklist

By completing Lab 2, you should be able to:

✅ Write OpenTofu/HCL infrastructure code
✅ Create reusable modules
✅ Deploy EC2 instances at scale
✅ Manage infrastructure state
✅ Version infrastructure code with Git
✅ Use AWS CLI effectively
✅ Understand tool trade-offs
✅ Implement disaster recovery
✅ Collaborate on infrastructure code
✅ Deploy production-ready infrastructure

📞 Quick Help

Getting Instance Information

# List all running instances in us-east-2
aws ec2 describe-instances \
  --filter "Name=instance-state-name,Values=running" \
  --region us-east-2 \
  --profile labs-devops_diallo \
  --query 'Reservations[*].Instances[*].PublicIpAddress' \
  --output text
 
# Get instance details from OpenTofu state
cd /home/sable/devops_base/scripts/live/sample-app
tofu state show module.local_module_instance.aws_instance.sample_app

Environment Setup

# AWS credentials location
cat ~/.aws/credentials | grep labs-devops_diallo -A 3
 
# Verify AWS CLI access
aws sts get-caller-identity --profile labs-devops_diallo
 
# Set default region
export AWS_REGION=us-east-2
export AWS_PROFILE=labs-devops_diallo
 
# List available configurations
ls -la /home/sable/devops_base/scripts/*/

🎓 Conclusion

Lab 2 Completed Successfully! 🎉

You’ve progressed from manual infrastructure management to enterprise-scale, version-controlled, modular infrastructure automation.

Key Achievement: Mastery of modern Infrastructure as Code practices

Ready for: Production deployments, team collaboration, multi-cloud strategies

Recommended Next: Advanced Terraform, Kubernetes, or CI/CD pipelines

Last Updated: November 27, 2025
Lab 2 Complete - All 12 Exercises Completed
All Documentation Created - Ready for Production Use

Vue Graphique

Liens retour