🧬 Copilot-LD

An intelligent agent leveraging GitHub Copilot and Linked Data

Deployment Guide

This guide covers how to deploy and run Copilot-LD in production using either Docker Compose or AWS CloudFormation. Choose the deployment method that best fits your infrastructure requirements.

Prerequisites

Note: For GitHub Actions workflows, ensure S3 access credentials are configured as repository secrets.

Docker Compose Deployment

The Docker Compose deployment provides a complete containerized stack with application load balancing, object storage, and all microservices.

Architecture Overview

The Docker Compose stack includes:

The ALB handles inbound web traffic with SSL termination (ports 80/443) and routes requests to backend services. The separate gateway provides outbound internet connectivity (port 3128) for services that need external API access (such as the LLM service for GitHub Copilot API) via the HTTPS_PROXY environment variable. This separation mirrors the AWS deployment architecture.

SSL Certificate Setup

Configure SSL certificates for secure HTTPS access. See the SSL Certificate Configuration section in the Configuration Guide for detailed setup instructions including self-signed certificate generation and production certificate installation.

Configuration

Configure environment variables and service settings. See the Configuration Guide for complete details on all environment variables and YAML configuration options.

Deploy the Stack

The platform uses a unified Docker build process with a single Dockerfile that handles all components (services, extensions, and tools). Building the containers requires a GitHub Personal Access Token (PAT) with at minimum the read:packages scope. Put the token in config/.build_token and deploy the stack: 

GITHUB_TOKEN=$(cat config/.build_token) docker compose build
docker compose up -d
docker compose ps

The first command builds all container images, the second starts the complete stack, and the third checks service status.

Access Points

Once deployed, access the system via:

AWS CloudFormation Deployment

Deploy Copilot-LD on AWS using ECS Fargate with CloudFormation for production-grade scalability and managed infrastructure.

Architecture Overview

The AWS deployment separates ALB and gateway responsibilities for optimal security and cost management:

This architecture provides production-grade SSL certificate management through AWS Certificate Manager while eliminating the expensive NAT Gateway. The gateway handles only egress proxy functionality, while the ALB handles only SSL offloading and ingress routing, keeping backend services isolated on private subnets.

AWS OIDC Setup for GitHub Actions

Before using the GitHub Actions workflows for automated deployment, you must configure AWS OpenID Connect (OIDC) to allow secure, keyless authentication from GitHub Actions to your AWS account.

1. Create GitHub OIDC Identity Provider in AWS

First, create an OIDC identity provider in your AWS account using AWS CLI: 

aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com \
  --thumbprint-list 6938fd4d98bab03faadb97b34396831e3780aea1 \
  --thumbprint-list 1c58a3a8518e8759bf075b76b750d4f2df264fcd

Or create via AWS Console:

Reference: GitHub Docs: Configuring OIDC in AWS

2. Create IAM Role for GitHub Actions

Create an IAM role that GitHub Actions can assume. Save this as github-actions-role.json: 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Principal": {
        "Federated": "arn:aws:iam::YOUR_ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com"
      },
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": ["sts.amazonaws.com"]
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": [
            "repo:copilot-ld/copilot-ld:*"
          ]
        }
      }
    }
  ]
}

Create the role and attach the necessary policies:

aws iam create-role \
  --role-name GitHubActions-CopilotLD-Demo-Deploy \
  --assume-role-policy-document file://github-actions-role.json

aws iam attach-role-policy \
  --role-name GitHubActions-CopilotLD-Demo-Deploy \
  --policy-arn arn:aws:iam::aws:policy/CloudFormationFullAccess

aws iam attach-role-policy \
  --role-name GitHubActions-CopilotLD-Demo-Deploy \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess

aws iam attach-role-policy \
  --role-name GitHubActions-CopilotLD-Demo-Deploy \
  --policy-arn arn:aws:iam::aws:policy/AmazonECS_FullAccess

aws iam attach-role-policy \
  --role-name GitHubActions-CopilotLD-Demo-Deploy \
  --policy-arn arn:aws:iam::aws:policy/IAMFullAccess

aws iam attach-role-policy \
  --role-name GitHubActions-CopilotLD-Demo-Deploy \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess

aws iam attach-role-policy \
  --role-name GitHubActions-CopilotLD-Demo-Deploy \
  --policy-arn arn:aws:iam::aws:policy/SecretsManagerReadWrite

aws iam attach-role-policy \
  --role-name GitHubActions-CopilotLD-Demo-Deploy \
  --policy-arn arn:aws:iam::aws:policy/CloudWatchLogsFullAccess

aws iam attach-role-policy \
  --role-name GitHubActions-CopilotLD-Demo-Deploy \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess

These attach CloudFormation permissions, EC2 permissions for network infrastructure deployment, ECS permissions for service deployment, IAM permissions for role creation, S3 permissions for data management, Secrets Manager permissions, CloudWatch Logs permissions for log group management, and ECR permissions for container registry management.

Security Note: For production environments, replace *FullAccess policies with more restrictive, principle-of-least-privilege policies tailored to your specific deployment needs.

3. Configure GitHub Repository Secrets

Add the following secrets to your GitHub repository (Settings → Secrets and variables → Actions):

4. Workflow Usage

The configured OIDC role enables secure access in deployment workflows:

For detailed CloudFormation deployment commands and parameters, refer to the actual workflow files in .github/workflows/. These workflows contain the most current deployment procedures and parameter configurations.

Reference: AWS Docs: Creating OIDC Identity Providers

Automated Deployment

Use the GitHub Actions workflows for automated deployment. The workflows handle all CloudFormation stack creation, parameter management, and deployment orchestration. See the workflow files for specific implementation details and current deployment procedures.

Deployment Order

Deploy the CloudFormation stacks in the following order using the GitHub Actions workflows:

Each stack outputs the necessary parameters for the next stack in the deployment chain. The registry stack provides the ECR registry URL used by network, services, and extensions stacks. The network stack provides VPC and subnet IDs required by the services stack.

Note: Container images must be available in ECR before deploying services. During initial deployment, you can either:

  1. Use the release workflow to build and push images to ECR automatically
  2. Manually push images to ECR repositories using docker tag and docker push commands after authenticating with aws ecr get-login-password

Container Registry Management

The ECR registry stack creates repositories for all platform components with automatic lifecycle policies:

To manually push images to ECR:

# Authenticate Docker with ECR
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com

# Tag and push an image
docker tag copilot-ld/agent:latest $AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/demo/agent:latest
docker push $AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/demo/agent:latest

Data Upload

Upload your processed knowledge base to S3 for deployment. For complete data management instructions including upload configuration, download utilities, and storage management, see the Data Management Utilities section in the Processing Guide.

Production Considerations

Security

Next Steps

Once deployment is complete, you can proceed with extending or customizing the system: