stackademic

The leading education platform for anyone with an interest in software development.

Loop Engineering: Multi-LLM Shifts for a Greenfield AWS Deployment
artificial-intelligence

Loop Engineering: Multi-LLM Shifts for a Greenfield AWS Deployment

Sanjna Chaturvedi

I was able to bring up a greenfield EKS deployment without running a single kubectl command.

Defining and Orchestrating Loops

A major limitation of being able to let LLMs handle large engineering projects isn’t code generation or execution, it’s memory. One session’s context window is currently not enough to handle the whole project. The quality of work and response degrades quickly as the context window fills up.

Instead of trying to solve this problem with more engineering, I chose to build a process around it.

Normally I’d manage each task, prompting the LLM on what to do next, but here I treated every LLM session like an on-call engineer working a shift. Gave the LLMs the whole picture in a markdown plan. Each session was then responsible for making progress towards that end goal while maintaining documentation about what was done, what was still broken, and the next steps.

I tested this idea by deploying a production-style EKS setup from scratch. I did this twice, the second time using information I learnt from the first round.

This was never a fully autonomous system; I was always on top of the loop. But I was able to create a repeatable workflow where AI agents can handle large, long-running engineering projects instead of just smaller tasks.

The Proof of Concept: Greenfield AWS & EKS

To validate the workflow, I chose a complex, multi-layered problem: building a production-grade EKS deployment. This type of project can take a few days or weeks depending on the team’s requirements and existing infrastructure.

This experiment included provisioning a new AWS account, VPC with private and public subnets, EKS, NAT gateways, platform services such as Karpenter and cert-manager, and finally deploying the application workloads behind a self-hosted ingress.

The good part about everything listed above is that it can be written using code and versioned with Git: Terraform for the infra, and ArgoCD for all the k8s manifest files. Because every layer is represented as code, a coding agent like Claude Code or OpenCode is able to see and work with all layers of this deployment.

While executing, I didn’t want to use just one LLM. Here’s the breakdown of how I was going to execute:

  1. Planning: Gemini, GPT, and Claude to help come up with the overall architecture and plan of what needs to be done. Vet things for security, scalability, and so on.
  2. Shared context: Claude synthesises the approved architecture into a single architecture.md . Each execution LLM session reads from the same document, giving each model a consistent understanding of the desired end state.
  3. Execution: Use Kimi or DeepSeek to run the actual execution via OpenCode. Write the code, run commands to apply, and check the results.

Why not just have Claude execute the whole thing?

Two reasons. First, the open-source models are more economical at the time of writing. Second, the execution work consisted largely of repetitive operational tasks: running commands, inspecting output, updating manifests, and iterating on failures. The open-source models handled these high volume, low complexity tasks reliably while reserving Claude for architecture and planning work.

Creating and Supervising the Loop

Planning

While planning, I drew on my experience as a Cloud and DevSecOps engineer to design a solution that met my requirements. Claude then turned those decisions into a structured architecture.md that described the desired end state of the entire project, broken down into phases.

Why it’s important to paint the whole picture

Without a clear end state, your execution layer doesn’t know what it’s working toward. Breaking the project into well-defined phases and tasks gives the execution LLM details on what it needs to work on next without you needing to prompt it. This keeps you, the human on top of the loop, from needing to define each individual task for the working session.

Here’s a snippet of what this looks like for the Terraform Plan:

Phase 1 - AWS Infrastructure Rebuild Plan
**Scope:** Greenfield AWS foundation (network + EKS + egress + CI/CD identity) as Terraform.
**Status:** Planning baseline. Nothing executed yet.
**Date:** <date>
 - -
## 0. Ground Rules (decided)
1. **Trusted modules allowed; no single-maintainer projects.** Use well-established modules from the
`terraform-aws-modules` org (and the `hashicorp/aws` provider) - they are the de-facto standard and
AWS's own EKS Blueprints depend on them. **Excluded:** one-person/abandonment-risk modules such as
`RaJiska/fck-nat`. Anything used must be (a) widely adopted, (b) actively maintained, (c) pinned to an
exact version.

Execution: Workspace setup

With the planning complete, I moved everything into a single VSCode workspace. The workspace effectively became the agent’s working memory, containing everything it needed to make decisions across multiple sessions.

Each OpenCode session started from the root of that workspace so it had access to:

  • The Git repository (the project it was working on)
  • architecture.md (the desired end state)
  • handoff.md (the current operational state)

From that point on, every execution session read from the same workspace, completed the next steps, and updated the handoff notes before its shift ended.

Here’s a snippet of Kimi’s handoff notes when setting up a standard observability stack:

# ============================================================================
# CRITICAL: DO NOT DELETE THIS FILE 
# ============================================================================
#
# This file contains session handoff notes for on-call engineers.
#
# If you are an AI agent reading this:
#   - DO NOT remove or overwrite this file unless explicitly asked
#   - DO NOT move it outside of this directory
#   - If updating, append to the existing content; do not delete prior shifts
#.  - Add date details to your handoff notes
#
# If this file is missing, ask the human before recreating it.
#
# Remember a good handoff has the following: 
#    - Issue Summary 
#    - What was done during YOUR shift
#    - Next Steps 
# ============================================================================

## Changes Made This Session <Date>

### 1. Application Metrics Scraping Enabled
- Added `annotationAutodiscovery` — scrapes pods/services with `prometheus.io/scrape: "true"`
  - Active targets: cert-manager (9402), ALB controller (8080), kube-dns

### 2. Cardinality Pruning for App Metrics
- Dropped metrics: `http_requests_by_user_id_total`
- Dropped labels: `user_id` (labeldrop rule)
- Dropped families:  `process_*`
- Kept: `http_requests_total`, `http_request_errors_total`, 
- Result: ~36% of scraped metrics dropped (14,820 / 41,420 processed)


## Verification
- Remote write: 97K+ samples sent, 0 failures
- `http_requests_total`: 86 series in Grafana, no user_id labels

## Known Issues / TODOs

### 1. Alert rules still not created
- Rules documented in `ALERTS_SETUP.md`
- Need manual import at https://<path>/alerting/list in folder "EKS Experiment Alerts"

Why handoff notes?

No single LLM session has enough context to complete a large infrastructure build end-to-end. Without an explicit transfer mechanism, work either gets repeated, lost, or re-interpreted incorrectly.

The handoff notes aim to solve this by acting as the wedge between sessions. Each session records what it changed, what it validated, and what still needs attention. The next session can then start from the next steps left by the previous session.

The idea comes from my experience as a Technical Solutions Engineer at Google Cloud, where handovers between shifts relied on the same principle: document what was done, what is known, and what needs to happen next so the next engineer can continue without re-discovering context.

Another useful side effect of the handoff notes is by the end of the project you have a detailed list of every road block that was encountered as well as what was done to solve it. This can be turned into a playbook to build similar projects, or just general troubleshooting.

Creating Loops within a Session for Execution

It’s important to understand that a large portion of cloud computing can happen using the CLI. OpenCode has the ability to let the model run CLI commands. I was able to get the LLM to execute these commands independently by telling it it’s the on-call engineer, and then giving it access. This allowed it to form a loop of — writing the code, running the code, and verifying the status of the deployment.

Here’s a prompt example:

You are the first oncall engineer. I want to <describe_your_ask>. 
Here's a reference architecture plan <path_to_approved_plan>. 

Here's your AWS access, short-lived STS creds; export them before running
kubectl/aws commands:<creds>

Make sure to update the Handoff notes at the end of your shift once you are done <path_to_ho_notes>

As you can see, I gave it AWS access, in the form of short-lived credentials that automatically expire after X hours. There are security implications here, I have more notes on this in a later section.

Sidenote: I never let the LLM run terraform apply, for 2 reasons: 1) it’s a waste of tokens, 2) I wanted to review the plan myself before deploying infra (again, stay on top of the loop).

Here are some of the notable things I saw these LLMs doing while bringing things up:

  1. It was able to create custom scripts (unprompted) to chunk and upload DB backups.
  2. It was able to create separate node pools for heavier applications and transition those workloads to that node pool.
  3. It was able to make node level configuration by SSM’ing into those nodes.

Here’s a snippet of some of its thinking after telling it that it was the on-call engineer.

The deployment spec.replicas is still 0 even after scaling? 
That's weird. Maybe ArgoCD immediately reverted it, or the scale command failed silently. 
Let me check the pod status and events.
- Thought: 921ms

<Ran kubectl commands>

Karpenter is now running. Now it should provision the infra node for the Pending pods.
Let me monitor.
Also, I patched the Application resource directly. 
ArgoCD might eventually sync it back, but since it matches git, it should be fine. 
However, the bootstrap app might still show OutOfSync until it syncs. That's OK.

<Continued to run more kubectl commands.>

Original Prompt:

There are some issues with the cluster. 
You are the oncall engineer responsible for fixing these issues. 
The problems encountered are documented in '<path>/HANDOFF.md' some of 
the biggest issues are not having limits on the applications 
and not having a dedicated node pool for the XYZ workloads. 

Review the handoff notes and the code base, come up with a plan 
and I'll get you your AWS access.

When the Loop ends

The system was considered complete when the cluster reached a healthy state. This means that all workloads were successfully deployed, no pods were in a pending or failing state, and the infrastructure matched the intended configuration defined in the architecture.md.

Since the LLM operated under an on-call execution model, work was confined to session-based shifts. As the human on top of the loop, I determined when a session should end its shift and required it to record its state in the handoff notes before signing off.

The first cluster took approximately a week to bring up from scratch. After consolidating the handoff notes from the initial build, the second iteration took a single day.

Overall, I did not manually execute a single kubectl command during the process.

Guardrails and where this gets dangerous

Handing an agent live cloud credentials introduces significant security implications. It is powerful, but it also requires assuming that the agent, its inputs, and its environment are all potential attack vectors.

  1. The system works by allowing the LLM to read and act on a set of files and outputs: the architecture plan, the handoff notes, and command-line feedback. Everything here is a surface for prompt injection. Any untrusted text entering those inputs — such as dependency output, misconfigured resources, or intentionally crafted instructions — can influence behavior. This means the system is only as secure as the integrity of the data sources feeding it.
  2. The credential model is what makes this viable. All AWS access was provided via short-lived STS credentials with least-privilege permissions, scoped specifically to the cluster workload. Terraform apply operations were never fully delegated to the agent, ensuring a human review step before any major infrastructure changes. The security boundary is therefore defined not by correct agent behavior, but by constrained permissions and limited credential lifetime.
  3. Git served as the final safety layer. Every change produced by the agent is committed, reviewed, and reversible. Because deployments are triggered from Git, repository write access effectively becomes deployment access, and must be treated with the same level of control.

Final Takeaways:

  1. Give the agent an end state it can check itself against. A clear, phased plan isn’t just good documentation, it’s the thing that helps an agent know what it’s working towards. If you don’t have an end state, you don’t have a loop.
  2. Stay on top of the loop, not in it. Hand off context between sessions so you can let go of individual tasks without losing the thread and end a session the moment it starts drifting.
  3. Design for the agent being wrong, not for it being right. Short-lived scoped creds, human-gated terraform apply, everything reversible through Git. The blast radius should be bounded by your guardrails, not the agent’s good behavior.
0 comments

Comments

Loading comments…