A developer vibe-coding a side project a dozen people will ever run, and a team keeping a ten-year-old enterprise system alive for another quarter, share almost no constraints worth naming, and most of the advice in circulation is really one of those two people telling the other how to live.

The AI development space changes so quickly it’s hard to tell what’s real and what’s hype. Further complicating things, an approach can be valid for a lot of other folks’ day to day, but not mine. It seems I regularly see a new project with 10K+ stars and say to myself, “Why would I ever need that?” The impetus for this post is my own desire to have some clarity on what seems to be working and when to apply it.

This post surveys current thoughts and patterns in AI engineering as of August 2026. It uses a number of videos from the AI Engineer World’s Fair conference in San Francisco, a conference I’ve found to be a good proxy for the current state of the conversation. It also includes a number of AI engineering blog posts from various companies, as well as posts from practitioners that I’ve found high signal.

My goal is to convey accurate information about the current landscape and to provide references of resources for others to learn from. I voice some personal opinions in order to spark discussion, but try to identify them as such. Let’s dive in.

The Agentic Era

“we haven’t even coalesced on a definition of what an agent is, even though we’re well into the agentic era.”

Across everything I’ve watched and read, I don’t know that I’ve seen even two people agree on the same definition of an Agent.

A few years ago most definitions of an Agent were roughly: LLM + memory + tools + planning + action. However, more recent definitions move the deterministic layer (tools, memory, etc) under the term harness. The definition of an Agent then becomes a harness plus the LLM it drives. Popular harnesses today include Claude Code, Codex, OpenCode, etc.

Splitting the harness from the LLM gives us a catch-all term to discuss that deterministic layer separate from the LLM. Folks using the same harness can author static configurations, allowing for version controlled experimentation or sharing.

At Scale

A major benefit of the Agent abstraction can be realized once you adopt the constraint of “one task per Agent”. Schroeder calls these domain specific Agents. Domain specific Agents have been shown to have better overall performance on their task, as well as a reduced cost.

In addition to improved cost and performance, this small change allows us to think of Agents as functions that can be tested, parallelized, looped, and scaled. Techniques that software engineers have spent decades utilizing and building infrastructure around.

Topologies for Agent orchestration seem varied, often customized for the organization or task at hand. A few companies have seized on this trend, such as Vercel or Anthropic, by creating Agent management SDKs, turning Agents into something you can install.

Others opt for bespoke in-house solutions. Cloudflare fans PR reviews to a panel of specialist Agents with a coordinating Agent managing them. DoorDash recently set up a cloud Agent platform, providing their developers with tools like sandboxes, an MCP gateway, and playbooks, letting teams run their own Agents.

Context is Everything

“the LLM is a new kind of a computer… it’s kind of like the CPU equivalent. The context windows are kind of like the memory, and then the LLM is orchestrating memory and compute for problem solving.”

An area of much discussion across all of these resources is the idea of curating the tokens you’re sending to the LLM to be highly relevant to the task and as few as possible. This process used to be called prompting, but today broadly falls under the term context engineering or context management. The rationale for context engineering can be seen across a few different categories.

Context Rot

Context Rot is the first of our categories, which was first introduced in a technical report by Chroma Context Rot: How Increasing Input Tokens Impacts LLM Performance. The basic idea behind Context Rot is that more tokens measurably “distract” the LLM, as its attention is spread thinner across the increased tokens.

This means even though a model may support a context window of a million tokens, you likely don’t want to be using even close to all of them.

“If you’re just getting started with AI, try to keep it around 100,000 tokens. For [a] larger million[-token] context window, we probably revise this up to like 200,000 tokens - but I’ve regularly tried to keep it under 60[k] for the hardest problems… One of the telltale signs that you’re in the dumb zone is… you’re 200,000 tokens in and the model’s finished some work and it’s trying to get the test to pass and it’s like not [passing].”

This quote caught me off guard the first time I heard it, but the more I thought about it, the more it made sense. While long chat sessions can be good, we’ve all had those conversations where it’s going great and then suddenly the LLM is forgetting things or hallucinating. I’ve actually added a warning to my Claude Code config that changes the context window to orange when I’ve reached 100k tokens, as a reminder to reset.

Another technique I’ve found useful for keeping chat sessions focused is heavy utilization of Claude Code’s subagents and dynamic workflows. I keep the main chat focused on the overall goal, i.e., “plan feature X”, and have it spawn agents for anything that might distract it from that task. This can include things like reading code, exploring data in Snowflake, checking latency in Datadog, or even web searches. The subagents come back with the information we need to make a decision, and the main chat stays on track.

Cost

The second category is the cost savings that come with using fewer tokens. Being deliberate about what gets sent to the model means less input cost as well as reduced back-and-forth cost (output).

Perhaps surprisingly, Rajkumar Sakthivel (Tesco) found that for their coding tasks ~90% of the tokens are input (big codebase/context in, small diff out). Despite the fact that output tokens are priced 4-8x higher than input tokens, reducing input tokens worked out to roughly ~61% of their dollar bill.

Where your tokens actually go Where your tokens actually go (Rajkumar Sakthivel, Tesco, 2:57)

Solutions In Practice

Skills

A popular solution to context management is skills. Skills have exploded onto the AI development scene and were the headline of several talks at the conference.

Skills are a markdown file spec created by Anthropic, that gives the LLM instructions on how to complete a task. Skills are an improvement over normal markdown files due to their utilization of progressive disclosure. Skills have a yaml metadata “frontmatter” that harnesses read, rather than the entire skill body. This metadata is then used to selectively load the skill content at runtime when the Agent deems it relevant.

“If you do something ≥ once a week, make it a skill… Think of skills as workflows written in markdown.”

Anecdotally, skills have been incredibly valuable to the ML search team as day-to-day workflows. This includes things such as PR reviewing, data exploration, working with Datadog, and even Optimizely experiment generation.

However, skills aren’t a silver bullet. Philipp Schmid of Google DeepMind found that while human-generated skills improved model performance on task, AI-generated skills performed worse on tasks than a model with no skills at all.

Perhaps dating myself, I see skills as the AI generation’s jQuery. Skills are an extremely useful abstraction for the problems we’re having now. However, I suspect that in five or ten years we’ll have different abstractions that can better model the nuances of working with Agents.

Compression

Another way to keep a lean context window is to utilize compression techniques to send the same information with fewer tokens. Headroom does this via a proxy server, inspecting your outbound payload for code and compressing it. In addition to compression, Tesco passes only function names and descriptions rather than the entire thing.

“Instead of sending whole files, the AI search[es] and index[es]. It gets back only [the] small piece of code it actually needs.”

Toon takes a different approach to compression, opting to replace JSON with a structured data format explicitly designed to use fewer tokens. Projects like AXI take this a step further, creating a set of rules for building CLI tools and MCP servers in a way that measurably uses fewer tokens.

Long Term Memory

“every company on this earth is about to need a brain - the memory layer that means you never have to re-ask what you knew.”

Since LLM sessions are stateless, important data must be persisted in some form. While sometimes a pain, this has the added benefit of allowing us to pick and choose only the most important information for the context window.

I’ve seen this show up in a few places as the “Company Brain”. Y Combinator believes all companies need to create a company brain. Cloudflare built a company brain when they realized that institutional knowledge “became harder to recover when people moved between teams.” Eugene Yan puts it simply: “Connect models to your organization’s context”.

The simplest way to do this is via a folder on your disk, some sort of Markdown vault (Obsidian, Tolaria, etc.), or per-project markdown documentation. However, the markdown-as-memory approach comes with its own pitfalls. Cloudflare notes that markdown memory “rots incredibly fast” and requires constant updates as things change. Personally I’ve had Claude recite “facts” to me on a number of occasions that can be traced to an outdated markdown file or code comment somewhere.

Another markdown pitfall is that it’s easy to load the whole document into context when only a small subset is required. Common ways to avoid this are to have the Agent use grep, tail, head or jq to parse relevant data from files. ast-grep is useful for codebases, allowing Agents to seek out specific programming language patterns. Cloudflare takes a different approach for the Codex brain, parsing relevant statements out and loading only those into Codex.

Non-markdown long term memory options are also gaining in popularity. These include projects such as Letta, Mem0, or Zep. The materials I reviewed for this post don’t really cover these. At a glance most seem to be abstractions on top of databases and vector embeddings that allow an Agent to access information semantically.

Loops

“My job is to write loops.”

I’m almost certain everyone reading this knows what a loop is. It’s so ingrained into our day to day that we overlook that almost everything can be reduced to a loop. The AI development space has been rediscovering loops lately.

I see AI Agent loops as largely falling under two categories: what I call the Agent loop and the Improvement loop.

Agent Loop

The Agent loop is about having a single Agent iterate until it has achieved a given task. This doesn’t necessarily have to be writing code. Uber Eats uses multi-modal Agent loops to augment product images.

Kyle Mistele of HumanLayer maps the Agent loop onto classical systems design control loop:

Agentic control loops Agentic control loops (Kyle Mistele, HumanLayer)

We give the Agent loop our desired state, which is then compared to the current state, generating a list of changes. The controller, usually heuristics or perhaps the Agent itself, selects a change to make. The Agent executes the change and commits it as the “new state”. The changes are run through a set of sensors (validators) and any violations are surfaced to the controller. A new Agent, with a fresh context window, picks up the next task and the loop repeats until the desired state is reached.

Giving an Agent loop a set of well-defined tasks and letting it run through them is an effective way to iterate towards a desired goal. As we saw in the previous section, a fresh context window helps keep the Agent task focused. However, if you don’t give the Agent some form of deterministic guardrails, it can quickly spiral out of control.

The Great Loops Debate has panelists argue points from opposite ends of the autonomy curve. Geoff Huntley (creator of the Ralph loop) and Ian Livingstone (Keycard) argue for the YOLO version of the Agent loop (Ralph Wiggum, /goal) where the Agent decides when the loop is completed. Dex Horthy (HumanLayer) and Greg Pstrucha (Sentry) argue that a human needs to decide when the loop is complete, citing non-deterministic LLM behavior. It’s worth noting that both sides agree that deterministic validation is important in agentic loops.

“I heavily exploit pre-commit hooks, folks… and I engineer in that back pressure by analyzing the work that is done.”

Personally, I think both types of loops have their place. Goal loops can be incredibly valuable for vibe coding one-off scripts or standing up a greenfield project. However, if I have to maintain critical systems or ship code at scale, I think I want a human in the loop.

Improvement Loop

The Improvement loop operates at a meta-level, across multiple iterations of an Agent. The output of the Agent loop is assessed on task performance and future iterations of the Agent loop are modified with the goal of improved performance on the task.

A real-world example of the Improvement loop is Uber Eats’ Agent tuner:

How Uber tunes an Agent How Uber tunes an Agent (Soumya Gupta & Jai Chopra, Uber)

Their Agent is fed a base prompt and human-labeled data. It performs a task, in this case choosing what images a user sees, and the output is scored against unseen samples. A different Agent suggests changes to the prompt for the initial Agent, and the loop starts over. This loop itself is part of a larger pipeline run on real-time user data, allowing Uber to ship improved Agent loops in near real-time.

Annabell Schäfer shows how Langfuse used an Improvement loop to have a large model (Opus 4.8) improve a smaller model’s (GPT-5.4 nano) performance on a classification task. The large model’s prompt changes are seeded via human domain knowledge.

Lilian Weng has an excellent survey of current research into Agent harness Improvement loops. Some papers continue refinement via prompt and context adjustments. Others move to an even higher level, experimenting with loops that modify harness code or model weights.

Taming the Stochastic Core

“People worry about hallucinations, but that’s the feature… We hallucinate in a way. We imagine things that may not exist, and then we turn them into reality. And that’s what large language models do.”

Code is deterministic. Given the same inputs you’ll get the same outputs. This is something software developers rely on every day: Does this code compile? Do my tests pass? The LLM core of an Agent is non-deterministic, making it difficult to guarantee it will do what you expect. As we’ll see, the AI development community utilizes both deterministic and non-deterministic methods together to keep Agents pointed in the right direction.

Verification should be multilayered Verification should be multilayered (Tariq Shaukat, Sonar, 12:06)

Tests, Linters, and Compilers

“I think of verification as a ladder. The bottom is cheap and deterministic; the top is expensive and requires judgement.”

Our existing deterministic verifiers allow us to perform a reliable “this must pass” check on LLM output. Since by definition deterministic checks are repeatable and reliable, any check that can be made deterministic in an Agent loop likely should be.

As we saw in the agent loop diagram above, the same compilers, tests, and linters that we use day to day are part of the Agent loop definition. Geoff Huntley recommends utilizing git pre-commit hooks as a window to catch violations, and echo “prompts” back to the Agent, letting it know where the boundary is.

Ontologies

An ontology is a deterministic rules layer that sits on top of a fact-based data graph. These rules define valid data types and relationships, giving a deeper semantic meaning to the data. Take this example of a teacher-student relationship:

“So if I say teaches has a domain of teacher, that means if I say ‘Bob teaches Scooter’ in my text, I can infer that Bob is a teacher. And if I say ‘all teachers are persons,’ then this statement lets me know, if I say ‘Bob teaches Scooter,’ now I know Bob is a person, Bob is a teacher. What about Scooter? If I say teaches has a range of student, that means the right side of the verb, then Scooter is a student. And now I have this extra information into my system.”

An Agent can construct a structured statement that is then deterministically fact-checked by the ontology. Coyle discusses two ontology specifications which were designed for the early semantic web over 20 years ago. RDFS defines a language for schema and vocabulary. OWL builds on top of RDFS adding richer ontology axioms.

Emil Eifrem of Neo4j posits that every business needs an ontology substrate with three pillars:

Ontology-based semantic layer, three pillars Ontology-based semantic layer, three pillars (Emil Eifrem, Neo4j, 6:12)

This ontology substrate prevents teams from needing to re-discover their information sources every time they build an Agent. It’s the data version of keeping your code DRY.

Both ontology talks outline two methods of building ontologies. The top-down approach is where domain experts define entities and their relationships. The bottom-up approach takes existing production execution traces and codifies them into ontologies.

Evals

“Code is provable, but when you start dealing with large code bases, software is not. It’s still very complex. It is still very messy.”

Evals are a set of tests you run on Agent outputs to measure how well a given Agent configuration is doing on a task over time. They play a critical part in the Improvement loop. Low initial evals are not necessarily bad. Rather, they highlight areas for improvement. With proper tuning an Agent can improve its eval pass rate over time.

Uber considers human labels the source of truth for use in their evals. They give human labelers an objective guideline for labeling data that is usually in a simple form (yes, no, unsure).

When building evals it’s important to use real-world data. This can include obvious examples like user signals, but can also include less obvious sources such as logs or system traces. Uber logs as much data as they can, allowing them to map production data to user segments where the Agent can improve.

A pitfall to watch out for with evals comes in the form of reward hacking, where the Agent learns how to game the metric rather than complete the task. This can even include scenarios such as Agents looking up prior runs to cheat (Schmid, 18:48). Lilian Weng’s article advises combatting reward hacking with a gating secondary “held-out” eval set that the Agent cannot see. Uber does something similar with a secondary recall guardrail metric, preventing the Agent from regressing in other areas to pass a specific eval.

Models Watching Models

One of the more popular forms of non-deterministic validation is using one LLM to grade the output of a different LLM against a given criterion. This process is often referred to as LLM as a Judge. Teams often reach for this form of verification when a grading task is complex or doesn’t necessarily have a verifiable correct answer.

LLM as a Judge shouldn’t be used lightly, however. Its non-deterministic nature means that a judge could pass an input one run and fail it the next. Cloudflare builds resilience into their judge architecture by having a panel of specialized judge Agents that feed verdicts to a coordinating Agent responsible for the final judgment. Judge model architectures also have the hidden overhead of needing an Improvement loop, evals, and data to ensure alignment to the task.

References

Talks

The Future Is Domain-Specific Agents

Justin Schroeder, StandardAgents · AI Engineer World’s Fair 2026 · https://www.youtube.com/watch?v=spNAUEgq_A8

The Great Loops Debate (panel)

Dex Horthy (HumanLayer), Geoff Huntley, Ian Livingstone (KeyCard), Greg Pstrucha (Sentry) · moderated by Allie Howe (Insecure Agents) · AI Engineer World’s Fair 2026 · https://youtu.be/c35YoMdnI78

We Cut 94% of AI Coding Tokens With a Local Code Index

Rajkumar Sakthivel, Tesco · AI Engineer World’s Fair 2026 · https://www.youtube.com/watch?v=dRmWYHuIJxM

Don’t Ship Skills Without Evals

Philipp Schmid, Google DeepMind · AI Engineer World’s Fair 2026 · https://www.youtube.com/watch?v=0vphxNt4wyk

Every company should have a Brain

Garry Tan, Y Combinator · AI Engineer World’s Fair 2026 · https://www.youtube.com/watch?v=eBUyTS7SzV4

Building Closed-Loop Evals for a Multimodal Agent at Scale

Soumya Gupta & Jai Chopra, Uber · AI Engineer World’s Fair 2026 · https://www.youtube.com/watch?v=31GUkCBD-Uc

Loop Engineering from First Principles

Kyle Mistele, HumanLayer · AI Engineer World’s Fair 2026 · https://www.youtube.com/watch?v=xIt_mTQp6mY

Stop Burning Tokens: Why self-improvement needs domain expertise first

Annabell Schäfer, Langfuse · AI Engineer World’s Fair 2026 · https://www.youtube.com/watch?v=eAXxdtNlK04

Why Agentic Systems Need Ontologies

Frank Coyle, UC Berkeley · AI Engineer World’s Fair 2026 · https://www.youtube.com/watch?v=Sir59K8ZDPU

Thinner Agents on a Smarter Substrate: The Ontology-based Semantic Layer

Emil Eifrem, Neo4j · AI Engineer World’s Fair 2026 · https://www.youtube.com/watch?v=VGN22pPpb-8

In the Land of AI Agents, the Verifiers Are King

Tariq Shaukat, Sonar · AI Engineer World’s Fair 2026 · https://www.youtube.com/watch?v=VrpEyglYgeU

Harness Engineering is not Enough: Why Software Factories Fail

Dex Horthy, HumanLayer · AI Engineer World’s Fair 2026 · https://www.youtube.com/watch?v=Ib5GBkD555M

I Run a Fleet of AI Agents Across Three Machines. Here’s What Broke.

Kyle Jaejun Lee, KRAFTON · AI Engineer World’s Fair 2026 · https://www.youtube.com/watch?v=4kYl2_mqmnQ

The Unreasonable Effectiveness of Separating the Task from the Model

Maxime Rivest & Isaac Miller · AI Engineer World’s Fair 2026 · https://www.youtube.com/watch?v=GgLQ02aO-hs

Full Workshop: Setting Yourself Up for Success

Jason Liu, OpenAI Codex · AI Engineer World’s Fair 2026 · https://www.youtube.com/watch?v=il1c1a2FufU

Active Graph Agent Runtime (BabyAGI 4)

Yohei Nakajima, Untapped Capital · AI Engineer World’s Fair 2026 · https://www.youtube.com/watch?v=khVX_BUnEwU

Written

Addy Osmani

Cloudflare

Shared by Nate Strandberg in #guild-ai on 2026-08-06 (thread).

DoorDash Engineering

  • Delegating Engineering Work to Cloud-Based Agents (Flux, DoorDash Engineering / @AIatDoorDash, 2026-08-11) · https://x.com/AIatDoorDash/status/2087284229751394705
    • In-house cloud agent platform: Firecracker microVM sandboxes, an MCP “Agent Gateway” with scoped per-playbook permissions, reusable YAML “playbooks” (mixing agentic and deterministic steps), and multi-surface invocation (Slack, GitHub, cron, CLI). DoorDash-reported scale: 130k automated engineering tasks/month, 25k+ code reviews/week. Thesis: writing code is mostly solved; the value is the harness/infrastructure around the agent. Started narrow with code review to earn trust.

Andrej Karpathy

Chroma

Eugene Yan

Boris Cherny

Lilian Weng

Kun Chen

Steve Yegge / Gas Town

Artificial Analysis

  • Artificial Analysis - independent, continuously-updated LLM model & provider benchmarks (Intelligence Index, price per token, speed/throughput, context window, open weights) · https://artificialanalysis.ai

Tools & Specs