- Published on
The Minimum to Be an AI Engineer: Evals First, Then Context Engineering, Then Fine-Tuning
- Authors

- Name
- Davidson Simões
- @davidsonsimoes
- Occupation
Software Developer (BR) @ Davidson / W2BIT
Table of Contents
- 1 TL;DR
- 2 The debate that prompted this post
- 3 How the role was defined
- 4 Evals are the actual minimum
- 5 Context engineering comes before weights
- 6 Where fine-tuning fits
- 7 Software engineering is still most of the job
- 8 A portfolio project that proves the minimum
- 9 What this means for product and TV teams
- 10 FAQ
- 11 Sources
The minimum to be an AI Engineer is not knowing how to fine-tune a model. It is being able to prove, with an evaluation you built, that the thing you shipped works, and being able to improve it with the cheapest lever available, which is almost always the context you give the model, not its weights. The primary sources agree: Anthropic's engineering team says evals are what turns "the agent feels worse" into something actionable; OpenAI's optimization guide puts evals before prompting and prompting before fine-tuning, and OpenAI is now winding down its fine-tuning platform for new users; and the 2023 essay that named the role quoted Andrej Karpathy saying one can be quite successful in it without ever training anything. Fine-tuning is a real skill. It is a specialization, not the entry ticket.
TL;DR
- Evals are the minimum. Define success criteria, build 20 to 50 tasks from real failures, grade with code where you can and a model where you must, read the transcripts.
- Context engineering is the first lever. Prompts, tools, retrieval and memory decide most behaviour and are cheaper to change than weights.
- Fine-tuning is a specialization. It pays off for format, style, rules and smaller models; 99% of the work is data, which a good eval system produces.
- Software engineering is most of the job. The role was defined as writing software around models.
The debate that prompted this post
On November 18, a Brazilian developer posted on X that you can only be an AI Engineer if you know, at a minimum, how to fine-tune and how to run an eval of what you built, "the minimum of the minimum". The pushback came fast and is worth quoting in substance: much of the work is solved with RAG and context engineering, an AI Engineer needs more software engineering than modelling, and fine-tuning is a plus rather than a floor. The author conceded he had set the bar too high. The thread is short, but it captures a disagreement that shows up in hiring conversations and in how teams plan their first AI features. The rest of this post checks both positions against what the model companies and the practitioners shipping with them recommend.

W2BIT's reading of the sources: the first layer is what you need to ship responsibly; the last one is where fine-tuning lives.
How the role was defined
The term AI Engineer was popularised by swyx in June 2023 in "The Rise of the AI Engineer". The essay describes a "shift right" of applied AI: tasks that used to need a research team now need API docs and an afternoon, and a new kind of software engineer sits on the application side of what it calls the API line, building on foundation models rather than training them. It quotes Karpathy's prediction that there will be significantly more AI Engineers than ML engineers and that one can be successful in the role without ever training anything. Its diagram was criticised for where it placed evals, and the essay agrees: model labs handle pretraining-scale data and general benchmarks, but "AI Engineers should certainly view product-specific data and evals as their job". The line is permeable; AI Engineers can go left to fine-tune and host models when it makes sense.
That framing already answers most of the debate. The role was defined by shipping software on top of models and owning the evidence that it works, not by training.
Evals are the actual minimum
Anthropic's engineering guide on evals for agents, published in January 2026, defines an eval as a test for an AI system: give it an input, apply grading logic to the output, measure success. Its argument for why evals are not optional is the one every team eventually lives through: without them, users report that the product feels worse after a change and the team is "flying blind", unable to tell a regression from noise. The advice for getting started is deliberately modest. Teams delay because they think they need hundreds of tasks; in reality, 20 to 50 simple tasks drawn from real failures is a good start, because early on every change has a large, visible effect. Write tasks two domain experts would grade the same way, and build balanced sets that test when a behaviour should and should not happen.
Three distinctions from that guide belong in any AI Engineer's vocabulary. Graders come in three kinds: code-based (string match, binary tests, static analysis, state checks), fast and objective; model-based (rubrics, pairwise comparison), which handle nuance but need calibration against humans; and human graders, the gold standard used sparingly. Evals come in two kinds: capability evals that start with a low pass rate and give the team a hill to climb, and regression evals that should stay near 100%. And because agents are non-deterministic, the metric depends on the product: pass@k, at least one success in k tries, versus pass^k, all k succeed, which is what a customer-facing agent needs. Anthropic's closing advice is the least glamorous and the most important: read the transcripts, because a 0% pass rate across many trials usually means a broken task, not an incapable agent.
Hamel Husain's "Your AI Product Needs Evals", from 2024, reaches the same place from the practitioner side. Unsuccessful LLM products, he writes, almost always share one root cause: a failure to build a robust evaluation system. He describes three levels, unit tests as assertions that run on every change, human and model evaluation on a cadence, and A/B tests for mature products, and a real estate assistant whose team maintains hundreds of such unit tests. His rule has no exceptions: you can never stop looking at data. Anthropic's platform documentation adds the front end: success criteria that are specific, measurable, achievable and relevant, with the difference between "the model should classify sentiments well" and an F1 target on a held-out set with a latency bound.
None of this requires a GPU. It requires product judgement, test-writing discipline and the willingness to read a lot of logs. That is why it is the minimum.
Context engineering comes before weights
The second thing the pushback got right is that most behaviour is decided by context. Anthropic's Applied AI team, in "Effective context engineering for AI agents", describes context engineering as the natural progression of prompt engineering: no longer finding the right words, but answering "what configuration of context is most likely to generate our model's desired behavior". Context is a finite resource with diminishing returns; long-context recall degrades in what they call context rot, so the goal is the smallest set of high-signal tokens that maximises the chance of the outcome you want.
The post is a practical curriculum on its own. System prompts should sit at the right altitude, neither brittle if-else logic nor vague guidance. Tools are the contract between the agent and its environment, and bloated, overlapping tool sets are one of the most common failure modes. Retrieval is moving from embedding-based, pre-inference RAG towards "just in time" strategies, where the agent keeps lightweight references and loads data with tools when needed, often in a hybrid with up-front retrieval. For long tasks, compaction, structured note-taking and sub-agents keep agents coherent beyond the window. Every item on that list is something you change in an afternoon and measure the same day with the evals from the previous section. Changing weights is neither.

The loop OpenAI and Anthropic both describe: evals set the baseline, context is the first lever, fine-tuning enters only when the evals justify it.
Where fine-tuning fits
OpenAI's model optimization guide lays out exactly this order. Its workflow starts by writing evals to establish a baseline, then prompting with relevant context, and only "for some use cases" fine-tuning, before running the evals again and repeating. The reasons it gives for fine-tuning are specific: more examples than fit in a context window, shorter prompts that save tokens and latency at scale, training on proprietary data, and making a smaller, cheaper model excel at a narrow task. Supervised fine-tuning covers classification, format and instruction-following fixes; direct preference optimization covers tone and style.
One fact in that guide tempers the "fine-tuning is the floor" claim. The page now says the evals and fine-tuning workflows are being moved into legacy documentation and that OpenAI is winding down its fine-tuning platform: it is no longer accessible to new users, and existing users can create jobs for the coming months. The skill has not disappeared, but it has moved: to open-weight models and libraries such as Hugging Face's TRL, whose trainers cover SFT, DPO, GRPO, reward modelling and distillation, and whose documentation points to a course on post-training.
Hamel Husain's framing of when to use it is the one we would teach: fine-tuning is best for learning syntax, style and rules, while techniques like RAG supply context and up-to-date facts. He also settles the order of operations: 99% of the labour in fine-tuning is assembling high-quality data that covers your product's surface area, and a solid eval system is already a data generation and curation engine. You cannot do fine-tuning well without first doing evals well, which is another way of saying which one is the minimum.
Software engineering is still most of the job
The reply that said an AI Engineer needs more software engineering is backed by the sources too. The founding essay's subtitle says the way to wield these capabilities is to "write software, and AI that writes software". Anthropic's eval guide describes the eval harness as infrastructure that isolates environments, records steps and aggregates results, and recommends running evals in CI on every change and model upgrade. Tools, in the context engineering post, are described like functions in a well-designed codebase: self-contained, robust to error, clear about their purpose.
The skill list we use at W2BIT when we discuss this role reads like a software curriculum with a model in the middle: tool and function calling, structured outputs, streaming responses, latency and cost optimisation, guardrails, memory systems, MCP, embeddings, chunking and reranking for retrieval. Each is ordinary engineering applied to a non-deterministic component, and each is only safe to change if an eval tells you what happened.
A portfolio project that proves the minimum
Our suggestion for someone who wants an AI Engineer role, built from the sources above:
- Pick one narrow, real task with a clear outcome, such as extracting structured fields from support tickets or routing TV-app crash reports to the right component.
- Write the success criteria first, in the specific and measurable form Anthropic's documentation describes, including latency and cost.
- Build 30 to 50 eval tasks from real or realistic failures, with code-based graders where possible and one calibrated LLM grader for the subjective part. Commit the harness and run it in CI.
- Reach a baseline with context alone: prompt, tools, retrieval, structured outputs. Record pass rate, cost per task and latency.
- Only then fine-tune a small open model with TRL, using the data your evals already curated, and report the same metrics on the same eval. If the gain is marginal, say so; a negative result with evidence beats a positive one without.
- Publish the transcripts and the write-up, including what you learned from the failures.

A project that shows evals, context engineering and fine-tuning in the order the sources recommend, with evidence at every step.
That project shows the minimum and the specialization in one repository, and the thing hiring managers find rarest: someone who can say, with numbers, whether their change helped.
What this means for product and TV teams
At W2BIT we build Smart TV apps with React and React Native, and AI features that reach a TV product, such as search, recommendations and assistants, run on constrained devices in front of users who will not retry. For a team like ours the minimum translates directly: an eval suite for each AI feature before it ships, context engineering as the default way to fix behaviour, and fine-tuning reserved for when a smaller model is the only way to meet a latency or cost budget. That is our interpretation, not a claim the sources make about TV.
FAQ
Do I need to know fine-tuning to be an AI Engineer?
Not as a minimum. The sources that define and guide the role put evals and context engineering first, and OpenAI's optimization guide treats fine-tuning as a step for some use cases after evals and prompting. It is a valuable specialization, especially for open-weight models, once you can measure what it changes.
What is the minimum eval setup for an AI feature?
Anthropic's guidance is 20 to 50 tasks drawn from real failures, with unambiguous success criteria, code-based graders where possible, a calibrated model grader where needed, and regular reading of transcripts. Hamel Husain adds unit-test-style assertions that run on every change, with human and model review on a cadence.
What is context engineering and how is it different from prompt engineering?
Anthropic describes context engineering as the progression of prompt engineering: curating the whole set of tokens the model sees, including system prompt, tools, retrieved data, message history and memory, rather than only wording the prompt. The goal is the smallest set of high-signal tokens that produces the desired behaviour.
When does fine-tuning make sense?
According to OpenAI's guide and Hamel Husain, when you need consistent format, style or rules, more examples than fit in a context window, shorter prompts at scale, or a smaller and cheaper model for a narrow task. RAG and context are the right tools for supplying facts and up-to-date information.
Sources
- JP (@mindofjota), post on X, November 18, 2026. https://x.com/mindofjota/status/2123178024439107690
- swyx, "The Rise of the AI Engineer," Latent Space, June 30, 2023. https://www.latent.space/p/ai-engineer
- Anthropic, "Demystifying evals for AI agents," January 9, 2026. https://www.anthropic.com/research/demystifying-evals-for-ai-agents
- Anthropic, "Effective context engineering for AI agents," September 29, 2025. https://www.anthropic.com/research/effective-context-engineering-for-ai-agents
- Anthropic, "Define success criteria and build evaluations" (Claude Platform documentation). https://docs.claude.com/en/docs/test-and-evaluate/develop-tests
- OpenAI, "Model optimization" (API documentation). https://developers.openai.com/api/docs/guides/model-optimization
- Hamel Husain, "Your AI Product Needs Evals," March 29, 2024. https://hamel.dev/blog/posts/evals/
- Hugging Face, "TRL - Transformers Reinforcement Learning" (documentation). https://huggingface.co/docs/trl/index