NineTwoThree has shipped 150+ AI projects with a 97% success rate. What drives that success is how we've restructured our software development lifecycle around AI agents, evaluation suites, and human judgment. Here's the actual process, from discovery to deployment, and why most companies get it wrong by bolting AI onto an old workflow.
In short:
Most companies take their current process and add an AI tool to it. They keep the same define, build, test, deploy stages and insert Claude or Copilot somewhere in the middle. We do the opposite. We look at the process first and figure out where AI actually helps.
The macro stages look the same on paper: you still define, build, test, and deploy. The work inside each stage changes fundamentally.
Traditional AI software development assumes deterministic outputs. The same input produces the same result. AI is probabilistic. The same prompt can produce different outputs on different runs. Quality depends on data as much as code, and models degrade over time as real-world data shifts underneath them.
McKinsey found that top-performing teams using AI in software development achieve 16 to 30 percent improvements in productivity, time to market, and customer experience, along with 31 to 45 percent gains in software quality. Those gains only materialize when teams restructure their workflows around AI. Teams that try to jam machine learning into an old SDLC mold are surprised when the model falls apart in production.
A randomized controlled trial by METR found that experienced developers using AI tools took 19 percent longer on their own repositories. Speed from AI tools isn't guaranteed.
It depends on how well the workflow is designed for probabilistic systems. We've seen this pattern across enterprise clients: the process looks the same, but the outputs are unreliable because the workflow wasn't built for AI.
Throughput is solved. AI agents can write code fast, and a junior engineer with a well-configured agent stack can produce more code in a day than a senior engineer could a year ago. Speed is no longer the bottleneck.
What matters now is discernment: the ability to consistently make high-quality judgment calls when there's no objective metric to tell you if you're right.
We think about discernment in three layers:
| Layer | What it is | Why agents can't replace it |
|---|---|---|
| Taste | The ability to look at a design, an architecture, or a codebase and feel whether it's good. When an agent generates five possible implementations, taste is what tells you which one to ship. | It's subjective, built from years of experience, and it's the thing AI agents can't replicate. |
| Judgment | What happens when you formalize taste. You take the intuition and translate it into deterministic instructions: a definition of done, a set of quality gates, a checklist of what "good" looks like for this specific task. | This is what makes agentic loops work. The agent follows your judgment to the letter, even without taste. |
| Human in the loop | The non-negotiable layer. There will always be a human making decisions, verifying outputs, and owning the result. | The question is where in the process that human sits, not whether one exists. |
Our team uses a two-sided loop to structure this. On one side, the agentic loop: investigate, implement, test, report. On the other, the human loop: decision-making, verification, approval, ownership. The two sides communicate through evidence.
The agent produces JSON files, test results, artifacts, screenshots. The human reviews the evidence and decides what happens next.
This is why principles are durable. One CTO, Pavel Kirillov, has been building software for over 20 years, mostly in .NET. He's now developing in React because he can direct agents to write the syntax. His experience transfers across languages and frameworks because the experience is about principles, not syntax.
As he puts it: we're stepping above syntax.
Our engagement model has three phases: Align, Validate, Build. We've used this process across 150+ projects, and it's the reason 24 of our last 27 projects have been ROI-positive.
Align is where we audit the client's workflows and data. We rank AI opportunities by ROI, not by what's technically impressive. This is where we figure out whether the client needs custom AI development or something simpler. We look at their data infrastructure, their team's existing skills, and where AI can create the most value.
Validate is a four to six week sprint where we prove the data works. We build a prototype, run evaluation suites, and lock the ROI math before committing to a full build. This is where most AI projects fail in traditional SDLCs: teams skip validation and go straight to building, then discover six months later that the data doesn't support the use case.
Build is the production phase. Clients always own the model. We build in their systems, with their data, and their engineers learn alongside ours. This is a non-negotiable principle for us. We wrote about this in detail in our guide to becoming an AI-native company.
We run a two-prong approach on every project. Product managers focus on how humans will interact with the AI system: the workflows, the UX, the change management. ML engineers focus on the model itself: the architecture, the training, the evaluation. Both prongs feed into each other. The PM's insights about user behavior inform the engineer's model design, and the engineer's understanding of model capabilities shapes what the PM designs for users.
Our engineers build systems that build code. They don't write code directly for most tasks. They configure stacks of skills, sub-agents, and flows that handle the implementation.
Each discipline has its own agent stack. Our mobile team has a different set of tools than our DevOps team or our ML team. Each engineer customizes their agent stack to match their expertise.
Here's a real example from a recent project. We split a single feature into 19 phases and 40 pull requests. The plan review happened five times.
The agent would produce a plan, the engineer would review it, reject it with feedback, and the agent would try again. This loop continued until the plan met the definition of done we'd set. Only then did the agent start implementing.
The definition of done is the critical piece. Before any agent starts work, we define what "done" looks like: which tests pass, which quality gates are satisfied, which edge cases are handled. The agent iterates until it meets those criteria. If it can't, it reports back with what it found and the engineer decides what to do.
By the time code reaches a human engineer for review, the agent has already written tests, passed quality gates, and produced evidence of its work. The engineer's job is to apply judgment, not to check syntax. This is why hiring ML engineers instead of vibe coding matters: you need people who can direct agents with expertise, not people who hope the agent gets it right.
Evaluation suites are the layer most companies skip, and they're the reason our AI products don't fall apart in production. Our ML team pioneered evaluation suites for the rest of the company. When we started using AI agents for non-ML work like web development, mobile, and DevOps, we brought the evaluation mindset with us. Now every discipline uses them.
Different AI products need different evaluation suites:
The suites measure whether the AI is doing what it's supposed to do, not just whether the code runs without errors.
We track evaluation rates as a KPI. Not just pass or fail, but how consistently the system meets the bar across different inputs and conditions.
This matters because AI outputs are probabilistic. You can't spot-check a few examples and call it done. You need systematic evaluation across the range of inputs the system will actually see in production.
We wrote about this in detail in our guide to testing AI products. The short version: if you're not running evaluation suites on your AI system, you're shipping blind. Data preparation is the other half of this equation. Bad data in, bad results out, no matter how good your evaluation suite is.
The stack is no longer rigid. Designers ship code. Product managers prototype. Engineers do design. The boundaries between functions are blurring because agents handle the syntax that used to define each role's territory.
Pavel, our .NET engineer now developing in React, is a good example. He doesn't know React syntax the way he knows .NET. He knows what good software looks like, and he can direct agents to write React code that meets his standards. His 20 years of engineering experience transfers because the experience is about principles, not syntax.
This is why bottom-up adoption works and executive mandates don't. When engineers experiment with agents on their own, they figure out what actually helps them work better. They build agent stacks that fit their workflow. When executives mandate AI tool adoption from the top down, engineers get tools that don't fit their process and they stop using them when nobody's watching.
We've seen this across enterprise clients. The teams that succeed with AI are the ones where a few engineers started experimenting, showed results, and the practice spread organically. The teams that fail are the ones where leadership bought a tool, mandated its use, and expected productivity gains to follow.
We retired most of the traditional engineering metrics. Lines of code, pull request counts, sprint velocity, story points: these all measure throughput, and throughput is solved. When agents write most of the code, counting lines is like counting words in a novel written by a ghostwriter. It tells you nothing about quality.
What we measure instead:
| Replacement metric | What it tells us |
|---|---|
| Roadmap progression | Whether we're shipping the things that matter to the client, on time. Our 98% on-time delivery rate comes from tracking this, not from tracking how fast individual engineers write code. |
| Evaluation rates | How consistently our test suites pass across different inputs and conditions. This tells us whether the AI system is reliable, not just whether it works on the happy path. |
| Quality of decisions | Whether the judgment calls our engineers make are leading to good outcomes. This is harder to measure than lines of code, but it's the metric that actually matters when agents handle implementation. |
Before this is a technology problem, it's an organizational change problem. We've seen this with every enterprise client we work with. Knowledge lives in people's heads.
Most companies have documentation, but it's unreliable. One client described their documentation system as a digital graveyard: full of information that might be right, might be outdated, and nobody can tell the difference. When you ask an AI agent to use that documentation, it can't distinguish between a current decision and a direction that was abandoned two years ago.
You need to structure unstructured knowledge before agents can use it. This means creating the right incentives for people to document their decisions, their reasoning, and their context. It means treating documentation as a first-class engineering deliverable, not a chore that gets done if there's time.
The teams that succeed with AI engineering are the ones that figure this out. They create systems where documenting decisions is part of the workflow, not a separate task. They make it easy for engineers to contribute knowledge. They start from the bottom up, with engineers who see the value and adopt the practice organically.
150+ projects shipped • 97% success rate • 24 of 27 ROI-positive • $20M in new revenue for Consumer Reports
Consumer Reports came to us with 90 years of product data and no way to search it effectively. We built the largest vector database we're aware of, conceptualized their entire archive into a single searchable model, and delivered $20 million in new revenue over two years.
The client owns the model. Their team learned how to maintain and extend it.
This is what our SDLC produces: AI systems that create measurable revenue, owned by the client, maintained by their team. These are systems that run in production and generate returns.
We've delivered 150+ projects with a 97% success rate. 24 of our last 27 projects have been ROI-positive. That track record comes from the process, not from the models.
If you're trying to figure out where AI fits in your development process, we can help. We audit your workflows, rank opportunities by ROI, and build production systems your team owns and maintains.
Work with NineTwoThree or explore our free resources for AI and machine learning.
