A fintech team burned $50,000 in API costs over six weeks — on a model that was simply wrong for the job. They were automating document verification. They reached for the flashiest frontier model available, ran it against unvalidated data, and watched the bill climb. The system never reached production. The story is more common than the industry admits.
Model selection has become the most expensive decision in AI projects. It is routinely made at the wrong time, for the wrong reasons, by people who haven’t yet defined what success looks like.
Listen To The Podcast Now!
Why This Keeps Happening
There’s a seductive logic to starting with the model. Demos are fast. Prompts are easy. You can have something that looks impressive in an afternoon — which is exactly the problem.
Teams skip past the harder questions: what exactly are we automating, on whose data, evaluated against what standard? McKinsey’s 2025 AI survey found that organizations reporting significant financial returns from AI were twice as likely to have redesigned workflows before selecting modeling techniques. Twice. The model is not the strategy. It’s a component of a strategy you haven’t built yet.
The market moving fast makes this worse. The cost of querying a GPT-3.5-equivalent model dropped from $20 per million tokens in November 2022 to just $0.07 by October 2024, per the Stanford HAI 2025 AI Index — a 99.6% reduction in two years. Teams who locked architecture assumptions around a specific model’s price point or capability ceiling found those assumptions obsolete before they shipped.
The Three Mistakes, in Order of How Often I See Them:
1. Picking the most powerful model when a smaller one would do
Frontier models are extraordinary. They are also priced for tasks that require extraordinary reasoning. Most production AI tasks don’t. Routing a support ticket, classifying an inbound lead, verifying that an email address is real — none of these require the same model you’d use to analyze a financial filing.
At Globussoftai, the production architecture pattern we return to repeatedly pairs a strong frontier model for generation with a smaller, task-specific classifier for routing and triage. The frontier model handles the hard calls. The classifier handles the volume. You pay frontier prices for frontier problems only. This isn’t a novel idea — it’s just one most teams don’t reach until after they’ve overspent.
2. Choosing between RAG and fine-tuning before understanding your data
This derails more projects than any other single decision. Teams read a blog post, commit to RAG, and six weeks later discover their documents chunk in ways that destroy retrieval quality. Or they commit to fine-tuning and find their labeled dataset is three hundred examples short of what stable results require. Both are avoidable.
The actual framework is less complicated than the debate suggests. RAG belongs on problems where the underlying information changes — policy manuals, product catalogs, support ticket histories. Fine-tuning belongs on stable formats with a fixed output set, like classifying support tickets into a known category structure. Critically: RAG accuracy is moved more by chunking strategy and embedding quality than by model choice. The retrieval pipeline is almost always the bottleneck. If you haven’t audited retrieval quality, arguing about which model to use is premature by at least a month.
3. Committing to a single model before you have an evaluation harness
This is the mistake that looks like discipline. Teams pick a model, tune prompts against it, ship. Three months later, the model updates, a competitor’s model outperforms it on their specific task, or usage scales to a point where cost becomes the constraint. Switching now means revalidating everything from scratch. Most teams don’t switch. That’s how vendor lock-in quietly happens.
The right sequence: build the eval harness first. A labeled evaluation harness requires 100–500 human-labeled examples built against real business criteria — not benchmark scores, not demo outputs, your criteria on your data. Once that harness exists, you can swap prompts and model choices in hours rather than weeks. The harness is what makes model selection a reversible decision instead of a permanent one.
What “Workflow First” Actually Means in Practice?
Frame every AI project with a specific outcome before touching a model. Not “we want to automate outbound sales.” Something like: “We want to reduce the time a sales rep spends researching a cold lead from forty minutes to under five, measured against a sample of 200 leads we’ve already worked.” That framing immediately forces four questions: Where does the lead data come from? How clean is it? What counts as a good output? Who checks it?
An outbound sales automation agent touches four distinct data sources — lead finding, email verification, personalized copy generation, and CRM sync — each carrying its own data quality problems. Until you’ve mapped those problems, you don’t know which model you need. You might need different models for different steps. The Globussoftai embedded pod supports multi-model usage across Claude, GPT, Gemini, and Qwen precisely because real production systems routinely use more than one.
And before the model question even becomes relevant: data readiness alone can consume 30–40% of a project’s total budget before a single line of model code is written. On a $200K engagement, that’s $60–80K spent cleaning, restructuring, and validating data. A CRM used for five years across two regional sales teams will almost certainly contain duplicate customer records — deduplication alone requires human judgment on ambiguous matches, business rules for which record survives, and a verification pass. Budget weeks, not days. A status field that started with four values can accumulate seventeen, with institutional knowledge of what codes 9 and 11 mean held by one person — who may have left. Model selection is downstream of all of this, every time.
The Practical Sequence
Here is what a disciplined AI project looks like before anyone argues about model choice:
- Define the outcome and the measurement. One sentence. One metric. “Reduce X by Y within Z timeframe.” If you can’t write it, you’re not ready to select a model.
- Audit the data. Scope weeks two through four to producing a data dictionary, a PII inventory, and a list of fields excluded from model input. PII surfacing in notes fields, support tickets, or email threads can trigger HIPAA, SOC 2, GDPR, or SEC compliance blockers that terminate a project entirely. Discovering this in week twelve is catastrophic and avoidable.
- Build 100–500 labeled examples against your real business criteria. Not samples from a cleaned demo dataset. Real data, real decisions, real edge cases — the ones your team actually argues about.
- Run a proof-of-concept on that real data. Six to ten weeks, low tens of thousands in cost. The milestone question at week six is whether the system works on live client data, not on a cleaned demo. If it only works on the demo, you don’t have a system — you have a slide.
- Select and compare models inside the harness. At this point, model selection is empirical. You test against your labeled set, you measure cost-accuracy tradeoffs, you pick. The conversation is short because the data is already there.
That sequence sounds slower. It isn’t. The alternative — pick a model, start building, discover your assumptions were wrong in month three — is what actually inflates timelines and costs in ways that never appeared in the original estimate.
Who Gets Into Trouble Most?
Founders and mid-market teams building their first serious AI system are the most exposed. They lack internal expertise to know what questions to ask before a vendor starts billing. Frontier-lab embedded programs exist for this problem — Microsoft Frontier, Google Cloud FDEs, Anthropic Solutions — but they’re priced at $500K–$2M per year per engineer, with contract minimums starting at $250K. That pricing brackets out most of the companies that actually need the help.
The regulatory dimension sharpens the exposure further. Under the EU AI Act, fines can reach €35 million or 7% of global turnover for high-risk violations. For European SMEs — already navigating AI adoption with limited internal expertise — premature model selection in a regulated context isn’t just expensive. It’s a liability event.
Take a Position Before You Pick a Model
Model selection should be the last technical decision you make, not the first. Outcome → data audit → evaluation harness → proof-of-concept → model comparison. Teams that invert this spend real money learning what the sequence would have told them for free.
If you’re planning an AI project and already debating which foundation model to use, that’s a sign you’ve jumped ahead. Step back. Define what success looks like on your actual data. The model choice will follow — and it will be a far cheaper conversation once you have a harness to test it against.
Get your AI project scoped correctly from the start with Globussoftai — embedded engineers who stay with you from the data audit through production, no offshore handoffs, and a first working agent shipped to real users within 30 days.
