"Just use LangChain."
"Just use LlamaIndex."
You hear versions of this whenever a team starts building an AI application.
Sometimes that advice is completely reasonable.
Both ecosystems can save real engineering time. They provide useful abstractions, integrations, retrieval components, agent patterns, observability hooks, and production-oriented tooling. Their current documentation explicitly addresses things like tracing, evaluation, deployment, persistence, guardrails, and production RAG.
So this is not an article arguing that these tools are bad.
It is an article about a more subtle mistake: confusing an application framework with a production architecture.
A framework can help you build the AI behavior. Your company still owns the system around it.
Frameworks solve real problems
There is a reason these libraries became popular.
Without them, teams repeatedly write glue code for:
- model calls
- prompt composition
- tool execution
- retrieval pipelines
- document loaders
- chunking
- output parsing
- agent loops
- state handling
- provider integrations
- tracing hooks
Using a mature abstraction can make a prototype faster and can keep common patterns consistent.
That is good engineering when the abstraction fits the problem.
The cost appears when a team assumes that because the framework has a component named "agent," "retriever," "memory," or "workflow," the surrounding production concerns are solved too.
They are not.
Production starts where the tutorial ends
A tutorial usually answers:
"Can the model call this tool?"
Production has to answer:
- Which users are allowed to call it?
- On which customer records?
- With which arguments?
- How is the request validated?
- What happens if the tool times out?
- What happens if the call succeeds but the response is lost?
- Can the action be retried safely?
- How is the action audited?
- Which model version made the decision?
- Which prompt version was used?
- What happens if the agent is halfway through when the process restarts?
The framework may help implement some of these patterns. Your application still needs to define them.
That is the hidden work behind "just use a framework."
Authentication and tenancy do not disappear
A business application needs identity.
If the system serves multiple customers, teams, legal matters, properties, departments, or workspaces, it also needs tenancy and authorization.
The AI layer must know not only who the user is, but what the user is allowed to retrieve and what actions they are allowed to perform.
This usually touches:
- authentication provider integration
- roles and groups
- tenant scoping
- row-level or document-level access
- API authorization
- tool permissions
- admin controls
- audit records
You cannot delegate this to a prompt.
"Only answer from documents the user can access" is not an access-control system.
The retrieval and tool layers need actual enforcement.
Data ingestion becomes its own product
RAG demos often start with a local folder.
Production knowledge systems need source lifecycle management.
That means:
- connectors
- incremental sync
- deletion propagation
- parsing failures
- retries
- version detection
- duplicate handling
- metadata normalization
- permission synchronization
- source health monitoring
A framework can provide loaders and index abstractions. The difficult part is deciding how your business data should behave.
If a contract is replaced, what happens to the previous version? If a user loses access in the source system, how quickly should the AI layer reflect that? If an ingestion job fails, who knows?
Those are application requirements, not library defaults.
Retrieval quality needs domain architecture
A generic retriever cannot know that your contract number should take priority over semantic similarity.
It cannot know that an HR policy applies only to one state unless you model that rule.
It cannot know that "active" means the most recent signed version rather than the most recently modified file.
It cannot know that one legal matter must never retrieve from another matter.
Production retrieval requires domain structure.
That may include deterministic lookup, metadata filters, lexical search, semantic search, reranking, parent-child relationships, SQL queries, and source-specific routing.
The framework can help orchestrate these pieces. It cannot infer your organization's truth model for you.
See Deterministic Retrieval and RAG Pipeline Design for the patterns we use around this problem.
Evals are not optional just because traces look good
Tracing is useful.
A beautiful trace does not tell you whether the system is getting better.
You need evaluation data that represents real work.
For RAG, that may include retrieval correctness, source correctness, citation quality, groundedness, abstention behavior, and permission boundaries.
For agents, evaluate tool selection, tool arguments, action sequence, approval behavior, recovery, and final business outcome.
Framework ecosystems increasingly provide evaluation tooling, which is helpful. The hard part remains creating the test set and defining what "correct" means for your workflow.
That work belongs to the product team.
Observability has a data-governance cost
Production AI needs traces.
But traces can contain sensitive data.
A trace may include:
- user prompts
- retrieved documents
- customer records
- tool arguments
- model outputs
- internal system identifiers
Before enabling every possible trace, decide:
- what should be logged
- what should be redacted
- where logs are stored
- who can access them
- how long they are retained
- whether production and development data are separated
The official tooling around modern AI frameworks increasingly supports observability and redaction patterns. That is useful. Your company still has to configure them according to its privacy and security requirements.
Agent memory can become accidental application state
"Memory" is one of the most overloaded words in AI systems.
Conversation history can be useful. Long-term user preferences can be useful. retrieval over prior interactions can be useful.
But business workflow state should not live only in agent memory.
If a case is awaiting approval, store that as a durable status. If a payment action was submitted, store the transaction reference. If a ticket was created, persist the ticket ID.
Do not rely on an agent transcript to tell you whether an irreversible action already happened.
Framework memory abstractions can support conversational context. They should not replace the application's database and state model.
Tool calling creates a new API surface
Every tool you expose to an agent is effectively an API designed for a probabilistic caller.
That API needs stronger validation, not weaker validation.
A tool should have:
- narrow purpose
- explicit schema
- server-side authorization
- input validation
- safe error behavior
- idempotency where needed
- structured outputs
- audit logging
Avoid generic tools that expose raw infrastructure unless the use case truly requires it.
The cost of "just add a tool" is the same cost as adding another privileged application capability.
Provider abstraction has limits
Frameworks make it easier to switch models.
That does not mean models are interchangeable.
Different models vary in:
- tool-use behavior
- structured-output reliability
- latency
- context handling
- instruction following
- cost
- multimodal capabilities
- safety behavior
- rate limits
A provider abstraction can normalize an API. It cannot normalize every behavioral difference.
If you switch a production agent from one model to another, run regression evaluations.
Treat model changes like dependency changes with user-visible consequences.
Upgrades have a real maintenance cost
Fast-moving AI libraries change quickly.
That is not inherently bad. The underlying field is changing quickly too.
But a production team needs an upgrade strategy.
Ask:
- Which versions are pinned?
- Who owns dependency updates?
- Which tests run before upgrades?
- Are deprecations monitored?
- How much framework-specific code have we written?
- Can critical business logic be tested independently of the framework?
The more your domain logic is tightly coupled to framework objects, the more expensive future migration becomes.
Keep important business rules in application code with clear interfaces where possible.
The framework is not your product boundary
A clean architecture usually separates:
Business domain
Customers, contracts, cases, properties, invoices, workflows, permissions, policies.
AI capabilities
Classification, extraction, retrieval, planning, generation, summarization.
Application services
Authentication, databases, integrations, queues, notifications, billing, approvals, audit.
Framework or model adapters
The specific orchestration and provider code used to implement the AI capabilities.
This separation makes the system easier to test and easier to change.
If the entire application is expressed as one giant agent graph, business logic can become difficult to reason about without executing the AI runtime.
Sometimes the cheapest architecture is less framework
For a simple workflow, direct code may be clearer.
Suppose the process is:
1. classify a support ticket 2. retrieve the customer record 3. retrieve an approved policy 4. draft a response 5. route for approval
You may not need an open-ended agent loop.
A deterministic workflow with a few model calls can be easier to test, cheaper to run, and easier to debug.
Use an agent when dynamic planning creates value.
Use a workflow when the path is known.
This distinction is more important than which library implements either pattern.
Sometimes the framework is absolutely worth it
Use a framework when it materially reduces complexity.
Good reasons include:
- you need multiple model integrations
- you need agent orchestration patterns
- you benefit from built-in tracing and evaluation
- the ecosystem has mature connectors you would otherwise maintain
- your team understands the abstractions
- the framework aligns with the product's actual behavior
The mistake is not adopting a framework.
The mistake is adopting it as a substitute for architecture.
Calculate the real cost correctly
When comparing "use a framework" with "build it ourselves," do not compare library code to custom library code.
Compare the full production system.
The cost categories include:
- engineering integration
- authentication and permissions
- data ingestion
- retrieval design
- testing and evals
- observability
- infrastructure
- operations
- security review
- failure recovery
- upgrades
- vendor dependencies
- model and token cost
- support tooling
- ongoing tuning
A framework may lower several of those costs. It does not make them zero.
Our position: use abstractions deliberately
We are not interested in framework purity.
If LangChain, LlamaIndex, or another tool solves a real problem cleanly, use it.
If a small amount of direct code is more reliable, use that.
If a workflow should be deterministic, do not make it agentic just because an agent abstraction is available.
If a vector database is unnecessary, do not add one because the RAG tutorial did.
Production architecture is the discipline of choosing the smallest set of abstractions that make the system easier to operate without hiding the business rules that matter.
For related work, see AI Architecture Review, AI Agent Orchestration, and RAG Knowledge Systems.
Next step: Talk to an engineer about applying this to your stack.