AI stacks
How do you know an AI system works in production?
The evaluation and observability stack is how you know the system holds up. Evals score it against known examples before release. Tracing, logging, and monitoring then show every step a request took in production, so when something goes wrong you can see where, not just that it did.
Every other stack on these pages can run and still be wrong. The evaluation and observability stack is the part that tells you whether it works, and it splits cleanly into two jobs: proving quality before you ship, and watching behavior after.
Before release, the tool is the eval: a set of example inputs paired with expected results, scored to produce a number you can track. Run it against each new version and you can see whether a prompt change, a new model, or a fine-tune actually helped or quietly regressed. Frameworks like Ragas exist to move teams from informal “vibe checks” to systematic evaluation loops, with metrics tuned for LLM and RAG output.[3] Evals are not a one-time gate; they are a fixture you keep running, because a model that scored well last month can drift when anything upstream changes. A regression is something you broke. Drift is something that broke around you: a provider updated its model, users started asking different things, the document set changed, and none of it touched your code. Evals, run against a fixed test set, catch the first. Only watching live traffic catches the second.
After release, the tools are tracing, logging, and monitoring. A trace records the full path one request took, every model call and tool call in order, so when an answer is wrong you can follow it back to the step that failed rather than guessing. Platforms like LangSmith provide this, framed as full visibility into an LLM application from individual traces up to production-wide metrics.[1] To keep traces consistent across different tools, OpenTelemetry maintains semantic conventions specifically for generative AI systems, a shared vocabulary for what a model call or tool call looks like in a trace.[2] Monitoring sits on top, watching cost, latency, and error rates over time and alerting when they move. Without this layer you ship blind; with it, a production problem becomes something you can locate instead of something you only hear about from a user.
Eval and observability tools
- Ragas ↗
Evaluation framework for LLM and RAG applications, turning vibe checks into scored loops.
- LangSmith ↗
Tracing and observability platform for LLM apps, from single traces to production metrics.
- OpenTelemetry GenAI ↗
Open semantic conventions so traces of model and tool calls are consistent across tools.
References
- LangSmith observability and tracing — LangChain
- OpenTelemetry semantic conventions for generative AI — OpenTelemetry
- Ragas: evaluation for LLM applications — Ragas
Common questions
- What is the difference between an eval and a trace?
- An eval is a graded test you run before shipping: a set of inputs with expected results, scored to give a number you can compare against the last version. A trace is a recording of a single real request after shipping, showing each model call and tool call it made. Evals tell you the system is good enough; traces tell you why a specific run went wrong.
- Why not just read the logs?
- Plain logs show isolated lines. A trace links every step of one request into a tree, so you can follow a bad answer back through the retrieval call, the tool call, and the model call that produced it. Standards like OpenTelemetry's GenAI conventions make those traces consistent across tools.