LangSmith Perceived Error judges cut eval costs 82 percent as Ornith-1.5 self-play models rival frontier agents and VS 2026 adds Copilot Git

LangSmith 82 Percent Cost Cut and Ornith-1.5 Self-Play Rival Frontier Agents

by

At a glance

  • LangSmith Tuned Evaluators launched today with a specialized Perceived Error judge that cuts production eval cost up to 82 percent while beating frontier models on accuracy.
  • Ornith-1.5 open-weight family dropped August 18, using self-generated tasks and scaffolds to match Claude Opus 4.8 on Terminal-Bench and DeepSWE at the 397B scale.
  • Visual Studio 2026 Insiders (August 18) adds a Copilot Git agent that explores PRs with navigable comments plus expanded Bring Your Own Key support for Anthropic, OpenAI, and Ollama.
  • OpenAI updated its Model Spec on August 18 with clearer teen-interaction principles, false-premise handling, and an explicit capabilities-and-limits section.

The agent engineering stack just got materially cheaper, more autonomous, and more native to the IDE. Production teams no longer have to sample a sliver of traces or pay frontier rates to know whether users are hitting silent failures. At the same time an emerging lab showed that a closed self-improvement loop—models proposing their own harder tasks, writing the harnesses, and rolling out solutions—can already rival closed frontier systems on the exact agentic and coding suites that matter to working engineers. Visual Studio is folding those same models and git context directly into the editor, while the largest closed provider is tightening the behavioral contract that every downstream app inherits. For builders this is not incremental news; it is the week the cost of reliability collapsed and the open-source self-play thesis started posting real numbers.

Top Stories

LangSmith Tuned Evaluators ship Perceived Error as a managed, 82-percent-cheaper production judge Practical dev impact: You can now attach a high-accuracy error detector to every production thread without writing prompts, managing judge models, or sampling traces. LangChain packaged a post-trained specialist that outperformed every frontier LLM-as-judge on Perceived Error while slashing inference cost 82 percent (up to 98 percent in some partner workloads). The evaluator automatically scores idle threads that contain at least two human-AI turns, attaches a score plus explanation as feedback, and surfaces the failures that never produce an explicit user rating or system exception. Teams attach it in one click on Plus and Cloud Enterprise plans; LangChain owns versioning, credentials, and infrastructure. Early users such as Vanta used it as an instant safety net while they built custom business evaluators.

Ornith-1.5 family releases MIT-licensed self-improving models that close the agentic coding gap Practical dev impact: You can now self-host a 397B MoE or a 9B mobile-quantized checkpoint whose self-generated curriculum already matches or beats much larger closed models on Terminal-Bench 2.1 and SWE-bench. Ornith AI extended its self-scaffolding loop so the model itself proposes novel, frontier-difficulty tasks, writes the evaluation harness, and generates the RL rollouts. The 397B variant scores 86.1 on Terminal-Bench 2.1 and 56.0 on DeepSWE—on par with Claude Opus 4.8—while the 9B dense model (plus GGUF/MLX/FP8 variants) runs on-device and still outperforms Gemma 4-31B on several coding suites. Weights, including quantized packs, are live on Hugging Face under MIT. This is the first widely available demonstration that an end-to-end self-improvement loop can produce production-grade agentic coders without a static human curriculum.

Visual Studio 2026 Insiders (18 August) adds Copilot Git agent and broader BYOK model support Practical dev impact: PR review and multi-model agent work now happen inside the IDE with one-click navigation to comments and files, plus first-class Anthropic, OpenAI, Ollama, and Microsoft Foundry keys. The August 18 Insiders drop (version 12113.136) lets the Git agent ingest a pull request from the Git Repository window, summarize discussion, and emit clickable links that jump straight to the relevant comment or file. Bring Your Own Key is now preview-default across Community, Professional, and Enterprise; it works with the new Agent (Preview) harness and supports custom endpoints for OpenAI-compatible and Ollama servers. Additional quality-of-life items include Hot Exit for unsaved work, Podman container attach, and File System View enabled by default. The update continues Microsoft’s push to make Copilot a first-class, model-agnostic teammate rather than an autocomplete add-on.

OpenAI refreshes Model Spec with teen-interaction rules, false-premise handling, and capability limits Practical dev impact: Any app that routes to GPT-5.x family models now inherits tighter, documented constraints on relational talk with teens and on how the model should flag its own limits. The 18 August Model Spec update adds explicit principles for appropriate relational interactions with users aged 13-17, clarifies how assistants must treat false or unsupported premises, removes leftover pre-reasoning guidance, and introduces a new “Be clear about capabilities and limits” section. The living document at model-spec.openai.com is the contract that ChatGPT, the API, and every downstream product must follow. Builders shipping consumer or teen-facing surfaces should re-read the spec before the next model family lands.

Practical Impact Analysis

The four stories converge on a single operational reality: the tax for knowing whether your agent is actually helping users just collapsed, while the cost of standing up a competitive open-weight alternative also dropped. LangSmith’s managed judge means you can now evaluate 100 percent of production threads instead of a 5-percent sample; the 82-percent cost cut is large enough that even high-volume support or coding agents become economically evaluable. That feedback loop is exactly what Ornith-1.5 automated at training time—models that generate their own harder tasks and harnesses will keep improving after you download the weights. The practical implication is that a small team can now run a nightly self-play cycle on the 35B or 9B Ornith checkpoint, feed the resulting traces into LangSmith, and let the Tuned Evaluator flag regressions without ever calling a frontier model.

Visual Studio’s Git agent and BYOK changes remove another friction layer: the same models you just evaluated can be dropped into the IDE as first-class agents that already understand your PR conversation. Combined with the OpenAI spec tightening, the week also raises the compliance bar—any teen-facing or high-stakes agent now has a clearer, auditable contract it must satisfy. The net effect is a narrower, cheaper, more inspectable path from prototype to production. Teams that still treat evaluation as a periodic, expensive ritual or that refuse to experiment with self-hosted self-improving weights will simply ship slower and more expensive agents than those who absorb today’s releases.

Recommended Tutorial Idea

Build a lightweight Perceived Error classifier that you can run on your existing agent logs today, then push the flagged conversations into a LangSmith dataset for the new Tuned Evaluator (or your own custom judge) to refine.

1. Export the last 200 production threads (or mock a few) as JSONL with fields `thread_id`, `messages` (list of role/content dicts). 2. Write a classifier prompt that looks for user corrections, repeated requests, contradictory answers, or unresolved outcomes—the same signals the official Perceived Error judge uses. 3. Batch the threads through a cheap, fast model (GPT-4o-mini, Gemini Flash, or a local Ornith-1.5-9B via Ollama). 4. Write the results plus the original thread to a LangSmith dataset so you can attach the official Tuned Evaluator or iterate on your own prompt. 5. Schedule the job nightly; any thread scored “error” becomes a few-shot example or a regression test.

python Recommended Tutorial Implementation 
from openai import OpenAI
import json

client = OpenAI()

CLASSIFIER_PROMPT = """You are a strict Perceived Error detector.
A thread has a Perceived Error if the user corrects the assistant,
repeats a request, the assistant contradicts itself, or the outcome
is clearly unresolved. Reply with ONLY a JSON object:
{"has_error": true/false, "reason": "one-sentence explanation"}"""

def classify_thread(messages: list[dict]) -> dict:
    convo = "\n".join(f"{m['role']}: {m['content']}" for m in messages)
    resp = client.chat.completions.create(
        model="gpt-4o-mini",

... click "Show full code" below to expand
▸ Show full code (32 lines) 
from openai import OpenAI
import json

client = OpenAI()

CLASSIFIER_PROMPT = """You are a strict Perceived Error detector.
A thread has a Perceived Error if the user corrects the assistant,
repeats a request, the assistant contradicts itself, or the outcome
is clearly unresolved. Reply with ONLY a JSON object:
{"has_error": true/false, "reason": "one-sentence explanation"}"""

def classify_thread(messages: list[dict]) -> dict:
    convo = "\n".join(f"{m['role']}: {m['content']}" for m in messages)
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": CLASSIFIER_PROMPT},
            {"role": "user", "content": convo},
        ],
        response_format={"type": "json_object"},
        temperature=0,
    )
    return json.loads(resp.choices[0].message.content)

# Example usage
with open("threads.jsonl") as f:
    for line in f:
        thread = json.loads(line)
        result = classify_thread(thread["messages"])
        if result["has_error"]:
            print(thread["thread_id"], result["reason"])
            # then client.create_example(...) into your LangSmith dataset

Grok Deep Dive

Today LangSmith made 100-percent production coverage economically viable with an 82-percent cheaper Perceived Error judge, Ornith-1.5 proved a fully self-generated curriculum can already match Claude Opus on Terminal-Bench and DeepSWE, Visual Studio folded those same models plus PR context into a first-class Git agent, and OpenAI tightened the teen and capability-limit contract every GPT app inherits. Design a closed-loop system that (1) uses an Ornith-1.5-35B checkpoint to propose the next harder coding or support tasks, (2) runs them through a LangGraph agent, (3) scores every thread with the new Tuned Evaluator plus the cheap classifier above, and (4) feeds the failures back as both RL signals and few-shot examples. Walk through the exact LangSmith + LangGraph + local-inference architecture, the nightly cron, the cost model versus calling Opus 5 on every trace, and the safety checks required by the updated OpenAI Model Spec if any traffic still hits GPT-5.x. Compare the resulting iteration speed and dollar-per-reliable-task against a classic “sample 5 percent and pay frontier judge” baseline.

Sources

Grok Deep Dive

Explore each Top Story in Grok — links open in a new tab. On phones, the same link may open the Grok app if you have it installed (via your device's normal link handling).

Article: LangSmith 82 Percent Cost Cut and Ornith-1.5 Self-Play Rival Frontier Agents

Privacy: links open grok.com in your session only. AIDevPulse does not run your prompts through our API.

Leave a Comment