CI/CD Flaky Test Triager
Choice root-cause + Noul retry-safety pattern for CI failure logs.
Architecture overview
Parses failing test logs to classify infra flakiness vs genuine regressions and recommend safe auto-retry.
Developers burn time re-reading CI logs or blindly re-running pipelines.
Typed classification supports GitHub Action branching; measure your own p50/p95 before promising SLOs.
Official API schema (field names from docs.typesafe.ai)
Source: TypeSafe HTTP API reference. Question IDs below this table are directory example compositions — you choose them.
| Field | Type | Range | Notes |
|---|---|---|---|
| state | string | object | array | Plain text or structured JSON | Content to evaluate. Shared across all questions in one request. |
| model | string | e.g. "jev-latest" | Optional; defaults to TypeSafe flagship alias when omitted in SDKs. |
| questions.<id>.type | "choice" | "score" | "noul" | Exactly one of three primitives | Question ID is chosen by you; answers return under the same keys. |
| questions.<id>.instructions | string | Natural-language judgment | The actual question sent for inference (IDs are not sent to the model). |
| questions.<id>.criteria (choice) | Record<option, string | null> | 1–255 options | Map of option key → rubric description. |
| questions.<id>.criteria (score) | string[] | ≥ 2 ordered levels | Ordered level descriptions; score is a weighted position along them. |
| questions.<id>.criteria (noul) | { true?: string; false?: string } | Optional | Optional clarification of yes/no meanings. Answer field is noul ∈ [0, 1]. |
| answers.<id> (choice) | { type, choice, probabilities, confidence } | choice ∈ criteria keys; confidence ∈ [0, 1] | Probabilities sum to 1 across options. |
| answers.<id> (score) | { type, score, legend, probabilities, confidence } | score may fall between levels | legend maps level index → description. |
| answers.<id> (noul) | { type, noul } | noul ∈ [0, 1] | Probability that the answer is yes. No separate confidence field. |
| usage | { input_tokens, output_tokens } | Non-negative integers | Token accounting for the request. |
Example composition for this Jev AI tools workflow
Sample state and question keys are illustrative compositions for this directory page — not a separate official product API. Primitives remain Choice / Score / Noul.
Log Snippet: "Error: connect ETIMEDOUT to external redis cluster after 30000ms. Test: test_session_expiry."
Runnable call examples
Endpoint: https://api.typesafe.ai/v1/systemone (official). Requires your own TYPESAFE_API_KEY.
curl
curl -X POST https://api.typesafe.ai/v1/systemone \
-H "Authorization: Bearer $TYPESAFE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"state":{"log":"ERROR_LOG"},"model":"jev-latest","questions":{"cause":{"type":"choice","instructions":"Likely failure classification","criteria":{"network_timeout":"Timeout","assertion_regression":"Regression","resource_exhaustion":"Resources","syntax_error":"Syntax"}},"can_retry":{"type":"noul","instructions":"Safe to auto-retry?"}}}'TypeScript
import { choice, noul, TypeSafeClient } from "@typesafe-ai/sdk";
const client = new TypeSafeClient();
const ci = await client.systemOne({
state: { log: errorLog },
model: "jev-latest",
questions: {
cause: choice("Likely failure classification", {
network_timeout: "Network / infra timeout",
assertion_regression: "Assertion regression",
resource_exhaustion: "Resource exhaustion",
syntax_error: "Syntax / compile error",
}),
can_retry: noul("Safe to auto-retry without code changes?"),
},
});Python
from typesafe_sdk import Choice, Noul, TypeSafeClient
with TypeSafeClient() as client:
ci = client.system_one(
state={"log": error_log},
model="jev-latest",
questions={
"cause": Choice(
instructions="Likely failure classification",
criteria={
"network_timeout": "Network / infra timeout",
"assertion_regression": "Assertion regression",
"resource_exhaustion": "Resource exhaustion",
"syntax_error": "Syntax / compile error",
},
),
"can_retry": Noul(instructions="Safe to auto-retry without code changes?"),
},
)Latency & cost (source attribution)
Official end-to-end latency range: ~70–500ms; many calls land near ~100ms from US West Coast (source: typesafe.ai / TypeSafe public materials, 2026-09). Official list price: $0.042 / 1M input tokens; output tokens free (source: typesafe.ai, as of 2026-09). Card latency figures are illustrative compositions within that published range — not independent lab measurements by this directory.
- Card illustration on this page: ~79ms (illustrative, within official range — not a lab run by jevaitools.com).
- Official list price: $0.042 / 1M input tokens; output tokens free (source: typesafe.ai, as of 2026-09).
This directory has not published an independent measurement script for this page. To measure yourself: call the official endpoint with your key, record wall-clock p50/p95 andusage.input_tokens, and keep the date of the run.
Comparison with generative LLMs on the same decision task
TypeSafe publishes workflow evaluations where Jev is compared with frontier LLMs on accuracy, cost, and latency (company materials, 2026). Those multipliers are vendor-reported ceilings, not results measured by this directory. Run the same questions through an LLM structured-output adapter on your labeled set before choosing a stack.
Suitable for
- CI bot comments that classify failures
- Auto-retry only when Noul exceeds a safe threshold
- Quarantine suggestions for known flaky suites
Not suitable for
- Patching the failing test
- Root-causing distributed systems without traces in state
- Security triage of secrets found in logs
Common failure modes
- Truncated logs hiding the real stack frame
- Retrying assertion failures that need code fixes
- One global threshold across all repositories
Other Jev AI Tools
This site is an independent third-party directory and is not affiliated with, endorsed by, or operated by TypeSafe AI.