Operations guide · independent

Self-hosting a decision model

Self-hosting is the reason several teams look at this category at all. Open weights mean the decision can run inside your own perimeter, offline, with data that never leaves your network — something a hosted-only model cannot offer.

What you get in exchange is an inference deployment: hardware, serving, monitoring, updates, and evaluation. This page covers the practical shape of that trade.

Why teams self-host

  • →Data residency, air-gapped infrastructure, or internal-only state that cannot go to a vendor API.
  • →Offline operation, or a decision that must sit next to the service consuming it.
  • →Fine-tuning on your own labels — the open-weight projects publish notebooks, adapters, or full training recipes.
  • →Stable high volume, where marginal cost can beat a per-token hosted rate once the deployment is amortised.

What you actually have to run

ModelSizePublished hardware note
Strands Decider 2B~2BLocal CPU or GPU; launch coverage reports ~106–153ms median on consumer hardware
Laya (English)421M encoderProject reports ~33ms for one question on an Nvidia T4, 7.2ms batched
Clef-flash9BLaunch coverage describes a single H200 for a self-hosted test
Clef27BSubstantially more memory than the flash variant; verify for your precision
Clef-omni30B-A3B MoERoughly 64 GB of GPU memory in bfloat16, per the model card summary

Every figure above is vendor- or project-reported, on different hardware and with different batching. None is a measurement taken by this site, and none is a like-for-like comparison.

Network and access

Local packages typically ship a small HTTP server for development. That server is a starting point, not a production endpoint: the Strands Decider 2B release notes explicitly warn that the bundled server binds to 127.0.0.1 with no authentication layer.

If the model lives on a private machine and the calling service lives elsewhere, a private mesh network such as Tailscale is a common way to reach it without exposing a port publicly. That is general infrastructure practice, not an integration shipped by any of these model projects. Add authentication, TLS, and rate limiting at the service boundary regardless of how the network is arranged.

The cost ledger

  • →Hosted: $0.042 per 1M input tokens with output free for Jev; $0.24 / $0.09 for hosted Clef / Clef-flash on Workers AI.
  • →Self-hosted: GPU or CPU time, memory, storage for weights, electricity, and the engineering time to keep it running.
  • →Community cost models have noted that once your own GPU hours are priced in, the hosted rate is hard to undercut at small volume.
  • →Hybrid is common: a hosted model for low-volume, fast-changing question sets and a self-hosted one for high-volume, stable decisions.

Frequently asked questions

Can I self-host Jev itself?

+

No. TypeSafe has not published weights. Self-hosting in this category means running Clef, Laya, Strands Decider 2B, or a community port — each with its own accuracy profile and question schema.

What is the smallest useful model to start with?

+

Strands Decider 2B is the lightest well-documented entry: roughly 2B parameters, Apache 2.0, a pip install, and CPU or GPU operation. Laya's English checkpoint is smaller still at 421M, but its short default context and fine-tuning expectation are the first constraints you will hit.

Do I need a GPU?

+

Not always. AWS says Strands Decider 2B runs on a local CPU or GPU, and Laya reports a T4 for its latency figure but is small enough to try on modest hardware. Clef-omni is the heavy end at roughly 64 GB of GPU memory in bfloat16.

How do I reach a self-hosted model from another machine?

+

Put an authenticated service in front of it and connect over a private network. Mesh VPNs such as Tailscale are commonly used to avoid exposing an inference port to the public internet. Neither the model projects nor this directory publish a turnkey integration for that.

Is self-hosting cheaper?

+

At small volume, usually not — the hosted rate is hard to beat once GPU time and operations are counted. At stable high volume, or when data residency is mandatory, the calculation changes. Cost the whole deployment, not just the token line.

Keep reading

Sources

Reviewed 2026-11-13. Vendor- and project-reported figures are labelled as such; none were independently reproduced by this site.

Operations guide · reviewed 2026-11-13 · first published 2026-11-13