| PolarSPARC |
Jev-Style Decision Model on Ollaya
| Bhaskar S | 09/26/2026 |
Overview
For the last few days, the web has been abuzz with a new model known as Jev. Note that it is *NOT* a frontier LLM model, but a fast Decision model (or a classifier) that outputs structured decisions (along with a confidence scores) for a given input state and a set of question(s). It does *NOT* generate any text like the frontier LLM models. In other words, Jev just evaluates a set of given question(s) against a provided state and returns the structured decision(s), along with their probability score(s).
Following the release of Jev, numerous open-weight model(s) have sprung up, along with an open source platform for serving these open-weight decision model(s) called Ollaya.
In this article, we will *ONLY* demonstrate the setup and usage on a open-weight decision model on the local machine using a Docker image of the Ollaya platform.
The Ollaya platform exposes an API endpoint, which enables developers to build business workflow applications that can interact with the decision model(s) to decide on how to progress along in the workflow.
Last but not the least, the Ollaya platform effectively leverages the underlying system resouces of the machine, such as CPU(s) and GPU(s), to efficiently and optimally run the decision model(s) for better performance.
Installation and Setup
The installation and setup will can on a Ubuntu 24.04 LTS based Linux desktop. Ensure that Docker is installed and setup on the desktop (see instructions).
Also, ensure that the command-line utilities, such as, skopeo, curl and jq are installed.
We will setup a folder structure by executing the following command in a terminal window:
$ mkdir -p $HOME/.ollaya/models
To check the latest version of the docker image for Ollaya, execute the following command in a terminal window:
$ skopeo list-tags docker://ghcr.io/ollaya-dev/ollaya | grep -i cuda | tac | head -5
The following should be the typical output:
"0.7.1-cuda" "0.7.0-cuda", "0.6.1-cuda", "0.6.0-cuda", "0.5.0-cuda",
At the time of this article, the latest image tag was 0.7.1-cuda. To pull the docker image with this tag for Ollaya, execute the following command in a terminal window:
$ docker pull ghcr.io/ollaya-dev/ollaya:0.7.1-cuda
The following should be the typical output:
0.7.1-cuda: Pulling from ollaya-dev/ollaya 6b37362b3da7: Pull complete 160464519af9: Pull complete 6de6155d2434: Pull complete 6bd4dccd26f8: Pull complete aa3bbac52a1f: Pull complete 4f4fb700ef54: Pull complete 7b31ff8718b2: Pull complete 66eb7ac6b0c8: Pull complete Digest: sha256:6a4f4b4dfa6992d569a270ab3785b2e8a556a455f63a496ec8ac7bd6f150ce70 Status: Downloaded newer image for ghcr.io/ollaya-dev/ollaya:0.7.1-cuda ghcr.io/ollaya-dev/ollaya:0.7.1-cuda
This completes all the system installation and setup for the Ollaya hands-on demonstration.
Hands-on with Ollaya
Assuming that the ip address on the Linux desktop is 192.168.1.25, start the Ollaya platform by executing the following command in the terminal window:
$ docker run --rm --name ollaya --gpus=all -e OLLAYA_MAX_LOADED_MODELS=1 -e OLLAYA_KEEP_ALIVE=-1 -p 192.168.1.25:11435:11435 -v $HOME/.ollaya/:/home/ollaya/.ollaya ghcr.io/ollaya-dev/ollaya:0.7.1-cuda
Note that one can optimize the Ollaya server instance by passing options via environment variables. The following table summarizes some of the commonly used configuration options:
| Option | Description |
|---|---|
| OLLAYA_NUM_PARALLEL | The maximum number of parallel requests each model can process at the same time. Default is 1 |
| OLLAYA_MAX_LOADED_MODELS | The maximum number of models that can be loaded concurrently provided they fit in available memory. Default is 1 for CPU and 3 for GPU |
| OLLAYA_KEEP_ALIVE | The maximum time models stay loaded in memory after last use (e.g., 5m, 1h, -1 for forever, 0 to unload immediately). Default is 5m |
| OLLAYA_MAX_QUEUE | The maximum number of requests Ollama will queue when busy before rejecting additional requests. Default is 512 |
The following should be the typical output on Linux:
2026-09-26T17:25:22.543181Z WARN ollaya_server::http: listening on 0.0.0.0:11435, which is reachable from other machines, without OLLAYA_API_KEY: anyone who can reach it can run, pull and delete models 2026-09-26T17:25:22.543190Z INFO ollaya_server::http: Ollaya is running address=0.0.0.0:11435 version="0.7.1" models=/home/ollaya/.ollaya/models
For the hands-on demonstration, we will download and use the Kev 0.8B open-weights decision model.
Open a new terminal window (referred to as T-1), execute the following docker command to download the Kev 0.8B decision model:
$ docker exec -it ollaya ollaya pull kev:0.8b
The following should be the typical output:
pulling manifest pulling ff892c703a72: 100% ||||||||||||||| 438 B/368 B 10.41 KiB/s pulling 25b60f5542bd: 100% ||||||||||||||| 9.91 MiB/9.91 MiB 41.30 MiB/s pulling c2b1e5a17d9c: 100% ||||||||||||||| 1.63 GiB/1.63 GiB 93.57 MiB/s pulling 9b908623acb1: 100% ||||||||||||||| 41.33 MiB/41.33 MiB 70.18 MiB/s pulling f400bd12802b: 100% ||||||||||||||| 2.01 MiB/2.01 MiB 16.59 MiB/s pulling 06b9509352d2: 100% ||||||||||||||| 19.06 MiB/19.06 MiB 47.93 MiB/s pulling 1eaff1136c67: 100% ||||||||||||||| 2.83 KiB/2.83 KiB 96.71 KiB/s pulling 8b2f8ed9ec53: 100% ||||||||||||||| 255 B/255 B 9.82 KiB/s pulling 8292f5d72c8e: 100% ||||||||||||||| 10.21 KiB/10.21 KiB 329.44 KiB/s verifying sha256 digest writing manifest success
Open another new terminal window (referred to as T-2) and execute the following curl command to list all the downloaded decision model(s):
$ curl -s http://192.168.1.25:11435/api/tags | jq
The following should be the typical output:
{
"models": [
{
"name": "kev:0.8b",
"model": "kev:0.8b",
"modified_at": "2026-09-26T17:49:57.101069929Z",
"size": 1822775186,
"digest": "8a0d4094df504ca7329bc999dabbb89c15ad5221f55728acde0ff61c2b3909f2",
"details": {
"parent_model": "",
"format": "onnx",
"family": "kev",
"families": [
"kev"
],
"parameter_size": "0.76B",
"quantization_level": "F32"
}
}
]
}
The Jev style decision models support the following three types of question types:
Noul :: Used for answering simple true or false questions
Choice :: Used for picking an option from a defined list of choices
Score :: Used for rating the question against a defined list of scores
The first use-case demonstrates the use of the Noul type. In the T-2 terminal window, execute the following curl command:
$ curl -s http://192.168.1.25:11435/api/decide -d '{
"model": "kev:0.8b",
"state": "It is very cold outside today",
"questions": {
"weather": {
"type": "noul",
"instructions": "is the user talking of the weather?"
}
}
}' | jq
The following should be the typical output:
{
"model": "kev:0.8b",
"answers": {
"weather": {
"type": "noul",
"noul": 0.9369
}
},
"usage": {
"input_tokens": 23,
"output_tokens": 0
},
"routing": null,
"state_truncated": false,
"done_reason": "decide",
"created_at": "2026-09-26T17:57:02.894384603Z",
"total_duration": 5379941736,
"load_duration": 5329975416,
"eval_duration": 49850722
}
The second use-case demonstrates the use of the Choice type. In the T-2 terminal window, execute the following curl command:
$ curl -s http://192.168.1.25:11435/api/decide -d '{
"model": "kev:0.8b",
"state": "Want to build a slick web app. Trying to decide which language to write it in",
"questions": {
"languages": {
"type": "choice",
"instructions": "Which language the web app should be written in?",
"criteria": {
"golang": "Preferred when building cloud platform tools",
"java": "Preferred for platform portability and server side",
"typescript": "Preferred when building web applications"
}
}
}
}' | jq
The following should be the typical output:
{
"model": "kev:0.8b",
"answers": {
"languages": {
"type": "choice",
"choice": "typescript",
"confidence": 0.3804,
"probabilities": {
"golang": 0.2334,
"java": 0.1797,
"typescript": 0.5869
}
}
},
"usage": {
"input_tokens": 62,
"output_tokens": 0
},
"routing": null,
"state_truncated": false,
"done_reason": "decide",
"created_at": "2026-09-26T17:58:02.156627978Z",
"total_duration": 5355350572,
"load_duration": 5304960640,
"eval_duration": 50276253
}
The final use-case demonstrates the use of the Score type. In the T-2 terminal window, execute the following curl command:
$ curl -s http://192.168.1.25:11435/api/decide -d '{
"model": "kev:0.8b",
"state": "Like to drink tea; it is my favortite beverage",
"questions": {
"beverage": {
"type": "score",
"instructions": "Help me decide a beverage?",
"criteria": ["Water", "Coffee", "Masala tea"]
}
}
}' | jq
The following should be the typical output:
{
"model": "kev:0.8b",
"answers": {
"beverage": {
"type": "score",
"score": 1.6658,
"confidence": 0.5848,
"legend": {
"0": "Water",
"1": "Coffee",
"2": "Masala tea"
},
"probabilities": {
"0": 0.0574,
"1": 0.2194,
"2": 0.7232
}
}
},
"usage": {
"input_tokens": 32,
"output_tokens": 0
},
"routing": null,
"state_truncated": false,
"done_reason": "decide",
"created_at": "2026-09-26T17:58:40.422488254Z",
"total_duration": 69171915,
"load_duration": 0,
"eval_duration": 69078596
}
Given this space is fast moving and the ollaya platform is evolving, you are likely to encounter the following error:
{
"error": "onnx runtime: Non-zero status code returned while running Scan node. Name:'node_scan__1' Status Message: /home/runner/work/ort-artifacts/ort-artifacts/onnxruntime/onnxruntime/core/framework/utils.cc:585 void onnxruntime::utils::UpdateWithParentStream(DeviceStreamCollection &, Stream *) Subgraph has nodes running on device: P while parent graph node running on device: \u0001, this is not supported yet.\n",
"code": "INFERENCE_FAILED"
}
In such a case, just restart the ollaya docker instance !!!
With this, we conclude the various demonstrations on using the Jev style decision models via the Ollaya platform locally !!!
References