Needle 3 – 8-29 MB basis mannequin for tiny units
Install the Python bundle. The inference engine is fetched as soon as from Hugging Face and cached; there’s nothing else to construct.
Needle reads your device descriptions to determine what to name and easy methods to fill arguments, so describing them nicely is the entire sport.
Simple: adorn a operate. The signature provides the argument sorts, the docstring is the device description, and run() completes the loop: the mannequin picks the decision, Needle executes your operate, feeds the outcome again, and returns the ultimate response with the executed device outcomes connected as outcomes.
import needle @needle.device def get_weather(metropolis: str): "Get the present climate for a metropolis." return {"metropolis": metropolis, "temp_c": 27, "sky": "clear"} agent = needle.Needle(instruments=[get_weather]) print(agent.run("what's it like in Lagos proper now?")["results"]) # [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]
Route by sample: when an outline can not enumerate each phrasing, give a device triggers, common expressions matched in opposition to every request. A match restricts the decode to the matched instruments and requires a name, so the request reaches the device you named as an alternative of being refused or misrouted, and the decision ships even under the boldness ground. A match restricts the entire flip, so a catch-all ought to exclude the nouns different instruments personal, e.g. ^(?![sS]*b(lights?|doorways?)b)[sS]*b(flip|swap)b[sS]*b(on|off)b; then “swap the fan on and dim the kitchen lights” nonetheless reaches each instruments.
from typing import Literal @needle.device(triggers=[r"b(turn|switch|power|flip)b.*b(on|off)b", r"btoggleb"]) def control_device(machine: str, motion: Literal["on", "off", "toggle"]): "Switch or toggle any named smart-home machine." return {"machine": machine, "motion": motion} agent = needle.Needle(instruments=[control_device, get_weather]) agent.full("toggle the storage door") # function_calls [{"name": "control_device", "arguments": {"device": "garage door", "action": "toggle"}}]
Extraction: to drag structured information out of textual content, declare the form and name extract(). Pass a Pydantic mannequin and also you get a typed object again.
from pydantic import BaseModel class Invoice(BaseModel): vendor: str whole: float due_date: str bill = needle.extract("Invoice from Acme Corp, $1,200.00, due 2026-09-01", Invoice) print(bill.vendor, bill.whole) # -> Acme Corp 1200.0
Every flip returns one JSON object:
{
"kind": "name",
"success": true,
"error": null,
"error_code": null,
"function_calls": [ { "name": "set_lights", "arguments": { "room": "living room", "on": true, "brightness": 30 } } ],
"reasoning": "'lounge' -> room; 'dim' -> on true, brightness 30",
"confidence": 0.94,
"prefill_tps": 4300.0,
"decode_tps": 850.0,
"peak_ram_mb": 28.5
}
Confidence gating and routing: each response carries a confidence rating from a calibrated head, and the engine already applies a ground of 0.1. Below it, the decision is withheld into suppressed_calls and function_calls is empty. Above it, the rating is yours to route on: act without delay when it’s excessive, present the decision and ask when it’s middling, and deal with an empty outcome as a refusal. A device with triggers all the time produces a name for an identical request, so the rating is what tells you whether or not to run it or verify it.
r = agent.full(user_text) calls = r["function_calls"] held = r["suppressed_calls"] if calls and r["confidence"] >= 0.7: execute(calls) # positive: act elif calls or held: verify(calls or held, r["reasoning"]) # uncertain: present the decision, ask else: say("I can not do this right here") # nothing to do: refuse
Writing instruments: the mannequin reads a schema actually, so a slender device with a plain description beats a broad one. One device per motion, described by the actions it covers (“Turn a room’s lights on or off”) quite than a class. Name enum choices after what a person says (motion: ["increase", "decrease"]) and hold synonyms within the description. Give a required argument a default when a request could go away it out; a required argument with no default and no proof within the request is withheld quite than guessed. Put worth codecs in descriptions ("City, ST", "e.g. T-1042"). Add triggers to intents that should all the time attain a device, and hold the toolset per flip small, since each further device is an opportunity to misroute.
Fine-tune: the Python bundle is the short path. LoRA on the frozen base on the full 20 layers, then a 4-bit .cact of any subnetwork that runs on the identical engine.
needle finetune information.jsonl --epochs 10 --out adapter.safetensors needle construct --lora adapter.safetensors --out tuned.cact needle construct --lora adapter.safetensors --platform linux-arm64 --layers 2 --out ./machine
