inference-sdk

Polyglot inference library for fully offline, text-only embedding and chat generation on CPU-only Linux, plus Windows and ARM64.

This repository hosts implementations in multiple languages. Java is first; Go follows. Both implementations produce wire-compatible artifacts and observable behavior.

Status

Language	Status	Path
Java	🚧 in development (Phase 1)	java/
Go	📋 planned	go/

Phase 1 is library-only — embedding via ONNX Runtime + bge-small-en-v1.5; chat generation via a forked llama.cpp Java binding + Qwen 2.5-0.5B-Instruct (default). HTTP/OpenAI-compatible layer is Phase 2.

Quick links

• docs/ARCHITECTURE.md — cross-language design
• docs/WIRE_FORMAT.md — JSON shapes shared across languages
• docs/MODEL_REGISTRY.md — canonical model IDs
• java/ — Java implementation

License

Apache 2.0 — see LICENSE.