|
1 | | -# BitNet.js |
| 1 | +<p align="center"> |
| 2 | + <img src="docs/assets/bitnet-stack.png" alt="BitNet-Stack" /> |
| 3 | +</p> |
2 | 4 |
|
3 | | -[](https://opensource.org/licenses/MIT) |
4 | | - |
| 5 | +<h1 align="center">BitNet-Stack</h1> |
5 | 6 |
|
6 | | -BitNet.js is the Node.js implementation of Microsoft's [bitnet.cpp](https://github.com/microsoft/BitNet) inference framework. This repository facilitates real-time interaction between a Node.js frontend and the bitnet 1-bit LLM model using **Socket.IO**. The app allows users to send queries to the BitNet LLM (Large Language Model) and receive responses line by line via a web interface. |
| 7 | +[](https://opensource.org/licenses/MIT) |
7 | 8 |
|
8 | | -## Working Example |
| 9 | +Run a small BitNet model on local machine with one Docker command, and chat in browser. |
9 | 10 |
|
10 | | -Web app will display the results in real-time. |
| 11 | +## What you get |
11 | 12 |
|
12 | | -### Docker App Run |
| 13 | +- **One Docker Compose command** — Build and start the LLM server with a single command. The image pulls the **official small BitNet GGUF** from Hugging Face (`BitNet-b1.58-2B-4T-gguf`, `ggml-model-i2_s.gguf`) so you do not manage weights by hand. |
| 14 | +- **Web interface** — Open the app in your browser, start a chat, and talk to the model from a simple page. |
| 15 | +- **Chat history in the browser** — Conversations are stored in **local storage** on your device. They stay until you clear them. |
| 16 | +- **Clear all chats** — Use the control in the UI to remove every saved thread and stop model sessions in one go. |
| 17 | +- **Session context** — Each chat keeps a **conversation session** with the model so follow-up messages stay in context until you start a new chat or clear data. |
| 18 | +- **Responsive replies** — Streaming output and a tuned server path so answers show up quickly as they are generated. |
| 19 | +- **Runs anywhere Docker runs** — On a laptop, a desktop, or a small server: clone the repo, run `docker compose`, open the URL. No extra install steps for Node or Python on the host. |
13 | 20 |
|
14 | | - |
| 21 | +## Working example |
15 | 22 |
|
16 | | -### Web Interface Interaction |
17 | | - |
18 | | - |
19 | | - |
20 | | - |
21 | | -## Features |
22 | | - |
23 | | -- **Web-based interface**: A simple frontend built with HTML and JavaScript to interact with the BitNet model. |
24 | | -- **Real-time communication**: Uses **Socket.IO** for bi-directional communication between the Node.js app and the Python-based BitNet model. |
25 | | -- **Dockerized environment**: Both the Node.js app and BitNet model run in separate Docker containers managed by **docker compose**. |
26 | | - |
27 | | -## Directory Structure |
28 | | - |
29 | | -```bash |
30 | | -bitnet.js/ |
31 | | -├── apps/ |
32 | | -│ ├── llm/ |
33 | | -│ │ ├── Dockerfile # Dockerfile for the BitNet model |
34 | | -│ │ ├── requirements-local.txt # List of packages that are used in Local Server |
35 | | -│ │ ├── run_model.py # Local Python Server to start Socket |
36 | | -│ ├── web/ |
37 | | -│ │ ├── Dockerfile # Dockerfile for the Node.js application |
38 | | -│ │ ├── app.js # Node.js app (Socket.IO client) |
39 | | -│ │ ├── index.html # Frontend (textarea for input, display for output) |
40 | | -├── docker-compose.yml # Compose file to run both containers together |
41 | | -├── README.md # Documentation |
42 | | -``` |
| 23 | + |
43 | 24 |
|
44 | 25 | ## Prerequisites |
45 | 26 |
|
46 | | -- **Docker**: Ensure Docker is installed. [Install Docker](https://docs.docker.com/get-docker/) |
| 27 | +- [Docker](https://docs.docker.com/get-docker/) (with Docker Compose) |
47 | 28 |
|
48 | | -## Setup Instructions |
| 29 | +## Quick start |
49 | 30 |
|
50 | | -### Step 1: Clone the Repository |
| 31 | +### 1. Clone the repository |
51 | 32 |
|
52 | 33 | ```bash |
53 | | -git clone https://github.com/stackblogger/bitnet.js.git |
54 | | -cd bitnet.js |
| 34 | +git clone https://github.com/stackblogger/BitNet-Stack.git |
| 35 | +cd BitNet-Stack |
55 | 36 | ``` |
56 | 37 |
|
57 | | -### Step 2: Build and Run Containers |
58 | | - |
59 | | -To build and start both the Node.js app and the BitNet model, use: |
| 38 | +### 2. Start the server |
60 | 39 |
|
61 | 40 | ```bash |
62 | 41 | docker compose up --build -d |
63 | 42 | ``` |
64 | 43 |
|
65 | | -This command will build the Docker images for both the **web** (Node.js) and **llm** (Python) containers, and run the services defined in the `docker-compose.yml` file. |
66 | | - |
67 | | -### Step 3: Access the Web Application |
68 | | - |
69 | | -After starting the containers, access the web interface by navigating to: |
70 | | - |
71 | | -`http://localhost:3000` |
72 | | - |
73 | | -You'll see a textarea where you can input your queries and send them to the BitNet model. |
74 | | - |
75 | | -### Step 4: Interact with the BitNet Model |
76 | | - |
77 | | -1. **Enter a query** in the provided textarea (e.g., "Why is the sky blue?"). |
78 | | -2. **Click Send**. Your query is transmitted to the BitNet model via Socket.IO. |
79 | | -3. The response will be returned **line by line** in real-time. |
80 | | - |
81 | | -## How it Works |
82 | | - |
83 | | -- The **Node.js application** (in `apps/web`) serves a simple web interface where users can input their queries. The application connects to the **Python backend** using Socket.IO. |
84 | | -- The **Python app** (in `apps/llm`) runs the BitNet model, processes the queries, and streams the response back to the Node.js client, word by word. |
85 | | -- Both applications are containerized and communicate through **docker compose**. |
86 | | - |
87 | | -### Node.js Frontend (app.js) |
88 | | - |
89 | | -The Node.js frontend sends user queries to the backend via a Socket.IO connection. Here's the simplified workflow: |
| 44 | +This builds the **LLM** image and starts one container. The model is downloaded during the image build (first time can take a while). |
90 | 45 |
|
91 | | -1. **User Input**: The user enters a query and clicks "Send." |
92 | | -2. **Socket Emission**: The query is sent to the backend using Socket.IO. |
93 | | -3. **Receive Data**: The response is received word by word and displayed on the page. |
| 46 | +### 3. Open the chat UI |
94 | 47 |
|
95 | | -### Python Backend (BitNet Integration) |
| 48 | +In your browser go to: |
96 | 49 |
|
97 | | -The Python backend runs the BitNet model and returns the response: |
| 50 | +**http://localhost:5001** |
98 | 51 |
|
99 | | -1. **Receive Query**: The Python app receives a query from the Node.js frontend. |
100 | | -2. **Model Inference**: The query is processed by the BitNet model. |
101 | | -3. **Stream Results**: The model's response is streamed word by word back to the Node.js client. |
| 52 | +(Port **5001** is mapped to the app inside the container on **5000**; change it in `docker-compose.yml` if you need another port.) |
102 | 53 |
|
103 | 54 | ## License |
104 | 55 |
|
|
0 commit comments