Uv & Apertus support

[loleg]

Add support for the uv package manager, and change from Llama to the Apertus model by default. The following environment variables (in the future, this should be a settings menu) allow changing this:

• ODE_AI_MODEL_NAME - the name of the model, as it should be shown in the UI
• ODE_AI_MODEL_URL - direct link to the download of the GGUF format file
• ODE_AI_MODEL_FILENAME - name of the file we are downloading

For example, to revert to the previous Llama 3 setup, use:

export ODE_AI_MODEL_NAME="Llama 3.2 3B (2.0GB)"
export ODE_AI_MODEL_URL="https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf?download=true"
export ODE_AI_MODEL_FILENAME="Llama-3.2-3B-Instruct-Q4_K_M.gguf"

In this PR, the default is set to the Bartowski quantization of Apertus.

Screenshot of Apertus running in Open Data Editor

Example dataset shown: ISIL library directory

[pdelboca]

Thanks for this! We're excited to test the Apertus model! It is definitely a nice replacement for Llama :D

I have also in my todo list to add uv support since the Makefile is too linux oriented and uv is just great.

[loleg]

Tested again yesterday, still no official update on llama-cpp-python. Works fine with the latest PR.

With the current release version of llama-cpp-python I get this error:

RuntimeError: Internal C++ object (PySide6.QtNetwork.QNetworkReply) already deleted.
Exception ignored in: <function LlamaModel.__del__ at 0x7f612e5d0a40>
Traceback (most recent call last):
  File "OKFN/opendataeditor/.venv/lib/python3.12/site-packages/llama_cpp/_internals.py", line 78, in close
    if self.sampler is not None:
       ^^^^^^^^^^^^
AttributeError: 'LlamaModel' object has no attribute 'sampler'
Traceback (most recent call last):
  File "/home/oleg/Localdev/OKFN/opendataeditor/ode/main.py", line 820, in on_llm_init_error

Even after bumping to a patched source version of the library, it is not yet working. But I'm going to try again once there's an October release.

[loleg]

@pdelboca do you have thoughts on how to support multiple models, at least the use of an alternative model via environment variable?

[pdelboca]

@loleg We used to have a dialog to select between different models. It should be fairly easy to restore it. Here is the PR in which we simplified the feature: https://github.com/okfn/opendataeditor/pull/1051/changes. The "backend" supports it, it just a matter of creating the UI interactions and adding some help text to guide the user on which to choose and why.

Some context

Supporting multiple models was removed for simplification and because it caused confusion on users regarding which one was best, what to choose, why is there an option, they didn't understand the concept of model and why they need to download, etc. Based on that experience we decided that for the user base of ODE having only one model is the best UX. We are more than happy to default to Apertus as it aligns perfectly with the tool and the philosophy.

[loleg]

"We are more than happy to default to Apertus as it aligns perfectly with the tool and the philosophy."

Can we have a call, try to get this pushed through ... if we're very quick, in time for Open Data Day?

[pdelboca]

@loleg sadly this period is a busy one for us so I don't think having something for the ODD on such short notice would be possible.

Currently we don't have capacity to tackle new things but we are more than happy to follow your lead if you have the time!

[loleg]

Happy Open Data Day! Let's get this merged 💪 🔜 🌱

[loleg]

I've updated the PR description to reflect current status. Besides testing for performance and stability, what I would suggest is tweaking the System Prompt and checking which size of Apertus we would like to default on:

• You had it on the quite small 2B version of Llama.
• See the Bartowski model card for a list of sizes and recommendations, they are however all at least double the size. I've suggested Q4_K_M, which requires at least 6GB of VRAM.
• Note that an unofficial 1.7B Apertus quantization is also available, and I know that they are working on an improved version.

[pdelboca]

@loleg Thanks for your work on this! I will take some time these week to review it :)