Epic: 0.5.0 Granite Library Support

[nrfulton]

Todo list in terms of PRs instead of issues:

• feat: update granite library examples to use Granite 4.1 3B adapters. #981 <- if this fails merge 986 then trying again
• fix: issues introduced by intrinsic changes #986
• release: bump minor version #977
• Execute checklist in Pre-flight checklist for Mellea support of granite-switch #96 #984 as soon as the switch model ids are ok to post publicly.

[jakelorocco]

Modified the description body to remove adding special support for granite given new info on training / support for interleaved documents in conversations.

[nrfulton]

copy/pasting old description here:

We need to marshal documents in a fairly precise way for Granite library models.

Background: what do models do special with documents?

What does Granite do that is Special?

Mostly Granite and Granite Library are the special cases.

Granite models treat documents in a special way. Documents must all be attached to the final user message using a kwarg outside of the conversation JSON. Per @frreiss, attaching documents to earlier messages isn't supported.

The granite library models represent documents in various ways. You always pass the documents to an intrinsic by attaching an extra kwarg to the API call (i.e., completely outside of the conversation JSON). From there, the intrinsic embeds the document in the conversation in one of four ways:

1. First Message String: all of the document texts are concatenated together and are sent as the first message in the conversation with role "user", like {"role": "user", "content": "[Document {id}]\n{doc 1 contents}\n[Document {id2}\n{doc 2 contents}..."}. If titles are provided, this method silently drops them. This is the default behavior; most do this.
2. JSON: Same as (1) in every way except that instead of concatenated the documents together as a string, a pretty-printed JSON array containing the documents is the contents of the first message. Each object contains the dodcument id and text. This method retains titles.
3. Roles: it looks like this is probably done like this: {"role": "Document 1", "content": "document text goes here"}. Title is dropped.
4. If the intrinsic doesn't specify docs_as_message, the docs are kept in the chat template arg "documents".

What do other models do that is special?

Usually nothing -- they're just trained to robustly handle throwing in the documents wherever. There are a couple of exceptions, designed to support platform ecosystems:

1. OpenAI and Anthropic allow you to upload documents to their platform and refer to those uploaded documents by a special URI. We don't know what happens after that; assume it's dropped into the context more-or-less directly.
2. Claude 3.5 sonnet (at least) allows you to attach base64 encoded documents to a message, just like you would do with an image (in the content of a message, something like {type: "file", file: ...}.

Scope

For now we'll focus on making Granite models behave correctly and ignore the OpenAI/Claude things for now.

Work Plan

1. Document gets an id and a title, both optional, both set to None by default.
2. We will retain the ability to attach documents to Messages.
3. We will also add a documents kwarg to mfuncs.chat and m.chat.
4. For non-granite models, we need to add documents to the jinja template for Message and "documents" to the Message template repr and use the message template repr to create the contents for a Message in backend? utils.
5. For granite and granite library models, we will do special handling of documents in a way that comports with best practices from the granite and granite library training teams. We definitely add a step after linearization of the context where we collect all of the documents from the Messages and remove those documents from the linearized messages (so that they aren't double-printed). This is done only for Messages and only for granite and granite library models (something like "granite" in self._model_id, probably). We'll then figure out what to do with them (where possible, add to special kwargs; for huggingface transformers backend, pass to tokenizer.from..., but it's unclear what to do for other backends. Currently not planning to have special handling for granite since granite does have conversations with documents interleaved. Will confirm before closing this issue.
6. (non-actionable observation, NB: all other models, and all other components passed to granite or granite library treat documents like any other key in the template repr dict. It's treated the same way as if it happened that template_repr.args["documents"] existed.)

There's some subtlety here, so we're also including the following defense:

1. If there's a key in TemplateRepresentation.args which is not in the jinja template, issue a warning. This can be done in a super hacky string inclusion checking way for now; we can do it properly later. This should also be done for the "special" fields on TemplateRepresentation (documents).

Accrued Technical Debt to be dealt with later

• We need to stop treating stdlib.chat.Message in a way that's not exactly the same as other Components. We are actually removing some special treatment by doing (4), but also adding new special treatment by doing 5... There will also a tiny bit of special treatment because of role which is assumed to be user for all other components, whichi s maybe the actual problem -- ie, Messages doesn't need special treatment, everyhting else needs to be able to say what role it wants to be in.

Alternatives

For the portion of step 5 that solves #953, we could instead just throw an exception or assert that the documents are only at the end.