Repository navigation
Add support for multimodal input #19
Description
Thanks a lot for making this package, I am keen to switching to this for a production app.
Many of the language model providers that AnyLanguageModel already supports have strong multimodal capabilities:
MLX: Supports vision-language models like Qwen2-VL, and other multimodal models
OpenAI: GPT-4o, GPT-4o-mini, and GPT-4-turbo all support vision
Currently, users cannot leverage these vision capabilities through AnyLanguageModel, which limits the library's usefulness for multimodal applications.
Use Cases:
- Image captioning and description
- OCR and document understanding
Activity
mattt commentedon Nov 5, 2025
Hi @noorbhatia. I agree that the Foundation Models framework's lack of support for multi-modality limits its usefulness. As an API compatibility layer, AnyLanguageModel is inherently a lowest common denominator; it's too bad that the limitations happen at Apple's API.
I suspect that Apple will update Foundation Models to support sending images with prompts in iOS 27. But I don't want to wait that long. I'm keen to support the extent of what's already possible with other language backends.
Kind of like how the Game of Thrones TV series ran past its source material while waiting for the next book to be written. There's a risk we write something that breaks the eventual canon — but hey, that's what deprecation warnings are for.
(Or maybe it's more like Borges, writing stories critiquing books that don't exist)
So let's put on our best Cupertino accent and guess at what that eventual API should be. Could it be as simple as adding a new case to
Transcript.Segment,.image(ImageSegment)?mattt commentedon Nov 5, 2025
@noorbhatia I just opened up #21 with an initial implementation. If you have a moment, please give the
mattt/image-contentbranch a try. I've tested with OpenAI and Anthropic adapters so far (I'm away from my main machine, and can't easily run on-device models)noorbhatia commentedon Nov 5, 2025
Thanks a lot for the initial implementation @mattt , I'll get back with updates!
noorbhatia commentedon Nov 6, 2025
Hi @mattt I got it working for MLX models, tested with Qwen2-VL-2B-Instruct-4bit. How do I go about raising a PR?
mattt commentedon Nov 6, 2025
@noorbhatia That's great to hear! I recently force-pushed a fix to the build errors when building for MLX. If you have any other changes to put on that, please open a PR against
mattt/image-contentor feel free to drop a diff as a comment in #21 and I can patch that myself.noorbhatia commentedon Nov 8, 2025
Hi @mattt , I had opened a PR #23
mattt commentedon Nov 9, 2025
@noorbhatia Thanks again for your help with this! I just merged #21 and cut 0.4.0. I look forward to implementing image support for CoreML and llama.cpp models (and fixing any bugs we find in MLX support) in follow-up PRs soon.