Multimodal capability

Author: fzowl

docs/api/vectorizer.rst

@@ -96,3 +96,15 @@ VoyageAITextVectorizer
 .. autoclass:: VoyageAITextVectorizer
    :show-inheritance:
    :members:
+
+
+VoyageAIMultimodalVectorizer
+============================
+
+.. _voyageaimultimodalvectorizer_api:
+
+.. currentmodule:: redisvl.utils.vectorize.multimodal.voyageai
+
+.. autoclass:: VoyageAIMultimodalVectorizer
+   :show-inheritance:
+   :members:

docs/user_guide/04_vectorizers.ipynb

@@ -4,31 +4,7 @@
    "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
-   "source": [
-    "# Vectorizers\n",
-    "\n",
-    "In this notebook, we will show how to use RedisVL to create embeddings using the built-in text embedding vectorizers. Today RedisVL supports:\n",
-    "1. OpenAI\n",
-    "2. HuggingFace\n",
-    "3. Vertex AI\n",
-    "4. Cohere\n",
-    "5. Mistral AI\n",
-    "6. Amazon Bedrock\n",
-    "7. Bringing your own vectorizer\n",
-    "8. VoyageAI\n",
-    "\n",
-    "Before running this notebook, be sure to\n",
-    "1. Have installed ``redisvl`` and have that environment active for this notebook.\n",
-    "2. Have a running Redis Stack instance with RediSearch > 2.4 active.\n",
-    "\n",
-    "For example, you can run Redis Stack locally with Docker:\n",
-    "\n",
-    "```bash\n",
-    "docker run -d -p 6379:6379 -p 8001:8001 redis/redis-stack:latest\n",
-    "```\n",
-    "\n",
-    "This will run Redis on port 6379 and RedisInsight at http://localhost:8001."
-   ]
+   "source": "# Vectorizers\n\nIn this notebook, we will show how to use RedisVL to create embeddings using the built-in text embedding vectorizers. Today RedisVL supports:\n1. OpenAI\n2. HuggingFace\n3. Vertex AI\n4. Cohere\n5. Mistral AI\n6. Amazon Bedrock\n7. Bringing your own vectorizer\n8. VoyageAI (text and multimodal)\n\nBefore running this notebook, be sure to\n1. Have installed ``redisvl`` and have that environment active for this notebook.\n2. Have a running Redis Stack instance with RediSearch > 2.4 active.\n\nFor example, you can run Redis Stack locally with Docker:\n\n```bash\ndocker run -d -p 6379:6379 -p 8001:8001 redis/redis-stack:latest\n```\n\nThis will run Redis on port 6379 and RedisInsight at http://localhost:8001."
   },
   {
    "cell_type": "code",
@@ -452,6 +428,30 @@
    "execution_count": null,
    "outputs": []
   },
+  {
+   "cell_type": "markdown",
+   "source": "#### Multimodal Embeddings\n\nVoyageAI also offers multimodal embedding models that can embed text, images, and video together. The `VoyageAIMultimodalVectorizer` supports:\n- **Text strings** - Plain text descriptions\n- **PIL Images** - Image objects loaded with PIL/Pillow\n- **Image URLs** - URLs pointing to images\n- **Video** - Video files (voyage-multimodal-3.5 only, requires voyageai>=0.3.6)\n\nAvailable models:\n- `voyage-multimodal-3` - Text and images\n- `voyage-multimodal-3.5` - Text, images, and video",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "source": "from redisvl.utils.vectorize import VoyageAIMultimodalVectorizer\nfrom PIL import Image\n\n# Create a multimodal vectorizer\nmultimodal_vo = VoyageAIMultimodalVectorizer(\n    model=\"voyage-multimodal-3\",\n    api_config={\"api_key\": api_key},\n)\n\n# Embed text only\ntext_embedding = multimodal_vo.embed(\n    content=[\"A description of a sunset over the ocean\"],\n    input_type=\"document\"\n)\nprint(f\"Text-only embedding dimensions: {len(text_embedding)}\")\n\n# Embed text with an image (uncomment to test with your own image)\n# image = Image.open(\"your_image.jpg\")\n# multimodal_embedding = multimodal_vo.embed(\n#     content=[\"A photo showing a beautiful sunset\", image],\n#     input_type=\"document\"\n# )\n# print(f\"Multimodal embedding dimensions: {len(multimodal_embedding)}\")\n\n# Batch embedding multiple contents\ncontents = [\n    [\"First text description\"],\n    [\"Second text description\"],\n    [\"Third text description\"],\n]\nembeddings = multimodal_vo.embed_many(contents, input_type=\"document\")\nprint(f\"Generated {len(embeddings)} embeddings\")",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": []
+  },
+  {
+   "cell_type": "markdown",
+   "source": "#### Video Embeddings (voyage-multimodal-3.5)\n\nThe `voyage-multimodal-3.5` model supports video inputs in addition to text and images. Videos must be loaded using VoyageAI's `Video` utility class.\n\n**Requirements:**\n- Model: `voyage-multimodal-3.5` only\n- Package: `voyageai>=0.3.6`\n- Max video size: 20 MB",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "source": "# Video embedding example (uncomment to test with your own video)\n# from voyageai.video_utils import Video\n# \n# # Create vectorizer with video-capable model\n# video_vo = VoyageAIMultimodalVectorizer(\n#     model=\"voyage-multimodal-3.5\",\n#     api_config={\"api_key\": api_key},\n# )\n# \n# # Load video using VoyageAI's Video utility\n# video = Video.from_path(\"your_video.mp4\", model=\"voyage-multimodal-3.5\")\n# \n# # Embed video with text description\n# video_embedding = video_vo.embed(\n#     content=[\"A video showing a cat playing with a toy\", video],\n#     input_type=\"document\"\n# )\n# print(f\"Video embedding dimensions: {len(video_embedding)}\")",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": []
+  },
   {
    "cell_type": "markdown",
    "metadata": {},

pyproject.toml

@@ -37,7 +37,7 @@ mistralai = ["mistralai>=1.0.0"]
 openai = ["openai>=1.1.0"]
 nltk = ["nltk>=3.8.1,<4"]
 cohere = ["cohere>=4.44"]
-voyageai = ["voyageai>=0.3.5"]
+voyageai = ["voyageai>=0.3.6"]
 sentence-transformers = ["sentence-transformers>=3.4.0,<4"]
 langcache = ["langcache>=0.9.0"]
 vertexai = [

redisvl/utils/vectorize/__init__.py

@@ -2,7 +2,12 @@
 from typing import Optional
 
 from redisvl.extensions.cache.embeddings import EmbeddingsCache
-from redisvl.utils.vectorize.base import BaseVectorizer, Vectorizers
+from redisvl.utils.vectorize.base import (
+    BaseMultimodalVectorizer,
+    BaseVectorizer,
+    Vectorizers,
+)
+from redisvl.utils.vectorize.multimodal.voyageai import VoyageAIMultimodalVectorizer
 from redisvl.utils.vectorize.text.azureopenai import AzureOpenAITextVectorizer
 from redisvl.utils.vectorize.text.bedrock import BedrockTextVectorizer
 from redisvl.utils.vectorize.text.cohere import CohereTextVectorizer
@@ -14,16 +19,21 @@
 from redisvl.utils.vectorize.text.voyageai import VoyageAITextVectorizer
 
 __all__ = [
+    # Base classes
     "BaseVectorizer",
+    "BaseMultimodalVectorizer",
+    # Text vectorizers
+    "AzureOpenAITextVectorizer",
+    "BedrockTextVectorizer",
     "CohereTextVectorizer",
+    "CustomTextVectorizer",
     "HFTextVectorizer",
+    "MistralAITextVectorizer",
     "OpenAITextVectorizer",
     "VertexAITextVectorizer",
-    "AzureOpenAITextVectorizer",
-    "MistralAITextVectorizer",
-    "CustomTextVectorizer",
-    "BedrockTextVectorizer",
     "VoyageAITextVectorizer",
+    # Multimodal vectorizers
+    "VoyageAIMultimodalVectorizer",
 ]