In today’s digital age, the demand for effective search functionality is higher than ever. Users expect applications and websites to provide seamless search experiences. To meet this demand, AI-powered search applications are on the rise. These applications leverage artificial intelligence to improve search results and enhance user experiences.
One such application is vector similarity search (VSS) in Redis, introduced by my colleague, Sam Partee. This innovative approach revolutionizes common search use cases. As Sam aptly puts it, “Finding new methods to improve search results is critical for architects and developers.”
For instance, in the world of eCommerce, AI-powered search can allow shoppers to browse product inventory with a visual similarity component. This brings online shopping one step closer to replicating the in-person experience. However, this is just the beginning. In this article, we will delve deeper into another common use case for AI-powered search: Document Search.
The Use Case: AI-Powered Document Search
Whether we realize it or not, document search and processing capabilities play a significant role in our daily lives. From searching for a long-lost text message on our phones to filtering spam emails, document search impacts our digital experiences. Businesses also rely on document search for information retrieval and content-based recommendations.
Traditional search, also known as lexical search, focuses on the intersection of common keywords between documents. However, this approach falls short when two documents convey the same meaning without sharing any keywords. For example, the sentences “The weather looks dark and stormy outside” and “The sky is threatening thunder and lightning” convey the same message, despite only two overlapping words.
To overcome this limitation, search has evolved to provide answers rather than just finding documents. Advances in natural language processing (NLP) and large language models have made it possible to bridge the lexical gap and uncover the semantic properties of text. This is where sentence embeddings come into play. These embeddings encode the “meaning” of unstructured data, allowing for more accurate search results.
Neural search, fueled by sentence embeddings, enables the computation of similarity metrics to find similar documents. This approach respects word order and understands the broader context beyond explicit terms. With neural search, a whole new range of powerful applications emerges, including question answering services, intelligent document search and retrieval, and insurance claim fraud detection.
Ready-made models, such as those provided by Hugging Face Transformers, fast-track the transformation of text into embeddings. However, customization and fine-tuning are often necessary to ensure high-quality results.
The Production Workflow
In a production software environment, document search requires a low-latency database capable of persisting all documents and managing a search index. This enables efficient nearest neighbors vector similarity operations between documents. RediSearch, a module built on top of Redis, extends these capabilities and seamlessly integrates with existing web request caching and online machine learning feature serving.
The production workflow for document search comprises several core components:
Document Processing
During this phase, documents are gathered, embedded, and stored in the vector database. This process occurs upfront before any client initiates a search and continues in the background to handle document updates, deletions, and insertions. Batch processing from a data warehouse and streaming data structures, such as Kafka or Redis Streams, facilitate efficient orchestration of the processing pipeline.
Advanced document processing services leverage high-throughput inference servers like NVIDIA’s Triton. These servers enable the deployment, running, and scaling of trained AI models on GPU or CPU hardware. Additional pre-processing steps, including embedding models to create vectors from text, are crucial, especially considering the source, volume, and variety of data.
Serving
When a client performs a search query, the query text is converted into an embedding, projected into the same vector space as the pre-processed documents, and used to discover the most relevant documents from the entire corpus. A robust vector database solution enables searches over hundreds of millions of documents in less than 100 milliseconds.
To illustrate the power of AI-powered document search, we developed redis-arXiv-search, a live demo built on top of the arXiv dataset. By converting paper abstracts into embeddings and storing them in RediSearch, this application allows users to search for papers using natural language queries.
The top search results reveal the semantic understanding of the embeddings. For example, a search query like “machine learning helps me get healthier” returns papers related to health outcomes and policy. Even complex queries like “Jay Z and Beyonce” yield relevant results related to music, celebrities, and Spotify, despite not explicitly mentioning these terms in the paper abstracts.
Scaling Embedding Workflows with GPU Acceleration
While the examples discussed so far demonstrate the power of AI-powered document search, real-world systems often deal with hundreds of millions or even billions of documents. Scaling the embedded workflows and ensuring timely search index building are critical.
To address these challenges, GPU acceleration becomes indispensable. Platforms like Saturn Cloud offer access to GPU hardware, enabling faster ad-hoc embedding workflows. Hugging Face Transformers can take advantage of GPU acceleration out-of-the-box, significantly speeding up text transformations. However, for production-scale use cases with massive amounts of data, a single GPU may not suffice.
To address this, the RAPIDS team at NVIDIA has developed open-source tools that allow engineers to leverage multi-node Dask clusters and cuDF data frames. These tools enable scaling out workloads across multiple GPUs, ensuring efficient data processing without requiring extensive knowledge of CUDA development.
AI-powered document search is revolutionizing information retrieval and opening up new possibilities in various domains. By harnessing the capabilities of AI and semantic understanding, businesses can enhance search experiences, provide accurate recommendations, and expedite fraud detection. With the right infrastructure and GPU acceleration, scaling document retrieval systems becomes a reality, propelling organizations to solve real-world problems with unstructured data.
Discover the power of AI in document search with Zenith City News. Check out our website for more insights and stay tuned for an upcoming hackathon co-hosted by Redis, MLOps Community, and Saturn Cloud from October 24 – November 4!
This article contains content adapted from the original article “AI powered document search” by Sam Partee.
