Why this is important: Our company specializes in AI applications and leverages DigitalOcean for all our AI infrastructure needs. We manage millions of vector records in our DigitalOcean Managed Database, currently utilizing the pgvector extension and Hnsw index type. However, we face significant challenges with our current setup: Our Hnsw indexes are approximately 50GB in size, consuming substantial RAM and making scalability a challenge. Index building is time-intensive, taking several days to complete, which hampers our operational efficiency. As we aim to triple our current indexing capacity to support business growth, it is crucial to adopt more cost-effective and faster technologies. pgvectorscale presents a viable solution that can help us reduce costs and enhance indexing speeds, which are essential for our continued growth and competitiveness. What is pgvectorscale: pgvectorscale is an extension that builds upon pgvector to offer higher performance in embedding search and more cost-efficient storage for AI applications, making it an ideal upgrade for our database systems. It includes key innovations such as: StreamingDiskANN : A new index type inspired by the DiskANN algorithm, which is a research output from Microsoft. Statistical Binary Quantization : Developed by researchers at Timescale, this advanced compression method improves upon traditional Binary Quantization techniques. Additional Information and Resources: Performance Metrics: According to Timescale, pgvectorscale achieves performance up to 28x faster than Pinecone, with significant cost reductions. More details can be found in their recent blog post: Timescale Blog: pgvector is now as fast as Pinecone at 75% less cost Source Code and Documentation: For technical details and implementation specifics, refer to the pgvectorscale GitHub repository: GitHub - pgvectorscale ---- We believe that integrating pgvectorscale into DigitalOcean's managed PostgreSQL offerings will not only benefit our operations but also enhance the service offering for other clients with similar needs. We request your consideration of this feature for a future release.
