René Larch
Why this is important:
Our company specializes in AI applications and leverages DigitalOcean for all our AI infrastructure needs. We manage millions of vector records in our DigitalOcean Managed Database, currently utilizing the
pgvector
extension and Hnsw index type. However, we face significant challenges with our current setup:
  • Our Hnsw indexes are approximately 50GB in size, consuming substantial RAM and making scalability a challenge.
  • Index building is time-intensive, taking several days to complete, which hampers our operational efficiency.
  • As we aim to triple our current indexing capacity to support business growth, it is crucial to adopt more cost-effective and faster technologies.
pgvectorscale
presents a viable solution that can help us reduce costs and enhance indexing speeds, which are essential for our continued growth and competitiveness.
What is pgvectorscale:
pgvectorscale
is an extension that builds upon
pgvector
to offer higher performance in embedding search and more cost-efficient storage for AI applications, making it an ideal upgrade for our database systems. It includes key innovations such as:
  • StreamingDiskANN
    : A new index type inspired by the DiskANN algorithm, which is a research output from Microsoft.
  • Statistical Binary Quantization
    : Developed by researchers at Timescale, this advanced compression method improves upon traditional Binary Quantization techniques.
Additional Information and Resources:
----
We believe that integrating
pgvectorscale
into DigitalOcean's managed PostgreSQL offerings will not only benefit our operations but also enhance the service offering for other clients with similar needs. We request your consideration of this feature for a future release.