Adding support for binary embeddings

[bhugueney]

Thank you for this most useful extension !
It seems that binary embeddings allow dramatic increase in performance for a small accuracy cost ( https://huggingface.co/blog/embedding-quantization#quantization-experiments ).
Various other vector DB allow to use them :

• Faiss Yes

• USearch Yes

• Vespa AI Yes

• Milvus Yes

• Qdrant Through Binary Quantization

• Weaviate Through Binary Quantization

It would be great if DuckDB vss could also support them efficiently.

EDIT pg_vecor also has it : https://github.com/pgvector/pgvector?tab=readme-ov-file#binary-vectors
Best Regards

[Maxxen]

Hello!

Yes, support for arrays of other types is planned, although binary vectors in particular might be slightly more complex since DuckDB itself doesn't really have a "bit" type, but it should be doable.

[bhugueney]

Hello!

Yes, support for arrays of other types is planned, although binary vectors in particular might be slightly more complex since DuckDB itself doesn't really have a "bit" type, but it should be doable.

Thank you for your interest.
I have high hopes for the binary vector as they provide considerable saving in memory and speed both because of data throughput (cf. memory) and simpler distance implementation
https://github.com/CountOnes/hamming_weight.
Best Regards.

Sorry, closed by mistake and couldn't see how to undo the closing.

[mitanshu7]

Would love this!

[isaaccorley]

+10000 for this -- it would make the size of the indexes significantly smaller and able to be served from cloud storage.

[zstatmanweil]

Yes, excited for support for arrays of other types. Thanks!

[cr458]

Hey guys just commenting to say we are super, super excited about this. We have some applications where we think duckdb outperforms other vector DBs from a cost/ease of use perspective, but the files are a little large and unwieldy. Any quantization level would be super useful for us!!