Add hierarchical structure to the graph index #402

marianotepper · 2025-03-13T21:59:18Z

This PR introduces a hybrid HNSW-DiskANN graph. From HNSW, we take the idea of using multiple layers. This adds robustness to particularly hard datasets. Each layer is a Vamana graph. The upper layers reside in memory while the base layer resides on disk (in DiskANN style). It also enable using a single layer. In that case, it is a plain Vamana graph.

…where the node is present

…ere added

…e test

vels to candidates pool in layer 0

… no dupes are submitted

# Conflicts: # jvector-base/src/main/java/io/github/jbellis/jvector/graph/disk/CommonHeader.java # jvector-base/src/main/java/io/github/jbellis/jvector/graph/disk/OnDiskGraphIndex.java # jvector-base/src/main/java/io/github/jbellis/jvector/graph/disk/OnDiskGraphIndexWriter.java # jvector-base/src/main/java/io/github/jbellis/jvector/quantization/BQVectors.java # jvector-examples/src/main/java/io/github/jbellis/jvector/example/Bench.java # jvector-examples/src/main/java/io/github/jbellis/jvector/example/Grid.java # jvector-examples/src/main/java/io/github/jbellis/jvector/example/IPCService.java # jvector-tests/src/test/java/io/github/jbellis/jvector/graph/Test2DThreshold.java # jvector-tests/src/test/java/io/github/jbellis/jvector/graph/disk/TestOnDiskGraphIndex.java

…om the same level. If the entrypoint has been deleted, the new one can be in level 0 if that's the new top level

…e live nodes in the bottom layer.

…t is smaller than layers.size() - 1

marianotepper · 2025-03-18T18:14:00Z

A point worth noting is that we recently updated the file format for graphs to V4 in main (#400). Since this PR should be merged fairly quickly after that one and V4 has not been included in a release yet, I opted for not bumping up the file version to V5 but to just update V4 instead. This was to avoid an unnecessary proliferation of versions that will complicate backwards compatibility in the future.

jkni

In general, this looks quite good to me. This PR is easy to understand if you're already well-versed in JVector internals, and I agree with the implementation choices. The vast majority of my comments are polish/style-related. Nice work! Let me know if any of my feedback is unclear.

EDIT: Can you also add some coverage of this new index functionality/API changes to README/UPGRADING?

jvector-base/src/main/java/io/github/jbellis/jvector/graph/GraphIndex.java

jvector-base/src/main/java/io/github/jbellis/jvector/graph/GraphIndexBuilder.java

jvector-base/src/main/java/io/github/jbellis/jvector/graph/GraphIndex.java

jvector-base/src/main/java/io/github/jbellis/jvector/graph/GraphSearcher.java

jkni · 2025-03-24T16:01:02Z

jvector-base/src/main/java/io/github/jbellis/jvector/graph/GraphSearcher.java

@@ -105,6 +111,10 @@ public static SearchResult search(VectorFloat<?> queryVector, int topK, RandomAc
        }
    }

+    public void setView(GraphIndex.View view) {


Let's add a Javadoc here with an example of when this would be/is used.

jvector-base/src/main/java/io/github/jbellis/jvector/graph/disk/CommonHeader.java

Co-authored-by: Joel Knighton <[email protected]>

# Conflicts: # jvector-tests/src/test/java/io/github/jbellis/jvector/graph/Test2DThreshold.java

Jonathan Ellis and others added 30 commits February 7, 2025 17:21

degen dataset, tested in memory

8572202

extract IntMap, removing @deprecated put()

5bc9c85

SparseIntMap

2cf61be

move javadoc into IntMap

2907a1a

multilayer part 1

ae2fb7a

almost builds

b6b4bf0

r/m cache stuff so it builds

b1e864c

fixes

d174556

don't try to write to disk, it doesn't work yet

b17226a

fixes

98934c1

Bypass the order check in NodeQueue.copyFrom when other.size == 1

2b8900a

Search all levels when building the graph but only update the layers …

08f848f

…where the node is present

tests build

b75000e

add support for copying from opposite-order queues

f2edbb5

need to update snapshot during construction to see other nodes that w…

1ef1010

…ere added

fix copyFrom better

0114f5a

put back .parallel in build()

4ac9144

address TODOs in GIB. short edges computation is gone

7c167a7

fix TODOs in GraphIndex

6f54bd3

refactor getNodes and the underlying IntMap methods

d46d8e1

tests build

807a6c7

fix NodeArray overload of addNode to add layers if needed

ddf8ffa

need to reset visited between levels

7770ec1

copy the vector being searched for, this was a pre-existing bug in th…

7aafffb

…e test

fix visited better and fix adding evicted notes from higher le

700bad1

vels to candidates pool in layer 0

cleanup

7bc2072

r/m printlns

a5f8d6b

insertSorted ignores duplicates by design, not correct to assert that…

909b782

… no dupes are submitted

acceptOrds should only be consulted in L0

66b807d

add multi-layer validateIndex

bdfbbcc

marianotepper added 4 commits March 13, 2025 13:59

Code cleanup in GraphIndexBuilder.java

a78bc95

Update Bench with new parameters

26b4930

Fix merging issues

85c4889

marianotepper self-assigned this Mar 13, 2025

marianotepper requested review from tlwillke and jkni March 13, 2025 22:00

marianotepper added 10 commits March 13, 2025 15:04

Fix merging issues in jmh

54eab1f

Add license notice to SparseIntMap.java

ababca3

Add license notice to IntMap.java

bcccf33

Make sure top level is printed by GraphIndex.prettyPrint

a5b8c25

When adding random nodes in removeDeletedNodes, make sure they are fr…

aef6fd9

…om the same level. If the entrypoint has been deleted, the new one can be in level 0 if that's the new top level

Remove redundant @test decorator

a913fed

In GraphSearcher.search, use all nodes in the top layers, only use th…

b53f165

…e live nodes in the bottom layer.

Fix OnDiskGraphIndex.getNodes to account for the new layout

04b002c

After deletions, OnHeapGraphIndex.getMaxLevel can have a maxLevel tha…

87deaa7

…t is smaller than layers.size() - 1

Use 4 neighbors in TestDeletions instead of 2

0553fd5

marianotepper marked this pull request as ready for review March 17, 2025 22:24

jkni requested changes Mar 24, 2025

View reviewed changes

marianotepper and others added 10 commits March 24, 2025 15:19

Update comment in GraphIndex

7321c8d

Co-authored-by: Joel Knighton <[email protected]>

Changes following the PR review

916c1c7

Add to the Javadoc of all constructors

9af51bf

Remove unused lock member

ffaafc1

Fix typo

88d2553

Add reason for deprecation

43a9c0d

Add javadocs to GraphIndex.java

57ebc52

Update layerInfo to contain the correct size for L0

1db1448

Add javadoc for GraphSearcher.usePruning

984cdf9

Merge branch 'main' into hnsw-3

a617b29

# Conflicts: # jvector-tests/src/test/java/io/github/jbellis/jvector/graph/Test2DThreshold.java

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add hierarchical structure to the graph index #402

Add hierarchical structure to the graph index #402

marianotepper commented Mar 13, 2025 •

edited

Loading

marianotepper commented Mar 18, 2025 •

edited

Loading

jkni left a comment •

edited

Loading

jkni Mar 24, 2025

Add hierarchical structure to the graph index #402

Are you sure you want to change the base?

Add hierarchical structure to the graph index #402

Conversation

marianotepper commented Mar 13, 2025 • edited Loading

marianotepper commented Mar 18, 2025 • edited Loading

jkni left a comment • edited Loading

Choose a reason for hiding this comment

jkni Mar 24, 2025

Choose a reason for hiding this comment

marianotepper commented Mar 13, 2025 •

edited

Loading

marianotepper commented Mar 18, 2025 •

edited

Loading

jkni left a comment •

edited

Loading