Skip to content

[VI] Vector Index Metadata persistence in the Data Dictionary #750

Description

@phulakun

MySQL Data Dictionary uses the mysql.indexes table to store index metadata. Vector index metadata should also be stored in this table, consistent with other index types. Each vector index should be represented by one row.

Proposed changes to mysql.indexes:

  1. Extend the enum fields
    1.1 Add VECTOR to the type enum:
    PRIMARY, UNIQUE, MULTIPLE, FULLTEXT, SPATIAL, VECTOR

    1.2 Add IVF to the algorithm enum:
    SE_SPECIFIC, BTREE, RTREE, HASH, FULLTEXT, IVF

  2. Store vector index properties
    Store vector index properties such as lists, distance, and other algorithm-specific parameters in the mysql.indexes.options column.
    These properties are part of the SQL definition of the vector index. Therefore, they should be persisted as part of the index metadata in mysql.indexes.options.

Example:

CREATE TABLE docs (
  id BIGINT PRIMARY KEY,
  embedding VECTOR(768),
  VECTOR INDEX idx_embedding (embedding)
    USING IVF (
      lists = 100,
      distance = EUCLIDEAN
    )
);

The corresponding row in mysql.indexes would contain:

id:                     <new DD object ID>
table_id:               <DD ID of test.docs>
name:                   idx_embedding
type:                   VECTOR
algorithm:              IVF
is_algorithm_explicit:  1
is_visible:             1
is_generated:           0
hidden:                 0
ordinal_position:       2, following the PRIMARY index in this example
comment:                ''
options:                lists=100; distance=EUCLIDEAN
engine:                 InnoDB
tablespace_id:          <tablespace DD ID, if applicable>

SELECT * FROM INFORMATION_SCHMEA.STATISTICS;

mysql> SELECT * FROM INFORMATION_SCHEMA.STATISTICS where table_Schema="test"\G
*************************** 1. row ***************************
TABLE_CATALOG: def
 TABLE_SCHEMA: test
   TABLE_NAME: docs
 INDEX_SCHEMA: test
   INDEX_NAME: idx_embedding
 SEQ_IN_INDEX: 1
  COLUMN_NAME: embedding
...
   INDEX_TYPE: VECTOR
....
   IS_VISIBLE: YES

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions