Included columns

[Sergei Iarymov (soyarym)]

Hi there!

Guys, I don't understand completely meaning of includedColumns. In the quick start guide:
To create a Hyperspace Index, specify a DataFrame along with index configurations. indexedColumns are the column names used for join or filter operations, and includedColumns are the ones used for project operations...
What's meaning of project operations?
For example, I have a dataframe with 1000 columns, and I want to create index by user_id and business_dt. Should I add all the rest columns as includedColumns? And how to do that simply?

Thanks a lot!

[EJ Song (sezruby)]

If you only want to use 10 of 1000 columns in your query, then you can just add those 10 columns as "included" columns.
(project operation means "select"(projection e.g. df.select("a","b"))
Hyperspace applies the index only if the index contains all required columns (filter condition columns & project columns) for the given query.

We will add the option to add all columns without specifying all columns, but for now you need to specify all columns if you want to add all columns to the index.

[Terry Kim (imback82)]

And how to do that simply?

You can work with schema of your DataFrame (e.g., df.schema.map(c => c.name)) to programmatically populate these columns.

[Sergei Iarymov (soyarym)]

Thanks a lot!