DuckDB Glue extension

Experimental extension that exposes an AWS Glue Data Catalog as a DuckDB catalog. It talks to Glue through the AWS SDK Glue client and works with Hive (Glue native) tables stored as parquet, csv, json or avro on S3. Tables of other formats that Glue registers (Iceberg, Delta, ...) are listed, with the columns Glue reports, but can not be read or written.

Use DDL/DML statements in this extension in production at your own risk. There is no guarantee that all incompatibilities between DuckDB and Trino/Spark engines writing to hive tables in glue have been discovered.

CREATE SECRET (TYPE S3, PROVIDER credential_chain, REGION 'eu-central-1');
ATTACH '<account_id>' AS my_datalake (TYPE GLUE);
SHOW ALL TABLES;
SELECT * FROM my_datalake.default.some_table;

Attach options:

option	description
`SECRET`	name of the s3/aws secret to take credentials from (default: default secret)
`REGION`	AWS region of the catalog (default: region of the secret)
`ENDPOINT`	Glue endpoint override, e.g. `http://localhost:5000` for a local moto server (default: AWS)
`DEFAULT_LOCATION`	optional S3 prefix for new databases and for tables created without an explicit location (takes precedence over the Glue database LocationUri)
`DEFAULT_SCHEMA`	Glue database to use as the default schema
`READ_ONLY`	refuse everything that changes the catalog: DML, DDL and the mutating `glue_*` partition functions

Reading

A table's format is read from its parameters (table_type, spark.sql.sources.provider, metadata_location) and otherwise from its InputFormat: Hudi tables (HoodieParquetInputFormat, ...) and symlink manifest tables (SymlinkTextInputFormat, e.g. a Delta _symlink_format_manifest) register a Parquet or text SerDe, so only the InputFormat tells them apart from a Hive table. They are listed but not read or written, like Iceberg and Delta.

The SerDe of the Glue table decides the reader: ParquetHiveSerDe reads with read_parquet (columns by name), LazySimpleSerDe and OpenCSVSerde with read_csv (columns by position), the Hive and the OpenX JsonSerDe with read_json (one object per line, keys by name ignoring case; SerDe properties see below) and AvroSerDe with read_avro (columns by name) from the avro extension, which is loaded on demand. Other SerDes (ORC, Ion, ...) are not supported. A parquet BYTE_ARRAY column without a STRING annotation (as Impala and older Hive writers produce) is read as UTF-8 text when Glue declares the column string (bytes that are not UTF-8 are an error), and as bytes when it declares it binary, whatever the session's binary_as_string. In a table that declares a binary column (or field), such bytes inside an array, struct or map column keep the escaped form the reader gives a BLOB. A float or double column may hold NaN wherever the file does not count its NaN values (nan_count, which most writers leave out): such a column's min/max never skip a row group for a filter that NaN could match. Text tables are read the way their SerDe reads them:

LazySimpleSerDe:
- The delimiter is field.delim, else serialization.format, else \001; a number from -128 to 127 is a byte code ('1' is \001, '9' a tab). NULL is serialization.null.format, \N by default. Fields are not quoted.
- timestamp.formats is not supported as the SerDe defines it (Joda-Time patterns, the longest match wins). Its patterns are hints: a timestamp field is trimmed and tried with try_strptime and each pattern strptime can express (y, M, d, H, h, m, s, SSS, a, E and quoted text), the first match in the list winning; then as milliseconds since the epoch if millis is listed; then with the usual cast. A field shorter than 8 characters is NULL, as in the SerDe. Patterns are separated by , (\, is a comma in a pattern); those with a time zone, a fraction other than SSS, or no year are ignored. Without the property, timestamps are read with the usual cast only.
OpenCSVSerde:
- separatorChar (,), quoteChar (") and escapeChar (\ when the table has none; see escapes below). An empty field is an empty string; there is no NULL.
- A boolean field is true when it is true in any case and false otherwise.
- A timestamp field is a number of milliseconds since the epoch (1579059880000 is 2020-01-15 03:44:40) and a date field a number of days since the epoch (18276 is 2020-01-15); text such as 2020-01-15 03:44:40 or 2020-01-15 does not parse.
- Escapes: without escapeChar a field is read with \ as the escape, as the SerDe reads it: the \ is dropped and the \ or quote character after it is kept, in a quoted field or not (\ is an empty field, a\x is ax, a\\b is a\b), and outside quotes it does not escape the separator (a\,b is the fields a and b). Doubled quotes are one quote either way. As read_csv splits the fields first, two cases differ: \" inside a quoted field ends the quote, so the field and the following lines run together until the next quote (the SerDe: "), and an unquoted field starting with \\ keeps one \ (the SerDe: none). An escapeChar the table records, such as the " of the tables this extension creates, is applied by read_csv (the SerDe reads " as \ too).
- Line breaks: the SerDe reads a file line by line (TextInputFormat), so a line break inside a quoted field splits the record into two broken rows, and a quote that is never closed (2,"abc) makes only that field NULL. The extension reads CSV as read_csv does: a quoted field continues across line breaks, so a quoted line break is read as one value, and the lines after an unclosed quote are read into that one value without an error. A quoted line break fails a scan of at most twice threads files, which reads each file with several threads (The parallel scanner does not support null_padding in conjunction with quoted new lines); with more files DuckDB reads one thread per file. SET hive_csv_parallel = false reads one thread per file in every scan.
Short rows are padded with NULL, extra fields are ignored, and a value that does not parse as its column type is NULL, except in OpenCSVSerde tables, where it fails the query unless use.null.for.invalid.data is true. Fields are converted with TRY_CAST (CAST in OpenCSVSerde tables without the property; OpenCSVSerde booleans, timestamps and dates as above), which differs from Hive in details: surrounding spaces are ignored, '1.5' in an int column is 2 (Hive: 1), and blank lines are skipped (Hive: a row of NULLs).
skip.header.line.count lines are skipped at the start of every file.
A line may be up to hive_csv_max_line_size bytes long (default 8000000; Hive has no limit); a longer line fails the query. Up to the default the reader's buffers and how a file is split between threads are unchanged; a larger limit makes each thread read at least that many bytes.
OpenX JsonSerDe: with ignore.malformed.json or use.null.for.invalid.data set to true the files are read with read_json(ignore_errors := true): a line that is not valid JSON is a row of NULLs and a value that does not fit its column is NULL; either property enables both. Other JsonSerDe properties that change how the files are read (OpenX mapping.* and dots.in.keys, Hive timestamp.formats) are not applied: the read succeeds and a warning is logged (CALL enable_logging(level := 'warning')).
JSON keys are matched to the columns ignoring (ASCII) case, as both SerDes do, at every nesting level; an OpenX table with case.insensitive=false matches them exactly. Keys of one object that differ only in case are duplicate keys.
Writes produce files that read back unchanged: LazySimpleSerDe values are unquoted and refused if they hold the delimiter or a line break, or equal the NULL string; OpenCSVSerde values are all quoted and refused if they hold a line break; timestamps and dates are written as milliseconds (truncated) and days since the epoch (an infinite value is refused), and a \ is doubled in a table without escapeChar, which reads with the \ escape. NULL is written as an empty field, which reads back as an empty string in a string column, false in a boolean column and NULL elsewhere only with use.null.for.invalid.data; otherwise reading it fails.
Not supported, for reads and writes: footer lines (skip.footer.line.count), header lines in JsonSerDe tables, nested columns, multi-byte quote or escape characters, and for LazySimpleSerDe serialization.escape.crlf, an escape.delim that is the delimiter or longer than one byte, a non-UTF-8 serialization.encoding, serialization.last.column.takes.rest, NUL or line-break delimiters and a delimiter in the NULL string. Writes need a single-byte delimiter.
LazySimpleSerDe with escape.delim (one byte): the escape character makes the next character part of the field, a delimiter included (a\,b is a,b, \\ is \); \N is still NULL. An escaped line break continues the field (Hive ends the record there). Writes escape the escape character and the delimiter in each value.
As in Hive, a table property overrides the SerDe property of the same name and keys are case-sensitive. Line counts and byte codes are DuckDB integer casts ('1.5' is 2). Partitions are read with the table's SerDe and SerDe properties: a scan that reaches a partition whose own InputFormat, SerDe or SerDe properties read its files differently (a csv partition of a parquet table, another delimiter) fails with an error naming the partition instead of reading it as the table's format. hive_scan() uses plain read_csv rules.

Every format is scanned through a custom MultiFileReader (HiveMultiFileReader) with these read semantics:

The data files are those below the location of every partition Glue lists (GetPartitions), at any depth, or below the table location for an unpartitioned table. Partition locations need not follow the <key>=<value> layout. Files and directories named _* or .*, *_$folder$ directory markers and objects the listing reports as zero bytes are skipped. When one partition's location lies inside another's, a file belongs to the deepest one. A location two or more partitions are registered at is refused when a query reads one of them and the location holds data files: Hive engines read such a file once for each partition, which DuckDB does not. s3a:// and s3n:// locations name the same bucket and key as s3://, and a location is read as registered, including a // in its path. A table without data files (just created) scans as empty.
A partitioned table with Athena partition projection (projection.enabled = true) is refused on read and write rather than scanned: Athena derives its partitions from the projection.* parameters and ignores the registered ones, which such a table usually has none of, so a scan would silently read nothing. To read it, set projection.enabled to false and register its partitions (ALTER TABLE ... ADD PARTITION, glue_add_partition).
Partition column values are the values Glue stores for the partition, not the directory names, typed as Glue's partition keys. Files are listed lazily: filters on partition columns, including those a join derives from its build side when the scan starts, are applied to the partition values first, so only the partitions a query reads are listed (EXPLAIN shows the partitions kept as Scanning Files). A filter the partition values decide is true of every row the scan then reads, and is dropped from the plan, as for read_parquet with hive partitioning. The partitions read below the table location are listed in whichever way takes fewer rounds of requests: the location once, recursively (one S3 request per 1000 keys, one after another), with the files matched to their partitions by prefix, or each partition on its own, several at the same time. On a tie the location is listed when that makes a tenth of the requests, so a table of up to 1000 partitions lists its location from 10 partitions read (or fewer, when they take more than one round). The estimate assumes one file per partition; SET hive_partition_listing_threshold = n replaces it with a fixed rule (the location from n partitions read). Partitions at custom locations are always listed on their own. They are listed as tasks on DuckDB's threads, as many at a time as SET threads. A directory is listed once per query, shared by every scan of it and the cardinality sample. With SET glue_cache_listings = true (default false) the listing is kept for later statements of the connection as well, held and cleared like the metadata cache (glue_metadata_cache_scope, writes through the extension, rollback, DETACH) — so files added any other way (another writer, COPY TO the table location) appear only after the next clear. Files removed any other way (compaction, an overwrite, dropping and recreating the table at the same location) are worse than unseen: scans under the cached listing fail trying to open the dead paths, or read the superseded files, until the next clear. It suits hosts that run one query per connection.
To estimate a scan's row count, planning lists one directory per table and query (the first partition a scan of the table reads, or the location of an unpartitioned table; when the scan lists the table location, the first page of that listing, which the scan then continues) and reads the row count of its largest file: the parquet footer, or the lines of a 64 KiB prefix for csv and json. Every scan of the table in the query scales that one measurement by the files and bytes of the partitions it reads: those the listing holds completely count with their own size, the others with the average of those. A scan that lists the same directory reuses the listing. Avro tables are not measured, so planning them lists nothing.
The Glue table definition and the partition list are fetched once per query and table, however often the query scans the table.
The schema is Glue's, data columns first and partition keys last, in PARTITIONED BY order. Files are matched by column name: a column a file does not have (added after the file was written) reads as NULL, a column with a different type in the file is cast, and file columns Glue does not list are ignored.
A partition declares its own columns, which keep the types they had when the partition was created when the table's columns change later. A scan that reads a column a partition declares with a type that can not be read as the table's type fails, naming the partition and both types. The columns are matched by name, by position for csv tables, and only those the query reads are checked. A partition type reads as the table's when ALTER COLUMN ... TYPE could have changed one into the other (integer widening, float to double, anything to string), when DuckDB casts it implicitly (for example int to double or decimal), or between decimals; bigint to int, double to int or string to int do not. Nested types are not checked.
Besides filename, the scan has three of the metadata columns of Trino's Hive connector, which SELECT * leaves out: "$path" (the data file, as filename), "$file_size" (BIGINT) and "$file_modified_time" (TIMESTAMP WITH TIME ZONE), as the listing reports them. hive_scan() has them too.

The SerDe of the Glue table decides the reader: ParquetHiveSerDe reads with read_parquet, LazySimpleSerDe and OpenCSVSerde with read_csv (columns by position, no header unless skip.header.line.count is 1, delimiter from field.delim / separatorChar, , otherwise) and JsonSerDe with read_json (one object per line, keys by name) and AvroSerDe with read_avro from the avro extension, which is loaded on demand. Other SerDes (ORC, Ion, ...) are not supported.

Compression: every csv and json file is read with the codec DuckDB tells from its name (.gz, .zst), whatever the table records, so a table whose files use more than one codec is read correctly and a stale write.compression or compressionType does not matter. The codec a table records is what writes to it use. Parquet and avro files carry their codec themselves.

Writing

CREATE SCHEMA [IF NOT EXISTS] ... [WITH (comment = '...', location = '...', <property> = '...')] creates a Glue database. comment is its Description, location its LocationUri (default <DEFAULT_LOCATION>/<schema>, or none when the catalog was attached without DEFAULT_LOCATION), and any other key a database parameter (Hive's DBPROPERTIES). A DEFAULT_LOCATION on ATTACH still decides where new tables go, over the database's LocationUri.
ALTER SCHEMA ... SET (<key> = '...', ...) merges options into the Glue database (UpdateDatabase), with the same keys as CREATE SCHEMA; ALTER SCHEMA ... RESET (<key>, ...) removes them. Keys are case-insensitive.
CREATE TABLE ... [PARTITIONED BY (col, ...)] [WITH (format = 'parquet' | 'csv' | 'json' | 'avro', location = '...', <property> = '...')] creates a parquet (default), csv (LazySimpleSerDe, , delimited, no header), json (JsonSerDe, one object per line) or avro (AvroSerDe) Hive table at location, else <DEFAULT_LOCATION>/<database>/<table>, else <database LocationUri>/<table>; without any of these the statement fails. Partition keys must be plain column names; they become Glue PartitionKeys and are listed last in the table's columns. Generated columns, column defaults and collated columns (COLLATE, also from a CREATE TABLE ... AS query) are refused. Unknown WITH keys are stored as Glue table parameters, except other engines' spellings of options this extension spells differently, which would do nothing there: partitioned_by, partitioning, partition_by, partition_columns, partition_keys and PartitionKeys (use PARTITIONED BY (...)), external_location (use location) and field_delimiter (use delimiter) are refused. Column types are stored as Hive types; DuckDB types without one are refused, e.g. UBIGINT, HUGEINT and TIMESTAMP_NS/_MS/_S (Hive's timestamp is TIMESTAMP, in microseconds). For csv, delimiter = '|' sets the field delimiter (field.delim), header = true makes every file start with a header line (skip.header.line.count), and quote = '"' / escape = '\' switch the table to OpenCSVSerde with separatorChar / quoteChar / escapeChar (the escape character defaults to the quote character). With SET glue_create_bucketed_tables = true, BucketColumns = ['col', ...], NumberOfBuckets = n and SortColumns = [{'Column': 'col', 'SortOrder': 1}, ...] (1 ascending, 0 descending) create a bucketed (clustered) table, Hive's CLUSTERED BY (...) SORTED BY (...) INTO n BUCKETS: bucket and sort columns are columns of the table that are not partition keys, and BucketColumns needs a positive NumberOfBuckets. The setting is off by default because DuckDB does not write to such a table (see below).
INSERT INTO and CREATE TABLE ... AS write files in the table's format into the table location (one file per partition touched, partition columns are not stored in the files; text files use the table's delimiter and NULL string, \N for LazySimpleSerDe unless serialization.null.format says otherwise, and at most one header line) and register new partition directories in Glue with BatchCreatePartition. New partitions get <key>=<value> directories; rows of an existing partition are written to its registered location, which may be any directory below the table location (e.g. <table>/2024/01). Inserting into a partition whose location is not below the table location, or whose location has an empty or . segment (//, /./) DuckDB can not write, fails, and so does inserting rows whose directory is the location of another partition; the rows of other partitions can still be inserted. Glue errors on single partitions that may pass (throttling, internal errors) are retried; when Glue still refuses a partition, the write is rolled back by deleting the files it wrote (the partitions it registered stay, empty), so the statement can be run again without adding its rows twice. Because the partition keys are the last columns of the table, INSERT ... VALUES without a column list must list them last. CREATE TABLE ... AS creates the Glue table when the statement starts executing, before the query runs (planning it, e.g. with EXPLAIN or PREPARE, creates nothing). It is refused when the location already holds data files, such as those of a dropped table, which would read as rows of the new table. WITH (overwrite = true) deletes those files instead, in one RemoveFiles call when the statement starts executing and before the table is created: they stay deleted if the statement then fails, and a failed delete may leave some of them (the table is then not created). It is refused when the location is, holds or lies inside the location of another table in the same Glue database, or of a partition of one (a partition can lie outside its table's location); tables in other databases, catalogs or accounts are not checked. CREATE OR REPLACE refuses overwrite. Files a scan skips (_SUCCESS, hidden files) are left. overwrite is not stored as a table parameter, and a CREATE TABLE without a query refuses it. Writes to bucketed (clustered) tables, i.e. tables with BucketColumns, are refused; they can be read. CREATE TABLE ... AS with the bucketing options is refused before the table is created. Like Athena, JSON tables are written only with the Hive JsonSerDe (org.apache.hive.hcatalog.data.JsonSerDe, what CREATE TABLE ... WITH (format = 'json') creates): INSERT into a table with the OpenX JSON SerDe (org.openx.data.jsonserde.JsonSerDe) is refused, it can be read. INSERT also requires the table's InputFormat to be one Hive uses for its file format (MapredParquetInputFormat, TextInputFormat or CombineTextInputFormat for csv and json, AvroContainerInputFormat) or empty: another engine's InputFormat means a layout DuckDB does not know, and files written into it could corrupt the table.
The files of one INSERT or CREATE TABLE ... AS are named <write id>_<file id>.<extension>, with a new write id (a UUIDv7) per statement. The write is registered with the Glue transaction when it starts and completed when its partitions are registered; a write that is still registered when the transaction ends belongs to a statement that failed, and the rollback deletes the files named after its write and, for CREATE TABLE ... AS, the Glue table it created, so the statement can be run again. To find the files, the rollback lists every object below the table's location, which takes one LIST request per 1,000 objects; it lists nothing, and deletes nothing, if the location contains a glob character (*, ?, [). A table CREATE TABLE ... AS created is kept if it holds files of another writer, or if another client dropped and created it again while the statement ran. This is best effort: a step that fails is logged as a warning (CALL enable_logging(level := 'warning') shows it). If the process dies, nothing is cleaned up: the files stay, and the rows in them appear once their partition is registered, by a later INSERT, MSCK REPAIR or a crawler. Other queries can read the files of an insert before it finishes.
ALTER TABLE ... ADD COLUMN (appended last, no defaults or collations), DROP COLUMN (not the last data column, not a partition key, bucket or sort column) and ALTER COLUMN ... TYPE (no collations) update the Glue definition with UpdateTable. Existing data files keep their types, so only widening type changes are allowed: integer widening (TINYINT to BIGINT), FLOAT to DOUBLE, and anything to VARCHAR; partition keys can not be retyped. ALTER COLUMN ... TYPE ... USING <expr> is refused, since the data files can not be rewritten.
Written files are compressed the way the table says: parquet with parquet.compression (and compression_level for zstd), csv and json with the codec the table records (gzip or zstd), named .csv.gz / .json.zst, and avro with avro.output.codec. A table without its format's codec parameter is written with write_compression, if it records one. Another codec is an error.
ALTER TABLE ... RENAME COLUMN is supported for csv tables, whose files are read by position. Parquet, json and avro files are matched by name, so a rename would make the existing data read as NULL and is refused (replace the column with glue_replace_columns to do that deliberately). Partition keys, bucket and sort columns can not be renamed.
COMMENT ON COLUMN sets the Glue column comment (IS NULL removes it); the comments of Glue columns show in duckdb_columns(). Comments on partition keys and on views are not supported yet.
ALTER TABLE ... SET (key = 'value', ...) and RESET (key, ...) change the Glue table parameters (Hive's TBLPROPERTIES) with UpdateTable: SET adds or overwrites the listed keys, RESET removes them, and every other parameter and the rest of the definition stay as they are. Values are stored as strings (compression_level = 4 becomes '4'); a key may be quoted ('parquet.compression' = 'ZSTD'). The parameters the table format is read from (table_type, spark.sql.sources.provider, metadata_location) can not be changed this way.
CALL glue_replace_columns('cat.db.t', {id: 'BIGINT', name: 'VARCHAR'}, comments := {id: '...'}) is Hive's ALTER TABLE ... REPLACE COLUMNS: it replaces all data columns of the table at once, which can also rename and reorder them. Types are DuckDB types, stored the way CREATE TABLE stores them; a column that stays (same name, compared case-insensitively) keeps its stored name and may only be widened, as with ALTER COLUMN ... TYPE. As in Hive, comments not given in comments are dropped; keep_comments := true keeps those of the columns that stay (a NULL in comments then removes one). The partition keys are kept and must not be listed; the bucketing and sort columns must be listed. It returns the columns as stored in Glue (Glue type names). The data files are not rewritten: parquet, json and avro files are matched by name (a renamed column reads as NULL), csv files by position, so a csv table keeps its number of columns, each may only be widened and giving it a new name renames it.
DROP TABLE and DROP SCHEMA delete the Glue entries but leave the data files in S3. Glue deletes all tables of a database when the database is dropped, so DROP SCHEMA refuses a database that still has tables or views unless CASCADE is given.

Glue has no transactions: DDL takes effect immediately, files are visible as soon as they are written, and nothing is rolled back on failure. DELETE, UPDATE and MERGE INTO are not supported. DDL (CREATE/ALTER/DROP of schemas, tables and views, CREATE TABLE ... AS, the partition SQL, or glue_add_partition, glue_alter_table, ...) and INSERT inside an explicit BEGIN transaction are an error that aborts the transaction; end it with COMMIT, ABORT or ROLLBACK and run the statement again. The glue_* functions that change Glue (glue_add_partition, glue_drop_partition, glue_rename_partition, glue_set_partition_location, glue_set_table_location, glue_replace_columns) name themselves in that error and say to run them as a statement of their own, CALL glue_replace_columns(...). This also applies to SELECT * FROM glue_replace_columns(...) from a client that runs each query inside a transaction of its own.

Within a DuckDB transaction (BEGIN … COMMIT) a table's definition and its list of partitions are fetched from Glue once and reused; the files inside those partitions are listed at each scan. So a partition added outside the transaction becomes visible only after the transaction ends, while a file added to an existing partition is seen by the next query (the scans of one query share a listing) — or, under glue_cache_listings, after the next cache clear.

With SET glue_metadata_cache_scope = 'connection' the cache also serves the connection's later transactions. It is per connection — two connections never share one — so another connection's write stays unseen until something clears this connection's cache: a write through it, a rollback, or DETACH.

Views

CREATE [OR REPLACE] VIEW [IF NOT EXISTS] db.v [(col, ...)] AS SELECT ... and CREATE SECURE VIEW create a Glue view (a table of type VIRTUAL_VIEW) marked as written by DuckDB. OR REPLACE updates it in place. Unqualified table names in the SELECT refer to the view's own database. The view's columns store their DuckDB types (e.g. HUGEINT, INTEGER), not Glue types.
CREATE VIEW ... WITH (DEFER_BINDING) is not supported yet: the tables a view reads must exist when it is created.
A column in the SELECT can not start with the name of another catalog the view reads (cat.db.t.col, * EXCLUDE (cat.db.t.col)), or with its own catalog's name as a row reference (cat.t), even when that name is a table, alias or column: use a table alias with another name.
With SET experimental_glue_view_external_catalog_types = ['duckdb', 'iceberg'] (empty by default; global only, so it applies to every connection and SET SESSION is refused), views can also read tables of catalogs of those types, stored as "external:<type>:<alias>".<schema>.<table> and read wherever a catalog of that type is attached under that alias, never from one in memory, temporary, system or attached hidden. A bare table name in a body written to Glue by other means is read from the view's own Glue database.
SHOW [ALL] TABLES, duckdb_views(), information_schema.views and duckdb_columns() list every view of a database, including views written by other engines. Only views written by DuckDB can be queried, replaced or dropped.
DROP VIEW [IF EXISTS] db.v deletes the view.
ALTER VIEW ... RENAME and COMMENT ON VIEW are not supported.

Reading without a catalog: hive_scan

The same scan is available as a table function, for parquet Hive tables that are not (or not yet) registered in Glue. The schema and the partitions are given as arguments; nothing is looked up in a catalog.

SELECT * FROM hive_scan('s3://bucket/warehouse/orders',
    schema := {id: 'INTEGER', amount: 'DOUBLE', dt: 'VARCHAR', country: 'VARCHAR'},
    partitions := [
        {dt: '2016-05-14', country: 'IN', location: NULL},
        {dt: '2016-05-16', country: 'IN', location: 's3://bucket/imports/INDIA_16_May_2016'}
    ]);

schema (required): a struct of column name to DuckDB type name, data columns and partition columns.
format: 'parquet' (default), 'csv', 'json' or 'avro'. For csv, header := true, delim := '|', quote := '"', escape := '\' and nullstr := '\N' describe the files (an empty field is NULL unless nullstr says otherwise, as in read_csv); csv columns are matched by position, json keys by name. For json, ignore_errors := true reads a line that is not valid JSON as a row of NULLs.
partitions: one struct per partition with a value for every partition key and an optional location; without a location the partition lives at <root>/<key>=<value>/.... The partition keys are the struct fields other than location, in that order, unless partition_keys := [...] names them. Without partitions the table is unpartitioned and all files below the root are read; partition_keys without partitions is an error, and partitions := [] with partition_keys reads no rows.
A root or partition without data files scans as zero rows.

Reading a scan's files with a standard reader (C++)

An engine that hands some of a Hive scan's files to machines without this extension (and without Glue access) can read them there with a standard reader. functions/hive_reader_call.hpp gives what that read needs from a bound scan, of a Glue table or of a hive_scan() call, using only core DuckDB types:

HiveScanGetReaderCall(get): the reader (read_parquet, read_csv, read_json or read_avro), its named parameters and the scan's data columns as expressions over the reader's columns. The parameters are those the scan bound the reader with (a text table's dialect, header lines and NULL spelling, hive_csv_parallel and hive_csv_max_line_size as the planning session had them), plus the data columns as schema for Parquet and Avro and hive_partitioning := false. A column of a SerDe text table is the field read as text and converted with TRY_CAST (CAST without use.null.for.invalid.data in an OpenCSVSerde table, lower(field) = 'true' for an OpenCSVSerde boolean), of the field wrapped in __internal_barrier, so the caller's optimizer does not push the conversion into read_csv, which would convert the text itself and strictly. A field of an OpenCSVSerde table without escapeChar that holds a \ is first passed through regexp_replace, which drops the escape.
HiveScanGetFilePartitionValues(get, path): the values of the partition a listed file belongs to, as constants of the partition columns' types. The path is the file's as the scan listed it, or another spelling of it that DuckDB's Path normalizes to it (s3://b//x or s3://b/./x for s3://b/x).

The caller reads a set of the scan's files as SELECT <columns>, <partition values> FROM <reader>([<files>], <parameters>), one such read per partition, UNION ALLed. Neither function calls Glue or needs a ClientContext. Both return nothing for a scan they do not recognize or that reads a column, a partition key included, as another type than the table declares (a type the optimizer pushed into the scan), and HiveScanGetReaderCall also for a Parquet table with a binary column, whose string columns are decoded file by file, and for a scan that reads with file credentials. Both only work when the caller and this extension are linked into the same binary.

EXPLAIN (SQL) renders a Hive scan as these reads, so the SQL it prints reads the scan's files without this extension (and SET debug_verify_statement = 'explain_sql_strict' runs a query as that SQL). The scan lists its files for it: a partition the filters pruned is not read, a filter the partition values decided is restated over the reads, and a scan without files is a relation without rows. A scan no reader call can read (a Parquet table with a binary column, or one whose query reads $path, $file_size, $file_modified_time or another virtual column) is rendered as the Glue table it reads, by name; a hive_scan() call then cannot be rendered.

File credentials

An extension can give the scans of a Glue table credentials of their own, for example credentials a governance service vends for that table. It registers a GlueFileCredentialsCallback (glue_file_credentials.hpp) as an ExtensionCallback, which this extension asks each time a scan of a Hive table is bound. The callback's extension must be linked into the same binary as this one (it is found with dynamic_cast); one LOADed as a separate library is not found. The callback answers with the name of a secret, typically one it created for the transaction (SecretPersistType::TRANSACTION). Every file the scan lists then names that secret in its options (ExtendedOpenFileInfo::SECRET_OPTION), and DuckDB opens it with that secret instead of the one that matches its path. Without a callback, or when it returns no name, the files are read as above.

The listings themselves are made by path: DuckDB's globs name no secret. A secret of the transaction takes part in the path lookup, so a listing under its scope is made with it. So does every other read by path under its scope in that transaction, e.g. a read_parquet of the table's files or a table without credentials at the same location.
One query lists a directory once, for every scan of it with or without credentials; only the opens differ.
A file naming a secret that does not exist (e.g. a transaction secret, after the transaction), or whose scope does not cover it, fails to open rather than being read by path.
glue_cache_listings does not keep the listings of a scan with credentials beyond the query.
A scan with credentials serializes the name of its secret; EXPLAIN (SQL) renders it as the Glue table by name.
INSERT into a table whose scans read with credentials, and glue_repair_table of one, are refused: they list or write its files by path. The check asks the callback's GetFileSecret, as a scan of the table would.

The tests under test/sql/file_credentials/ give tables credentials with the test extension glue_test_credentials (test/extension/glue_test_credentials/), which must share a binary with this extension; extension_config.cmake links both into unittest.

Partitions

DuckDB has no ALTER TABLE ... PARTITION syntax, so the Hive partition statements are table functions. The partition is given as a struct naming every partition key; values are stored as strings in Glue, in partition key order. The table name may be partially qualified ('db.t', or 't' after USE cat.db); it is resolved like in a query.

function	Hive statement
`glue_partitions('cat.db.t')`	`SHOW PARTITIONS`: one row per registered partition, a typed column per partition key plus `location` and `partition_values`, the strings Glue stores (`__HIVE_DEFAULT_PARTITION__` for NULL); a typed column is NULL both for the NULL partition and for a value the key's type cannot hold, `partition_values` tells them apart. Both are numbered like a duplicate result column (`location_1`) when a partition key has their name
`glue_get_partition_response('cat.db.t', {dt: '2016-05-14', country: 'IN'})`	`DESCRIBE FORMATTED ... PARTITION (...)`: the partition's location, SerDe, columns and parameters, and its complete Glue definition in `response`
`CALL glue_add_partition('cat.db.t', {dt: '2016-05-14', country: 'IN'}, location := 's3://...', if_not_exists := false)`	`ALTER TABLE ADD [IF NOT EXISTS] PARTITION (...) [LOCATION ...]`; without `location` the partition lives at `<table location>/dt=2016-05-14/country=IN`
`CALL glue_drop_partition('cat.db.t', {dt: '2016-05-14', country: 'IN'}, if_exists := false)`	`ALTER TABLE DROP [IF EXISTS] PARTITION (...)`; the data files stay in S3
`CALL glue_rename_partition('cat.db.t', {dt: '2016-05-14', country: 'IN'}, {dt: '2016-05-15', country: 'IN'})`	`ALTER TABLE PARTITION (...) RENAME TO PARTITION (...)`; changes the values, keeps the location
`CALL glue_set_partition_location('cat.db.t', {dt: '2016-05-14', country: 'IN'}, 's3://...')`	`ALTER TABLE PARTITION (...) SET LOCATION '...'`
`CALL glue_set_table_location('cat.db.t', 's3://...')`	`ALTER TABLE SET LOCATION '...'`; existing partitions keep their locations, new ones land under the new location
`CALL glue_repair_table('cat.db.t')`	`MSCK REPAIR TABLE`: registers the Hive `<key>=<value>` partition directories under the table location that are not in Glue yet; one row per added partition (a typed column per partition key plus `location`)

The functions that change Glue are called with CALL, outside a BEGIN transaction, since Glue has no transactions (see Writing).

The Hive SQL forms are available as well, through the glue_hive_ddl grammar extension the extension registers. Grammar extensions are switched on per connection:

SET active_grammar_extensions = ['glue_hive_ddl'];
ALTER TABLE my_datalake.default.orders ADD IF NOT EXISTS
    PARTITION (dt = '2016-05-14', country = 'IN')
    PARTITION (dt = '2016-06-02', country = 'IN') LOCATION 's3://bucket/imports/INDIA_02_June_2016/';
ALTER TABLE my_datalake.default.orders DROP IF EXISTS PARTITION (dt = '2016-05-14', country = 'IN'), PARTITION (dt = '2016-05-15', country = 'IN');
ALTER TABLE my_datalake.default.orders PARTITION (dt = '2016-05-15', country = 'IN') RENAME TO PARTITION (dt = '2016-05-16', country = 'IN');
ALTER TABLE my_datalake.default.orders PARTITION (dt = '2016-05-16', country = 'IN') SET LOCATION 's3://bucket/other/';
ALTER TABLE my_datalake.default.orders SET LOCATION 's3://bucket/orders_v2/';

Actions can be chained in one statement (ADD PARTITION (...) LOCATION '...' ADD PARTITION (...) ...). The statement becomes CALL glue_alter_table(table, [actions]): every action is checked against Glue before any is applied, so a statement that fails changes nothing, and consecutive adds go out as one BatchCreatePartition call. The checks fetch the partitions they look at with BatchGetPartition, and consecutive drops go out as BatchDeletePartition calls of up to 25 partitions, so dropping partitions needs glue:BatchGetPartition and glue:BatchDeletePartition. The table name may be partially qualified; it is resolved like in a query.

glue_partitions lists every registered partition with a typed column per partition key, its location and partition_values, the strings Glue stores. A typed column is NULL for the NULL partition (Glue stores __HIVE_DEFAULT_PARTITION__) and for a value the key's type cannot hold, such as abc registered by another tool on an integer key; the latter fails every filtered read of the table, and partition_values is how to find it, one condition per partition key:

SELECT location, partition_values
FROM glue_partitions('my_datalake.default.orders')
WHERE (dt IS NULL) <> (partition_values[1] = '__HIVE_DEFAULT_PARTITION__')
   OR (country IS NULL) <> (partition_values[2] = '__HIVE_DEFAULT_PARTITION__');

glue_repair_table lists the table location once (never from a listing kept by glue_cache_listings) and registers the partitions Glue does not have yet with BatchCreatePartition. A partition is a directory with one <key>=<value> component per partition key, in partition key order (keys compare case-insensitively); it is registered at that directory, with the values unescaped and cast to the key types (year=07 is 7). Directories that do not follow this scheme, whose values do not cast, that are already the location of a partition, or that hold no file a scan would read (only _/. prefixed, _$folder$ or empty objects) are ignored. It never removes partitions. A table with another engine's InputFormat over a Hive SerDe (one INSERT refuses too) is refused: its files are not laid out the way Hive writes them.

Listing the partitions of a table - glue_partitions, and planning a scan of a partitioned table - pages through Glue's GetPartitions. A scan fetches them when planning first needs them (pruning, cardinality), not at bind, so DESCRIBE, CREATE VIEW and PREPARE make no GetPartitions call; every scan of the table in a query shares one fetch. The pages are asked for in parallel with Glue's Segment API: glue_get_partitions_segments requests run at the same time, each over a segment of the partitions that does not overlap with the others. 0, the default, uses 8 requests against AWS and 1 against a Glue compatible server given with ENDPOINT (moto ignores Segment and answers every segment with the whole table, so the partitions a segment already returned are dropped); 10 is the maximum Glue accepts. The responses include every partition's storage descriptor with its columns, which the scan checks the partitions it reads against.

Against AWS, listing the 2526 partitions of a TPC-H SF1 lineitem took ~2.1s before and ~0.45s with the default of 8 segments; SET glue_get_partitions_segments = 1 restores one request at a time.

Inspecting tables

Every table entry carries its format in duckdb_tables().tags['table_type'] (HIVE, ICEBERG, DELTA, HUDI, SYMLINK or UNKNOWN). The full Glue definition of a table is available through a table function:

SELECT * FROM glue_get_table_response('my_datalake.default.some_table');
SELECT response.StorageDescriptor.Location FROM glue_get_table_response('my_datalake.default.some_table');

It returns one row with the classification, the Glue table type, location, SerDe, columns, partition keys and parameters as columns, plus the complete Glue Table object as a VARIANT in response. A partially qualified name ('default.some_table', or 'some_table' after USE my_datalake.default) is resolved like in a query, through the table's catalog entry, so for a table DuckDB can not read (e.g. an unsupported column type) give the fully qualified name.

glue_get_partition_response('<table>', {dt: '2024-01-01', ...}) does the same for one partition of a Hive table: its values, location, SerDe, columns and parameters as columns, plus the complete Glue Partition object in response. The partition is given like for the other partition functions, as a struct naming every partition key.

SELECT location, response.StorageDescriptor.InputFormat
FROM glue_get_partition_response('my_datalake.default.some_table', {dt: '2024-01-01'});

glue_get_database_response('<catalog>.<database>') does the same for a Glue database (a DuckDB schema): its description, location and parameters as columns and the complete Glue Database object in response. An unqualified '<database>' is resolved like in a query, through the search path.

HTTP transport and logging

The AWS SDK's Glue calls are routed through DuckDB's HTTP layer (httpfs), so they honor DuckDB's proxy and certificate settings and appear in the HTTP log:

SET redact_http_logs = false; -- header values (incl. x-amz-target) are redacted by default
CALL enable_logging('HTTP', storage='memory');
-- ... run queries ...
SELECT request.type, request.url, request.headers['x-amz-target'], response.status FROM duckdb_logs_parsed('HTTP');

SET glue_network_calls_via_duckdb = false switches back to the SDK's own HTTP client (not under WebAssembly, where the SDK has none).

glue_requests() lists the Glue requests in the HTTP log, retries and pages included: operation (the X-Amz-Target, e.g. AWSGlue.GetPartitions), status, reason, start_time, duration_ms and the connection_id, transaction_id and query_id that made them. Only requests sent through DuckDB's HTTP layer are logged.

The GlueScan log type (INFO) records what each scan of a Hive table does, one entry per event, with the scan_id that numbers the scans of a query from 1:

partitions: the table's partitions were loaded, taking duration_ms;
plan: of partitions, partitions_read are read, with listings S3 listings; detail is root when they are listed in one listing of the table location, else partitions;
listing: one listing (detail: table, root or partition) of partitions_read partitions found files data files in duration_ms;
sample: the cardinality sample listed files files and measured one in duration_ms (files is NULL when nothing was listed: avro tables are not sampled). The scan reuses that listing, so a listing of the same directory then takes next to no time. Every scan of the table in the query shares the one sample.

CALL enable_logging(['HTTP', 'GlueScan'], storage = 'memory');
-- ... run queries ...
SELECT operation, count(*), sum(duration_ms) FROM glue_requests() GROUP BY operation;
SELECT "table", scan_id, event, partitions, partitions_read, listings, files, duration_ms, detail
FROM duckdb_logs_parsed('GlueScan');

The GlueRetry log type (WARNING) records each retry the AWS SDK's retry strategy decides on for a Glue API call, before the wait, with either HTTP client: the error that caused it (e.g. ThrottlingException), its HTTP status, the retry number (1 for the first) and the request_id of the failed attempt. status is NULL when the request got no response, error and request_id when the SDK has none (e.g. a dropped connection). The retry limit is glue_max_retries, or 10 by default. A retry the SDK makes to correct clock skew does not go through the retry strategy and is not logged.

CALL enable_logging('GlueRetry', storage = 'memory');
-- ... run queries ...
SELECT error, status, count(*) FROM duckdb_logs_parsed('GlueRetry') GROUP BY ALL;

Testing

The tests are written against two --test-config files, which decide where the catalog and the storage are:

test/configs/local_glue.json: moto serving the Glue API and SeaweedFS serving S3, both from scripts/docker-compose.yml, which also creates the bucket, the Glue database default and the bucketed table default.fixture_bucketed_multi (bucket columns without a NumberOfBuckets, which the extension does not create).
test/configs/cloud_glue.json: a live AWS Glue Data Catalog, with credentials from the AWS credential chain.

A config creates the S3 secret (on_init) and sets GLUE_CATALOG_ID, GLUE_ENDPOINT and DEFAULT_S3_LOCATION, which the tests use in their ATTACH; tests are skipped without a config (require-env GLUE_CATALOG_ID). Tests under test/sql/cloud/ read tables of the live account that the tests do not create and only run with the cloud config. The local config also sets GLUE_FAULT_ENDPOINT: scripts/glue_fault_proxy.py in front of moto, which answers GetTables on a database named fault_access_denied* / fault_invalid_input* with that Glue error, and carries out CreateDatabase / CreateTable on a fault_already_exists* database before answering AlreadyExistsException (what it creates differs from what was asked for), and refuses the first attempt of each request on a fault_throttled* database with ThrottlingException, letting the SDK's retries through. A table named fault_denied_* is one the caller may not describe, as Lake Formation leaves it: GetTable on it fails with AccessDeniedException and GetTables leaves it out. Tests that need the proxy are skipped without it. It also sets GLUE_LINK_PROXY_ENDPOINT: scripts/resource_link_proxy.py in front of moto, which answers for the database link_db as Glue does through a database resource link whose target is target_db.

make glue-fixture        # docker compose up (creates the bucket and the 'default' database)
make test-local          # unittest --test-config test/configs/local_glue.json 'test/sql/*', with retries
make glue-fixture-down

AWS_PROFILE=... AWS_CONFIG_FILE=~/.aws/config AWS_SHARED_CREDENTIALS_FILE=~/.aws/credentials make test-cloud

Both targets set AWS_EC2_METADATA_DISABLED=true: the test runner hides ~/.aws, and without a region from the environment or a profile the AWS SDK asks the EC2 instance metadata service for one, which off EC2 hangs for minutes per client. A test config can not export process environment variables, so this stays on the command.

make test-local runs the tests through DuckDB's duckdb/scripts/ci/run_tests.py (Python 3.10+; pick the interpreter with PYTHON=python3.14), one at a time in batches of TEST_BATCH_SIZE (10) tests per process, and reruns a failing batch up to twice: against the local servers a read right after a write occasionally comes back with no rows. Every retry is reported in the output. TEST_BUILD=release runs the release build instead of relassert. Every process first installs the loadable extensions from build/<type>/repository; on Linux the debug info of relassert makes them ~1.5 GB each, so CI tests a make release EXT_RELEASE_FLAGS=-DFORCE_ASSERT=1 build (assertions and sanitizers, no debug info).

The Cloud Glue tests workflow (.github/workflows/CloudGlueTests.yml) runs the whole suite, on a release build, against a live catalog. It runs on every push to main; when such a run fails, its job summary lists the failing tests and the merges since the last passing run on main. On a pull request it runs when a maintainer adds the cloud-tests label, and again on every push to a labeled PR whose author is a member or collaborator of the repository. Since the PR's code runs with the AWS credentials, read an outside contributor's PR before labeling it, and remove and re-add the label to test their later commits. A maintainer can also start it by hand (gh workflow run CloudGlueTests.yml -f pr_number=<n>, or gh workflow run CloudGlueTests.yml --ref main without a PR). Starting it by hand or through the label is refused for anyone below the maintain role. A run tests the PR's head commit (or the branch's) as it was when the run started, and reports the result as a Cloud Glue tests status on that commit. Runs share the catalog, so they go one at a time; GitHub keeps only the newest waiting run, which is why a failing run on main compares with the last passing one rather than the previous commit. Every run first deletes all Glue databases of the account and creates an empty default. The catalog, region and AWS credentials come from the variables and secrets of the cloud-glue environment, listed at the top of the workflow. The tests write to the bucket duckdb-glue-ci-<account id>-<region>, which the workflow creates if needed, under duckdb-aws-glue-cloud-tests/ci-<run id>-<attempt>/, which it deletes when the job ends, whether the tests pass or not; a lifecycle rule on duckdb-aws-glue-cloud-tests/ expires anything a failed run leaves behind after 2 days.

Every test creates the tables it needs and writes under its own {TEST_DIR} prefix, so runs do not interfere with each other; make glue-fixture-down throws the containers and their data away.

The benchmarks under benchmark/ read from the same local servers. They build their tables in the load step and use debug_fs_delay_mean_ms to add latency to every file open and read, standing in for the S3 round trip the local SeaweedFS does not have (make glue-fixture first; the benchmark runner needs a build with BUILD_BENCHMARK=1):

AWS_EC2_METADATA_DISABLED=true ./build/relassert/benchmark/benchmark_runner benchmark/heavily_partitioned_table.benchmark

benchmark/tpch/<sf>/<format>/ runs the 22 TPC-H queries against Hive tables of that format in the Glue database bench_tpch_<sf>_<format> (lineitem and orders partitioned by 10-day buckets of l_shipdate and o_orderdate) and checks the answers: sf1/parquet/ at SF1, sf0.1/csv/, sf0.1/json/ and sf0.1/avro/ at SF0.1 (the Glue database names spell the scale factor sf1, sf0_1, ...). The queries in benchmark/tpch/queries/ are DuckDB's with filters on the bucket columns added next to the date filters. The first run generates the data with dbgen and writes it with CTAS, which takes a while; later runs reuse duckdb_benchmark_data/glue_tpch_<sf>_<format>.duckdb, which make glue-fixture removes:

AWS_EC2_METADATA_DISABLED=true ./build/relassert/benchmark/benchmark_runner 'benchmark/tpch/sf1/parquet/.*'

benchmark/tpcds/<sf>/<format>/ does the same for the 99 TPC-DS queries, sf1/parquet/ at SF1 and the other formats at SF0.01 (sf0.01/<format>/), with the Glue database bench_tpcds_<sf>_<format>: the fact tables are partitioned by 10-day buckets of their date key (ss_sold_date_bucket, sr_returned_date_bucket, cs_sold_date_bucket, cr_returned_date_bucket, ws_sold_date_bucket, wr_returned_date_bucket, inv_date_bucket, each (date_sk - 2440588) // 10, the same buckets as the TPC-H tables), the dimension tables are not. The queries in benchmark/tpcds/queries/ are DuckDB's with filters on the bucket columns added where a fact table's date key is joined to a filtered date_dim. Its cache is duckdb_benchmark_data/glue_tpcds_<sf>_<format>.duckdb:

AWS_EC2_METADATA_DISABLED=true ./build/release/benchmark/benchmark_runner 'benchmark/tpcds/sf1/parquet/.*'

Both suites share a template per suite (benchmark/tpch/tpch.benchmark.in, benchmark/tpcds/tpcds.benchmark.in) that takes FORMAT, SF and SF_NAME. The TPC-DS load needs a build without assertions (BUILD_BENCHMARK=1 make release): its fact tables have rows with a NULL date key, and writing such a partition trips a D_ASSERT in DuckDB's partitioned copy (see the header of benchmark/tpcds/tpcds.benchmark.in for a repro without Glue).

benchmark/{tpch,tpcds}/sf10/parquet/ run the same queries at SF10 (the TPC-DS load writes ~4 GB to S3), and benchmark/{tpch,tpcds}/sf0.01/parquet/ at SF0.01, which has as many partitions and files as SF1 but almost no rows, so its timings are mostly the Glue calls, the S3 listings and the file opens. Neither runs in CI. (TPC-H q17 fails at SF0.01: DuckDB's answer file for it is empty, the query returns one NULL.)

Both loads create their Glue tables with CREATE TABLE IF NOT EXISTS ... AS, so changing a load has no effect while the tables are in Glue: rebuild the fixture with make glue-fixture-down && make glue-fixture first.

benchmark_runner runs the benchmarks below its own root (the repository), not the working directory. DuckDB's own TPC-H and TPC-DS benchmarks on native tables, the reference for these, run with --root-dir duckdb ('benchmark/tpcds/sf1/.*' --sf 10 for TPC-DS at SF10).

.github/workflows/Regression.yml builds the benchmark runner for a PR and for its merge base once, then compares their timings (5 runs each) in two jobs of about three minutes, so it runs only a few join-heavy TPC-H and TPC-DS queries. The parquet job runs benchmark/*.benchmark, the parquet benchmarks of benchmark/pushdown/ and benchmark/optimizer/ and TPC-DS q64 at SF1. The csv/json job runs the csv and json benchmarks of benchmark/pushdown/ and benchmark/optimizer/, TPC-H q09, q18 and q21 at SF0.1 and TPC-DS q14, q64, q72 and q78 at SF0.01 on csv tables. The other queries and the json and avro TPC benchmarks only run locally.

Building

VCPKG_TOOLCHAIN_PATH='<path to vcpkg>/scripts/buildsystems/vcpkg.cmake' GEN=ninja make relassert
ASAN_OPTIONS=detect_container_overflow=0 ./build/relassert/test/unittest test/sql/test_glue_attach.test

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
benchmark		benchmark
docs		docs
duckdb @ 925bcdf		duckdb @ 925bcdf
extension-ci-tools @ c286b12		extension-ci-tools @ c286b12
scripts		scripts
src		src
test		test
vcpkg_ports/aws-c-io		vcpkg_ports/aws-c-io
.clang-format		.clang-format
.clang-tidy		.clang-tidy
.editorconfig		.editorconfig
.gitignore		.gitignore
.gitmodules		.gitmodules
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
CMakeLists.txt		CMakeLists.txt
LICENSE		LICENSE
Makefile		Makefile
extension_config.cmake		extension_config.cmake
vcpkg.json		vcpkg.json

Navigation Menu

Repository navigation

About

Repository files navigation

DuckDB Glue extension

Reading

Writing

Views

Reading without a catalog: hive_scan

Reading a scan's files with a standard reader (C++)

File credentials

Partitions

Inspecting tables

HTTP transport and logging

Testing

Building

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

DuckDB Glue extension

Reading

Writing

Views

Reading without a catalog: hive_scan

Reading a scan's files with a standard reader (C++)

File credentials

Partitions

Inspecting tables

HTTP transport and logging

Testing

Building

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages