# MariaDB ↔ DuckDB Storage Engine: Architecture

## Overview

DuckDB is embedded into MariaDB as a **storage engine plugin** (`ha_duckdb`). MariaDB owns metadata (`.frm` files), SQL parsing, optimization, and the client protocol. DuckDB owns data storage, columnar analytical execution, and MVCC transactions. The two systems communicate through MariaDB's standard handler API and the `select_handler` interface for query pushdown.

```
┌───────────────────────────────────────────────────────────┐
│                     MariaDB Server                        │
│                                                           │
│  SQL Parser → Optimizer → Executor                        │
│       │            │          │                            │
│       │     ┌──────┘          │                            │
│       │     │  select_handler │  handler API               │
│       │     │  (pushdown)     │  (row-by-row)              │
│       ▼     ▼                 ▼                            │
│  ┌─────────────────────────────────────────┐              │
│  │        ha_duckdb  (storage engine)      │              │
│  │                                         │              │
│  │  DDL convertors ──┐                     │              │
│  │  DML convertors ──┼── SQL generation    │              │
│  │  select_handler ──┘                     │              │
│  │                                         │              │
│  │  DuckdbThdContext (per THD)             │              │
│  │   └─ duckdb::Connection                 │              │
│  │       └─ transaction, session state     │              │
│  │                                         │              │
│  │  DuckdbManager (singleton)              │              │
│  │   └─ duckdb::DuckDB instance            │              │
│  │       └─ catalog, buffer pool, WAL      │              │
│  └─────────────────────────────────────────┘              │
└───────────────────────────────────────────────────────────┘
```

## Plugin Lifecycle

### Initialization (`duckdb_init_func`)

1. Registers handlerton callbacks: `create`, `commit`, `rollback`, `close_connection`, `drop_database`, `create_select`, `create_unit`.
2. `DuckdbManager::CreateInstance()` opens a single `duckdb.db` file in the MariaDB datadir, creating the `duckdb::DuckDB` instance.
3. Registers SQL macros for MySQL compatibility (`adddate`, `datediff`, `curdate`, `convert_tz`, `substring_index`, `strcmp`, etc.).
4. Registers C++ UDFs via `register_mariadb_compat_functions()` — `mid`, `oct`, `bin`, `hex`, `locate`, `addtime`, `subtime`, `regexp_replace`, etc.
5. Registers cross-engine scan infrastructure: `_mdb_scan` table function + replacement scan callback.

### Shutdown (`duckdb_deinit_func`)

`DuckdbManager::Cleanup()` destroys the DuckDB instance and closes the database file.

---

## Query Processing Paths

### Path 1: SELECT Pushdown (`select_handler`)

Primary path for analytical queries. MariaDB hands the **entire SELECT** to DuckDB.

```
MariaDB parser
  → optimizer calls hton->create_select / hton->create_unit
    → can_pushdown_to_duckdb(): at least one DuckDB table?
      → creates ha_duckdb_select_handler
        → init_scan():
           1. Takes original SQL text from THD::query()
           2. Rewrites MariaDB-specific syntax for DuckDB:
              • GROUP BY ... WITH ROLLUP  →  GROUP BY ROLLUP(...)
              • CONVERT(expr, TYPE)       →  CAST(expr AS TYPE)
              • CURRENT_TIME()            →  current_time
              • STRAIGHT_JOIN             →  JOIN / CROSS JOIN
              • REGEXP / RLIKE            →  ~ / !~
              • LIMIT offset,count        →  LIMIT count OFFSET offset
              • Strips hints: HIGH_PRIORITY, SQL_NO_CACHE, FORCE INDEX, etc.
           3. Sends rewritten SQL to duckdb::Connection::Query()
        → next_row(): reads DataChunk row-by-row, converts Value → Field
        → end_scan(): releases result
```

#### Single-SELECT pushdown eligibility

The single-SELECT handler declines pushdown when MariaDB marks the top-level
`SELECT_LEX` with `UNCACHEABLE_SIDEEFFECT`. The regular handler path (`rnd_*`)
is used instead. MariaDB sets this flag for:

- reads and assignments of user variables (`@var`, `@var := expr`);
- reads of session or global system variables (`@@session.var`, `@@global.var`);
- `LAST_INSERT_ID()`, `BENCHMARK()`, `SLEEP()`, and `LOAD_FILE()`;
- named-lock functions (`GET_LOCK()`, `IS_FREE_LOCK()`, `IS_USED_LOCK()`,
  `RELEASE_LOCK()`, and `RELEASE_ALL_LOCKS()`);
- `SELECT ... INTO OUTFILE`, `INTO DUMPFILE`, or `INTO` variables;
- the `PROCEDURE` clause.

`UNCACHEABLE_RAND` is a separate flag and does not by itself disable DuckDB
pushdown. This eligibility check currently applies to the single-SELECT factory;
the whole-unit `UNION`/`EXCEPT`/`INTERSECT` factory does not perform the same
side-effect check.

### Path 1a: Cross-Engine Queries

When a SELECT mixes tables from **different engines** (DuckDB + InnoDB, etc.):

```
ha_duckdb_select_handler::init_scan()
  → registers non-DuckDB tables in a thread-local registry
  → sends SQL to DuckDB
    → DuckDB cannot find the table in its catalog
      → replacement scan callback redirects to _mdb_scan('table_name')
        → mdb_scan_function():
           • creates FiberScanState
           • in fiber: new THD, synthetic SELECT * FROM table WHERE …,
             mysql_execute_command()
           • MariaDB result rows → DataChunk via select_to_duckdb interceptor
           • yield() when DataChunk is full, continue() for next chunk
```

The fiber runs on the same OS thread as DuckDB. TLS (`current_thd`, `THR_KEY_mysys`) is swapped around fiber spawn/continue calls.

#### Pushdown modes: `duckdb_cross_engine_ryow`

The session variable `duckdb_cross_engine_ryow` (default `OFF`) selects, per
query, how external (non-DuckDB) tables are read. The flag is captured once in
`init_scan()` (`register_cross_engine_ryow()`), and the replacement scan then
redirects to one of two table functions.

| Aspect | `duckdb_cross_engine_ryow = OFF` (default) | `duckdb_cross_engine_ryow = ON` |
|---|---|---|
| Table function | `_mdb_scan` | `_mdb_scan_direct` |
| Scan mechanism | fiber runs a synthetic `SELECT … WHERE …` via `mysql_execute_command()` | direct `ha_rnd_init` / `ha_rnd_next` on the parent handler |
| Transaction | separate background-THD transaction | parent THD's transaction |
| Visibility | committed data only — the fiber runs in a *separate transaction* (background THD), so the parent's uncommitted writes are **not** visible (isolation follows the server default) | **Read-Your-Own-Writes**: the parent's uncommitted writes are visible |
| MariaDB optimizer for the external table | yes — range/index access, ICP (`idx_cond_push`), range-filter pushdown, index choice | no — plain full table scan |
| DuckDB `filter_pushdown` | `true` — DuckDB removes the predicates from the plan; the fiber applies them via the WHERE registered by `make_cond_for_table()` | `false` — DuckDB keeps the `Filter` operator above the scan and applies predicates itself |
| DuckDB `projection_pushdown` | `true` | `true` |

Why `filter_pushdown` differs: `PhysicalTableScan` returns the table function's
chunk **without re-applying** `table_filters`. When `filter_pushdown = true`
the optimizer strips those predicates and the scan is solely responsible for
them. The direct `ha_rnd_next` path does not evaluate DuckDB `input.filters`,
so `_mdb_scan_direct` sets `filter_pushdown = false` to keep correctness via
DuckDB's own `Filter` operator. Projection pushdown stays enabled in both modes
(the direct path honors `column_ids` through the read set).

Tradeoffs of `ON`: RYOW at the cost of losing the external engine's range/index
access and `idx_cond_push` for that table (it becomes a full scan). Note that
`cond_push` (engine condition pushdown) is unavailable for InnoDB/Aria/MyISAM/
RocksDB regardless of mode — only ICP is, and only in the `OFF` (fiber) path.

#### Future: index access / `idx_cond_push` in the RYOW path

The RYOW path currently loses row-count reduction because it does a full
`ha_rnd_next` scan. This can be improved without giving up RYOW or correctness:

1. `input.filters` (a DuckDB `TableFilterSet`) is already available in
   `mdb_scan_init_global`. Inspect it for sargable predicates
   (`CONSTANT_COMPARISON`, `IS NULL`, `IN`) on a key prefix of some index of
   the external `TABLE`.
2. Replace `ha_rnd_init` with `ha_index_init(keyno)` + `ha_index_read_map()`
   and an `end_range`, iterating with `ha_index_next()`. This recovers range
   access — the larger win — and needs no server `Item` tree.
3. Optionally, for residual index-column predicates, synthesize an `Item`
   expression from the `TableFilter`s over the table's `Field*` and call
   `tbl->file->idx_cond_push(keyno, item)`, so the engine checks them at the
   index level.

Key constraint: keep `filter_pushdown = false` even after this work. In DuckDB's
model the function must apply pushed filters *entirely* (no re-check), so a
partial/best-effort translation would be unsafe. Treating `input.filters` purely
as a seek/ICP *hint* — while DuckDB's `Filter` stays the source of truth — keeps
correctness independent of translation quality.

Caveats: `Item` synthesis needs a THD + memory root (use `tbl->in_use`);
constant/collation coercion between the DuckDB value and the MariaDB `Field`;
only AND-of-per-column predicates are sargable (OR is not); the parent handler
is mid-statement (opened and RDLCK-locked), so switch cleanly between rnd/index
scans and keep a single active scan; InnoDB refuses ICP on indexes with virtual
columns (`idx_cond_push` returns the condition), leaving a residual that must
fall back to DuckDB's `Filter`.

### Path 2: DDL via Handler API

```
MariaDB executor calls handler method
  → ha_duckdb::create() / delete_table() / rename_table() / truncate()
    → DDL convertor generates SQL for DuckDB
      (CreateTableConvertor, RenameTableConvertor, etc.)
    → SQL sent to DuckDB Connection
```

ALTER TABLE uses inplace ALTER:
```
check_if_supported_inplace_alter() → HA_ALTER_INPLACE_NO_LOCK
commit_inplace_alter_table()
  → AddColumnConvertor / DropColumnConvertor / ChangeColumnConvertor /
    ChangeColumnDefaultConvertor / ChangeColumnForPrimaryKeyConvertor
  → all generated operations execute in one explicit DuckDB transaction
```

Table partitioning is disabled with `HTON_NO_PARTITION`; both
`CREATE TABLE ... PARTITION BY` and `ALTER TABLE ... PARTITION BY` are rejected
before MariaDB creates a partition handler or starts the table-copy protocol.

DROP DATABASE: `duckdb_drop_database()` → `DROP SCHEMA IF EXISTS "db"`.

### Path 3: Row-by-Row DML

```
ha_duckdb::write_row() / update_row() / delete_row()
  → duckdb_register_trx(thd)
  → if BatchState == NOT_IN_BATCH:
      InsertConvertor / UpdateConvertor / DeleteConvertor
        → generates SQL: INSERT INTO … VALUES (…) / UPDATE … / DELETE …
        → duckdb_query(connection, sql)
  → if BatchState == IN_INSERT_ONLY_BATCH or IN_MIX_BATCH:
      DeltaAppender::append_row_insert/update/delete()
        → DuckDB Appender API → temp table → flushed at commit
```

### Path 4: Direct UPDATE/DELETE (Statement Pushdown)

For single-table statements with simple WHERE (`HA_CAN_DIRECT_UPDATE_AND_DELETE`):
```
ha_duckdb::direct_delete_rows() / direct_update_rows()
  → flush_appenders()          // ensure DuckDB sees consistent data
  → take original SQL from thd_query_string()
  → send entire statement to DuckDB
  → read affected-row count from result
```

### Path 5: Table Scan Fallback (rnd_*)

When `select_handler` is not applicable:
```
rnd_init() → SELECT * FROM "schema"."table" → sent to DuckDB
rnd_next() → reads DataChunk chunk-by-chunk, converts values
rnd_end()  → releases QueryResult
```

---

## Transaction Coordination

```
MariaDB THD
  │
  ├─ external_lock(F_WRLCK/F_RDLCK)
  │    └─ duckdb_register_trx()
  │         ├─ trans_register_ha(stmt)   // register with MariaDB TC
  │         ├─ trans_register_ha(all)    // if explicit transaction
  │         └─ DuckdbThdContext::duckdb_trans_begin()  // BEGIN in DuckDB
  │
  ├─ write_row / update_row / …
  │
  ├─ duckdb_prepare()                   // flush appenders
  │
  ├─ duckdb_commit()
  │    ├─ flush_appenders()             // safety net if prepare was skipped
  │    └─ DuckdbThdContext::duckdb_trans_commit()
  │
  └─ duckdb_rollback()
       └─ DuckdbThdContext::duckdb_trans_rollback()
```

XA transactions are rejected (`reject_xa_if_active`).

---

## Batch DML (DeltaAppender)

For bulk INSERT (and mixed INSERT/UPDATE/DELETE within a single transaction):

1. Creates a **temporary table** cloning the target schema + 3 auxiliary columns (`#mdb_delete_flag`, `#mdb_row_no`, `#mdb_trx_no`).
2. Writes rows via the `duckdb::Appender` API (much faster than per-row SQL INSERT).
3. On `flush()` (at commit/prepare): executes `INSERT INTO target SELECT … FROM tmp` and/or `DELETE FROM target WHERE pk IN (SELECT pk FROM tmp WHERE delete_flag)`.
4. The temp table lives in the connection-local temporary catalog — invisible to other sessions.

---

## SQL Translation Layer

### DDL Convertors (`convertor/ddl_convertor.cc`)
Translate MariaDB DDL → DuckDB DDL:
- Data type mapping (MariaDB types → DuckDB types)
- Expression defaults (read from `TABLE_SHARE::vcol_defs` binary blob, since the Item tree is invalid at `ha_duckdb::create()` time)
- PRIMARY KEY → NOT NULL constraints (DuckDB has no indexes)
- ADD/DROP/CHANGE COLUMN, RENAME TABLE

### DML Convertors (`convertor/dml_convertor.cc`)
Generate INSERT/UPDATE/DELETE SQL from MariaDB row buffers.

### MySQL Compat Functions (`runtime/duckdb_mysql_compat.cc`)
~1000 lines of C++ scalar UDFs registered in DuckDB for MariaDB SQL compatibility: `mid`, `oct`, `bin`, `hex`, `locate`, `addtime`, `subtime`, `regexp_replace`, `field`, `elt`, etc.

### DuckDB Value → MariaDB Field (`convertor/duckdb_select.cc`)
`store_duckdb_field_in_mysql_format()` converts DuckDB `Value` objects to MariaDB `Field::store()` calls when reading query results.

---

## Source Code Map

```
storage/duckdb/
├── ha_duckdb.h/cc                — handler class, handlerton, DDL/DML entry points
├── ha_duckdb_pushdown.h/cc       — select_handler: SELECT/UNION pushdown + SQL rewriting
├── duckdb_udf.cc                 — UDF plugin registration
│
├── convertor/
│   ├── ddl_convertor.h/cc        — CREATE/ALTER/DROP/RENAME SQL generation
│   ├── dml_convertor.h/cc        — INSERT/UPDATE/DELETE SQL generation
│   └── duckdb_select.h/cc        — DuckDB Value → MariaDB Field conversion
│
├── runtime/
│   ├── duckdb_manager.h/cc       — DuckdbManager singleton, DuckDB instance lifecycle
│   ├── duckdb_context.h/cc       — DuckdbThdContext: per-THD connection + transactions
│   ├── duckdb_query.h/cc         — duckdb_query() wrappers
│   ├── duckdb_mysql_compat.h/cc  — C++ UDFs for MySQL function compatibility
│   ├── cross_engine_scan.h/cc    — _mdb_scan table function + replacement scan
│   ├── fiber_context.h/c         — stackful coroutine (fiber) primitives
│   ├── fiber_scan.h/cc           — fiber-based MariaDB scan for cross-engine queries
│   └── delta_appender.h/cc       — batch DML via Appender API + temp tables
│
├── common/
│   ├── duckdb_config.h/cc        — global sysvar definitions
│   ├── duckdb_charset_collation.h/cc
│   ├── duckdb_timezone.h/cc
│   ├── duckdb_types.h/cc         — DatabaseTableNames, path parsing
│   ├── duckdb_log.h/cc           — logging control
│   ├── duckdb_handler_errors.h   — error code constants
│   └── row_helpers.h             — byte-level row reading utilities
│
├── cmake/                        — DuckDB build integration (ExternalProject_Add)
└── third_parties/duckdb/         — upstream DuckDB submodule (v1.5.5)
```
