Add database engine plugins (external engines)

[asmyasnikov]

This PR resolves issue #4158 and adds support for database engine plugins: external processes that implement a single Parse RPC and allow sqlc to work with databases that are not built-in (e.g. CockroachDB, TiDB, or custom SQL dialects). The plugin contract is deliberately minimal: no AST, no compiler in the middle, and a straight path from plugin output to codegen.

Motivation

1. sqlc is widely used, and requests to add support for new databases come in regularly: feat(clickhouse): add ClickHouse database engine support #4244, feat(sqlserver): add Microsoft SQL Server support #4243, YDB support (draft) #4090, Add ClickHouse Engine Support to sqlc #4220, Adding support for CockroachDB #4009.

2. Supporting new database engines increases maintenance burden on core maintainers (parsers, catalogs, dialects, and compatibility for each engine).

3. Exposing a way to add external database engines (Refactor project for easy adding support of new database engine without sqlc core changes #4158) lets the community ship new backends without growing sqlc’s core, and lets users adopt sqlc for more SQL dialects.

4. What should an engine plugin do?

   • 4.1) Validate the query. If the query is invalid, the plugin returns an error with a clear message. This is straightforward.
   • 4.2) Parse the query — but into what? If the plugin were to return an AST in sqlc’s format (i.e. by making internal/sql/ast public), we risk that a new dialect cannot be expressed with the existing AST node types. This PR therefore does not have the plugin pass an AST into sqlc’s core.
   • 4.3) Instead, the plugin parses the SQL and returns to the core the query parameters and the types and names of the result columns. That is enough for type-safe codegen.
   • 4.4) Optional behavior (wildcard expansion, etc.) is left to the plugin. To support it, the core passes either schema.sql or database connection parameters into the plugin.
   • 4.5) The core then feeds the plugin’s output directly into the codegen plugin. So an external database engine plugin bypasses the compiler and goes straight to code generation.

With this design, the engine plugin system can support many SQL dialects, while validation and enrichment of SQL (e.g. * expansion) are the plugin’s responsibility, not the core’s.

---

Pipeline: built-in engine vs external plugin

The choice between "built-in engine" and "external plugin" is made once per sql[] block in internal/cmd/process.go, based on engine: in config: if it is sqlite, mysql, or postgresql → built-in path; if it is a name listed under top-level engines: → plugin path. (Vet has no plugin logic; for plugin-engine blocks it fails with "unknown engine".)

flowchart TB
  subgraph input["input"]
    schema[schema.sql]
    config[sqlc.yaml]
    queries[queries.sql]
  end
  subgraph sqlc_core["SQLC"]
    direction TB
    subgraph builtin_flow["Built-in Engine processing"]
      direction TB
      parser["Parse (postgresql, mysql, sqlite)"]
      ast[(AST)]
      catalog[(Catalog)]
      compiler[Compiler]
      convert_compiler_results_to_codegen[Queries + types]
      parser --> ast
      parser --> catalog
      schema --> parser
      queries --> parser
      ast --> compiler
      catalog --> compiler
      compiler --> convert_compiler_results_to_codegen
    end
    subgraph plugin_flow["Plugin Engine processing"]
      direction TB
      adapter["Engine process runner"]
      ext_parse["call Parse from external Engine"]
      convert_plugin_response_to_codegen["Queries + types"]
      adapter --> |"ParseRequest"| ext_parse
      schema --> adapter
      queries --> adapter
      ext_parse --> |"ParseResponse"| convert_plugin_response_to_codegen
    end
    branch --> builtin_flow["Built-in"]
    branch --> plugin_flow
  end
  config --> branch{"Engine Type"}
  convert_plugin_response_to_codegen --> codegen
  convert_compiler_results_to_codegen --> codegen
  subgraph output["Codegen"]
    codegen[Codegen plugin]
  end

Loading

• Built-in path: Parser → AST, schema → Catalog; Compiler uses both and produces Queries + types. Codegen receives that as usual.

• Plugin path: Engine process runner sends one ParseRequest per query file: sql is the entire contents of that file, plus schema_sql or connection_params. The plugin returns ParseResponse.statements — one Statement per query block in the file (each with name, cmd, sql, parameters, columns). Each statement is turned into a compiler query and passed into the same ProcessResult/codegen path, so N blocks → N codegen queries (N helpers). No AST, no compiler — validation and enrichment are the plugin's job.

• No intermediate AST: the plugin returns already “resolved” data (SQL text, parameters, columns).

• No compiler for the plugin path: type resolution, * expansion, and validation are the plugin’s job. sqlc does not run the built-in compiler on plugin output.

• Data from the plugin is passed through to the codegen plugin via a thin adapter that maps each Statement to a compiler query (name, cmd, sql, parameters, columns). N statements → N queries → N generated helpers.

So: for external engines, the pipeline is schema + full query file → engine plugin (Parse) → N statements → N codegen queries, with no AST and no compiler in between.

Call flow (built-in vs plugin)

The split between built-in and plugin engines happens in internal/cmd/process.go inside processQuerySets(), which branches on config.IsBuiltinEngine(combo.Package.Engine).

• Built-in path: For engine: sqlite | mysql | postgresql, processQuerySets calls parse() (defined in internal/cmd/generate.go). parse() calls compiler.NewCompiler(sql, combo, parserOpts). NewCompiler in internal/compiler/engine.go has only three cases (SQLite, MySQL, PostgreSQL) and default: return nil, fmt.Errorf("unknown engine: %s", conf.Engine); there is no plugin branch and no FindEnginePlugin in the compiler.

• Plugin path: For any other engine: (e.g. a name defined under engines: in config), processQuerySets calls runPluginQuerySet() in internal/cmd/plugin_engine.go. parse() and NewCompiler are not used; the engine process runner talks to the external plugin and passes the result into the same codegen pipeline.

• Vet: The vet command uses its own path and calls parse() / NewCompiler(), so vet fails for plugin engines with "unknown engine" (there is no plugin-specific branch in vet).

Summary: The branch and orchestration live in process.go. Built-in logic goes through generate.go → compiler/engine.go; plugin logic is in plugin_engine.go.

No intermediate AST for external plugins

The plugin does not return an AST:

• Request: entire query file content (sql) + schema (or connection). One Parse call per query file.
• Response: repeated Statement statements. Each Statement has: name, cmd (enum: :one / :many / :exec / etc.), sql, parameters, columns. The plugin returns one Statement per query block in the file.

The plugin is the single place that defines how the query is interpreted. sqlc does not parse or analyze that SQL again; it maps each Statement to a codegen query and forwards them. N blocks in the file → N statements → N generated helpers.

---

No compiler for external plugins

The built-in compiler (catalog, type resolution, validation, expansion of *) is not used for external engine plugins:

• The plugin is responsible for:
  • Resolving parameter and column types (using schema or DB).
  • Expanding SELECT * if desired.
  • Emitting whatever shape of parameters and columns the codegen expects.
• sqlc does not run the compiler on plugin output; it passes that output through to codegen. So “compiler” is only in the built-in-engine path.

---

What is sent to and returned from the plugin

Invocation: one RPC, Parse, over stdin/stdout (protobuf).
Example: sqlc-engine-mydb parse with ParseRequest on stdin and ParseResponse on stdout.

Call pattern: sqlc issues one Parse call per query file. For each file (e.g. query.sql), the full file contents are sent in a single request. The plugin returns one Statement per query block in that file.

Sent to the plugin (ParseRequest)

Field	Description
sql	Entire contents of one query file (all blocks, including -- name: X :cmd comments).
schema_sql	(optional) Contents of the schema file(s), e.g. concatenated schema.sql.
connection_params	(optional) DSN + options for “database-only” mode when schema is taken from the DB.

Exactly one of schema_sql or connection_params is used per request, depending on project configuration.

Returned from the plugin (ParseResponse)

Field	Description
statements	One Statement per query block in the request.

Each Statement has:

Field	Description
name	Query name (from -- name: GetUser etc.).
cmd	Command/type: enum Cmd (CMD_ONE, CMD_MANY, CMD_EXEC, CMD_EXEC_RESULT, CMD_EXEC_ROWS, CMD_EXEC_LAST_ID, CMD_COPY_FROM, CMD_BATCH_EXEC, CMD_BATCH_MANY, CMD_BATCH_ONE). Matches sqlc’s :one, :many, :exec, etc.
sql	Processed SQL for that block (as-is or with * expanded).
parameters	Parameters for this statement (name, position, data_type, nullable, is_array, array_dims).
columns	Result columns for this statement (name, data_type, nullable, is_array, array_dims, optional table/schema).

There is no top-level sql / parameters / columns; the response is only statements. N blocks in the file → N statements → N codegen queries (N helpers).

Helpers in pkg/engine for plugin authors

To split the query file and parse "-- name: X :cmd" lines in the same way as built-in engines, the engine package provides (optional) helpers that delegate to internal/metadata:

• CommentSyntax — which comment styles to accept (Dash, SlashStar, Hash).
• ParseNameAndCmd(line, syntax) — parses a line like "-- name: ListAuthors :many" → (name, cmd Cmd, ok).
• QueryBlocks(content, syntax) — splits file content into []QueryBlock (Name, Cmd, SQL).
• StatementMeta(name, cmd, sql) — builds a *Statement with name/cmd/sql set; the plugin fills parameters and columns.
• CmdToString(c) / CmdFromString(s) — convert between enum Cmd and strings like ":one", ":many".

---

How the schema is passed into the plugin

Schema is provided to the plugin in one of two ways, via ParseRequest.schema_source:

1. Schema-based (files)

   • sqlc reads the configured schema files (e.g. schema: "schema.sql") and passes their contents as schema_sql (a string) in ParseRequest.
   • The plugin parses this SQL (e.g. CREATE TABLE ...) and uses it to resolve types, expand *, etc.

2. Database-only

   • When schema is not from files, sqlc can pass connection_params (DSN + optional extra options) in ParseRequest.
   • The plugin connects to the DB and uses live metadata (e.g. INFORMATION_SCHEMA / pg_catalog) to resolve types and columns.

So: schema is either “schema.sql as text” or “connection params to the database”; the plugin chooses how to use it.

---

Changes in sqlc.yaml

New top-level engines

Plugins are declared under engines and referenced by name in sql[].engine:

version: "2"

engines:
  - name: mydb
    process:
      cmd: sqlc-engine-mydb
    env:
      - MYDB_DSN

sql:
  - engine: mydb
    schema: "schema.sql"
    queries: "queries.sql"
    codegen:
      - plugin: go
        out: db

• engines: list of named engines. Each has name and either process.cmd (and optionally env) or a WASM config.
• sql[].engine: for that SQL block, use the engine named mydb (which triggers the plugin) instead of postgresql / mysql / sqlite.

So the only new concept in config is “define engines (including plugins) by name, then point sql[].engine at them.” Schema and queries are still configured per sql[] block as today.

---

Who handles sqlc placeholders in queries

Support for sqlc-style placeholders (sqlc.arg(), sqlc.narg(), sqlc.slice(), sqlc.embed(), etc.) is entirely up to the plugin:

• The plugin receives the full query file content (including those macros) in ParseRequest.sql.
• It can parse and interpret them and reflect the result in parameters (and, if needed, in sql or in how it uses schema). There is no separate “sqlc placeholder” pass in the core for the plugin path.
• If the plugin does not handle a placeholder, that placeholder will not be turned into proper parameters/columns by sqlc; the pipeline does not add a generic placeholder expander for external engines.

So: the database engine plugin is responsible for understanding and handling sqlc placeholders for its engine.

---

Summary for maintainers

• One Parse call per query file: Parse(full_query_file_content, schema_sql | connection_params) → ParseResponse{ statements }.
• Response: only repeated Statement statements. Each Statement has name, cmd (enum), sql, parameters, columns. No top-level sql/parameters/columns.
• N blocks → N statements → N codegen queries (N helpers).
• No AST, no compiler on the plugin path; each Statement is mapped to a compiler query and passed to codegen.
• Schema is passed either as schema_sql (file contents) or as connection_params (DSN) in ParseRequest.
• Config: engines[] + sql[].engine: <name>; existing schema / queries / codegen stay as-is.
• Placeholders: handled inside the plugin; core does not add a generic layer for external engines.
• Helpers in pkg/engine (ParseNameAndCmd, QueryBlocks, StatementMeta, Cmd enum) reuse internal/metadata so plugin behavior matches built-in engines.

This keeps the plugin API small and leaves type resolution and dialect behavior inside the plugin, while still allowing sqlc to drive generation from a single, well-defined contract.