Repository navigation
Multiple FORMATs merge instead of last-one-wins - #572
Conversation
|
In general this looks good, but I do have some concerns over how defaults are handled during overrides. For the column type specifically we would have retarget Something akin to this is happening in renaming because a default value is deduced from the content ( will be a no-op. Probably an edge case but super surprising if encountered. The better approach is for both structs to encode the content as an |
|
For this piece: I'd rather require For this piece: Yeah I agree |
|
hmm... I think I disagree... The default is on the column, not on the query. So However, if we have already given the column a type from a prior FORMAT clause then the column will respond with that This is not how it is directly implemented, I know, but that is how I think about it |
A FORMAT clause that never names a wildcard template now leaves the
field None instead of storing the default "{}", so merging stacked
FORMAT clauses on one column only overrides a template a clause
actually set. parse_renaming_clause returns Option<String> for the
template to keep an explicit * => '{}' distinguishable from no
wildcard, letting a later clause reset an earlier template.
A FORMAT clause without a target identifier (or with the new ANY keyword) now makes no claim about which table section its columns belong to: Format.target is Option<ColumnSection>, merging stacked clauses lets concrete BODY/STUB targets win over ANY, and unclaimed columns resolve to the body in setup_formats. Previously a bare FORMAT counted as a BODY claim and would silently pull columns out of the stub when combined with FORMAT STUB.
|
The solution I've taken is to have the |
FORMAT * sets a wildcard selection, expanded to the data's columns in setup_formats via the new expand_format_columns helper (which also absorbs SPAN id expansion and unknown-column validation). Wildcard clauses merge like any other, so an explicit clause can still refine a single column after a catch-all FORMAT *.
|
I'm now also allowing |
|
We can merge this into the main PR if fine with you |
* Table plumbing (#550) * Rename render::Spec to ResolvedPlot Frees up the `Spec` name for future enum meaning both table and plot output kinds. * Add Spec enum for Plot/Table plumbing Parse query / build AST will now produce Spec::{Plot,Table}. Up- and downstream plumbing still missing. * Wire TABULATE into grammer. Wire TABULATE into the grammar Adds tabulate_statement/tabulate_keyword to grammar.js, parallel to VISUALISE, so build_ast constructs real Spec::Table values instead of the type existing but never being reachable from source text. TABULATE FROM reuses the same restricted single-source grammar as VISUALISE FROM (no joins/comma-lists) via a shared single_source_from rule; Table itself stays fieldless, so a parsed FROM source is discarded rather than stored. Generalizes the SQL/Spec boundary detection that VISUALISE previously owned alone: SourceTree::extract_visualise() (now extract_spec(), to match the Spec enum), extract_sql()'s boundary check, and check_no_double_from() all recognize TABULATE too, so a TABULATE-only query no longer has its keyword silently swallowed into "SQL" text. Renames Validated::has_visual to has_spec for the same reason. Table's own validation/execution paths remain known gaps (see inline "Known gap" comments and golden tests in validate.rs/execute/mod.rs) — this is grammar and parse-tree plumbing only. * Add source field to Table Table's first real field: source: Option<DataSource>, populated from TABULATE FROM the same way Plot.source is from VISUALISE FROM (same single_source_from grammar node, same parse_data_source parsing). New build_tabulate_statement mirrors build_visualise_statement's own inline single_source_from handling rather than sharing a helper with it — the extraction is only 3 lines at 2 call sites, and the other existing parse_data_source caller (mapping_clause's layer_source) uses a different field name, so a shared wrapper or a field-name parameter would've been a new pattern not otherwise used in this file. build_ast's TABULATE arm now checks table.source.is_some() directly instead of querying the CST, so the "FROM after a trailing SELECT" validation is symmetric with VISUALISE's own check — only the error message text is still duplicated between the two arms (left as a TODO). * Share FROM-injection between VISUALISE and TABULATE in extract_sql extract_sql's "append SELECT * FROM <source>" rewrite was anchored to visualise_statement only, so a TABULATE FROM query lost its source entirely from the extracted SQL text. Both statement kinds now get the same treatment, via a query explicitly anchored to visualise_statement/tabulate_statement (rather than a bare single_source_from match) so this can't silently start matching some unrelated future use of that node. TABULATE FROM x and SELECT * FROM x TABULATE now extract to the identical SQL, mirroring the equivalence VISUALISE FROM already has with a bare SELECT. Updates the validate.rs golden test that was locking in the old (gap) behavior; the remaining gap — no actual validation of a Table's contents — is unchanged and still documented. * Add ResolvedTable and ResolvedSpec plumbing ResolvedTable (table, body, sql, warnings) and ResolvedSpec (Plot(Box<ResolvedPlot>) | Table(ResolvedTable)) are the execution-time counterparts to Table and Spec, mirroring the existing ResolvedPlot shape. ResolvedTable's impl is colocated in reader/spec.rs rather than a new file, since it's small enough not to warrant one yet. body's type (a plain DataFrame) is explicitly marked provisional, not a settled design — it's enough to design the execution plumbing against, but will likely need to become a table-specific representation once real table writers exist. Not yet wired up: Reader::execute() still returns Result<ResolvedPlot> and Writer::render() still takes &ResolvedPlot. ResolvedTable::new has no call site yet (temporary dead_code warning, same as ResolvedPlot:: new's own single-call-site pattern) until the execute/table.rs resolver and the Plot/Table dispatch in execute_with_reader exist — that dispatch and the Reader/Writer signature change land together as one atomic step, since every caller does execute() then render() back to back. * Add resolve_table_with_reader execute::table::resolve_table_with_reader(query, reader) -> Result<ResolvedTable> is the Table-side substitute for prepare_data_with_reader + execute_with_reader combined — Table has no per-layer/scale/facet resolution step to justify a separate intermediate struct the way Plot's PreparedData does, so both collapse into one function here. Parses, takes the first Table spec, gets SQL via extract_sql() (now shared with VISUALISE), runs it via reader.execute_sql(), and wraps the result into ResolvedTable. Confirms end to end that "TABULATE FROM x" and "SELECT * FROM x TABULATE" resolve identically, not just that they extract to the same SQL text. Known gap, called out in the doc comment: unlike the Plot pipeline, setup statements (INSTALL, LOAD, SET, etc.) ahead of a TABULATE aren't executed here yet. Not yet wired up: no caller exists for this function until the Reader::execute()/Writer::render() dispatch step lands. * Wire ResolvedSpec into Reader::execute()/Writer::render() Reader::execute() now returns Result<ResolvedSpec> and dispatches (via renamed execute_with_reader, which peeks at the query's first Spec) to either resolve_plot_with_reader (renamed from the old execute_with_reader) or resolve_table_with_reader. Writer::render() takes &ResolvedSpec and returns a clean WriterError for the Table case — no writer renders tables yet. Both were tried as a single default-method consolidation first; abandoned once it hit a real constraint, not a style choice: a default method can't unsize &Self into &dyn Reader without a Self: Sized bound, which would remove execute from dyn Reader's vtable. execute() stays a required method, with the 8 implementations as one-liners. Updates ~60 call sites accordingly. Tests route through the new as_plot()/as_table()/into_plot()/into_table() accessors (mirroring Spec's own API) to keep exercising ResolvedPlot-specific assertions while still passing the original ResolvedSpec through to render(); each test binds `let plot = spec.as_plot().unwrap();` once rather than repeating the chain per line. Real call sites (ggsql-cli, ggsql-jupyter, the doc-example harness) each get a deliberate, clear error/skip for the Table case instead of a blind unwrap; ggsql-wasm needed no changes. Adds one integration test confirming the dispatch itself (not just its two pieces) routes correctly and that rendering a table fails cleanly rather than panicking. * Split Writer::write/validate into Plot/Table-specific methods write_plot/validate_plot are now the real, required methods; write_table is a new default method (returns WriterError) that render() dispatches to for ResolvedSpec::Table, mirroring how write_plot handles the Plot case. Every writer name now says which side it's for, matching the naming clarity already applied on the Reader side (resolve_plot_with_reader/resolve_table_with_reader). * Add minimal HtmlWriter stub HtmlWriter is the first writer to actually implement write_table: a bare <table>, no styling/headings/spanners/footnotes, since Table has no fields to describe those yet. Proves the Table -> ResolvedTable -> Writer plumbing end to end; not the real grammar-of-tables output. Deliberately does not reuse ggsql-jupyter's existing dataframe_to_html (used for its pure-SQL fallback display) - that function is built around DataFrame specifically, and ResolvedTable.body's type is itself still provisional, so tying a new writer to it now would be reuse for its own sake rather than because the shapes are known to match long-term. Has its own small escape_html for the same reason; noted as a near-term follow-up that this duplicates logic already present in ggsql-jupyter (and, more loosely, ggsql-cli's doc-example harness) and is worth consolidating into a shared utility in core ggsql. No feature flag of its own (pure string formatting, no new dependencies) - reachable wherever the writer module already is. * Consolidate `escape_html` across crates * Wire ggsql-cli to HtmlWriter; unblock writer module from vegalite ggsql exec/run now accept --writer html, rendering a TABULATE result via HtmlWriter. render_spec matches on ResolvedSpec::Plot/Table directly instead of rejecting Table upfront; Plot-only pre-checks (metadata, empty-layer) now only run for the Plot case, and the Text-output message no longer hardcodes "Vega-Lite JSON" now that PNG and HTML share the same output path. Wiring HtmlWriter through the CLI kept needing awkward per-call-site vegalite gates, because lib.rs's pub mod writer was entirely gated on `feature = "vegalite"` even though the Writer trait, WriterOptions, PngWriter and HtmlWriter have no real dependency on it. Fixed the root cause instead of accumulating workarounds: the gate is removed; VegaLiteWriter still gates itself internally, untouched. That fix surfaced one real, previously-silent test gap: data.rs's test_ribbon_transposed_vegalite_encoding used VegaLiteWriter unconditionally inside a test module gated only on duckdb+builtin-data. Given its own #[cfg(feature = "vegalite")]. * Wire TABULATE output through HtmlWriter in ggsql-jupyter A TABULATE query previously reached the hardcoded VegaLiteWriter and errored. ExecutionResult gains a Table variant carrying pre-rendered HTML; the executor dispatches on ResolvedSpec (Table -> HtmlWriter, Plot -> VegaLiteWriter) and display_data wraps the HTML directly, with no Positron-specific sizing since a plain <table> needs none. * honour setup statements * dedup error message efforts * Validate TABULATE has a data source before executing A bare TABULATE with no FROM and no preceding SQL previously reported valid: true from validate() and only failed once SQL actually ran, since Table specs were silently dropped from validation entirely. build_ast's specs are now split into both plots and tables so this is caught up front, mirroring resolve_table_with_reader's existing rejection of the same query at execution time. * Fix bugs and gaps found in a pre-emptive table_plumbing review Ran the code-review skill against main...HEAD before sending the branch out for real review; this addresses the fixable findings. extract_sql/extract_spec scoped their FROM-injection query to the whole document root instead of the statement Reader::execute() is actually resolving, so `TABULATE VISUALISE FROM sales DRAW point` silently borrowed the VISUALISE's FROM for the source-less TABULATE instead of erroring. Added first_stmt (returns the first visualise_statement/tabulate_statement node) and scoped the FROM query to it. A query mixing VISUALISE and TABULATE silently resolved only the first spec and dropped the rest with no diagnostic. validate() now warns whenever a query has more than one such statement, threaded through into ResolvedPlot/ResolvedTable's existing warnings() and surfaced via ggsql-cli's stderr and ggsql-jupyter's tracing log. The "use --writer html" hint for a TABULATE query only fired for svg/pdf/hep. All eight Plot-only writers (including the default, vegalite, and the four raster writers) now route through require_plot, so the hint is consistent regardless of --writer. build_ast collected visualise_statement/tabulate_statement nodes via two find_nodes calls plus a sort_by_key; a single alternation query now yields them already in document order in one tree traversal. tree-sitter-ggsql/CLAUDE.md's grammar description omitted TABULATE from the top-level query rule; it now names tabulate_statement explicitly and flags it as a work in progress. Two other findings from the review — HtmlWriter's duplication of ggsql-jupyter's dataframe_to_html, and the missing CHANGELOG entry — were reviewed and left as-is: both are intentional given the Table feature is still incomplete. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> * Table labels (#551) * Replace ResolvedTable's DataFrame body with a resolved cell layout TableCell (execute/table.rs) positions each column label and data value on an inclusive row/column grid, tagged ColumnLabel/Body (TableCellKind, naming borrowed from R's gt) so a writer can tell them apart without relying on position. ResolvedTable now holds Vec<TableCell> instead of a DataFrame; nrow()/ncol() are computed from cells rather than stored, so there's one source of truth for the table's shape. Writer::write_table drops its Table parameter entirely: Table's only field (source) is already consumed before cells exist, so there's nothing left for a writer to use it for. HtmlWriter, ggsql-cli and ggsql-jupyter are updated to match. * Split table-cell construction into pure column-label/body/composer steps create_column_labels and create_body each build their half of the layout independently, both numbering rows from 0 — neither knows where it sits relative to the other. compose_cells is a pure function of the two (no DataFrame/SQL knowledge), computing the row offset from the column labels' actual extent and shifting the body via TableCell::offset_rows/offset_cols rather than each caller touching top/bottom or left/right separately. This is what lets more intermediate composers (header, footer, ...) join compose_cells later as Table grows, and keeps cells_from_dataframe useful as a place to test offset arithmetic in isolation from any real query — added layout_tests covering create_column_labels, create_body, and compose_cells (including a label section spanning more than one row, to pin down that the offset comes from the labels' real extent rather than an assumed single row) using the df! macro, no duckdb feature or Reader needed. * Add TABULATE LABEL clause for custom column display labels tabulate_statement now accepts an optional label_clause (reused unchanged from VISUALISE), so TABULATE FROM sales LABEL id => 'ID' parses. Table.labels: Labels (not Option<Labels> — an empty Labels already means "no overrides", so there's no third state an Option would distinguish, and every consumer's logic is the same either way). build_tabulate_statement now mirrors build_visualise_statement's structure: a process_tab_clause function parallels process_viz_clause, ready to grow an arm per future TABULATE clause (FACET/SCALE). execute::table::create_column_labels resolves each column's content against Table.labels with three outcomes: a column absent from labels keeps its name, an explicit LABEL col => NULL empties the cell, and LABEL col => 'text' overrides it. Verified end-to-end via a real ggsql exec --writer html invocation, plus a unit test covering all three outcomes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Reject resolved table layouts with overlapping cells Groundwork ahead of column spanners, the feature expected to make cell overlap actually reachable: validate_overlaps walks each cell's grid footprint into a position set and errors on the first collision. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Track resolved column name/label as TableColumn ahead of spanners create_table_columns is now the one place a TABULATE LABEL clause gets resolved; create_column_labels and create_body both build cells off its order instead of the DataFrame's raw column order, so a future spanner- driven column rearrangement reaches cell positions without either function changing. create_body also now looks columns up by name and fetches each column's array once (outside the row loop) rather than once per cell. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> * Table spans (#553) * Add TABULATE SPAN clause grammar and AST construction SPAN groups columns under one spanner: `SPAN label ACROSS col, ... [SETTING ...]`, where label is a string (possibly empty) or NULL (suppress the cell, keep the grouping) — mirrors label_assignment's own string/NULL value shape. tabulate_statement now takes a repeated tab_clause (LABEL and SPAN, any order), the same "any order, repeated" shape viz_clause gives VISUALISE's own clauses. Table gains a `spans: Vec<Spanner>` field and Spanner::validate_settings, built the same way a GeomTrait declares default_params() — a static name/default/constraint list validated through the existing validate_parameter — so SPAN's SETTING keys (gather, level) get the same type checking layer settings already do. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Resolve TABULATE SPAN into positioned spanner cells Wires SPAN into table resolution: build_cells validates each spanner's settings, reorders columns so gather-enabled spanners become contiguous (gt's tab_spanner(gather = TRUE) default), assigns each spanner a header row (explicit SETTING level pins directly, others greedily bump past any column-set intersection, then levels are compacted), and RLE-builds one TableCell per contiguous run of a spanner's columns — fragmenting a non-contiguous spanner into several cells. NULL-labeled spanners are filtered out before level assignment so they group columns without consuming a header row. Spanner-specific resolution (reordering, level assignment, cell construction) lives in the new sibling module execute/table_spanner.rs rather than growing table.rs further, mirroring how Plot's own resolution logic is split across schema.rs/casting.rs/layer.rs/scale.rs/position.rs/ cte.rs. validate() now also runs Spanner::validate_settings per SPAN clause, the same way layer SETTINGs are validated, so a bad SPAN setting is caught before execution. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Render spanner rows with colspan in HtmlWriter write_table now buckets cells by is_header() into header/body row groups, wrapping them in <thead>/<tbody> (each omitted when empty), and renders each row through the new render_row: cells sharing a top walk every column position, emitting a colspan attribute via TableCell::width() and filling any gap (a spanner not reaching every column, or one fragmented by another spanner's occupancy) with a synthesized empty cell of the same kind, so a row always reaches the table's full column count. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Merge repeated TABULATE LABEL clauses and suppress an all-NULL label row process_tab_clause was overwriting table.labels on every LABEL clause instead of merging, unlike the VISUALISE-side handler for the same node — a second LABEL clause silently dropped every override from the first. create_column_labels now also returns no cells at all when every column's label is empty (only reachable via LABEL col => NULL, or => '', on every column, since an unlabeled column keeps its non-empty name), so a wholly suppressed label row is omitted entirely rather than rendered as a row of blank header cells. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Document TABULATE and SPAN New doc/syntax/clause/tabulate.qmd covers TABULATE's FROM/LABEL/SPAN clauses, wired into the navbar and syntax sidebar in _quarto.yml. ggsql.xml gains TabulateClause and SpanClause highlighting contexts, mirroring how DRAW/PLACE/SCALE/PROJECT/FACET/LABEL/VISUALISE already switch contexts on their own keyword. TABULATE is reachable from every existing context (a new TABULATE statement can start after any VISUALISE clause), but SPAN is only reachable from TabulateClause, SpanClause and LabelClause, since it's a TABULATE-only clause and TABULATE is mutually exclusive with VISUALISE within one statement. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Teach HtmlWriter to render and consume rowspan cells render_cell now emits a rowspan attribute alongside colspan, and render_row takes the set of columns a rowspan from an earlier row already occupies, skipping them (no cell, no filler) rather than double-rendering. occupied_columns_per_row computes that set for every row in one pass over cells. write_table also now checks that every header cell sits above every body cell, a requirement of splitting output into separate <thead>/<tbody> blocks that isn't a general TableCell invariant. Nothing in the resolution pipeline produces a multi-row TableCell yet — this is writer-side groundwork, exercised directly with hand-built cells, ahead of compose_header actually merging a spanner-less column's label into a rowspan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Stretch an unspanned column's label over its header gap compose_header now grows a column's label cell upward into every consecutive spanner row above it that has no spanner covering that column, stopping at the first row that does — a rowspan on the label instead of a separate blank filler cell for that gap. Deliberately diverges from gt: gt only ever stretches into the single row immediately above the labels, even when rows further up are also empty for that column. Verified directly against gt's own output. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Align spanner level assignment and add SPAN id references to gt assign_spanner_levels now matches gt's real resolve_spanner_level() rule: a spanner's level is one more than the highest level of any already-assigned spanner it shares a column with, rather than the lowest level free of overlap. Confirmed against gt's own source; it can use more header rows than strictly necessary for a chain of pairwise-but-not-all conflicting spanners, which is deliberate parity with gt, not a compaction gap. New SETTING id => 'foo' on a SPAN clause lets a later SPAN's ACROSS list reference an earlier spanner by id, folding its columns in alongside any literal ones (SPAN 'H1' ACROSS q1, apr after SPAN 'Q1' ... SETTING id => 'q1'). Table::resolve_spanner_ids expands these (only backward references resolve; a forward reference or typo falls through to the existing unknown-column error) and rejects a duplicate id; it lives on Table rather than in execute::table_spanner so validate() can call it without depending on execute. A separate check, folded into create_spanners since it needs the real column list, rejects an id that collides with an actual column name. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> * Table format (#554) * Unify SPAN/PARTITION BY/FACET column lists into a shared column_list rule SPAN's columns, PARTITION BY's columns, and FACET's row/column vars were four identical `identifier, (',' identifier)*` grammar rules with matching duplicate extraction logic in builder.rs. Collapse them into one column_list rule and one parse_column_list() function. project_aesthetics stays separate since it names aesthetics, not columns, despite the identical shape. Also correct two CLAUDE.md files that claimed the generated tree-sitter parser files are committed to git — they're gitignored and regenerated by the Rust build script instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add TABULATE FORMAT clause grammar and AST construction FORMAT configures cell formatting for a group of columns: FORMAT col, ... [SETTING ...] [RENAMING ...], repeatable like SPAN. SETTING's semantics are deferred — parsed and stored but not yet validated or applied. RENAMING reuses SCALE's exact clause (scale_renaming_clause renamed to the generic renaming_clause) so FORMAT gets identical explicit-value and wildcard-template renaming behavior for free. Table gains a `formats: Vec<Format>` field. Format stores value_mapping/ value_template rather than reusing Scale's label_mapping/label_template names, since Table::labels already claims "label" for column headers. Both fields share one default_template() helper in format.rs instead of each struct defining its own serde default function. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Apply TABULATE FORMAT's RENAMING to resolved table cell values Wires Format.value_mapping/value_template into actual cell display: a new table_format module replaces each FORMAT-covered column in the resolved DataFrame with its resolved display text before column/cell layout runs, so create_body's ordinary value_to_string rendering already reflects any RENAMING and never needs to know FORMAT exists. A later FORMAT clause wins over an earlier one naming the same column; an unknown column errors. format.rs gains resolve_column_values, which works directly off a column's typed Arrow values rather than boxing each row into an ArrayElement first (the path apply_label_template uses for Scale's much smaller break lists) — avoiding that allocation for what can be a much larger number of table rows. compute_numeric_precision now takes plain f64s so both call sites can share it without going through ArrayElement, and the string/numeric column conversion in format_dataframe_column is factored out into column_to_strings for both to reuse. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Resolve FORMAT's hjust SETTING and render it in HtmlWriter Validates FORMAT's SETTING against a real FORMAT_PARAMS list (previously accepted-and-ignored): hjust takes "left"/"right"/"center"/"centre" or a 0-1 number, via a new ParamConstraint::string_option_or_number shared with any future param needing the same keyword-or-continuous shape. TableColumn and TableCell gain a `properties: Parameters` field. resolve_column_properties resolves each column's hjust exactly once, standardised to a number (left/center/right = 0.0/0.5/1.0; an explicit number passes through; an absent setting defaults from the column's Arrow dtype, numeric right, else left) so every writer buckets one representation instead of re-implementing keyword handling. Column-label cells inherit their column's properties, matching gt's own convention. HtmlWriter reads a cell's resolved hjust and renders it as an inline `style` attribute (text-align, plus font-variant-numeric: tabular-nums when right-aligned) through one translation function, so a future property is one more mapper rather than a change to render_cell itself. The numeric bucketing (0.25/0.75 thresholds) matches VegaLiteWriter's own convert_hjust, so hjust means the same alignment in every writer. Since dtype-derived defaults apply to every TABULATE output regardless of FORMAT, several pre-existing HtmlWriter/CLI/Jupyter tests asserting bare `<th>`/`<td>` strings needed updating; loosened them (and the new cell tests) to check tag content and alignment keywords rather than exact attribute strings, so future properties don't compound into unreadable literals. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Document TABULATE FORMAT and add its doc-site syntax highlighting Adds a FORMAT section to doc/syntax/clause/tabulate.qmd (clause syntax, SETTING's hjust, RENAMING linked to SCALE's own RENAMING/break-formatting docs rather than duplicated) plus a runnable example combining hjust alignment with null/template RENAMING against ggsql:penguins. doc/ggsql.xml gains a FORMAT keyword and FormatClause highlighting context, mirroring SpanClause, so the doc site's ```ggsql fences highlight it correctly; verified with a full quarto render of the page (after rebuilding and reinstalling the ggsql-jupyter kernel, since the previously-installed one predated the FORMAT grammar). CHANGELOG.md's existing TABULATE bullet list gains a FORMAT entry at the same one-line-per-clause detail level as LABEL/SPAN. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Plumb per-column and per-row properties through ResolvedTable/Writer Table writers now receive resolved TableColumn/TableRow slices alongside cells, so a future whole-column setting like FORMAT's width can be read once per column instead of duplicated across every cell. Also drops ResolvedTable's vestigial table: Table field, made dead once cells/columns absorbed everything it used to carry. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add FORMAT's width SETTING and render it as an HTML <colgroup> Adds a numeric-with-unit ParamConstraint (px/%) for FORMAT's width SETTING, and has HtmlWriter emit a <colgroup> with a per-column width style, mirroring gt's own colgroup output (checked directly against gt's source: width is the only thing it ever puts there). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> * Table titelature (#563) * Plumb TableCellClass and add HtmlWriter's inline/class CSS mode Cells now carry structural (row/col_heading/spanner/spanner_outer) and alignment classes through ResolvedTable, and HtmlWriter renders them either as class names against a <style> block or folded into inline style attributes, selected via a css_mode writer option. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add TABULATE LABEL title/subtitle/caption, generalize TableCellClass into TableClass VISUALISE's LABEL clause already supports the reserved title/subtitle/caption keys; TABULATE's LABEL now does too, reusing the same grammar. Reserved keys always win over a same-named column. Title/subtitle render as heading rows (a new row-scoped TableClass::Heading, alongside the renamed, now more general TableClass) above the column labels; caption renders as a real HTML <caption> element, pulled out of the row grid by HtmlWriter. Also reworks HtmlWriter's row rendering: a Section type (execute::table) bundles a layout part's cells with one TableRow per row it occupies, built correctly from the start rather than derived after the fact; HtmlWriter decides every row's Slot (real cell, synthesized filler, or rowspan-skip) for the whole table in one pass (build_all_slots) instead of separately bookkeeping occupied columns and re-deciding per row. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Restructure execute::table into its own module directory table.rs/table_spanner.rs/table_format.rs move from flat siblings of execute/table.rs to execute/table/{mod,spanner,format}.rs — unlike Plot's own flat-sibling schema.rs/casting.rs/layer.rs/scale.rs/position.rs/cte.rs (each reachable from more than one place), spanner and format are used exclusively by table itself, so nesting them under it narrows their visibility from pub(crate) to pub(super) instead of leaking crate-wide. Also splits the cell-layout construction (Section, build_cells and its create_* helpers, TableColumn/TableRow) out of table/mod.rs into its own table/layout.rs, since that was the dominant share of the old file (and the part most likely to keep growing) — table/mod.rs now holds just the resolve_table_with_reader entry point and the TableCellKind/TableClass/ TableCell public data model. Section stays pub(super) rather than fully private, since spanner::create_spanners builds one too, and a future stub/stubhead module is expected to need the same access. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Make TableRow track header/body, require columns/rows in write_table HtmlWriter used to sort cells into <thead>/<tbody> off each cell's own TableCellKind, even though header-ness is really a row-level property enforced by the writer's own "header above body" check. TableRow now carries is_header, decided once per row where it's built (create_heading/ create_column_labels/create_spanners), and HtmlWriter splits rows via a new split_rows() helper instead of re-deriving the answer per cell. Along the way: captions are pulled out of the grid before ncol/nrow are computed rather than filtered out later; header_rows/body_rows switched from BTreeSet to Vec (insertion is already ascending, no membership queries); render_row takes &TableRow instead of Option<&TableRow>, with a debug-only assertion cross-checking it against its own cells' kind. Since ResolvedTable's columns/rows are always Some in practice (the one constructor always builds both), write_table's columns/rows parameters are now plain slices instead of Option<&[_]>. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> * Table stub (#568) * Add TABULATE column selection TABULATE now requires a column selection right after the keyword: `*` for every column, or a comma-separated list of columns to include, optionally renamed with AS (TABULATE bill_len, bill_dep AS Depth). Grammatically this reuses mapping_list, the same shape VISUALISE's global mapping already uses. Table::selection captures the selection as raw source text rather than a parsed structure, since it's structurally a SQL select-list; a new build_table_sql (execute/table/mod.rs) splices it into the generated SQL as the outer projection, letting the backend do the actual column picking/renaming/reordering instead of reimplementing that in Rust. Breaking: bare TABULATE (no selection) is now a parse error. Existing queries need an explicit `*` or column list. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Table stub Adds the first slice of gt's stub concept to TABULATE: row labels and the stubhead, via FORMAT's new optional target. FORMAT [BODY | STUB] <column>, ... marks one or more columns as the table's stub instead of its regular body; BODY (the default) is rarely written. Grammar gains format_target_identifier; Format/TableColumn both gain a target: ColumnSection field (Body/Stub, with an is_stub() method used everywhere instead of raw equality checks). Target follows the same last-clause-wins rule as every other FORMAT field. A SPAN can never include a column any FORMAT clause ever targeted at STUB, regardless of whether a later clause retargeted it back to BODY - Table::validate_span_stub_boundary enforces this from both validate() and real execution (setup_columns), since validate() alone doesn't gate resolve_table_with_reader. Stub columns are moved to the front by move_stub_columns, preserving whatever relative order they already had (their DataFrame/SQL projection order) rather than FORMAT-declaration order - R's gt package doesn't support multi-column row labels either, so a precise ordering rule isn't worth the complexity yet. create_row_labels/create_stubhead split off from create_body/create_column_labels, each partitioning the same columns list by is_stub(); unlike the spanner/column-label split, this is a horizontal (column-range) rather than vertical (row-range) split, so both return a plain Vec<TableCell> that build_cells merges into one Section with a single shared rows vec, instead of two Sections stitched with rowbind. resolve_column_properties now defaults hjust to right for a stub column regardless of dtype. HtmlWriter gains TableCellKind::StubHead/StubRowLabel and TableClass::StubHead/Stub (the latter has no gt equivalent), but needed no other changes - render_cell/build_all_slots/render_row all key off TableCellKind::is_header() rather than an exhaustive match, so the new kinds render correctly automatically. Verified end-to-end with a real render test. doc/syntax/clause/tabulate.qmd documents the target, its last-wins semantics, and that a stub column is otherwise ordinary (LABEL/ SETTING/RENAMING all apply the same way). Also relocates the WIP TABULATE changelog entry, previously sitting under the already-released 0.5.0 section, into [Unreleased] where ongoing work belongs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix stub/spanner filler misclassification in tables FORMAT STUB combined with any SPAN left every spanner-row gap under the stub column to be inferred by the HTML writer from a row neighbor, which could pick the wrong kind (e.g. Spanner instead of Filler, or vice versa) once the stub head's stretch made it the row's leftmost real cell. Move filler synthesis into execute (TableCellKind::Filler, built by fill_spanner_gaps) so every grid gap is resolved with full structural knowledge instead of guessed from a sibling cell in the writer. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add default table styling and fix Quarto table-processing conflict Fill in real CSS declarations for every TableClass, add TableClass::Table and TableClass::TableBody as the <table>/<tbody> elements' own styling hooks (with no TableCell/TableRow to carry them), add ColHeadingRow for the column-label row's <tr>, and centre spanner labels by default. The HTML writer's <table> now also opts out of Quarto's HTML table processing via data-quarto-disable-processing — verified against a real `quarto render` of doc/syntax/clause/tabulate.qmd, since the initial "false" value (copied from a non-representative gt sample) turned out to mean "don't disable," the opposite of what was intended. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Default stub column labels to blank A FORMAT STUB column previously showed its own name in the stub head by default, same as any body column. That reads as a stray label for what is a row-label header, not a data column header. LABEL still overrides it the same way it does for any other column. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> * Finishing touches for tables (#571) * Change SPAN's label from a string to an identifier SPAN <identifier> ACROSS <columns> replaces SPAN <string> ACROSS <columns>; quote the identifier with backticks or double quotes to include spaces or other non-identifier characters. Drops the present-but-blank ('') label state, since neither bare nor quoted identifiers can be empty. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Give SPAN a real id, drop SETTING id, and consolidate spanner validation SPAN's leading identifier is now the spanner's id (referenceable from a later SPAN's ACROSS list or a LABEL <id> => ... override), not a bare label. SETTING id => '...' is gone; SPAN NULL leaves a spanner anonymous with an internally generated id. All spanner validation/resolution now lives on Table/Spanner (Table::resolve_spanners orchestrates settings, ACROSS-id folding, the STUB boundary, column-existence, and LABEL overlay; the per-span checks are Spanner methods). This also fixes a real bug: unknown-column checking used to live only in gather_columns, which SETTING gather => false skipped entirely, silently producing no error and no cell. Column-reordering (gather_columns/reorder_table_columns/setup_columns) is now infallible, since Table::resolve_spanners validates upfront. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add SPAN <columns> UNDER <id> as an alternative to SPAN <id> ACROSS <columns> Both forms produce the same id/column_list shape in the grammar (just in a different order), so builder.rs needs no changes — verified by a test asserting the two forms parse to equal Spanners (Spanner now derives PartialEq for that comparison). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Document SPAN's UNDER form in the TABULATE docs Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Let FORMAT reference a SPAN id in place of its columns Resolve spans before FORMAT so a FORMAT column entry can name a SPAN id and apply to every column it covers, mixed freely with literal column names in the same clause. Extends the STUB/SPAN collision guard to expand SPAN ids the same way, so FORMAT STUB <span-id> is checked against real columns instead of the literal id text. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Render TABULATE cells in the visual_test doc harness doc/syntax/clause/tabulate.qmd now has runnable TABULATE examples, which the harness was reporting as failed since Outcome had no table variant. Add Outcome::Table, rendered via HtmlWriter, whose own inline <style> already covers table styling. Full 260-cell corpus now runs with 0 problems, down from 6. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Point prepare_data_with_reader's TABULATE error at Reader::execute() prepare_data_with_reader only ever prepares Plot data (PreparedData is Plot-shaped), so a TABULATE-only query still errors here — but now names the entry point that actually handles it instead of the generic "no specifications" message. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Emit scope attributes on HtmlWriter header cells ColumnLabel/StubHead get scope="col", Spanner gets "col" or "colgroup" depending on colspan, and StubRowLabel becomes header-shaped and gets scope="row" (rendered as <th> inside its body row rather than <td>). Drops render_row's debug_assert_eq! cross-checking TableCellKind::is_header against TableRow::is_header, since a body row can now legitimately mix Body and header-shaped StubRowLabel cells. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * update syntax highlighting * De-duplicate named columns in mixed TABULATE selections A selection like TABULATE foo, bar, * passed through to SQL verbatim, so the wildcard re-selected foo and bar. Table.selection is now parsed into ordered items (named vs wildcard) and the result frame is rebuilt: named columns stay at their written position, the wildcard contributes the rest. Only a bare reference suppresses the wildcard's copy; a renamed item's source column still surfaces once, unrenamed. Output columns that still share a name after this (repeated alias, rename colliding with a wildcard column) are an error instead of silently shadowing. * Run side-effect DML ahead of TABULATE queries A leading CREATE (or INSERT/UPDATE/DELETE) before a TABULATE was silently skipped: the Table pipeline ran only execute_setup_statements, never cte::extract_side_effects, so a temp table it created didn't exist when the TABULATE query read it. resolve_table_with_reader now runs side-effect DML after setup statements, mirroring prepare_data_with_reader. * Let a TABULATE rename suppress its wildcard duplicate SelectionItem::Column now records the source column it reads from, so `bill_len AS foo, *` drops the wildcard's own copy of bill_len instead of showing it twice under both names. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix spanner header styling in TABULATE HTML output Spanner cells carried either `Spanner` or `SpannerOuter` depending on level, but gt puts both on every spanner cell — `SpannerOuter` on the `<th>` and `Spanner` on a div wrapping the label, whose 5px padding and underline were therefore missing on topmost-level spanners. Spanner rows also missed the `ColHeadingRow` treatment, so they drew no border above the header and a stray 2px rule between levels. Mirror gt: apply both spanner classes unconditionally with `Spanner` routed to an inner div (new `INNER_CLASSES` split in `render_cell`), and give spanner rows the new row-scoped `SpannerRow` class (hiding their bottom border) alongside `ColHeadingRow`. * Rename SpannerOuter to Spanner and Spanner to SpannerLabel * write docs * update SKILL.md * Editorialise comments for brevity --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> * Rename `ACROSS` to `OVER` * propagate doc suggestions * Apply batched suggestions from code review Co-authored-by: Thomas Lin Pedersen <thomasp85@gmail.com> * apply more suggestions * move out params into top-level file * move labels to top level * Update doc/vendor/SKILL.md Co-authored-by: Thomas Lin Pedersen <thomas.pedersen@posit.co> * code formatting * move table cell types out of execute into table::layout TableCell, TableCellKind and TableClass are resolved-layout vocabulary with no DataFrame dependency, so they live in src/table/layout.rs rather than under execute. count_cell_rows/cols and standardise_hjust move along (execute's format.rs now imports the latter from crate::table). * move TableColumn and TableRow into table::layout Like the cell types before them, both are resolved-layout vocabulary with no DataFrame dependency — only the builder functions that produce them (reading names, dtypes and values out of the DataFrame) belong in execute. * move cell/section assembly out of execute into table::layout Section, rowbind, compose_header, the heading/caption/column-label builders and validate_overlaps all operate on cells and columns, never on a DataFrame — only build_cells and the create_table_columns/create_body builders that feed them do, and those stay in execute. Moved items are pub(crate) so execute's build_cells can keep orchestrating them. * move spanner code into the table module spanner.rs operates on TableColumns and Spanners and emits Section cells — no DataFrame contact — so with the Section assembly now in table::layout, the whole file transplants from execute with only import-line changes. The Spanner spec type, its impl and SPAN_PARAMS move along from table/mod.rs, mirroring how plot/ keeps spec types in their own submodules. * move extract_heading_labels and move_stub_columns into table::layout Both are DataFrame-free — one pops heading keys out of a Labels map, the other partitions TableColumns — completing the layout-function migration: execute/table now only holds code that reads or transforms the DataFrame itself. * subsume Section into TableLayout, bundling columns with rows and cells * route ResolvedTable and Writer::write_table through TableLayout * move row partitioning and header-order validation into TableLayout TableRow gains an is_caption flag with caption()/body() constructors, so TableLayout::header_and_body_rows() can partition rows purely from row flags and validate() can enforce the caption invariants (one row, last row, never a header, exactly one Caption cell). validate() splits into per-invariant helpers; the header-above-body check moves up from HtmlWriter, which now sizes its grid from the partition and ncol(). * document TableLayout in the export list, API reference and changelog * move slot grid construction into TableLayout::slots_by_row Slot and build_all_slots move from HtmlWriter to the layout itself: the grid is sized from header_and_body_rows()/ncol() and caption cells are filtered internally, so write_table no longer clones every cell up front or computes nrow/ncol itself. Since the slots borrow the layout's cells, hjust folding moves to render time via a new read-only TableCell::hjust_class(), which discretise_hjust now delegates to. * bundle TableCell kind and classes into named constructors * Route TABULATE derived-table wraps through sql::select_from Replaces hand-written wrapping SQL with sql::select_from, gaining dialect-correct quoting of the __table__ alias and CTE hoisting on dialects that disallow CTEs in derived tables. Also adds a SPAN parser test pinning that quoted identifiers (id and OVER columns) are unquoted. * minor cosmetic polishes * Multiple `FORMAT`s merge instead of last-one-wins (#572) * colums that are targetted by multiple formats merge them * Make Format::value_template optional A FORMAT clause that never names a wildcard template now leaves the field None instead of storing the default "{}", so merging stacked FORMAT clauses on one column only overrides a template a clause actually set. parse_renaming_clause returns Option<String> for the template to keep an explicit * => '{}' distinguishable from no wildcard, letting a later clause reset an earlier template. * Add ANY target to FORMAT clause, defaulting to no section claim A FORMAT clause without a target identifier (or with the new ANY keyword) now makes no claim about which table section its columns belong to: Format.target is Option<ColumnSection>, merging stacked clauses lets concrete BODY/STUB targets win over ANY, and unclaimed columns resolve to the body in setup_formats. Previously a bare FORMAT counted as a BODY claim and would silently pull columns out of the stub when combined with FORMAT STUB. * Allow FORMAT to target all columns with * FORMAT * sets a wildcard selection, expanded to the data's columns in setup_formats via the new expand_format_columns helper (which also absorbs SPAN id expansion and unknown-column validation). Wildcard clauses merge like any other, so an explicit clause can still refine a single column after a catch-all FORMAT *. * use new label method * dedup spanner columns * Guard against `SETTING level => null` panic * fix git shennanigans with test folder * Add note for reserved word * restructure `SelectionItem` * change default writer mode to inline * wasm emits tables * add caption-side: bottom * Use counter for anonymous span ids * rework unit number constraint * improve formatting * Harden TableLayout cell/row invariants Clamp the full-width cell constructors against ncol == 0 and judge rowbind emptiness by rows rather than cells. * Address FORMAT review comments Document that a leading any/body/stub in a FORMAT column list parses as a target (quote the column instead), with a corpus test for the quoted form, and give hjust a null declared default — the real default is resolved per column type. * Validate only the resolved first spec validate() checked the first plot and the first table while execution resolves only specs[0], so a bad setting in an ignored statement flipped valid to false. Also: a specific error for 'TABULATE FROM t' (FROM lexes as a column name) and a kind-neutral no-spec error from Reader::execute. * Fix CLI html writer surface, add Writer::render_plot --writer help listed no options for html (it takes css_mode); -o x.html wrote a bare fragment without a charset meta; and a VISUALISE query aimed at the html writer failed without a hint. The new default Writer::render_plot(&ResolvedPlot) removes the write_plot(plot(), data()) unpacking at call sites. Also adds a changelog line for query warnings now always printing to stderr. * Drop the text/plain placeholder from Jupyter table output 'ggsql table' told non-HTML frontends nothing; omit the entry until a plain-text table writer exists to fill it. * improve accessibility features * Apply batched suggestions from code review Co-authored-by: Thomas Lin Pedersen <thomasp85@gmail.com> * fill test gap for `validate_stub_boundary()` * sync #609 with new spec * soothe clippy --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Thomas Lin Pedersen <thomasp85@gmail.com> Co-authored-by: Thomas Lin Pedersen <thomas.pedersen@posit.co>
This PR implements suggestion here: #558 (comment)
@thomasp85 if you want this now rather than later, feel free to merge