Skip to content

The Frontend IR ​

A portable JSON representation of a GoFish chart specification, captured at the source level (chart(data).flow(...).mark(...)) before macro expansion and elaboration. Three consumers:

  • Olli — accessibility adapter. Walks the IR to expose mark boundaries, labels, and axes to assistive technology. Its existing Bluefish adapter has to capture imperative execution to reconstruct this; a declarative IR sidesteps that.
  • The Python wrapper. Builds the same IR and ships it across anywidget. The schema package makes it official; the JS deserializer is shared. See The Jupyter Bridge & RPC for the transport and § Bridge extensions below for the Python-side sentinels that extend the canonical schema.
  • Future internal tooling — debuggers, alternative renderers, parity-test harnesses.

What ships in v0 ​

ArtifactPath
Construct descriptor table (single-source field lists)packages/gofish-ir/src/frontend/descriptors.ts
Schema types + validator + canonical examplespackages/gofish-ir/src/frontend/
JSON Schema (Draft 2020-12)packages/gofish-ir/dist/frontend/v0.json (build artifact)
JS-side emitter (Serialize.toJSON, ChartBuilder.toJSON())packages/gofish-graphics/src/serialize/toJSON.ts
JS-side deserializer (Serialize.renderIR, buildIR, mapMark, …)packages/gofish-graphics/src/serialize/fromJSON.ts
Deserializer factory table, keyed by the descriptor tablepackages/gofish-graphics/src/serialize/registry.ts
Generated Python factory layer (checked in, CI freshness-checked)packages/gofish-python/gofish/_generated.py (from scripts/generate.ts)
Generated Python strategy families (checked in, CI freshness-checked)packages/gofish-python/gofish/{tile,overlap,curve,bin,calendar,shape,place}.py (same script)
Hand-written Python residue (dispatch, bridge, DataFrame conversion), emits IR validated against schemapackages/gofish-python/gofish/ast.py

v0 matches the existing widget wire format exactly — lowercase type discriminators, __combinator flag on combinator-form marks, channel slots accept the existing strings/numbers/sentinels. The design improvements summarized in § Future evolution (PascalCase rename, ChannelExpr-only IR, __combinator removal, per-stage sibling schemas) are deferred to subsequent breaking releases.

Dumping the IR from a spec ​

JavaScript:

ts
import { chart, spread, rect, Serialize } from "gofish-graphics";
import { Frontend } from "gofish-ir";

const c = chart(data)
  .flow(spread({ by: "lake", dir: "x" }))
  .mark(rect({ h: "count" }));

// Two call shapes, both returning Promise<FrontendIRDocument>:
const doc = await c.toJSON(); // method on ChartBuilder and on a .layer(...) chain
const doc2 = await Serialize.toJSON(c); // any root: a chart, a chain, a mark

Serialize.toJSON takes anything a root holds (below): a chart, a .layer(...) chain, a layer([...]) of charts or marks, or a bare mark.

toJSON is async because combinator-form marks may carry their child list as a Promise<Mark[]> (e.g. from map(...) helpers); the emitter resolves these to walk into them.

Python (via the wrapper):

python
from gofish import chart, spread, rect
from gofish.ast import ir_payload

builder = chart(data).flow(spread(by="lake", dir="x")).mark(rect(h="count"))
doc = builder.to_ir()                   # the document, tables as JSON rows
shipped, callbacks = ir_payload(builder)  # as a host ships it: tables as base64 Arrow

to_ir() returns the same envelope as JS toJSON (irVersion, ir, root, and tables as JSON rows), on a chart, a chain or a mark alike. A host ships the same document with its tables as base64 Arrow, beside the Python callbacks the IR names (§ One renderer, two hosts).

Validate either against the schema:

ts
import { Frontend } from "gofish-ir";
const result = Frontend.validate(doc);
if (!result.valid) console.error(result.errors);

The validator has one mode: an unknown field is an error, wherever it sits.

The document at a glance ​

A document is a wrapper that names the schema version, the stage, the root, and the rows of its charts:

ts
type FrontendIRDocument = {
  irVersion: 0;
  ir: "gofish-frontend";
  $schema?: string; // optional canonical URL
  root: MarkIR;
  tables?: Record<string, Row[]>; // by table id
};

A root holds anything a mark position holds (MarkIR, below), by one rule, so a chart crosses the same way wherever it sits. The shapes mirror the fluent builder:

  • ChartIR, { type: "chart", data, operators?, mark, options?, zOrder?, name? }, for chart(...).
  • LayerChainIR, { type: "layer-chain", tiers }, for a chart(...).layer(...) chain. Each tier is in a mark position. The first is a chart; a later tier is often an empty chart() scope, or a bare mark: a relational line() that connects the previous tier's marks, or an annotation such as text(...). The deserializer rebuilds it through the real LayerBuilder, so JS owns the chain's tier wiring and its render logic (inferred axis titles, the root tier's axes).
  • The layer combinator mark, { type: "layer", __combinator: true, options?, children, relate? }, for layer([...]) of charts or marks.
  • Any other mark, such as a bare rect or a spread([...]).

So "layer" names one thing, the combinator, and the chain has its own type. The type is named for what it is, the .layer(...) chain; its tiers are the tiers that previous-tier data (below) speaks of.

A chart's data is one of three:

  • {type: "table", id} names rows of the chart's own. The rows travel beside the IR, keyed by that id, in the document's tables: JSON rows in a document JS toJSON or Python to_ir() writes, or base64 Arrow when a Python host ships it, which the host decodes before rebuilding (§ One renderer, two hosts). The ids are the writer's own, unique within one document; both writers mint them in walk order (t0, t1, ...), and the same rows object, used by two charts, is one table.
  • {type: "select", layer, mode?} reads the marks a sibling named (ref(name), selectAll(name)).
  • {type: "previous-tier"} marks an empty chart() scope: its rows are the enclosing scope's. Inside a layer-chain that is the previous tier's marks; in a mark position it is the rows of the partition the chart sits in. A chain whose first tier is an empty scope binds that tier the same way, so a chain in a mark position draws its partition's rows. The deserializer maps it to the JS PREVIOUS_LAYER_MARKS sentinel so the real LayerBuilder.wireTiers() derives the auto-naming + selectAll wiring at resolve time. The producer's auto-minted layer name never appears in the IR (mirroring how a relational mark's zBelow-by-default paint order stays a resolve-time constraint rather than a serialized field).

There is no inline form: a chart with rows has a table, by one rule wherever it sits, so a nested chart's rows cross as Arrow too, with their column types (#1088). Operators are a flat list (derive, resolve, join, filter, spread, stack, group, scatter, table, log, treemap, pack). treemap's tile is a strategy object made by a call in the Tile family (Tile.squarify({ ratio }), Tile.slice(), Tile.dice(), Tile.binary(), Tile.voronoi({ seed }) in both languages). Every strategy has one shape, { kind, ...params }, which is both the JS value and the wire form, e.g. { "kind": "squarify", "ratio": 1 }, and the JS layout dispatches on kind. scatter's overlap (the Overlap family) and a line or ribbon curve (the Curve family: { "kind": "arc", "direction": "down" }) work the same way; a bare curve name is not a curve. So does enclose's shape (the Shape family: { "kind": "blob", "convexity": 0 }). enclose's avoiding names nodes, so on the wire it is a NodeSelectionIR: a list of { "type": "ref", "selection": "b" } refs, or every node of a named layer as { "type": "select", "layer": "nodes", "mode": "all" }, the SelectRefIR chart data takes for selectAll (one shape in schema.ts and the JSON Schema, written by selectRefIR in toJSON.ts and read by refOfSelect in fromJSON.ts, and by _select_ref in Python, for both uses). Only fields whose declared type holds nodes are converted, wherever the descriptor table puts them, including inside a strategy's params: mapHeldNodes in descriptors.ts walks a field's type beside its value (object fields, list items, the union branch the value's kind names) and hands each node-holding part to a converter. The emitter's converter (nodesToIR in toJSON.ts) writes a ref by a string name, a .relate() operand, or a node with a string name as a ref by that name, and one selectAll as its select; an unnamed node, a token or path ref, a single ref, and a selectAll inside a list have no wire form and throw, naming the fix. The reader's converter (resolveNodeRefs in fromJSON.ts) turns each back into a live ref. A relational mark's combinator form carries its IR tag (tagCombinator), and a ref child is written as a RefMarkIR, so an enclose(opts, [ref("a")]) inside a chart serializes. JS does not serialize a .relate() callback (it is a closure), so a relate clause's IR comes from Python. A label's position takes a position string or a strategy of the Place family ({ "kind": "along", "at": 0.6, "orient": "sloped" }), and its avoiding is a NodeSelectionIR like enclose's. A .label() chain is not a construct with a descriptor, so the emitter writes it with wireLabels in toJSON.ts: each call's accessor, and of its options only the keys LABEL_OPTIONS declares, kept by the same wireFields that wireOpts uses, with the nodes in them converted by the same mapHeldNodes walk. The reader rebuilds them with the same resolveHeldNodes over LABEL_OPTIONS in fromJSON.ts. The families are namespaces in the two surfaces only; the wire carries the plain objects. Their kinds and params are declared once, in the STRATEGIES table (below). Note join inlines its right-hand table as JSON rows, so unlike derive it round-trips without a bridge. filter round-trips the same way when its predicate is a field predicate: field(name).between(lo, hi, { closed }) returns a row predicate that also carries its description, so the operator goes on the wire as { "type": "filter", "predicate": { "field": "day", "between": [100, 120], "closed": "right" } }. The predicate is its own value, not an op in the field expression's ops. A filter over a hand-written JS predicate has no wire form and emits the opaque derive. A derive's schema (derive(fn, { schema }), Python derive(fn, schema={...})) is plain data, the wire form of a chart's schema, and the registry passes it to the rebuilt derive, which applies it to the rows the Python callback returns. A chart's schema crosses as the record of traits each entry stands for, never as the entry a user wrote: JS toJSON writes a chart's with schemaToWire (schema.ts), JS derive keeps its own in that form from the moment it is built, and Python's chart and derive write theirs with _schema_wire, so a Unit, a unit's name and a column record { name, unit } reach the wire the same from both languages. A unit's traits ride in its HasUnit: Unit("USD", { traits: [Trait.summable()] }) is { "HasUnit": { "unit": "USD", "symbol": "USD", "traits": { "CanSum": {} } } }, Unit("°C") has "traits": {}, and the name "km²" alone has no traits, since what it can do is unknown. A unit's domain and traits do not reach the wire as such, only the traits they expand into: the Likert unit Unit({ domain: Domain.discrete(LEVELS), traits: [Trait.diverging()] }) is { "HasLevels": { "levels": [...] }, "HasOrder": {}, "HasMidpoint": { "at": 2.5 } }, with no HasUnit, since it has no name. (A derive once also carried a measure provenance map for the rows of the bin() data helper; both are gone, #1058.) A binned key, field("date").bin(Calendar.month), is an op in the field expression's ops: { "op": "bin", "partition": { "kind": "month", "step": 1 } }, where the partition is a Calendar value's wire form, a Bin kind that bins one field, { "kind": "step", "size": 0.5 } or { "kind": "thresholds", "thresholds": 20 } or { "kind": "quantiles", "quantiles": 10 } (PartitionIR, one union told apart by kind), or a list of Calendar values for a product (.bin([Calendar.year, Calendar.month])), and an op with no partition bins into about 10 cells. A chain, .bin(Calendar.year).bin(Calendar.month), is one bin op per .bin, in order. The op takes no other key, so the old flat spelling, { "op": "bin", "thresholds": 20 }, is an error, not the default partition, and so is the untagged { "step": 0.5 }. A partition with a JS .format(fn) has no wire form, and serializing it is an error. Marks are a tree — leaves (rect, circle, blank, ellipse, petal, text, image, polygon, plus the Python-bridge mark-fn), combinators (with __combinator: true and a children array — layer, spread, stack, arrow, position, line, ribbon, enclose, treemap, pack, and the Porter-Duff family), refs, the two self-discriminating wrapper marks offset and cut (below), a whole ChartIR, or a LayerChainIR. The __combinator flag stays: line, ribbon, enclose, spread, stack, treemap and pack are both leaf marks or operators and combinators, so their type alone does not say which. enclose is a leaf mark (the bag form) and a combinator, and its leaf mark may also stand in operators as a flow step (flowStep: true on LEAF_MARKS.enclose, isFlowStepMark). There its IR is the mark's own, modifiers included. The emitter writes it with markToIR (it finds the mark through the step expandComposedOperator made), and the deserializer rebuilds it with mapMark, and .flow() expands the result into its flow step. A mark position (a chart's mark, a combinator child, a chain tier, a cut source, a drawing .relate(...) clause, and the root) may hold a chart or a chain, by one rule, because JS allows one wherever a mark sits: .mark(chart(...)), .mark(chart().flow(...).mark(...).layer(...)), layer([rect(...), chart(...)]). The deserializer (mapMark) rebuilds the same chart(...) builder or LayerBuilder chain a JS author writes there, and the mark machinery (resolveMarkResult, which resolves every mark position against its partition's rows) binds an empty scope to the partition's rows, as for the JS object. A layer of charts is the layer combinator mark whose children are ChartIRs, like any combinator, at the root too. Both emitters write the same thing: Python's walk (to_ir()) of a chart, a chain or a mark, and JS toJSON, where .mark(chart(...)) and .mark(chain) store the builder itself as the chart's mark. A nested chart is a pure spec, so it needs no call back into Python, and the notebook widget draws it. position sets a single child's min-corner (x, y) in the parent's coordinates and draws nothing of its own, added for the bluefish Topology story's combinator trees.

Operators and marks may also carry translate: {x?, y?}. This is canonical frontend IR, not a Python-only bridge sentinel: it records the structural .translate({x?, y?}) modifier and the JS deserializer reapplies it as a runtime chain.

offset — { type: "offset", x?, y?, children: [<node>] } — wraps a single child and shifts it by (x, y) render-pixels without moving the bounds it advertises to its parent; it maps to the public offset operator.

cut — { type: "cut", source: <mark>, dir, size?, inset? } — slices a single source mark into N clipped sub-shapes along dir. size is a field-name string (per-row weights) or an array of absolute-pixel numbers and datum() flex-weights; omitted means equal slices. It has two deserialization surfaces over one JS core, dispatched by context so extent resolution (flexbox sizing, absolute-vs-weight mixing, measure-unit checks) lives in ONE place, JS-side:

  • used as a chart .mark(...) → the expand-mark form (cutMark / source.cut(opts)), so a chart flow treats it as an expand mark;
  • used as a combinator child (inside a Spread/Stack children array) → the deserializer expands it in place into its N slices — the pure cut(source, opts) returns an array of slice marks that combinators accept directly as children (see mapMarkChildren in fromJSON.ts).

Refs in a callback's input (#591, #1046) — a Python callback isn't limited to receiving plain data rows. Its input may hold GoFishRefs: a mark function over a bag of refs (the .layer(chart().flow(group(...)).mark(fn)) shape that labels a bar chart's totals), a mark function over rows whose field resolve filled with a ref (LabeledChart's r.ref), or a zOrder function over the bag of refs a mark is bound to. A GoFishRef is a class instance over the render's layer registry, not JSON, so one rule covers every bridge call: callBridge in registry.ts (encodeInputRefs) replaces each ref, at any depth of the input's lists and plain-object rows, with a BridgeRef, its index i in one flat inputRefs list. A node the input holds many times is one index, and its datum crosses once, as refs[i] beside the rows, encoded by the same rule. The walk is gofish-ir's Frontend.walk, the one the non-finite codec uses: a list or a row is copied only when it holds a ref, and a copied list keeps its column types, so an input with no refs crosses as it is. A ref is a value of its own type on the wire, the Arrow extension type gofish.ref (storage {index}, named in the field's metadata), in both codecs (arrowTransport.ts, arrow_utils.py), so no row is read as a ref and a ref adds no keys to the rows beside it; a column that holds refs and other values is a type conflict, as a string and a number are. On the Python side, answer turns every index, wherever it sits in the rows, into one _InputRef (_decode_input_refs in gofish/ast.py), whose .datum reads naturally and which pluck reads through. The rule runs both ways: a reply's tables may hold the refs the call was given (a derive that keeps the rows resolve filled, a mark function's chart over them), which Python writes back as BridgeRefs (_encode_input_refs) and callBridge resolves to the live refs. The callback may return a ChartBuilder, a .layer(...) chain, a layer([...]) of charts or a bare Mark. The reply is a document like the top-level one, through the bridge's one call (DeriveBridge.call): its root is the mark-position IR of what the callback returned, and its tables are the ones that IR's charts name, with ids of the reply's own, decoded. mapMark rebuilds the root like any other mark position, with the reply's tables and this call's input refs (a chart nested in the reply keeps both, since they belong to the reply's document). Python records the callbacks the reply names in the reply's own payload, so the host registers them before JS calls one. A bare mark lets the callback embed one of its _InputRef arguments directly in the returned layout (mirroring JS spread({...}, [d[0], text(...)]), or a row's ref, spread([r["ref"], text(...)])). mapMark resolves any {"type": "ref", "__inputRef": i} node found inside that returned mark tree back to inputRefs[i] — the original live GoFishRef from the closure, not a reconstructed stand-in, so the ref's identity (and anything already registered against it, e.g. a name from an earlier tier) survives. A .name(...) call on the Python _InputRef (#556) rides along as a name field on the node and is applied to the resolved ref before it's returned, since GoFishRef.name() mutates in place — this is how a per-slice label overlay (Cut.stories.tsx::ImageCutWithLabels) can .relate(...) against a ref it only received through the bridge. Each call asks Python again and builds its own mark, so an __inputRef always resolves against that call's own refs.

A mark applied to its input. A JS mark is a function, and a mark function often applies the mark it builds to its own rows, (d) => circle({ r: ..., fill: "species" })(d), so a field name in a channel reads d and binds its scale. A Python Mark is callable the same way: mark(d, key) returns a copy of the mark that carries its input, and the IR writes it as the mark's __datum (and __key) bridge fields, which MARK_BASE_FIELDS declares for leaf and combinator marks. mapMark applies them once, after it builds the mark of any kind: the position holds a mark that ignores the rows it is offered and calls the built mark with __datum and __key, which is what the JS expression mark(d, key) puts there. The same fields carry a treemap combinator's children, each applied to its own row. The input is data that JSON holds; Python raises when it holds a mark or a ref, and a ref, a cut and an offset (nodes, not mark functions) raise when called.

The plain leaf-form ref node carries names the same way: RefMarkIR declares an optional name, emitted by Python's _RefProxy.to_dict(payload) when the ref was renamed and re-applied by the deserializer's mapMark via the same mutate-in-place GoFishRef.name(). This is what lets a named ref stand-in, such as ref(token).name("a"), serve as a constraint operand from Python exactly as from JS: without the wire name, the reconstructed ref would not answer to "a" and the constraint would fail to resolve. A ref has no paint order of its own: a JS GoFishRef has no .zOrder(), so RefMarkIR declares no zOrder, and a Python ref's z_order raises, naming the mark that draws the node or a .relate(...) z constraint as the place to order it.

There is no connect field on the wire. .layer(...) — the one way to overlay a connector — always serializes as an ordinary tier of a layer-chain: an empty chart() scope crosses as a ChartIR whose data is {type: "previous-tier"} (above), and a bare relational mark used directly crosses as that mark's ordinary MarkIR — by, where present, is just another field in the line/ribbon mark's own option bag, the same as stroke or opacity, not a separate IR shape. Elaboration (naming the previous tier's mark, wiring the next tier to selectAll(thatName), and defaulting the connector to zBelow) is entirely JS-side, done at resolve time by LayerBuilder.wireTiers(): no name is minted or leaked into the JSON, and the zBelow default is applied as a paint-order constraint rather than serialized as a field, so it composes with any explicit .zOrder(...) or .relate(...) the reader adds on top.

A chart's coordinate transform rides the IR as a small spec the deserializer maps back to the JS factory by type — e.g. { type: "polar", innerRadius, centralAngle, baselineAngle, direction, center }. fromJSON.ts rebuilds it with rebuild("coord", type, opts), the same call that rebuilds an operator or a mark (§ Modularity), from the spec without its type. An options-object factory takes the options whole (Coord.polar(opts)), so a parameterized polar/clock (donut hole, partial fan, start angle) round-trips without per-option plumbing, and the descriptor's positional field goes first (Coord.geo(opts.projection, rest)). An unknown type throws. polar and clock share one field group and differ only in its baselineAngle / direction defaults, so the type tag is what keeps the angular convention across the wire when those options are left out.

Channel values (h, w, fill, …) accept bare primitives (the shorthand path) or one of three explicit tagged objects:

  • field(name) → {type: "field", name} — per-row accessor, scaled.
  • datum(x) → {type: "datum", datum: x, measure?} — inline value, scaled.
  • literal(x) → {type: "literal", value: x} — inline constant, not scaled.

These three mirror Vega-Lite's field / datum / value trichotomy.

Non-finite numbers ​

JSON has no Infinity, -Infinity or NaN. JSON.stringify writes them as null, and Python's json.dumps writes a bare Infinity, which JSON.parse rejects. So the IR carries each one as a tagged object, the canonical form of MongoDB Extended JSON:

json
{ "$numberDouble": "Infinity" }
{ "$numberDouble": "-Infinity" }
{ "$numberDouble": "NaN" }

It is one mechanism for every number in the document (an option such as noise's smoothing, a channel value, a data row), at the serialization boundary, with no per-field rule:

  • A writer encodes the whole document as it makes it: JS toJSON (through Frontend.encodeIR, gofish-ir's frontend/nonFinite.ts) and Python (_json_value in gofish/ast.py, through which to_ir(), ir_payload and every bridge reply write their document). Python's rule also writes a numpy scalar as the Python value it stands for and a date or a datetime (a pandas Timestamp too) as its ISO 8601 string, as JSON writes a JS Date, so a document is JSON in both hosts.
  • The tag is part of the JSON encoding, so a reader decodes it where the JSON encoding put it, in Serialize.readIR (fromJSON.ts), which parses JSON text if it is given text and decodes the tags. A JSON document is read whole. A document a Python host ships is JSON except its tables, which are Arrow, so decodeDocument (widget-src/arrowBridge.ts) reads its JSON part through readIR and its tables from Arrow, and never walks a decoded row again; the bridge reads each reply the same way. buildIR (and so renderIR) and the other reconstruction functions (mapMark, mapOperator, ...) take a document as read.
  • The validator accepts the tagged Infinity and -Infinity wherever it expects a number, through one check (isIRNumber), and the JSON Schema has one shared Number def that every number field points to. A tagged NaN where a number is expected is an error: a NaN option is always a bug. Data rows are untyped, so a NaN there round-trips.

The tag was chosen over protobuf's JSON mapping (a bare string "Infinity" in a number field) because a string already means a field name in a channel value, so a bare string would be ambiguous there; a tagged object is not, in any position.

.relate() clauses ​

A layer combinator mark carries its .relate() callback as relate: { scope, clauses } (RelateIR): a scope id for the call and the callback's clauses in order, already evaluated on the authoring side. A clause is a ConstraintIR ({ type, options?, refs }) or a MarkIR that draws (arrow, enclose, a relational line, ...). An operand is the same value in both places, a RelateOperandIR{ type: "ref", selection: "name", scope }: a RefMarkIR whose scope is required, as a constraint's refs entry and as a child of a drawing clause. The validator checks each refs entry with the same ref walker, so an error names the entry's path. Python serializes a constraint's operands as the ref marks they are (_refs_ir). Its scope is the id of the .relate() call that bound it, which may be an enclosing layer's call when the clause holds a nested related layer. Python mints the id per call (relate-<uuid4>), so ids from separate mark-function replies never clash. The reader binds each id to the operands of the layer instance whose .relate() it is rebuilding: GoFishNode.relate hands the callback its environment, and relateCallback extends IRContext.relateScopes with it. The context is passed on to every ref the clauses hold, and an operand becomes that environment's RelateOperand for its name, or its ref (relateOperandFor). A name the call did not bind becomes an operand that means no node, reported by name where it is used. The two are told apart by refs: a constraint always carries it and a mark never does (isConstraintIR, which the validator uses too). Both readers (fromJSON.ts and the parity harness, which both go through fromJSON.ts) rebuild a constraint with the same constraintFromIR and hand the list back through the callback, so the layer schedules the clauses exactly as it would for a JS author. The field was called constraints, and held constraints only, before the .constrain() → .relate() rename.

A constraint is a descriptor kind of its own, "constraint", and CONSTRAINTS (descriptors.ts) is its table in DESCRIPTOR_TABLES, one entry per type: align's x/y, distribute's dir, spacing, order and glue, position's x/y/anchor, nest's padding, and no options for zAbove/zBelow. As a combinator mark's do, its fields sit under options on the wire, so acceptedFields("constraint", type) is the set of its option keys and the validator walks a clause's options with the same code that walks a combinator mark's. A key the entry does not declare is an error, and the JSON Schema's ConstraintIR is generated from the same table. The wire type position is both a combinator mark and a constraint, two constructs, so every lookup is by (kind, type): the docs name the constraint Constraint.position (DESCRIPTOR_NAMESPACES), and its Python core is _position_constraint_opts. distribute's spacing has spread's spacing type, one FieldType both entries name, so a Spacing strategy in a constraint is checked as it is in a spread. The Python Constraint.* methods keep their refs-first signatures and build their options with the generated _<type>_constraint_opts cores, so a nested value is routed through _to_wire there too. Options that only JS sets (spread's origin, measure and hasOrder on its distribute; partition's region and scatter's override on a position) are not declared. A position's x/y is a pixel or a DatumValue (the datum(v) wire shape, an AUTHORED_REFS entry), or a two-item tuple of them. A descriptor's requireOneOf (["x", "y"] for align, position and nest) asks for at least one of those keys, and only a constraint sets it: the validator checks it in its walk of a nested options object (a null key is not given), the JSON Schema writes it as an anyOf of required, and the generated Python core raises Constraint.<type> requires at least one of x, y. nest's two refs stay a hand-written check. constraintFromIR resolves the operands and calls rebuild("constraint", ...) (see "The way back" below), and throws on a type it does not know.

A worked example ​

ts
chart(seafood)
  .flow(spread({ by: "lake", dir: "x" }))
  .mark(rect({ h: "count", fill: "species" }).name("bars"))
  .render(container, { w: 500, h: 300, axes: true });
json
{
  "irVersion": 0,
  "ir": "gofish-frontend",
  "root": {
    "type": "chart",
    "data": { "type": "table", "id": "t0" },
    "options": { "w": 500, "h": 300, "axes": true },
    "operators": [{ "type": "spread", "by": "lake", "dir": "x" }],
    "mark": {
      "type": "rect",
      "name": "bars",
      "h": "count",
      "fill": "species"
    }
  },
  "tables": {
    "t0": [
      /* seafood */
    ]
  }
}

The bare "count" and "species" strings use the shorthand path; the runtime resolves them as field accessors. Writing h: field("count"), fill: field("species") would emit the explicit tagged-object form instead.

A combinator-form example ​

ts
chart(data).mark(
  layer([
    rect({ w: 100, h: 40, fill: "steelblue" }),
    text({ text: "label", fontSize: 14 }),
  ])
);
json
{
  "irVersion": 0,
  "ir": "gofish-frontend",
  "root": {
    "type": "chart",
    "data": { "type": "table", "id": "t0" },
    "operators": [],
    "mark": {
      "type": "layer",
      "__combinator": true,
      "children": [
        { "type": "rect", "w": 100, "h": 40, "fill": "steelblue" },
        { "type": "text", "text": "label", "fontSize": 14 }
      ]
    }
  },
  "tables": {
    "t0": [
      /* ... */
    ]
  }
}

The __combinator: true flag tells the deserializer to dispatch this node through the combinator factory registry (layer, spread, arrow, position, line, ribbon, treemap, pack, Porter-Duff) rather than the leaf-mark registry — same type discriminator namespace, different code path.

The descriptor table — one authored source for construct field lists ​

Before the change described in this section, a construct's field list (what keys rect or spread accept, which are required, what they default to) was hand-duplicated across four places: the TS type in schema.ts, the field checks in validate.ts, the $defs in jsonSchema.ts, and the Python factory in ast.py. They drifted — OPERATOR_TYPES listed "treemap" while the OperatorIR union and the JSON Schema enum omitted it, and the hand-written Python rect() exposed rs=/ts= kwargs that don't exist anywhere in JS (the real names are rSize/thetaSize at the time; they serialized, passed the open-world validator, and were silently dropped at render).

descriptors.ts collapses three of those four into one authored table: one entry per construct (operator, leaf mark, combinator mark, coord transform) listing its fields in a small type DSL (t.string, t.number, t.enum(...), t.literal(v) for exactly one value, such as the false in an axis title's string | false, t.channel(...) for a ChannelValue slot, t.ref("AxesOptions") for a pointer at a named type, and so on — see the file's t/ch exports). Shared field groups (boxDims, the ten closed x/y/w/h box channels plus the open dims bag keyed by axis name; paint, the five paint channels) are declared once and pulled into a mark's entry by reference, so most mark entries list only the fields genuinely their own.

Three smaller tables sit beside the construct entries. OPTION_TYPES declares the nested option objects a field points at by name, in the same type DSL: today AxesOptions (a boolean, or {x, y}), AxisOptions (a boolean, or {title, side, label, levels}), AxisLabelOptions ({angle, align}, the value of an axis's label), AxisInterval ({min, center, max, size, embedded}), AxisDimsValue (a channel value or an AxisInterval, the value of a dims entry), FieldPredicate ({field, between, closed}, the predicate of filter), Calendar, Partition and StructBin (the kinds valid on each key, below), and one entry per strategy family. A t.ref(name) resolves against it first, so the validator, the JSON Schema, and the Python generator all read one declaration of each. ChartOptions is one of them: t.object(CHART_OPTIONS), the chart-level options (w, h, coord, color, axes, legend, padding, schema), mirroring the JS ChartOptions. The validator walks ChartIR.options as t.ref("ChartOptions"), the JSON Schema emits its $def with the others, the Python generator builds _chart_opts from CHART_OPTIONS, and the docs build the chart options table from it (::: gofish-ref ChartOptions).

STRATEGIES declares the strategy families whose values cross the wire: Tile, Overlap, Curve, Bin, Calendar, Shape and Place. For each it lists the kinds, each with its params (types, defaults and docs), and the presets: a factory that makes a kind with some params already set, such as Overlap.sina(), which is noise with smoothing: "silverman". Each family derives its OPTION_TYPES entry, a union of one object per kind with a literal kind, so the strategy options are t.ref("Tile"), t.ref("Overlap"), t.ref("Curve"), t.ref("Shape") and t.ref("Place"). A kind may take a param by position (positional: "size"): Bin.step(0.5) in JS, Bin.step(0.5) with a positional-only parameter in Python, and { kind: "step", size: 0.5 } on the wire either way. A construct descriptor has the same field: COORDS.geo takes projection by position (Coord.geo("mercator", { lon })). The Bin kinds bin one of two keys (BIN_KINDS): step, thresholds and quantiles cut the line of one field, and hex and voronoi divide the plane of two. OPTION_TYPES.StructBin is the kinds of the plane, derived from the one Bin entry, so a struct's bin op takes a StructBin, and the JS layout checks a struct's bin with checkStructBin(value, where) and a field's with checkPartition(value, where), each of which names the right key for a kind of the other one.

A field's bin op takes a Calendar value, { kind: "month", every? } or { kind: "week", every?, start? } or { kind: "weekday", start? } (a weekday has no step), or a Bin kind that cuts one field's line. The Calendar family declares the Calendar kinds, one per level, and OPTION_TYPES.Calendar is its union. OPTION_TYPES.Partition is one tagged union of the Calendar kinds and the Bin kinds step, thresholds and quantiles, so the validator, the JSON Schema and the Python generator each read a field's partition by its kind, and the JS layout checks one with checkPartition(value, where). An axis's levels entries take OPTION_TYPES.Calendar.

OPTION_TYPES.Length is a length (#1176 step 6): { type: "length", percent, px? } or { type: "length", fr }, what JS percent(n) / fr(k) and Python's LengthValue dicts serialize to (LengthIR in schema.ts). A field takes a length exactly when its type names t.ref("Length") (lengthSize for a box's w/h, spacing on spread, stack and table, inset on rect). The JS factories read that from the table (checkLengths in gofish-graphics/src/ast/length.ts), so a length in any other field throws rather than being dropped by the wire's key filter.

A kind with no required param is a value as it is, and calling it sets options (#1133): Curve.monotone is Curve.monotone(), and Calendar.month({ every: 3 }) sets the month's step. Neither the wire nor the table has a second form for this. The bare kind is the value its call with no arguments makes, made callable, in both languages: in JS a function with that value's own keys and prototype and a toJSON() that returns it (bareKind, gofish-graphics/src/ast/strategyKind.ts), and in Python a dict subclass holding that dict with a __call__ (StrategyKind, gofish/_strategy.py, which the generated family modules apply to each factory with no required param). JSON has no functions, so the JS side reads a bare kind as its value once, where an option enters, at any depth of the options' plain objects and arrays (withKindValues in chart(data, options), createOperator, createMark and createRelationalMark, checkBinStep for a .bin(...) partition, and levelsFromOption for an axis's levels, which render options reach without passing a chart). The emitter, which drops every function as a callback, the shared encoder (encodeIR), and checkStrategy then see only values. A Python bare kind is a dict, so _to_wire and json.dumps read it as one. A param may carry bounds, t.num({ min: 0, finite: true }) for a pixel padding, t.num({ exclusiveMin: 0, finite: true }) for a hex radius, t.num({ exclusiveMin: 0, exclusiveMax: 1 }) for a quantile's probability, or t.num({ min: 1, integer: true }) for a Calendar step or a count of thresholds, and a list may carry a least length, t.array(t.num({ finite: true }), { minItems: 1 }) for a list of edges that must not be empty, and may hold no item twice, uniqueItems: true for a list of quantile probabilities. The validator, the JSON Schema (minimum, exclusiveMinimum, exclusiveMaximum, integer, minItems, uniqueItems) and the Python generator check them. When a value matches no branch of an untagged union, and only one branch is of the value's kind (a number for Bin.hex's radius: number | { x, y }), the validator reports that branch's own findings (expected a number above 0). A union whose branches are all tagged by the same field is checked against the one branch its value's tag names: a strategy's kind, or the type of a hand-authored ref that declares a tag in AUTHORED_REFS ("field" for FieldAccessor, "struct" for StructAccessor). So partition's by, a field accessor or a struct, reports what is wrong with a bad bin, not that no shape matched. The JS layout checks a strategy where it reads it, with checkStrategy(family, value, where) from validate.ts, the same walk the validator runs over a whole document. Every JS-side check is that one walk (walkOrThrow), so it throws the validator's own message: checkPartition, checkStructBin, and checkOption(name, value, where), which checks a named option object of OPTION_TYPES (layout checks each axis's options against AxisOptions with it, so a JS caller and a Python one get the same errors). So an unknown kind, an undeclared param, and a value out of bounds each fail in one place, whether the strategy came from a factory, a hand-written object, or Python IR. A JS check of an authored value passes over a ref whose authored value holds nodes (holdsNodes in AUTHORED_REFS): enclose's avoiding, typed t.ref("NodeSelectionIR"), holds the nodes themselves in JS, not their IR, so enclose checks it as it reads the nodes. Coord and Color are families on the two surfaces but not in this table: a coordinate transform is a JS object of functions, rebuilt from a type-tagged config (COORDS), and a color scale is tagged by _tag and takes its one argument by position.

schema.ts keeps the TypeScript types of the strategies, TileIR, OverlapIR, CurveIR, BinIR, CalendarPartitionIR, ShapeIR and PlaceIR, by hand, and each family's JS type is one of them (Curve.Curve is CurveIR; Shape.Shape is ShapeIR). descriptors.test.ts checks that their kinds and params agree with STRATEGIES. Every family is closed: there is no public way to add a kind, so every strategy can cross the wire (user-defined strategies are designed in #1101).

A named type may say which Python class builds a value of it (pyClass): FieldPredicate here, and FieldAccessor, StructAccessor and ChunkKey in AUTHORED_REFS, the list of refs to hand-authored shapes (a struct key, { type: "struct", fields: { x, y }, ops? }, is walked by its own walker, whose bin op is checked against OPTION_TYPES.StructBin). ChunkKey is the { type: "chunk", size } that chunk(size) builds, the by key that groups consecutive rows; a split operator's by takes a field name, a FieldAccessor, or a ChunkKey. Such a value already carries its wire keys, so the Python generator passes it through. An authored ref may instead say that its value holds nodes (holdsNodes: NodeSelectionIR). The generator passes such a value through unchecked, and Mark.to_dict writes each mark in an option as its IR (_marks_in_options in ast.py: a select_all(name) becomes the select form, any other mark its to_dict(payload), so the payload collects its callbacks too). One function, pyType in descriptors.ts, gives the Python type of a field: the generated factory signatures (the strategy modules' too) annotate with it and the Python docs tables print it, so the two agree. A ref prints as its pyClass when it has one, else as the Python type it stands for (AxesOptions is bool | dict). A strategy family's entry names its Python namespace (pyFamily: Tile, Overlap, Curve): the docs print that name, since users write Tile.squarify(...), and a signature annotates the value it makes (dict), since a namespace is not a type.

What's still authored, not in the table: the envelope (ChartIR/LayerChainIR/DataIR/MarkIR union, ChannelValue, ConstraintIR, LabelIR, TranslateIR) and cut/offset/ ref — these are structural or recursive shapes rather than flat field bags, and stay hand-written in schema.ts and jsonSchema.ts (the parts of those files the doc comment marks as "stays hand-written below"). ConstraintIR stays authored in schema.ts as a union by type, which descriptors.test.ts checks against CONSTRAINTS; its JSON Schema is generated.

A dims option is t.record(t.ref("AxisDimsValue")): each value is a bare ChannelValue or an interval object with only min/center/max/size/embedded keys. The two are told apart by a tag. A channel value that is an object always carries one (type for field(...), datum(...), and literals, or the __gofish_lambda bridge sentinel), so an untagged plain object is an interval. validate.ts exports that test as isAxisInterval, and its channel check rejects an untagged object, so the generic union walk reads an untagged object only as an interval and reports a misspelled anchor or a mistyped one. The renderer's dims.ts imports isAxisInterval and the key list AXIS_INTERVAL_KEYS, so the wire and the renderer share one definition; a test checks that the key list matches the AxisInterval declaration. The keys of dims are axis names that only mean something inside the enclosing coordinate space, so the wire keeps them open and carries them verbatim; the same goes for spread/stack's dir, which is a plain string on the wire.

Six consumers read the table:

  • validate.ts interprets it generically — a single walk over each descriptor's resolved fields instead of a per-type imperative switch. Operators, leaf marks, and a combinator mark's options are all checked the same way: an unknown field or a wrong-shaped known one is an error. Leaf marks used to only warn, during a rollout that ended once every Python story validated with no warnings. The Python bridge fields the renderer reads (__scope, __datum, __key) are declared in MARK_BASE_FIELDS like any other field, so nothing is exempt by name.

  • jsonSchema.ts builds one $def per operator (SpreadOperator, TableOperator, …) and one per leaf mark (RectMark, TextMark, …) from the table (buildOperatorDefs() / buildLeafMarkDefs()), and one per named option type (buildOptionTypeDefs()), merged into the hand-written $defs object. These $defs stay additionalProperties: true: the published schema keeps the open wire contract, and rejecting an unknown field is validate.ts's job.

  • The JS emitter (toJSON) filters what reaches the wire through it. A factory tags its options as the caller passed them; wireOpts keeps only the keys acceptedFields(kind, type) lists, and drops any function value, since a callback has no JSON form. So a key the descriptor does not know never reaches the IR, and a construct with no descriptor (such as time.transition) has no IR form: toJSON throws.

    acceptedFields (in descriptors.ts) is the one statement of which keys an options object takes on the wire: the descriptor's fields plus the base fields that sit in the same object. An operator's and a leaf mark's options are spread onto the node, so all of OPERATOR_BASE_FIELDS or MARK_BASE_FIELDS sit beside them. A combinator mark nests its options under options, where only COMBINATOR_OPTIONS_BASE_FIELDS (debug) ride; its other base fields sit on the node. The emitter keeps these keys, and the validator checks an operator or leaf-mark node against them. On a leaf mark, the base fields that Mark methods set (name, label, relate, zOrder, hidden, translate) are checked by their own walkers, so the check of the mark's channels skips them. The emitter writes each field(...) or datum(...) instance, and each strategy kind used bare (Curve.monotone), in its plain toJSON() form, so the document it returns validates as the wire carries it. It does this in the same pass that encodes non-finite numbers (Frontend.encodeIR), which shares unchanged subtrees, so table rows are not copied, and leaves a value whose toJSON() is a primitive (a Date, a Temporal value) as it is.

  • The JS deserializer (registry.ts) rebuilds a wire type through the factory its descriptor names — see § Modularity below.

  • gofish-python/scripts/generate.ts emits the mechanical part of the Python wrapper from the same table — see § Generating the Python factory layer below.

  • The docs site generates every API reference page's options table from it. A page under apps/docs/docs/{js,python}/api/ writes ::: gofish-ref rect under its ## Parameters heading, and the container (docs/.vitepress/markdown-it-gofish-ref.ts) renders the construct's doc line plus an Option/Type/Default/Description table — JS field names and a TS-ish type on a JS page, snake case kwarg names (from pyKwarg, see § Generating the Python factory layer) and a Python type on a Python page, with the shared groups (boxDims, paint) folded into a collapsed block. So an option's one-line description lives in its doc string and reaches the reader and the generated Python docstring from there. apps/docs/scripts/check-api-coverage.mjs closes the loop in CI: every construct in the table has a page in both languages, and every name a page references is a real construct.

The JSON Schema ​

The schema is Draft 2020-12. Its envelope (Root, ChartIR, LayerChainIR, DataIR, ChannelValue, LabelIR, and friends) is hand-written; the per-operator, per-leaf-mark and per-constraint $defs are generated from descriptors.ts at call time (see the previous section) and merged in. It lives in source at packages/gofish-ir/src/frontend/jsonSchema.ts and is emitted as a JSON artifact during build to packages/gofish-ir/dist/frontend/v0.json. See Full JSON Schema for the rendered document.

The high-level structure:

jsonc
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id":     "https://gofish.graphics/schema/frontend/v0.json",
  "type":    "object",
  "required": ["irVersion", "ir", "root"],
  "additionalProperties": false,
  "properties": {
    "irVersion": { "const": 0 },
    "ir":        { "const": "gofish-frontend" },
    "$schema":   { "type": "string" },
    "root":      { "$ref": "#/$defs/Root" },
    "tables":    { /* rows by table id */ }
  },
  "$defs": {
    "Root":       { "$ref": "#/$defs/MarkIR" },
    "ChartIR":    { /* type, data, operators, mark, options, zOrder, ... */ },
    "LayerChainIR": { /* type, tiers (first a ChartIR, then MarkIRs) */ },
    "DataIR":     { "oneOf": [/* table, select, previous-tier */] },
    "OperatorIR": { /* GENERATED oneOf: SpreadOperator | StackOperator | ... | TreemapOperator */ },
    "MarkIR":     { "oneOf": [LeafMarkIR, CombinatorMarkIR, RefMarkIR, OffsetMarkIR, CutMarkIR, ChartIR, LayerChainIR] },
    "LeafMarkIR": { /* GENERATED oneOf: RectMark | CircleMark | ... */ },
    "LabelIR":      { "oneOf": [/* boolean shorthand, array of {accessor, position, fontSize, ...} specs */] },
    "ConstraintIR": { /* GENERATED oneOf: AlignConstraint | DistributeConstraint | ... */ },
    "ChannelValue": { "oneOf": [/* primitives, field, datum, literal, bridge sentinels */] },
    "ChartOptions": { /* GENERATED from CHART_OPTIONS: w, h, coord, color, axes, ... */ },
    "AxesOptions":  { /* GENERATED from OPTION_TYPES, as are AxisOptions, AxisInterval, AxisDimsValue, FieldPredicate */ }
  }
}

The validator at validate.ts covers the same shapes, generically interpreting the descriptor table as described above, plus the structural checks for the hand-authored parts (e.g. table.by requires {x, y}). A field typed with a named option type, such as the axes override on spread/stack/scatter, is walked by the same generic interpreter against its OPTION_TYPES entry, and so is a chart's options, against CHART_OPTIONS. A nested object rejects a key it does not declare. A tagged union (every branch an object with a literal kind, as a strategy family is) is walked by the branch its kind picks, so an unknown kind gets its own error. Any other union accepts a value when any branch does, each branch checked into a probe context of its own, which is why the JSON Schema writes a union as anyOf, not oneOf.

Generating the Python factory layer ​

The treemap drift mentioned above ran the opposite direction from what you'd guess: treemap isn't a stray entry that needs deleting from OPERATOR_TYPES — it's confirmed as a genuine dual-form construct (a real Python story sizes with h: "fare" through the .flow() operator form), so the fix added a TreemapOperator member to the OperatorIR union and its JSON Schema enum, matching what descriptors.ts already modeled.

packages/gofish-python/scripts/generate.ts imports the same descriptors.ts table (via the gofish-ir/frontend package export, so it needs pnpm --filter gofish-ir build to have run first) and emits gofish/_generated.py and the family modules gofish/tile.py, gofish/overlap.py, gofish/curve.py, gofish/bin.py and gofish/calendar.py (from STRATEGIES) and gofish/coord.py (from COORDS) — checked into the repo, with a CI freshness check (pnpm --filter gofish-python gen then git diff --exit-code) rather than a build-time step, matching the "commit the generated Python" norm Altair and Plotly.py both follow.

Python kwargs are in snake case, while the wire stays in camel case. One function in descriptors.ts, pyKwarg(fieldName), gives each field's Python name: the field name in snake case (strokeWidth becomes stroke_width, emX becomes em_x), with a trailing underscore when that is a Python keyword (from becomes from_). Every generated function lists its (wireKey, pyName) pairs and builds its IR dict under the wire key, so rect(stroke_width=2) serializes as {"strokeWidth": 2}, and rect(strokeWidth=2) is a TypeError. The docs options tables call the same function. Hand-written wrappers take **options and pass them to a generated core, so none renames by hand: .label(accessor, **options) on marks and operators calls _label_opts, built from LABEL_OPTIONS (the options of a LabelSpecIR, which the JSON Schema's LabelIR and the validator also read), and line/ribbon/enclose call _line_opts/_ribbon_opts/_enclose_opts. The operator form of enclose reuses _enclose_opts, since its fields are the mark's: a childless enclose(**opts) is a _FlowStepMark, a bag mark that .flow() and compose() keep as a step. Its IR is the mark's in both places, modifiers included, and every modifier returns the same class, so a chain such as .name(...).label(...) keeps the step.

A nested option dict follows the same rule, by its declared type. The generator compiles a field's type into a small Python literal that records only its key structure: for an object, each Python key (from pyKwarg) paired with its wire key and the shape of its value. A field whose type has no option keys compiles to nothing and its value passes through as is. The generated module holds one entry per named option type (_OPTION_TYPES) and one interpreter, _to_wire, which every generated function calls on such a field. A declared camelCase key would be snake_case in the dict; no nested option type declares one today, so chart(data, axes={"x": {"label": {"angle": 45}}}) serializes as {"axes": {"x": {"label": {"angle": 45}}}}. An undeclared key, such as a flat "label_angle", is a TypeError that names the expected keys. Python's chart() now goes through a generated _chart_opts core built from CHART_OPTIONS, so an unknown chart keyword is a TypeError too. The axes option of .render(...), on a chart, a mark, or a layer, goes through _to_wire as well (in the widget constructor), so it is spelled and checked as in chart().

The conversion is driven by the declared type, never by the dict itself, so dicts whose keys are data keep them: a record type's keys (a schema keyed by column name, the axis names of dims) are never renamed, and a field typed any (color=Color.palette({...}) keyed by category, coord) passes through whole. A channel value built by field(...) or datum(...) is a dict already in wire form, so _to_wire passes it through by its class wherever it appears. That is how a dims entry works: field("x") passes through, and any other dict is an interval whose keys are checked (dims={"theta": {"width": 2}} is a TypeError). Two rules keep this honest. A union may have only one branch that a dict could match (a channel branch does not count, for the reason just given), or generation fails, since _to_wire would have to guess. A union whose dict branches mix named types with a Python class and types without one is told apart by class: the generator emits ("class", ((py_class, shape), ...), rest), an instance of a class takes its type's shape, and any other value takes rest. So spread's spacing, a number, a length or a Spacing, walks a LengthValue (from percent(n) or fr(k)) as a Length and a plain dict as a Spacing; a length written by hand as a plain dict, or a chunk(5), is a TypeError there. The other exception is a tagged union: when every dict branch is an object whose kind field is a literal or enum, and no two branches share a kind value (a strategy family: treemap's tile, {kind: "squarify", ratio?} or {kind: "slice"} or ...; scatter's overlap; a curve), the generator emits a ("tagged", "kind", {kind_value: branch_shape}) shape, and _to_wire picks the branch by the dict's kind. A missing or unknown kind, or a key that branch does not declare (ratio on slice), is a TypeError. And a t.ref must name either an OPTION_TYPES entry or one of the few refs the generator lists as already in wire form (today FieldAccessor, built by field(...), and ChunkKey, built by chunk(...)), or generation fails, so a new nested type has to be declared before Python can take it.

It emits:

  • Closed-signature leaf mark factories (rect, circle, ellipse, petal, text, image, polygon, blank) — pure kwargs-collection plus wire-key rename, with docstrings from each field's doc. An undeclared kwarg is a TypeError on all of them; none takes **kwargs.
  • The five compositing operators and other combinator-only marks — each named by its wire type, which is its public name (union, intersect, subtract, exclude, atop).
  • _opts(...) -> dict cores for the dual-form constructs (spread, stack, scatter, group, table, treemap, line, ribbon, layer, pack), with separate combinator-form cores for spread, stack, and treemap (their combinator entries add key; spread and stack also include the whole boxDims group, since JS Spread spreads its FancyDims into its box), and _chart_opts for chart() and the chart-tier layer([...]) — just the kwargs→dict half. The polymorphic operator-vs-combinator dispatch stays hand-written in ast.py, calling into these generated cores. A field whose type is a number with bounds (enclose's clearance, pack's spacing: t.num({ min: 0, finite: true })) is checked at the call when it is given, by the same checks a strategy factory runs on its params (below): enclose(clearance=-1) raises ValueError, as JS's enclose({ clearance: -1 }) throws at render. Every generated function does this, the cores and the leaf and combinator-only factories.
  • One module per strategy family in STRATEGIES (gofish/tile.py, gofish/overlap.py, gofish/curve.py, gofish/bin.py, gofish/calendar.py, gofish/shape.py, gofish/filter_.py), bound in __init__.py under the family name (Tile, Overlap, Curve, Bin, Calendar, Shape, Filter): one factory per kind and per preset, named by pyKwarg (Curve.catmull_rom, Overlap.separate). A module whose name is an operator's gets a trailing underscore, as a keyword kwarg does: gofish.filter is the filter operator, so the Filter family is gofish/filter_.py. A factory with no required param is decorated with bare_kind, so it is also the value its call with no arguments makes. Each returns {"kind": ..., **params} with snake_case param keys, and the option it is passed to renames them to wire keys through the family's _OPTION_TYPES entry, like any nested option dict, so Curve.perfect_arrows(pad_end=4) reaches the wire as {"kind": "perfectArrows", "padEnd": 4}. Each factory checks its params at the call, with checks the generator writes from each param's type (pyParamChecks in generate.ts): a value of the wrong Python type raises a TypeError, and a value out of bounds or not in an enum raises a ValueError that names the call and the kwarg (Overlap.separate(padding=...) must be >= 0, got -1). An optional param is checked when it is given; a required one always, so None is a TypeError (Bin.step(None)). An unknown kwarg is a TypeError from the signature, which for a bare kind lists its options (Calendar.month(step=3) names every=...). A dict param (Bin.hex's radius={"x": 2, "y": 500}) has its keys and each value checked the same way, and a table param (an array of records: Bin.voronoi's seeds) is read first with to_records, so a dataframe works too, and goes on the wire as dict rows. These are the same constraints checkStrategy checks on the JS side, read from the same table, so Python users see the error at the line that made it.
  • The Coord family module, gofish/coord.py, bound in __init__.py as Coord, written by the same family-module writer (writeFamilyModule) with one factory per COORDS entry (Coord.polar). Three things differ, each a parameter of the writer. A coord dict is tagged by type, not kind. It carries the camelCase wire keys itself, since a coord option is typed any and _to_wire does not route it. And no coord is a bare kind, as no JS coord is. Each factory checks its params at the call as a strategy factory does (Coord.geo("robinson") raises a ValueError), takes the descriptor's positional field as a positional-only parameter and every other option as keyword-only, as JS does (Coord.geo("mercator", lon=(-170, -30)), Coord.polar(inner_radius=0.5)), and puts a tuple on the wire as a list.

derive/resolve/join/filter (real RPC-bridge/ref-shape/DataFrame/ predicate logic) and field/datum/normalize/repeat/ref/select_all (not in the descriptor table at all) stay fully hand-written in ast.py, as does the Color family module (color.py), alongside the builder chain, _RefProxy, DatumValue arithmetic, and the widget/RPC layer — see Design space: generating the Python wrapper for the full hand-written-residue accounting and what's still deferred (the .layer()/relate-ref-walk follow-ups).

Generating this layer fixed real drift along the way: the hand-written rect() had exposed phantom rs=/ts= kwargs (see the descriptor-table section above) that the generator's output doesn't have; text() lost a phantom fontWeight and a phantom label kwarg that don't exist on the JS factory, and gained the box-dims channels it was missing.

Modularity — the registry pattern ​

Each operator and leaf-mark factory takes an optional serialize config; the factory attaches the produced value's wire form with withWire (ast/wire.ts). The emitter (toJSON) reads the tag at walk time. Adding a new operator is a one-line config change to its existing factory call, not a switch-statement edit.

ts
// graphicalOperators/spread.tsx
export const spread = createOperator<any, SpreadOptions>(Spread, {
  split: ({ by }, d) => /* ... */,
  channels: { w: "size", h: "size" },
  axisFields: ({ by, dir }) => /* ... */,
  serialize: "spread",                 // <-- new
});

// shapes/rect.tsx
export const rect = createMark(Rect, { w: "size", h: "size", /* ... */ }, "rect");
//                                                                          ^^^^^^

Combinator-form marks (spread([m1, m2]), Porter-Duff, etc.) tag with __combinator: true and stash the child marks on the tag so the emitter can walk them. Untagged operators emit as opaque {type: "derive"}; untagged marks throw. The tag holds the options as the caller passed them, and the emitter filters them through the descriptor (see the descriptor-table consumers above), so a factory never lists its own fields.

The way back is one table. FACTORIES in registry.ts maps each wire type to the public factory that rebuilds it, and rebuild(kind, type, opts, { children, bridge }) calls it: it returns undefined unless the descriptor table declares that type for that kind, and the kind decides the call. An operator, a leaf mark or a coord transform is factory(opts); a combinator mark is factory(opts, children). A descriptor's positional field is taken out of the options and passed first (geo(projection, { lon, lat })); when that field is required and missing, rebuild throws and names it (coord geo: missing required field "projection"). A dual-form construct such as spread or line therefore has one entry for both forms, and each compositing wire type is its factory's name (intersect to intersect, and so on). The coord transforms (polar, clock, wavy, geo) are entries of the same table. Only an OPERATOR_BUILDERS entry sees the bridge.

A constraint is called as a combinator mark is, factory(opts, refs) with its refs as the children, but from CONSTRAINT_FACTORIES rather than FACTORIES, since its types are a namespace of their own (position is also a combinator). That table is typed by ConstraintIR: each entry passes the wire options to its Constraint factory as they are, so an option the factory retypes is a compile error. A rename is one too: beside the table, Record<UnknownWireOption, never> must accept {}, and fails naming the key (Property '"distribute.order"' is missing) when a wire option is not a key of its factory's options. zAbove and zBelow take their two refs as (a, b), so their entries adapt.

Five operators keep a hand-written builder in OPERATOR_BUILDERS, because their IR is not their factory's options object: derive (it calls a Python lambda through the bridge and types the returned rows with its schema), resolve (the IR names a layer, the factory takes a selection), join and log (their factories take positional arguments), and filter (the IR describes a field predicate, the factory takes the predicate function, which fieldPredicate builds from that description). mark-fn, cut, offset and ref are rebuilt structurally in fromJSON.ts's mapMark. The serialize test fails when a descriptor has no factory or a factory has no descriptor, looking a constraint up in CONSTRAINT_FACTORIES.

User-defined custom marks via the no-channels createMark((data, props) => …) overload are an open question (deferred to v0.1+ — Olli treats them as opaque semantic boundaries via the name field).

Future evolution ​

v0 publishes the existing widget wire shape so consumers can ship against a typed schema today. Subsequent breaking releases will layer in the design improvements:

  • PascalCase rename ("Rect" over "rect", etc.) — matches ESTree/Babel convention; one lockstep migration across Python + JS + Olli per the no-back-compat policy.
  • __combinator removal — discriminate operator-form vs combinator-form by position (inside operators[] vs inside the mark tree's children) rather than a flag.
  • ChannelExpr-only IR — the fluent API would desugar all shorthand strings to field() at construction time; the IR sees only the canonical tagged-object form.
  • Multi-stage sibling schemas — distinct frontend / core / rendered schemas in the same package, with a one-way elaborate transform between them. Mirrors Vega-Lite → Vega and Lean Syntax → Expr. Reserved as namespace; nothing ships under Core or Rendered yet.
  • Inline meta? annotations — per-node optional slot for later-pass info (underlying-space classification, source positions, scale resolution). Open-typed; the slot is reserved in v0, unset by emitters.

Bridge extensions (Python widget) ​

The Python wrapper emits a few sentinels that are not part of the public schema — they extend it for the round-trip across anywidget:

SentinelMeaning
{__gofish_lambda}A Python callable; the JS deserializer wires it to an async accessor.
{__gofish_token}A hygienic-name token; resolved via a per-render token map.
__scope: trueThe @mark decorator's scope-wrap signal.
__datum / __keyA mark applied to its input, Python mark(d, key) (JS mark(d, key)).

A {__gofish_lambda} sentinel may sit at any depth of a channel option, not only as a top-level channel: for each option whose descriptor type can hold a channel (carriesChannel), the generated Python factory wraps every callable it finds in the option's plain dicts and lists (dims={"r": {"size": lambda d: ...}}), and unwrapOpts resolves the sentinels at any depth of both mark and operator options. When the Python Mark or Operator is built, one walk of its options (_scan_options) records the accessors per option and raises a TypeError for a callable in any other option (spread(by=lambda d: ...)), since the JS side resolves accessors only in channels. Serializing reads that record, so only the options that hold one are walked again. The Python option walks share one copy-on-write walk (_nonfinite.py), which keeps unchanged values as the same objects. In a channel, the accessor JS builds is resolved in its batch form (RESOLVE_ROWS): channel inference is synchronous, so the mark factory and the operator factory first resolve every Python accessor in a channel over the rows they are about to infer from (resolveChannelAccessors in channels.ts), with one bridge call per accessor for all of its rows, and inference reads the resolved values (#1080). That works the same in every channel kind, size and position included. A JS async (d) => ... accessor is not supported in a channel: only Python accessors are resolved this way. When no channel holds a Python accessor, nothing is awaited or copied.

Called per row, the accessor throws: it has only its batch form. The other place the IR uses it is a mark's zOrder: a mark field (MARK_BASE_FIELDS.zOrder) is a number or a {__gofish_lambda} handle. Python's .z_order(fn) writes the handle, withMarkMeta in fromJSON.ts passes the accessor to the mark's .zOrder(...), and the mark resolves it over its inputs the way a channel is resolved over its rows. The operator factory hands the mark every group of its split first (inputsResolver in createOperator.ts), so the accessor makes one bridge call for all of them, and each instance reads its own value back, as it would from a JS .zOrder((d) => ...). A mark applied outside an operator resolves it over its one input. That input is often a bag of refs (a ribbon over selectAll(...)), which crosses by the ref rule above. The Python callback is an accessor (_accessor_reply), so its reply is a one-value table.

Python's datum(x) emits the canonical {type: "datum", datum: x} shape directly — no bridge sentinel needed.

Olli and other pure-JS consumers don't see these — they're a FrontendIRWithBridge extension declared in the Python widget code (see The Jupyter Bridge & RPC).

One walk collects the callbacks ​

The IR names a Python callback by lambda id in three places: a derive operator's lambdaId, an accessor's {__gofish_lambda} sentinel, and a {type: "mark-fn", lambdaId} mark. Python's one IR walk (to_dict(payload) in gofish/ast.py, which every node takes) records each callback in payload.callbacks at the moment it writes the callback's id, so the walk that writes the IR is the one that collects what the IR names; no second walker can miss a place. Each callback is recorded in its answering form: it takes the rows JS sends and returns a reply's root and its own payload. A derive's or an accessor's reply names the table of the list it returned; a mark function's reply is the walk of what it drew, so its payload carries the tables and callbacks of that drawing. The same payload holds the tables (payload.table(data)), so a chart's rows and its callbacks are found by one rule wherever the chart sits.

A host keeps every callback it has registered, so a reply's callbacks need names that do not pile up. A mark function's code makes its marks afresh on each call, with fresh lambda ids, and it is called once per partition, again for each view of a widget. So a reply's payload names each callback it records by the call's scope and its walk order, not by its lambda id: the scope is a hash of the callback's id and the rows it was called with (answer). The same call names the same callbacks, and registering them replaces the ones it registered before, so the registry holds one set per distinct call, which a document's fixed rows bound. A root's callbacks keep their lambda ids.

One renderer, two hosts ​

Two hosts render Python IR: the notebook widget (packages/gofish-python/widget-src/index.ts, gofish/widget.py) and the Python parity harness (tests/harness/main.ts, tests/scripts/derive-server.py). Both call the same function, Serialize.renderIR(doc, container, renderOptions, bridge) in fromJSON.ts, which rebuilds the document's root as a mark position (buildIR, then mapMark) and renders it. buildIR takes the one document shape, FrontendIRDocument, with its tables as rows. Every rebuild threads one context object (IRContext: the bridge, the token resolver, and the current document's tables and a mark function's input refs), so a chart rebuilds by one rule wherever it sits. Both hosts share everything but transport:

  • The document. Python's ir_payload ships the document with its tables as base64 Arrow IPC by id (arrow_utils.b64_tables), beside the callbacks it names. The widget syncs it in its doc trait; the derive server answers /load with it. Arrow is only an encoding of tables: each host decodes them with the one codec, decodeDocument (widget-src/arrowBridge.ts, which the harness imports as the gofish-python/arrowBridge package export), before it calls renderIR; it reads the rest of the document as JSON (readIR). A JS document's tables are already rows. A chart that names a table its document does not carry is an error, and the validator reports it too, at any depth.
  • The one bridge call. DeriveBridge has one method, call(lambdaId, rows), for every callback kind, and every caller reaches it through callBridge in registry.ts, which encodes the refs the rows hold (see Refs in a callback's input above). Both hosts build it with the same codec, arrowBridge(send): the rows go out as base64 Arrow, and the reply is a document of the same shape, read by decodeDocument. Its root says what the callback produced: a table reference, {type: "table", id}, to the list a derive or an accessor returned (callForRows in registry.ts reads it), or the mark-position IR a mark function drew. Only send differs: the widget sets its call_request trait and waits for call_response, and the harness POSTs to the derive server's /call/<id>. On the Python side both run answer(callbacks, lambda_id, rows_b64), which decodes the ref sentinels in the rows, calls the callback, registers the callbacks its reply names (by the call's scope, § One walk collects the callbacks), and ships the reply as ir_payload ships a root. So the widget answers mark functions and accessors as the harness does (#1049).
  • The codec. A table is a list, so in both languages and both directions it is an Arrow table with one column, item, whose type is the type of the list's items: a struct column for records (its fields are the records' columns), a scalar column for values (an accessor's), a list column for groups (a derive's). Data flows as lists, so nothing inspects an item's shape: the decode always reads column 0 as the list, and the encode builds the item type from the values by one rule at every depth. Python wraps a dataframe's columns as one struct column without copying them (_records_table in arrow_utils.py), and a list goes through pa.array of its items. The JS decode reads a struct column a field at a time, from its child vectors (listOf in arrowDecode.ts), so a table of records costs what one column per field cost.
  • Column types cross both ways. The decode attaches each record field's type with Serialize.setColumnTypes, and the encode reads the types the list carries (getColumnTypes) to pick each column's Arrow type. So a column keeps its declared type across a round trip, and nothing guesses a type from the values. A Python integer column decodes as HasInteger and goes back as Int64; a time goes back as a millisecond timestamp in its unit's zone; a number column JS authored is Float64, because JS has one number type, so {x: 2} reaches Python as the float 2.0. The types ride the array, so a split leaf, a filter's result, and a derive's result whose values still fit keep them (copyColumnTypes, applySchema in schema.ts), and the partition a bridge call sends carries its columns' types: a derive sends its input as it is, and a Python accessor's batch call sends the rows inference reads with their types (the mark's rows, an operator's input and its split leaves, or the rows a group's refs reach, rowsReached), never a fresh untyped copy. A list inside a record carries its own types, for the fields of its records. A schema entry the reader declares for a column replaces its decoded type, integer included.
  • Values. The decode converts each column once by its Arrow type: a tz-aware timestamp becomes epoch milliseconds and marks the column as a time (Unit.time({ zone }) in its zone); a naive timestamp or a date is a wall-clock value, so it becomes an ISO string without an offset and is marked Unit.time() (UTC), which applySchema reads in the zone the chart declares for the column, as it reads the same string from JS data. The decode attaches types and converts no time to an instant itself: every reader of decoded rows runs applySchema with its own schema before it reads a value. A table is chart data; an accessor's values go through applySchema where the deserializer reads them, and a derive types its result with its own schema as it does a JS function's, so a derive's result and an accessor's result hold epoch milliseconds. A derive's lambda always gets a list: data flows as a list, and a split with no by hands on [row]. A list becomes a plain array, a struct a plain object, a 64-bit integer a JS number, and a null stays null. No schema names a value inside a list or a struct, so a time there decodes to epoch milliseconds (a naive timestamp or a date read in UTC, as a chart with no zone reads it), and a list of structs, a list of rows, carries its own column types (Unit.time({ zone }) in the timestamp's zone, or UTC, and HasInteger). A float NaN stays NaN: GoFish never reads NaN as missing. Only a pandas DataFrame's NaN crosses as null, because pandas defines NaN as the missing value of its float columns and pyarrow's from_pandas follows that rule. A column whose rows mix types (a string in one, a number in another) is a loud error in either encode (to_arrow_table, buildArrowTable), naming the column, never a silent coercion.
  • Render options. These are the JS .render(container, options) options: w, h, axes, legend, debug. The widget reads them from its traits, which the Python .render(...) call sets; the harness reads the story's render-options dict. axes and legend stay unset unless the caller passed them, so a chart's own option (or a layer chain's root tier's) decides.

The root has no cases of its own: a layer-chain rebuilds the real LayerBuilder wherever it sits, and a layer([...]) of charts is the JS layer(options, children) combinator with the layer's options as they are, whose relate clauses are rebuilt once and handed to .relate(...). A child named with chart(...).name(n) carries name on its ChartIR, which chartFromIR applies with ChartBuilder.name, so the clauses can refer to it. Python's layer([...], **options) is always that combinator mark and takes exactly JS layer's options; legend is a render option in both languages.

Prior art ​

The schema-shape and modularity decisions draw on four sources:

  • ESTree and Babel @babel/types — the tagged-union JSON ast pattern and the defineType registry. Babel's registry primitive in packages/babel-types/src/definitions/utils.ts is the model for the serialize config on createOperator / createMark. ESTree's universal type: string discriminator is the convention v0 follows (lowercase to match the existing widget wire format; PascalCase is the v0.1+ target).

  • Vega-Lite — the closest precedent for a JSON chart spec. Lessons stolen: separate authoring / runtime schemas (vega-lite-schema.json vs vega-schema.json) — informs the multi-stage sibling-schemas plan. $schema URL versioning. Lessons rejected: signature-key discrimination (VL's negative hasProperty checks in src/transform.ts:680 are brittle) and closed mark sets (VL marks are a fixed const-object; GoFish's registry pattern is open).

  • GHC's Trees-That-Grow — pass-parameterized in-memory AST. Lesson: even GHC doesn't serialize its phase-tagged HsSyn; it serializes a separate flatter IR (.hie files). The IR you emit is not the AST you keep. TTG's per-constructor extension fields inform the inline meta? slot. The phantom Pass type parameter is rejected for a JSON wire format where TypeScript can't enforce exhaustiveness anyway.

  • Lean 4 Syntax / TSyntax — pre-elaboration Syntax and post-elaboration Expr are structurally distinct inductives, not one type with a phase flag. The macro-expansion boundary is a type-level cut. Same conclusion as Vega-Lite reached from the opposite side: multi-stage means sibling schemas.