Skip to content

derive ​

Transforms data before it reaches the next operator or mark. The function receives the current data group and returns a new one.

js
gf.chart(seafood, { axes: true })
  .flow(
    gf.derive((d) => d.filter((row) => row.species === "Salmon")),
    gf.spread({ by: "lake", dir: "x" })
  )
  .mark(gf.rect({ h: "count", fill: "steelblue" }))
  .render(root, { w: 400, h: 250 });

Signature ​

ts
derive(fn);
derive(fn, { schema });

Parameters ​

fn (required, (rows: T[]) => U[] | Promise<U[]>) receives the rows at this point in the flow (inside .flow(), the current group's rows) and returns the list of rows the rest of the pipeline sees. It returns a list even for one row ((d) => [d[0]]). It may be async. The options:

Transforms the data with a function, derive(fn). A function does not serialize: the IR carries a Python bridge handle in its place.

OptionTypeDefaultDescription
schemaRecord<string, any>Column types of the result, keyed by column name, as in a chart's schema, e.g. Unit({ domain: Domain.discrete(levels), traits: [Trait.ordered()] }) or Unit.time(). They override the types the result keeps from its input or infers, and convert values (an ISO string in a time column becomes an instant).

schema takes the same column types as chart's schema.

Column types of the result ​

Each column of the returned rows is typed like chart data:

  • A column whose values still fit the type it had in the input keeps that type. A time column of unchanged instants stays a time, and an ordered column whose values are still text or numbers keeps the order, so a value outside its levels is an error where the order is used.
  • A column of Date values is a time in UTC.
  • Any other column has no type. A derive that rewrites a date to "Mar" makes the column plain text.
  • A schema entry overrides these for its column and converts the values as a chart's schema does: an ISO 8601 string in a Unit.time() column becomes an instant.

A result that is one object, not an array, is typed as one row the same way. A column of Date values that had a time type in the input keeps it, time zone included. Fitting reads the values, not what they mean: a derive that turns a time column into plain numbers, such as years, keeps the time type, because any number is a valid instant (#1089). Give the column its type with derive(fn, { schema }).

filter carries the input's types over as they are, since its rows are the input's rows.

ts
// Monthly highs, months in calendar order
derive(monthlyHighs, {
  schema: {
    month: Unit({ domain: Domain.discrete(MONTHS), traits: [Trait.ordered()] }),
  },
});

Examples ​

ts
// Filter before spreading
.flow(
  derive(d => d.filter(row => row.year === 2020)),
  spread({ by: "category",  dir: "x" })
)

// Compute a per-group sum (after spread, d is scoped to one group)
.flow(
  spread({ by: "category",  dir: "x" }),
  derive(d => [{ ...d[0], total: sumBy(d, "value") }])
)

// Reshape wide-to-long
.flow(
  derive(d => d.flatMap(row => [
    { ...row, measure: "a", value: row.a },
    { ...row, measure: "b", value: row.b },
  ]))
)

Data helpers ​

Helpers for the function derive takes:

  • normalize(rows, field): each row with field divided by the column's total.
  • repeat(row, field): the row, row[field] times.
  • quantileRows(rows, { x, density, n }): n rows at the quantiles of a density curve, one at each probability (i + 0.5) / n. rows is a grid of one curve, each row a position x and the density there, in any order, with no two rows at one position (call it after the split into curves). The density is integrated by the trapezoid rule. Each row made holds x and every column whose value is the same in all of rows (a group's month). Bin.quantiles and field(x).median() over the result read the curve instead of the grid. It stands in for first-class distributions (#980), which will replace it.
ts
// The quantiles of a drawn density, cut into ten equal-mass cells
.flow(
  derive((rows) => quantileRows(rows, { x: "x", density: "pdf", n: 1000 })),
  partition({ by: field("x").bin(Bin.quantiles(10)), dir: "x" })
)
.mark(circle({ cx: field("x").median(), r: 3 }))

Units: keeping axes shared across a transform ​

A channel that encodes a column carries the column's unit. GoFish uses units to decide when two axes may share a scale. A column with no declared unit has an unknown unit, which shares an axis with any column, so the columns a derive computes (lo, hi, a box plot's quartiles) share an axis with no annotation. Two different declared units on one axis are an error. See Unit.

  • A column's type (its declared quantity and unit, HasQuantity and HasUnit) rides the data array, so a column the input declares keeps it through derive.

  • An arbitrary derive can declare the unit of a column it makes with its schema:

    ts
    derive(quartiles, {
      schema: { lo: Unit("USD"), hi: Unit("USD") },
    });

An axis is titled by the quantities of its columns (each column's name, unless its schema declares a quantity), then a declared unit in parentheses: "Pay (USD)". See Schema.

If an axis combines two different units, you have two remedies:

  1. If the units really are the same, declare the same unit for both columns.
  2. If the units really differ, give the inner chart an explicit w/h so it becomes a self-contained scale region and never shares that axis.