Browse documentation

#Building Blocks

Columns of the Data Frame are described by the Schema - it owns their names, order, types and nullability. A Row carries the values for one record, keyed by column name. A group of Rows is called Rows, represented by the Rows class, and every Rows carries the one Schema that describes it.

Let's look at the following example:

<?php

declare(strict_types=1);

use function Flow\ETL\DSL\{bool_schema, int_schema, row, rows, schema, str_schema};

$rows = rows(
    schema(int_schema('id'), str_schema('name'), bool_schema('active')),
    row(['id' => 1, 'name' => 'user_01', 'active' => true]),
    row(['id' => 2, 'name' => 'user_02', 'active' => false]),
    row(['id' => 3, 'name' => 'user_03', 'active' => true]),
    row(['id' => 4, 'name' => 'user_04', 'active' => false]),
);

Rows are the main data structure in Flow ETL, they're used to represent data in the data frame. Extractors are yielding Rows and Loaders are saving Rows.

The same can be achieved using the following code:

<?php

declare(strict_types=1);

use function Flow\ETL\DSL\array_to_rows;
use function Flow\ETL\DSL\bool_schema;
use function Flow\ETL\DSL\int_schema;
use function Flow\ETL\DSL\schema;
use function Flow\ETL\DSL\str_schema;

$rows = array_to_rows(
    [
        ['id' => 1, 'name' => 'user_01', 'active' => true],
        ['id' => 2, 'name' => 'user_02', 'active' => false],
        ['id' => 3, 'name' => 'user_03', 'active' => true],
        ['id' => 4, 'name' => 'user_04', 'active' => false],
    ],
    schema(int_schema('id'), str_schema('name'), bool_schema('active')),
);

#Column Types

Every column is described by a Definition, built with the matching *_schema() DSL function. A definition owns the column name, its Flow Type, nullability and metadata.

A column holds exactly one type, so union_schema() accepts only null|T - a nullable column - and refuses every other union. Declare the widest common type with str_schema(), or json_schema() when the shape is genuinely dynamic.

The schema is declared, never guessed: array_to_rows() takes it as its second argument and a Hydrator turns the raw values into Rows against it, casting each one to the type its column declares. A value the declared type refuses aborts the batch with a SchemaMismatchException naming the column and its row.

Schema inference belongs to the readers, which sample a source and derive a schema from it before any row flows - see Schema.

Found a typo or an outdated section? Edit this page on GitHub ↗