Examples

Partitioning

Description

{column} placeholders in the destination path name files and directories after the partition values, instead of Hive column=value directories. A partition column without a placeholder still gets a column=value directory.

output/blue/PRODUCT02.csv
output/green/PRODUCT01.csv
output/red/PRODUCT01.csv
output/red/PRODUCT02.csv

Reading with the same pattern restores color and sku from the path, and filter() on them prunes files. The layout is not self-describing: a plain glob such as output/**/*.csv reads the data without the partition columns.

Documentation

Partitioning is declared on the loader, and lets a reader skip whole directories.

Code

<?php

declare(strict_types=1);

use function Flow\ETL\Adapter\CSV\{from_csv, to_csv};
use function Flow\ETL\DSL\{data_frame, from_array, overwrite, partition_by, ref, to_output};

require __DIR__ . '/vendor/autoload.php';

data_frame()
    ->read(from_array(
        [
            ['id' => 1, 'color' => 'red', 'sku' => 'PRODUCT01'],
            ['id' => 2, 'color' => 'red', 'sku' => 'PRODUCT02'],
            ['id' => 3, 'color' => 'green', 'sku' => 'PRODUCT01'],
            ['id' => 4, 'color' => 'blue', 'sku' => 'PRODUCT02'],
        ]
    ))
    ->write(to_csv(__DIR__ . '/output/{color}/{sku}.csv')->saveMode(overwrite())->partitionBy(partition_by(ref('color'), ref('sku'))))
    ->run();

data_frame()
    ->read(from_csv(__DIR__ . '/output/{color}/{sku}.csv'))
    ->write(to_output(truncate: false))
    ->run();

Contributors

Built in the open.

Join us on GitHub