Examples

Partitioning

Description

Partition into a flat directory structure using {column} placeholders in the destination path instead of Hive-style column=value directories. Every partition consumed by a placeholder becomes part of the file or directory name; remaining partitions still create column=value directories.

output
├── blue
│   └── PRODUCT02.csv
├── green
│   └── PRODUCT01.csv
└── red
    ├── PRODUCT01.csv
    └── PRODUCT02.csv

Reading with the same placeholder pattern recreates the partitions from the path, including support for partition pruning with filterPartitions(). Keep in mind that this layout is not self-describing - a plain glob like output/**/*.csv will read the data but won't recognize any partitions.

Documentation

Code

<?php

declare(strict_types=1);

use function Flow\ETL\Adapter\CSV\{from_csv, to_csv};
use function Flow\ETL\DSL\{data_frame, from_array, overwrite, ref, to_output};

require __DIR__ . '/vendor/autoload.php';

data_frame()
    ->read(from_array(
        [
            ['id' => 1, 'color' => 'red', 'sku' => 'PRODUCT01'],
            ['id' => 2, 'color' => 'red', 'sku' => 'PRODUCT02'],
            ['id' => 3, 'color' => 'green', 'sku' => 'PRODUCT01'],
            ['id' => 4, 'color' => 'blue', 'sku' => 'PRODUCT02'],
        ]
    ))
    ->partitionBy(ref('color'), ref('sku'))
    ->mode(overwrite())
    ->write(to_csv(__DIR__ . '/output/{color}/{sku}.csv'))
    ->run();

data_frame()
    ->read(from_csv(__DIR__ . '/output/{color}/{sku}.csv'))
    ->write(to_output(truncate: false))
    ->run();

Contributors

Built in the open.

Join us on GitHub