Examples

Batching

Description

Load all rows into memory and process them at once. This is useful for debugging and working with small datasets.

Warning: Avoid using collect with large datasets as it loads everything into memory. For controlled memory consumption, use batchSize instead.

Documentation

Batch size decides how many rows travel through the pipeline together.

Batch Processing

Code

<?php

declare(strict_types=1);

use function Flow\ETL\DSL\{data_frame, from_array, to_output};

require __DIR__ . '/vendor/autoload.php';

data_frame()
    ->read(from_array([
        ['id' => 1, 'name' => 'John'],
        ['id' => 2, 'name' => 'Doe'],
        ['id' => 3, 'name' => 'Jane'],
        ['id' => 4, 'name' => 'Smith'],
        ['id' => 5, 'name' => 'Alice'],
    ]))
    ->collect() // alternatively we can also use ->batchSize(-1)
    ->write(to_output(truncate: false))
    ->run();

Contributors

Built in the open.

Join us on GitHub