Performance Optimizations

From July 10 to 12, we organized the first Flow PHP hackathon together with Józef (@stloyd) and Aleksander (@MrHDOLEK).

We had one goal: improve general Flow performance. We created a dedicated GitHub label for the event that groups together all the work we pushed through those 3 days. Those changes also pushed me to redesign Flow's serialization/hydration architecture and invent a new binary data format, called Floe.

GitHub Label: hackaton-07-2026

Benchmarks

Before I explain what and how we changed, take a look at the numbers.

Benchmarking Method
100k rows, same dataset files, same machine, PHP 8.3 + opcache, phpbench (mode of 3 iterations, 1 warmup). Delta = (1.0 - 0.41.1) / 0.41.1; negative is better.
1.0 pipelines read input from Floe format, 0.41.1 from Parquet (Floe did not exist yet); timings include input reading.

<?php

use function Flow\ETL\DSL\df;

df()
    ->read(from_xxx())
    ->batchSize(1000)
    ->run();

DataFrame Operations

Data Formats & Storage

Parquet engines: both versions ship php and arrow engines. 1.0 pins the engine explicitly per benchmark; 0.41.1 auto-selects (arrow whenever ext-arrow is loaded), so its php rows were measured with ext-arrow disabled and its arrow rows with ext-arrow loaded. On 0.41.1 the selected engine applies to both the input read and the write inside the write benchmark; on 1.0 the write benchmark reads its input from Floe.

Floe

A floe (or ice floe) is a large, flat sheet of ice floating on the ocean, a river, or a lake.

Floe is a dedicated, immutable binary data format created to speed up serialization/hydration of Flow Rows. It combines ideas from Parquet & Avro, optimized for PHP and Flow Rows.

Floe is a row-based binary format, with metadata stored as JSON in the footer. It is self-descriptive, which means the format provides a schema (also in the footer) that supports deeply nested data structures like lists/rows/maps.

Floe also supports schema evolution without breaking backward compatibility. This means we can add new columns between write batches, as long as those columns are nullable.

Encoders and Hydrator

Introducing Floe was just the first step to optimize performance. It allowed us to replace the old Serializer based on PHP's serialize & unserialize functions.

The diagram below shows how Flow used to read and convert data from datasets into Rows, and then back to the format expected by the destination storage.

As you can see, EntryFactory was responsible for encoding the values coming from a data source and casting them to a specific schema, and then each loader converted them back to whatever format the destination storage expected.

That was one of our biggest bottlenecks. Because of how generic it was, EntryFactory always cast data into a specific format, regardless of which data format it was reading from. You might ask: what is wrong with this approach? Most data formats like CSV, JSON or PostgreSQL return data as strings or simple scalar values anyway.

That is correct, but there are also data formats that are self-descriptive (they come with a schema), like Parquet or Floe. For those, casting — especially of nested data structures like lists / structures / maps — was redundant. That redundancy is not so painful when working with a few thousand rows, but once we move beyond 100k, the pain becomes real.

So with our new architecture, each data format provides its own dedicated Encoder/Decoder, which internally decides how to handle data just after reading it from the source and just before we write it to the destination.

<?php

declare(strict_types=1);

namespace Flow\ETL\Row;

/**
 * @template TPhysical
 */
interface Encoder
{
    /**
     * @param list<TypedRowValues> $batch
     *
     * @return list<TPhysical>
     */
    public function encode(array $batch): array;

    /**
     * @param list<TPhysical> $batch
     *
     * @return list<RawRowValues>
     */
    public function decode(array $batch): array;
}

Once our data is encoded into its plain PHP representation, we pass it to the Hydrator, whose only goal is to create the Rows instances based on the passed RawRowValues.

<?php

declare(strict_types=1);

namespace Flow\ETL\Row;

use Flow\ETL\Rows;
use Flow\ETL\Schema;

interface Hydrator
{
    /**
     * @param list<RawRowValues> $batch
     */
    public function cast(array $batch, ?Schema $schema = null): Rows;

    /**
     * @return list<TypedRowValues>
     */
    public function dehydrate(Rows $rows): array;

    /**
     * @param list<RawRowValues> $batch
     */
    public function hydrate(array $batch, ?Schema $schema = null): Rows;
}

This separation also allowed us to create two instances of the Hydrator: one based on PHP that internally uses EntryFactory, and a Rust-based hydrator available when flow-php/flow-ext is installed.

• PhpRowHydrator
• NativeRowHydrator

The Hydrator is not the only place where we managed to leverage the PHP extension. We used a similar pattern (php / native) for Floe's dedicated Encoder, AdaptiveFloeEncoder, which let us squeeze out even more performance.

Caching / Sorting / Grouping / Joining

Since all the mechanisms mentioned above need to dump Rows from memory to disk (or other storage) at some point, Floe became the default serialization format, which allowed us to significantly improve the performance of those operations.

But we had to slightly change the Serializer contract, which previously attempted to serialize/unserialize any object. Now it focuses only on Rows, which can really represent any data structure.

<?php

declare(strict_types=1);

namespace Flow\Serializer;

use Flow\ETL\Rows;
use Flow\Filesystem\DestinationStream;
use Flow\Filesystem\SourceStream;
use Flow\Serializer\Exception\SerializationException;

/**
 * @internal
 */
interface Serializer
{
    /**
     * @throws SerializationException
     */
    public function serialize(Rows $rows, DestinationStream $destination): void;

    /**
     * @throws SerializationException
     */
    public function unserialize(SourceStream $source): Rows;
}

The new serialization contract also expects source/destination streams to read from and write to.

Bucketing

The final step of this refactoring was improving bucketing (breaking the dataset into smaller buckets to reduce memory consumption).

Sort / Join / Group By - all 3 are directly affected by those changes, and can now operate on datasets of any size. I believe there is still room for optimization, especially around Sort, which currently still takes around 100mb of RAM to sort 100k rows.

This is also not the final performance we are aiming for. All of the benchmarks above are executed in a single process. True scalability will come with parallel processing, and that's another area where Floe will shine, since we'll be able to move (shuffle) data between workers faster.

Using PHP's native serializer, 100k rows produced a 1.5GB string; with Floe we managed to drop that to 36MB — and this is still before compression!

The goal is to implement different compression algorithms and let the Floe reader/writer use them, like Snappy, which is pretty much the industry standard today.

The end goal

These numbers might not feel very impressive yet, but joining 100k rows in under 3 seconds with only 30MB of RAM might already be a good reason to move some processes outside the database and work on plain files.

Flow's true performance boost will come with parallel processing, which will make speed depend on another variable — the number of processes, and maybe even nodes, that can process ETL pipelines in parallel.

That's our final goal, and I hope it will make Flow a real alternative to other data processing frameworks.