Browse documentation

#Sort

#Sort Algorithm

The sort algorithm is configured through config_builder()->sort() - all options live on the algorithm's own builder:

• external_sort() - default; buffers rows in memory, sorts them and spills each sorted run as a bucket through a BucketsStorage (the same abstraction used by join and group by), then k-way merges the runs back into one sorted stream. Memory usage is bounded by one run.
• memory_sort() - buffers the whole dataset and sorts it in one pass; fastest, but everything must fit in RAM. It has no options.

<?php

data_frame(
    config_builder()->sort(memory_sort())
)
    ->read(from_parquet('orders.parquet'))
    ->sortBy(ref('total')->desc())
    ->run();

#External sort options

<?php

data_frame(
    config_builder()
        ->sort(
            external_sort()
                ->runSize(10_000)               // rows buffered and sorted in memory before spilled as one run
                ->bucketsCount(100)             // how many runs are merged at once (merge fan-in)
                ->batchSize(1000)               // rows per spill/output batch
                ->filesystemProtocol('file')    // filesystem protocol used for the default FilesystemBuckets
        )
)
    ->read(from_parquet('orders.parquet'))
    ->sortBy(ref('total')->desc())
    ->run();

The storage implementation is swapped with external_sort()->storage(BucketsStorage $storage).

While runs are spilled, each bucket is announced downstream as a single metadata row (BucketShape: _bucket_id, _bucket_total_rows). Those rows are an internal pipeline detail consumed by the merge processor - they never appear in the sorted output.

#Example

<?php 

data_frame()
    ->read(from_sequence_number('id', 1, 10))
    ->sortBy(ref('id')->desc())
    ->collect()
    ->write(to_output(false))
    ->run()

Output:

+----+
| id |
+----+
| 10 |
|  9 |
|  8 |
|  7 |
|  6 |
|  5 |
|  4 |
|  3 |
|  2 |
|  1 |
+----+
10 rows

Found a typo or an outdated section? Edit this page on GitHub ↗