Flow PHP

FloeExtractor implements BatchableExtractor, Extractor, FileExtractor, MetadataColumnsExtractor, SplittableExtractor

FinalYes

Interfaces

BatchableExtractor
A source that decides how many rows it puts in one Rows: withBatchSize() bounds what the source builds.
Extractor
FileExtractor
MetadataColumnsExtractor
SplittableExtractor

Methods

__construct()  : mixed
batchSize()  : int<1, max>
extract()  : Generator<Signal|null, void>
Yields rows in batches. When the source is a BatchableExtractor a batch holds between 0 and that extractor's own batchSize() rows; the size is chosen by the extractor and MAY vary between batches. Consumers must not assume a minimum size, a constant size, or a non-empty batch.
isRepeatable()  : bool
True when extract() can run more than once and yield the same rows each time, so a build-time pass may read the source before the run without consuming it. A one-shot stream returns false; a wrapper answers for what it reads.
partitionSchema()  : Schema
partitionTypes()  : static
prunes()  : bool
reader()  : FileSplitReader
The read of this source with its state resolved (schema, options, inference): what a worker runs for one split, and what the in-process scan delegates to.
refresh()  : void
Drops the listing and everything derived from it, so the next run lists the source again. A run calls it when it ends; a plan that does not run (explain(), schema()) keeps what it listed for the run that follows.
schema()  : Schema
Answers before extract() runs, so every batch it yields carries this shape. Takes no FlowContext: a source that needs the pipeline's context to describe itself has not moved the answer to bind time.
source()  : Path
splits()  : array<int, FileSplit>
Byte windows over every admitted file; a section belongs to the window holding its midpoint.
statistics()  : Statistics
What the source knows about the data before a single row is read: how many rows, how many bytes. A source that knows nothing returns `new Statistics()`. Describes the source with NO pushdown applied - a limit or a partition filter the plan pushed is the plan's to account for.
unionByName()  : self
Reconcile every listed file's footer instead of trusting the first one.
withBatchSize()  : static
Mutates and returns the same instance, so one extractor shared across from_all() children shares its size.
withMetadataColumns()  : static
withSchema()  : static
Declares the shape every yielded batch must carry. A source that describes itself uses this instead of its own description, and values are cast to fit.

Methods

__construct()

public __construct(Path $path[, Codec $codec = new NoopCodec() ][, int $chunkSize = 65536 ][, Filesystem $filesystem = new NativeLocalFilesystem() ]) : mixed
Parameters
$path : Path
$codec : Codec = new NoopCodec()
$chunkSize : int = 65536
$filesystem : Filesystem = new NativeLocalFilesystem()

batchSize()

public batchSize() : int<1, max>
Return values
int<1, max>

extract()

Yields rows in batches. When the source is a BatchableExtractor a batch holds between 0 and that extractor's own batchSize() rows; the size is chosen by the extractor and MAY vary between batches. Consumers must not assume a minimum size, a constant size, or a non-empty batch.

public extract(Backend $backend, Criteria $criteria) : Generator<Signal|null, void>
Parameters
$backend : Backend
$criteria : Criteria
Return values
Generator<Signal|null, void>

isRepeatable()

True when extract() can run more than once and yield the same rows each time, so a build-time pass may read the source before the run without consuming it. A one-shot stream returns false; a wrapper answers for what it reads.

public isRepeatable() : bool
Return values
bool

reader()

The read of this source with its state resolved (schema, options, inference): what a worker runs for one split, and what the in-process scan delegates to.

public reader() : FileSplitReader
Return values
FileSplitReader

refresh()

Drops the listing and everything derived from it, so the next run lists the source again. A run calls it when it ends; a plan that does not run (explain(), schema()) keeps what it listed for the run that follows.

public refresh() : void

schema()

Answers before extract() runs, so every batch it yields carries this shape. Takes no FlowContext: a source that needs the pipeline's context to describe itself has not moved the answer to bind time.

public schema() : Schema

Called once per RUN - every run plans afresh - so it MUST be idempotent and cheap on repeat: memoise what it sniffs, as CSVExtractor and ArrayExtractor do.

Return values
Schema

splits()

Byte windows over every admitted file; a section belongs to the window holding its midpoint.

public splits(Criteria $criteria, int $targetSplitBytes) : array<int, FileSplit>
Parameters
$criteria : Criteria

the read being cut, as the plan hands it to every source today: a file source prunes its listing by the path filter, so a pruned file is never cut; any other source reads only its limit; the criteria's own split is ignored

$targetSplitBytes : int

advisory: the caller's preferred split size, not a guarantee

Return values
array<int, FileSplit>

statistics()

What the source knows about the data before a single row is read: how many rows, how many bytes. A source that knows nothing returns `new Statistics()`. Describes the source with NO pushdown applied - a limit or a partition filter the plan pushed is the plan's to account for.

public statistics() : Statistics

Called AT MOST ONCE per run, and only when something needs the answer - so it must be lazy, never computed in the constructor - and it MUST be idempotent and cheap on repeat: memoise what it reads, the way schema() does.

A file source may only spend what the run already spends: it may complete a listing it must produce anyway and read metadata from the files schema() already opens, and it may extrapolate from those. It must not open an extra file or make an extra remote call. A database source may ask the planner for an estimate (EXPLAIN without ANALYZE), never run the query.

Return values
Statistics

unionByName()

Reconcile every listed file's footer instead of trusting the first one.

public unionByName([bool $union = true ]) : self
Parameters
$union : bool = true
Return values
self

withBatchSize()

Mutates and returns the same instance, so one extractor shared across from_all() children shares its size.

public withBatchSize(int $batchSize) : static
Parameters
$batchSize : int
Return values
static

withMetadataColumns()

public withMetadataColumns(bool $addMetadataColumns) : static
Parameters
$addMetadataColumns : bool
Return values
static

withSchema()

Declares the shape every yielded batch must carry. A source that describes itself uses this instead of its own description, and values are cast to fit.

public withSchema(Schema $schema) : static
Parameters
$schema : Schema
Return values
static
On this page

Search results