SplittableExtractor extends Extractor
Methods
- extract() : Generator<Signal|null, void>
- Yields rows in batches. When the source is a BatchableExtractor a batch holds between 0 and that extractor's own batchSize() rows; the size is chosen by the extractor and MAY vary between batches. Consumers must not assume a minimum size, a constant size, or a non-empty batch.
- isRepeatable() : bool
- True when extract() can run more than once and yield the same rows each time, so a build-time pass may read the source before the run without consuming it. A one-shot stream returns false; a wrapper answers for what it reads.
- reader() : Reader
- The read of this source with its state resolved (schema, options, inference): what a worker runs for one split, and what the in-process scan delegates to.
- schema() : Schema
- Answers before extract() runs, so every batch it yields carries this shape. Takes no FlowContext: a source that needs the pipeline's context to describe itself has not moved the answer to bind time.
- splits() : array<int, Split>
- Where this source may legally be cut. Called at most once per run, before any worker starts. The splits cover every row exactly once, in source order; each is read through the scan of its kind - a FileSplit through Extractor::extract($backend, $criteria->withSplit($split)).
- statistics() : Statistics
- What the source knows about the data before a single row is read: how many rows, how many bytes. A source that knows nothing returns `new Statistics()`. Describes the source with NO pushdown applied - a limit or a partition filter the plan pushed is the plan's to account for.
- withSchema() : static
- Declares the shape every yielded batch must carry. A source that describes itself uses this instead of its own description, and values are cast to fit.
Methods
extract()
Yields rows in batches. When the source is a BatchableExtractor a batch holds between 0 and that extractor's own batchSize() rows; the size is chosen by the extractor and MAY vary between batches. Consumers must not assume a minimum size, a constant size, or a non-empty batch.
public
extract(Backend $backend, Criteria $criteria) : Generator<Signal|null, void>
yield from is allowed only as the whole body: it routes send() into the delegate, so the delegate sees
Signal::STOP and this generator ends with it. A generator that does anything after delegating forwards with
foreach + send().
$criteria is what the plan pushed into this read. A source returns only rows matching every one of $criteria->conjuncts; only a FileExtractor ever receives any. The limit and the path filter are hints: the plan still enforces the limit, and the predicate of every path filter is also one of the conjuncts or stays in a Filter node, so a source that reads or lists more is still correct. Batches built on another backend than $backend are adopted by the runtime. A record that cannot become a row is yielded as its CorruptedRecordException, in source order; the runtime routes or refuses it.
Parameters
Return values
Generator<Signal|null, void>isRepeatable()
True when extract() can run more than once and yield the same rows each time, so a build-time pass may read the source before the run without consuming it. A one-shot stream returns false; a wrapper answers for what it reads.
public
isRepeatable() : bool
Return values
boolreader()
The read of this source with its state resolved (schema, options, inference): what a worker runs for one split, and what the in-process scan delegates to.
public
reader() : Reader
Return values
Readerschema()
Answers before extract() runs, so every batch it yields carries this shape. Takes no FlowContext: a source that needs the pipeline's context to describe itself has not moved the answer to bind time.
public
schema() : Schema
Called once per RUN - every run plans afresh - so it MUST be idempotent and cheap on repeat: memoise what it sniffs, as CSVExtractor and ArrayExtractor do.
Tags
Return values
Schemasplits()
Where this source may legally be cut. Called at most once per run, before any worker starts. The splits cover every row exactly once, in source order; each is read through the scan of its kind - a FileSplit through Extractor::extract($backend, $criteria->withSplit($split)).
public
splits(Criteria $criteria, int<1, max> $targetSplitBytes) : array<int, Split>
Parameters
- $criteria : Criteria
-
the read being cut, as the plan hands it to every source today: a file source prunes its listing by the path filter, so a pruned file is never cut; any other source reads only its limit; the criteria's own split is ignored
- $targetSplitBytes : int<1, max>
-
advisory: the caller's preferred split size, not a guarantee
Return values
array<int, Split>statistics()
What the source knows about the data before a single row is read: how many rows, how many bytes. A source that knows nothing returns `new Statistics()`. Describes the source with NO pushdown applied - a limit or a partition filter the plan pushed is the plan's to account for.
public
statistics() : Statistics
Called AT MOST ONCE per run, and only when something needs the answer - so it must be lazy, never computed in the constructor - and it MUST be idempotent and cheap on repeat: memoise what it reads, the way schema() does.
A file source may only spend what the run already spends: it may complete a listing it must produce anyway and read metadata from the files schema() already opens, and it may extrapolate from those. It must not open an extra file or make an extra remote call. A database source may ask the planner for an estimate (EXPLAIN without ANALYZE), never run the query.
Return values
StatisticswithSchema()
Declares the shape every yielded batch must carry. A source that describes itself uses this instead of its own description, and values are cast to fit.
public
withSchema(Schema $schema) : static
Parameters
- $schema : Schema