Extractor
Interfaces
- BatchableExtractor
- A source that decides how many rows it puts in one Rows. withBatchSize() bounds what the source
BUILDS; batches() re-slices what a source already emitted and cannot lower its peak.
- FileExtractor
- InfersSchema
- LimitPushDown
- A source that can read fewer rows when the plan has a LIMIT above it. This is an optimization
hint, never a guarantee: the limit operator stays in the plan and enforces the exact count, so an
implementation that yields more rows than asked - or ignores the hint - is still correct.
- MetadataColumnsExtractor
- OverridingExtractor
- Extractors implementing OverridingExtractor interface overrides one or more extractors.
- RewindableExtractor
- A rewindable extractor can be read more than once and yield the same rows each time, so a build-time
pass may scan it before extraction without consuming it.
- SelfDescribingFile
Classes
- ArrayExtractor
- BatchByExtractor
- BatchExtractor
- CacheExtractor
- ChainExtractor
- CollectingExtractor
- DataFrameExtractor
- FileColumns
- Both read paths take the tail from one instance of this, in one order, so they cannot disagree.
- FileConstants
- FilesExtractor
- InferredRows
- The gate places a refused row in its batch; only the source knows where that batch started, so a row past a
bounded sample is reported here at its position in the whole source, with the way out.
- InMemoryRows
- MemoryExtractor
- PartitionColumns
- PartitionTypes
- Types for partition columns discovered from a path.
- PathPartitionsExtractor
- Repeatability
- RowsExtractor
- SequenceExtractor
- SourceFile
- The partition values come from the path, so a read knows them without opening the file.
- SpilledRows
- A source that can be read only once, made replayable: every row is written to a temporary
schema-less spill file as it streams past, and the replay reads the file back. The source is
advanced exactly once, ever; the spill it produced can be replayed as often as the caller likes,
and lives until this object is collected.
Namespaces
- SequenceGenerator
Traits
- Batches
- DeclaresPartitionTypes
- FileReading
- MetadataColumns
- PathFiltering
- PushesLimit
Enums
- Signal
- SpillState
- The lifecycle of one SpilledRows. Split into states rather than carried as booleans because the
reachable-but-unnamed fourth state - "spilled K of N rows, then the source threw" - is where a bool
pair silently re-streams a half-consumed source and loses rows.