ColumnIndex
Optional statistics for each data page in a ColumnChunk.
Forms part the page index, along with OffsetIndex.
If this structure is present, OffsetIndex must also be present.
For each field in this structure,
Properties
- $_TSPEC : mixed
- $boundary_order : int
- Stores whether both min_values and max_values are ordered and if so, in which direction. This allows readers to perform binary searches in both lists. Readers cannot assume that max_values[i] <= min_values[i+1], even if the lists are ordered.
- $definition_level_histograms : null|array<string|int, int>
- Same as repetition_level_histograms except for definitions levels.
- $isValidate : mixed
- $max_values : array<string|int, string>
- $min_values : array<string|int, string>
- Two lists containing lower and upper bounds for the values of each page determined by the ColumnOrder of the column. These may be the actual minimum and maximum values found on a page, but can also be (more compact) values that do not exist on a page. For example, instead of storing "Blart Versenwald III", a writer may set min_values[i]="B", max_values[i]="C".
- $nan_counts : null|array<string|int, int>
- A list containing the number of NaN values for each page. Only present for columns of physical type FLOAT or DOUBLE, or logical type FLOAT16.
- $null_counts : null|array<string|int, int>
- A list containing the number of null values for each page
- $null_pages : array<string|int, bool>
- A list of Boolean values to determine the validity of the corresponding min and max values. If true, a page contains only null values, and writers have to set the corresponding entries in min_values and max_values to byte[0], so that all lists have the same length. If false, the corresponding entries in min_values and max_values must be valid.
- $repetition_level_histograms : null|array<string|int, int>
- Contains repetition level histograms for each page concatenated together. The repetition_level_histogram field on SizeStatistics contains more details.
Methods
- __construct() : mixed
- getName() : mixed
- read() : mixed
- write() : mixed
Properties
$_TSPEC
public
static mixed
$_TSPEC
= array(1 => array('var' => 'null_pages', 'isRequired' => true, 'type' => \Thrift\Type\TType::LST, 'etype' => \Thrift\Type\TType::BOOL, 'elem' => array('type' => \Thrift\Type\TType::BOOL)), 2 => array('var' => 'min_values', 'isRequired' => true, 'type' => \Thrift\Type\TType::LST, 'etype' => \Thrift\Type\TType::STRING, 'elem' => array('type' => \Thrift\Type\TType::STRING)), 3 => array('var' => 'max_values', 'isRequired' => true, 'type' => \Thrift\Type\TType::LST, 'etype' => \Thrift\Type\TType::STRING, 'elem' => array('type' => \Thrift\Type\TType::STRING)), 4 => array('var' => 'boundary_order', 'isRequired' => true, 'type' => \Thrift\Type\TType::I32, 'class' => '\Flow\Parquet\ThriftModel\BoundaryOrder'), 5 => array('var' => 'null_counts', 'isRequired' => false, 'type' => \Thrift\Type\TType::LST, 'etype' => \Thrift\Type\TType::I64, 'elem' => array('type' => \Thrift\Type\TType::I64)), 6 => array('var' => 'repetition_level_histograms', 'isRequired' => false, 'type' => \Thrift\Type\TType::LST, 'etype' => \Thrift\Type\TType::I64, 'elem' => array('type' => \Thrift\Type\TType::I64)), 7 => array('var' => 'definition_level_histograms', 'isRequired' => false, 'type' => \Thrift\Type\TType::LST, 'etype' => \Thrift\Type\TType::I64, 'elem' => array('type' => \Thrift\Type\TType::I64)), 8 => array('var' => 'nan_counts', 'isRequired' => false, 'type' => \Thrift\Type\TType::LST, 'etype' => \Thrift\Type\TType::I64, 'elem' => array('type' => \Thrift\Type\TType::I64)))
$boundary_order
Stores whether both min_values and max_values are ordered and if so, in which direction. This allows readers to perform binary searches in both lists. Readers cannot assume that max_values[i] <= min_values[i+1], even if the lists are ordered.
public
int
$boundary_order
= null
$definition_level_histograms
Same as repetition_level_histograms except for definitions levels.
public
null|array<string|int, int>
$definition_level_histograms
= null
$isValidate
public
static mixed
$isValidate
= false
$max_values
public
array<string|int, string>
$max_values
= null
$min_values
Two lists containing lower and upper bounds for the values of each page determined by the ColumnOrder of the column. These may be the actual minimum and maximum values found on a page, but can also be (more compact) values that do not exist on a page. For example, instead of storing "Blart Versenwald III", a writer may set min_values[i]="B", max_values[i]="C".
public
array<string|int, string>
$min_values
= null
Such more compact values must still be valid values within the column's logical type. Readers must make sure that list entries are populated before using them by inspecting null_pages.
For columns of physical type FLOAT or DOUBLE, or logical type FLOAT16, NaN values are not to be included in these bounds. If all non-null values of a page are NaN, then a writer must do the following:
- If the order of this column is TYPE_ORDER, then a column index must not be written for this column chunk. While this is unfortunate for performance, it is necessary to avoid conflict with legacy files that still included NaN in min_values and max_values even if the page had non-NaN values. To mitigate this, IEEE754_TOTAL_ORDER is recommended.
- If the order of this column is IEEE754_TOTAL_ORDER, then min_values[i] and max_values[i] of that page must be set to the smallest and largest NaN values as defined by IEEE 754 total order.
For columns of physical type INT96, the writer must do the following:
- If the order of this column is not INT96_TIMESTAMP_ORDER, then a column index must not be written for this column chunk.
- If the order of this column is INT96_TIMESTAMP_ORDER, the min_values[i] and max_values[i] of that page must be set to the smallest and largest values as defined by the INT96 chronological timestamp ordering.
$nan_counts
A list containing the number of NaN values for each page. Only present for columns of physical type FLOAT or DOUBLE, or logical type FLOAT16.
public
null|array<string|int, int>
$nan_counts
= null
If this field is not present, readers MUST assume that there might be NaN values in any page.
$null_counts
A list containing the number of null values for each page
public
null|array<string|int, int>
$null_counts
= null
Writers SHOULD always write this field even if no null values are present or the column is not nullable. Readers MUST distinguish between null_counts not being present and null_count being 0. If null_counts are not present, readers MUST NOT assume all null counts are 0.
$null_pages
A list of Boolean values to determine the validity of the corresponding min and max values. If true, a page contains only null values, and writers have to set the corresponding entries in min_values and max_values to byte[0], so that all lists have the same length. If false, the corresponding entries in min_values and max_values must be valid.
public
array<string|int, bool>
$null_pages
= null
$repetition_level_histograms
Contains repetition level histograms for each page concatenated together. The repetition_level_histogram field on SizeStatistics contains more details.
public
null|array<string|int, int>
$repetition_level_histograms
= null
When present the length should always be (number of pages * (max_repetition_level + 1)) elements.
Element 0 is the first element of the histogram for the first page. Element (max_repetition_level + 1) is the first element of the histogram for the second page.
Methods
__construct()
public
__construct([mixed $vals = null ]) : mixed
Parameters
- $vals : mixed = null
getName()
public
getName() : mixed
read()
public
read(mixed $input) : mixed
Parameters
- $input : mixed
write()
public
write(mixed $output) : mixed
Parameters
- $output : mixed