Content Class Referenceabstract

Abstract superclass of all array node types (flat hierarchy). Any Content can be nested within any other Content. More...

#include <Content.h>

Inheritance diagram for Content:
BitMaskedArray ByteMaskedArray EmptyArray IndexedArrayOf< T, ISOPTION > ListArrayOf< T > ListOffsetArrayOf< T > None NumpyArray Record RecordArray RegularArray UnionArrayOf< T, I > UnmaskedArray VirtualArray

Public Member Functions

 Content (const IdentitiesPtr &identities, const util::Parameters &parameters)
 Called by all subclass constructors; assigns identities and parameters upon construction. More...
 
virtual ~Content ()
 Empty destructor; required for some C++ reason. More...
 
virtual bool isscalar () const
 Returns true if the data represented by this node is scalar, not a true array. More...
 
virtual const std::string classname () const =0
 User-friendly name of this class, including integer-type specialization. More...
 
virtual const IdentitiesPtr identities () const
 Optional Identities for each element of the array (may be nullptr). More...
 
virtual void setidentities ()=0
 Assign a surrogate index of Identities to this array (in-place). More...
 
virtual void setidentities (const IdentitiesPtr &identities)=0
 Assign a specified set of Identities to this array (in-place). More...
 
virtual const TypePtr type (const util::TypeStrs &typestrs) const =0
 High-level Type describing this array. More...
 
virtual const FormPtr form (bool materialize) const =0
 Low-level Form describing all the features of this array except the actual data buffers (Index, Identities, NumpyArray::ptr, and RawArray::ptr). More...
 
virtual kernel::lib kernels () const =0
 Returns the kernel library enum for all nested ptr_lib within the array's tree structure. If different libraries are associated with nodes of the same tree, this returns kernel::lib::size to indicate that it is "mixed". More...
 
virtual void caches (std::vector< ArrayCachePtr > &out) const =0
 Accumulates all the unique #ArrayCache objects from nested #VirtualArray nodes. (Uniqueness is determined by pointer value.) More...
 
virtual const std::string tostring_part (const std::string &indent, const std::string &pre, const std::string &post) const =0
 Internal function to build an output string for tostring. More...
 
virtual void tojson_part (ToJson &builder, bool include_beginendlist) const =0
 Internal function to produce a JSON representation one node at a time. More...
 
virtual void nbytes_part (std::map< size_t, int64_t > &largest) const =0
 Internal function used to calculate nbytes. More...
 
virtual int64_t length () const =0
 The number of elements in the array. More...
 
virtual const ContentPtr shallow_copy () const =0
 Copies this node without copying any nodes hierarchically nested within it or any array/index/identity buffers. More...
 
virtual const ContentPtr deep_copy (bool copyarrays, bool copyindexes, bool copyidentities) const =0
 Copies this node and all nodes hierarchically nested within it, optionally copying the associated arrays, indexes, and identities, too. More...
 
virtual void check_for_iteration () const =0
 Performs up-front validity checks on an array so that they don't have to be checked in getitem_at_nowrap for each item. More...
 
virtual const ContentPtr getitem_nothing () const =0
 Internal function to get an empty slice (with the correct type). More...
 
virtual const ContentPtr getitem_at (int64_t at) const =0
 Returns the element at a given position in the array, handling negative indexing and bounds-checking like Python. More...
 
virtual const ContentPtr getitem_at_nowrap (int64_t at) const =0
 Returns the element at a given position in the array, without handling negative indexing or bounds-checking. More...
 
virtual const ContentPtr getitem_range (int64_t start, int64_t stop) const =0
 Subinterval of this array, handling negative indexing and bounds-checking like Python. More...
 
virtual const ContentPtr getitem_range_nowrap (int64_t start, int64_t stop) const =0
 Subinterval of this array, without handling negative indexing or bounds-checking. More...
 
virtual const ContentPtr getitem_field (const std::string &key) const =0
 This array with the first nested RecordArray replaced by the field at key. More...
 
virtual const ContentPtr getitem_field (const std::string &key, const Slice &only_fields) const =0
 
virtual const ContentPtr getitem_fields (const std::vector< std::string > &keys) const =0
 This array with the first nested RecordArray replaced by a RecordArray of a given subset of keys. More...
 
virtual const ContentPtr getitem_fields (const std::vector< std::string > &keys, const Slice &only_fields) const =0
 
virtual const ContentPtr getitem (const Slice &where) const
 Entry point for general slicing: Slice represents a tuple of SliceItem nodes applying to each level of nested lists. More...
 
virtual const ContentPtr getitem_next (const SliceItemPtr &head, const Slice &tail, const Index64 &advanced) const
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
virtual const ContentPtr getitem_next_jagged (const Index64 &slicestarts, const Index64 &slicestops, const SliceItemPtr &slicecontent, const Slice &tail) const
 Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis. More...
 
virtual const ContentPtr carry (const Index64 &carry, bool allow_lazy) const =0
 Returns an array of the same type with elements filtered, rearranged, and possibly duplicated by the carry array of integers. More...
 
virtual const std::string purelist_parameter (const std::string &key) const
 The parameter associated with key at the first level that has a non-null value, descending only as deep as the first RecordArray. More...
 
bool purelist_isregular () const
 Returns true if all nested lists down to the first RecordArray are RegularArray nodes; false otherwise. More...
 
virtual bool dimension_optiontype () const
 Returns true if this dimension has option-type; false otherwise. More...
 
virtual int64_t purelist_depth () const =0
 The list-depth of this array, not counting any contained within a RecordArray. More...
 
virtual const std::pair< int64_t, int64_t > minmax_depth () const =0
 Returns (a) the minimum list-depth and (b) the maximum list-depth of the array, which can differ if this array "branches" (differs when followed through different fields of a RecordArray or UnionArray). More...
 
virtual const std::pair< bool, int64_t > branch_depth () const =0
 Returns (a) whether the list-depth of this array "branches," or differs when followed through different fields of a RecordArray or UnionArray and (b) the minimum list-depth. More...
 
virtual int64_t numfields () const =0
 The number of fields in the first nested tuple or records or -1 if this array does not contain a RecordArray. More...
 
virtual int64_t fieldindex (const std::string &key) const =0
 The position of a tuple or record key name if this array contains a RecordArray. More...
 
virtual const std::string key (int64_t fieldindex) const =0
 The record name associated with a given field index or the tuple index as a string (e.g. "0", "1", "2") if a tuple. More...
 
virtual bool haskey (const std::string &key) const =0
 Returns true if the array contains a RecordArray with the specified key; false otherwise. More...
 
virtual const std::vector< std::string > keys () const =0
 A list of RecordArray keys or an empty list if this array does not contain a RecordArray. More...
 
virtual const std::string validityerror (const std::string &path) const =0
 Returns an error message if this array is invalid; otherwise, returns an empty string. More...
 
virtual const ContentPtr shallow_simplify () const =0
 Returns an equivalent array simplified at one level only using simplify_optiontype if an option-type array and simplify_uniontype if a union-type array. More...
 
virtual const ContentPtr num (int64_t axis, int64_t depth) const =0
 The length of this array (as a NumpyArray scalar) if axis = 0 or the lengths of subarrays (as an array or nested array) if axis != 0. More...
 
virtual const std::pair< Index64, ContentPtroffsets_and_flattened (int64_t axis, int64_t depth) const =0
 Returns (a) an offsets Index and (b) a flattened version of the array at some axis depth. More...
 
virtual bool mergeable (const ContentPtr &other, bool mergebool) const =0
 Returns true if this array can be merged with the other; false otherwise. More...
 
virtual bool referentially_equal (const ContentPtr &other) const =0
 Returns true if this array has all the same buffers and parameters as other; false otherwise. More...
 
virtual const std::pair< ContentPtrVec, ContentPtrVecmerging_strategy (const ContentPtrVec &others) const
 Partitions this array plus a list of others into a head sequence and a tail sequence: More...
 
virtual const ContentPtr reverse_merge (const ContentPtr &other) const
 Merges a single other with this array in reverse order: other first, this last. More...
 
const ContentPtr merge (const ContentPtr &other) const
 An array with this and the other concatenated (this first, other last). More...
 
virtual const ContentPtr mergemany (const ContentPtrVec &others) const =0
 Returns an array with this and the others concatenated (in order, this first, others last). More...
 
virtual const SliceItemPtr asslice () const =0
 Converts this array into a SliceItem that can be used in getitem. More...
 
virtual const ContentPtr fillna (const ContentPtr &value) const =0
 Returns this array with None values replaced by a given value. More...
 
virtual const ContentPtr rpad (int64_t target, int64_t axis, int64_t depth) const =0
 If axis = 0, returns a view of this array padded on the right with None values to have a minimum length; otherwise, returns an array with nested lists all padded to the minimum length. More...
 
virtual const ContentPtr rpad_and_clip (int64_t target, int64_t axis, int64_t depth) const =0
 If axis = 0, returns a view of this array padded on the right. More...
 
virtual const ContentPtr reduce_next (const Reducer &reducer, int64_t negaxis, const Index64 &starts, const Index64 &shifts, const Index64 &parents, int64_t outlength, bool mask, bool keepdims) const =0
 This array with one axis removed by applying a Reducer (e.g. "sum", "max", "any", "all). More...
 
virtual const ContentPtr sort_next (int64_t negaxis, const Index64 &starts, const Index64 &parents, int64_t outlength, bool ascending, bool stable) const =0
 This array sorted. More...
 
virtual const ContentPtr argsort_next (int64_t negaxis, const Index64 &starts, const Index64 &shifts, const Index64 &parents, int64_t outlength, bool ascending, bool stable) const =0
 This array sorted indices. More...
 
virtual const ContentPtr localindex (int64_t axis, int64_t depth) const =0
 A (possibly nested) array of integers indicating the positions of elements within each nested list. More...
 
virtual const ContentPtr combinations (int64_t n, bool replacement, const util::RecordLookupPtr &recordlookup, const util::Parameters &parameters, int64_t axis, int64_t depth) const =0
 Tuples or records of all n-tuple combinations of list items at some axis depth. More...
 
const std::string tostring () const
 Returns a string representation of this array (multi-line XML). More...
 
const std::string tojson (bool pretty, int64_t maxdecimals, const char *nan_string=nullptr, const char *infinity_string=nullptr, const char *minus_infinity_string=nullptr, const char *complex_real_string=nullptr, const char *complex_imag_string=nullptr) const
 Returns a JSON representation of this array. More...
 
void tojson (FILE *destination, bool pretty, int64_t maxdecimals, int64_t buffersize, const char *nan_string=nullptr, const char *infinity_string=nullptr, const char *minus_infinity_string=nullptr, const char *complex_real_string=nullptr, const char *complex_imag_string=nullptr) const
 Writes a JSON representation of this array to a destination file. More...
 
int64_t nbytes () const
 The number of bytes contained in all array buffers, Index buffers, and Identities buffers, not including the lightweight node objects themselves. More...
 
const ContentPtr reduce (const Reducer &reducer, int64_t axis, bool mask, bool keepdims) const
 This array with one axis removed by applying a Reducer (e.g. "sum", "max", "any", "all). More...
 
const ContentPtr sort (int64_t axis, bool ascending, bool stable) const
 This array with one axis sorted by applying a sorting algorithm. More...
 
const ContentPtr argsort (int64_t axis, bool ascending, bool stable) const
 This array indices with one axis sorted by applying a sorting algorithm. More...
 
const util::Parameters parameters () const
 String-to-JSON map that augments the meaning of this array. More...
 
void setparameters (const util::Parameters &parameters)
 Assign all the parameters for this array node (in-place). More...
 
const std::string parameter (const std::string &key) const
 Get one parameter from this array node. More...
 
void setparameter (const std::string &key, const std::string &value)
 Assign one parameter for this array node (in-place). More...
 
bool parameter_equals (const std::string &key, const std::string &value) const
 Returns true if the parameter associated with key exists and is equal to value; false otherwise. More...
 
bool parameters_equal (const util::Parameters &other, bool check_all) const
 Returns true if all parameters of this array node are equal to the other parameters. More...
 
bool parameter_isstring (const std::string &key) const
 Returns true if the parameter associated with key is a string; false otherwise. More...
 
bool parameter_isname (const std::string &key) const
 Returns true if the parameter associated with key is a string that matches [A-Za-z_][A-Za-z_0-9]*; false otherwise. More...
 
const std::string parameter_asstring (const std::string &key) const
 Returns the parameter associated with key as a string if parameter_isstring; raises an error otherwise. More...
 
const ContentPtr merge_as_union (const ContentPtr &other) const
 Concatenates this array with other by creating a UnionArray instead of actually merging the data. This does not call UnionArray::simplify_uniontype. More...
 
const ContentPtr rpad_axis0 (int64_t target, bool clip) const
 Internal function to handle the axis = 0 case of rpad and rpad_and_clip. More...
 
const ContentPtr localindex_axis0 () const
 Internal function to handle the axis = 0 case of localindex. More...
 
const ContentPtr combinations_axis0 (int64_t n, bool replacement, const util::RecordLookupPtr &recordlookup, const util::Parameters &parameters) const
 Internal function to handle the axis = 0 case of combinations. More...
 
virtual const ContentPtr getitem_next (const SliceAt &at, const Slice &tail, const Index64 &advanced) const =0
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
virtual const ContentPtr getitem_next (const SliceRange &range, const Slice &tail, const Index64 &advanced) const =0
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
virtual const ContentPtr getitem_next (const SliceEllipsis &ellipsis, const Slice &tail, const Index64 &advanced) const
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
virtual const ContentPtr getitem_next (const SliceNewAxis &newaxis, const Slice &tail, const Index64 &advanced) const
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
virtual const ContentPtr getitem_next (const SliceArray64 &array, const Slice &tail, const Index64 &advanced) const =0
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
virtual const ContentPtr getitem_next (const SliceField &field, const Slice &tail, const Index64 &advanced) const
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
virtual const ContentPtr getitem_next (const SliceFields &fields, const Slice &tail, const Index64 &advanced) const
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
virtual const ContentPtr getitem_next (const SliceMissing64 &missing, const Slice &tail, const Index64 &advanced) const
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
virtual const ContentPtr getitem_next (const SliceJagged64 &jagged, const Slice &tail, const Index64 &advanced) const =0
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
virtual const ContentPtr getitem_next_jagged (const Index64 &slicestarts, const Index64 &slicestops, const SliceArray64 &slicecontent, const Slice &tail) const =0
 Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis. More...
 
virtual const ContentPtr getitem_next_jagged (const Index64 &slicestarts, const Index64 &slicestops, const SliceMissing64 &slicecontent, const Slice &tail) const =0
 Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis. More...
 
virtual const ContentPtr getitem_next_jagged (const Index64 &slicestarts, const Index64 &slicestops, const SliceJagged64 &slicecontent, const Slice &tail) const =0
 Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis. More...
 
const int64_t axis_wrap_if_negative (int64_t axis) const
 Internal function defining the negative axis handling for many operations. More...
 
virtual const ContentPtr copy_to (kernel::lib ptr_lib) const =0
 Recursively copies components of the array from main memory to a GPU (if ptr_lib == kernel::lib::cuda) or to main memory (if ptr_lib == kernel::lib::cpu) if those components are not already there. More...
 
virtual const ContentPtr numbers_to_type (const std::string &name) const =0
 Change the leaf types to 'totype'. More...
 
virtual bool is_unique () const =0
 Returns 'true' if all components of the array are unique. More...
 
virtual const ContentPtr unique () const =0
 Returns sorted unique values. More...
 
virtual bool is_subrange_equal (const Index64 &start, const Index64 &stop) const =0
 Returns 'true' if subranges are equal. More...
 

Protected Member Functions

const std::string validityerror_parameters (const std::string &path) const
 Internal function to determine if a particular set of parameters are allowed on a particular node (centralized by parameter, not by class). More...
 
const ContentPtr getitem_next_array_wrap (const ContentPtr &outcontent, const std::vector< int64_t > &shape) const
 Internal function to wrap putative getitem output with enough RegularArray nodes to satisfy a given shape. More...
 
const std::string parameters_tostring (const std::string &indent, const std::string &pre, const std::string &post) const
 Internal function to convert parameters into a string fragment for tostring. More...
 
kernel::lib kernels_compare (kernel::lib from_index, const ContentPtr &content) const
 Internal function to support most of the logic in kernels. More...
 

Protected Attributes

IdentitiesPtr identities_
 See identities. More...
 
util::Parameters parameters_
 See parameters. More...
 

Detailed Description

Abstract superclass of all array node types (flat hierarchy). Any Content can be nested within any other Content.

Constructor & Destructor Documentation

◆ Content()

Content ( const IdentitiesPtr identities,
const util::Parameters parameters 
)

Called by all subclass constructors; assigns identities and parameters upon construction.

◆ ~Content()

virtual ~Content ( )
inlinevirtual

Empty destructor; required for some C++ reason.

Member Function Documentation

◆ argsort()

const ContentPtr argsort ( int64_t  axis,
bool  ascending,
bool  stable 
) const

This array indices with one axis sorted by applying a sorting algorithm.

This operation is implemented on each node through argsort_next.

Parameters
axisThe axis to sort. Negative axis counts backward from the deepest levels (-1 is the last valid axis).
ascendingIf true, the values will be sorted in an ascending order.
stableIf true, the values will be sorted by a stable sort algorithm to maintain the relative order of records with equal keys (i.e. values).

◆ argsort_next()

virtual const ContentPtr argsort_next ( int64_t  negaxis,
const Index64 starts,
const Index64 shifts,
const Index64 parents,
int64_t  outlength,
bool  ascending,
bool  stable 
) const
pure virtual

This array sorted indices.

The user's entry point for this operation is argsort.

Parameters
negaxisThe negative axis: -axis. That is, negaxis = 1 means the deepest axis level.
startsStaring positions of each group to combine as an Index. These are downward pointers from an outer structure into this structure with the same meaning as in ListArray.
shiftsPer-element adjustments that allows for variable-length lists with axis != -1 and for missing values (None).
parentsGroups to combine as an Index of upward pointers from this structure to the outer structure to sort.
outlengthThe length of the array, after the operation completes.
ascendingIf true, the values will be sorted in an ascending order.
stableIf true, the values will be sorted by a stable sort algorithm to maintain the relative order of records with equal keys (i.e. values).

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ asslice()

virtual const SliceItemPtr asslice ( ) const
pure virtual

◆ axis_wrap_if_negative()

const int64_t axis_wrap_if_negative ( int64_t  axis) const

Internal function defining the negative axis handling for many operations.

Returns a non-negative equivalent axis if unambiguous and passes the negative axis through if ambiguous. A negative axis can be ambiguous if the list-depth "branches" in a RecordArray or a UnionArray with different contents having different depths. As an operation descends through the nodes of an array, it repeatedly calls axis_wrap_if_negative until an unambiguous non-negative axis can be identified.

This allows a RecordArray or UnionArray with different depths to accept axis = -1 as the last axis, regardless of how deep that is in different record fields or union possibilities.

◆ branch_depth()

virtual const std::pair<bool, int64_t> branch_depth ( ) const
pure virtual

Returns (a) whether the list-depth of this array "branches," or differs when followed through different fields of a RecordArray or UnionArray and (b) the minimum list-depth.

If the array does not contain any records or heterogeneous data, the first element is always true and the second is simply the depth.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ caches()

virtual void caches ( std::vector< ArrayCachePtr > &  out) const
pure virtual

Accumulates all the unique #ArrayCache objects from nested #VirtualArray nodes. (Uniqueness is determined by pointer value.)

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ carry()

virtual const ContentPtr carry ( const Index64 carry,
bool  allow_lazy 
) const
pure virtual

Returns an array of the same type with elements filtered, rearranged, and possibly duplicated by the carry array of integers.

The output has the same length as the carry index, not the array that is being manipulated. For each item i in carry, the output is array[index[i]].

This operation is called take in NumPy and Arrow, although this carry is a low-level function that does not handle negative indexes and is not exposed to the Python layer. It is used by many operations to pass filters/rearrangements/duplications from one typed array node to another without knowing the latter's type.

Taking getitem_at_nowrap as a function from integers to the array's item type, A: [0, len(a)) → T, and the carry array's Index64::getitem_at_nowrap as a function C: [0, len(c)) → [0, len(a)), this method represents function composition, A ∘ C: [0, len(c)) → T.

Note
If the carry array has duplicate elements, the array is only duplicated one level deep. For instance, on a ListArray, only the starts and stops get duplicated, not the content (and similarly for all other node types).

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ check_for_iteration()

virtual void check_for_iteration ( ) const
pure virtual

Performs up-front validity checks on an array so that they don't have to be checked in getitem_at_nowrap for each item.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ classname()

virtual const std::string classname ( ) const
pure virtual

◆ combinations()

virtual const ContentPtr combinations ( int64_t  n,
bool  replacement,
const util::RecordLookupPtr recordlookup,
const util::Parameters parameters,
int64_t  axis,
int64_t  depth 
) const
pure virtual

Tuples or records of all n-tuple combinations of list items at some axis depth.

For example, the n = 2 combinations at axis = 0 of

[a, b, c, d, e]

would be

[[(a, b), (a, c), (a, d), (a, e)],
[(b, c), (b, d), (b, e)],
[(c, d), (c, e)],
[(d, e)]]

and the n = 3 combinations at axis = 1 of

[[a, b, c, d], [], [e, f], [g, h, i]]

would be

[[[(a, b, c), (a, b, d), (a, c, d)], [(b, c, d)]],
[],
[],
[[(g, h, i)]]]
Parameters
nThe number of items in each tuple/record.
replacementIf true, the tuples/records are allowed to include the same item more than once, such as (a, a, a) and (a, a, b). In the above examples, replacement = false. (If the output of this function is thought of as the "upper triangle" elements of the Cartesian product of the input with itself n times, the replacement parameter determines whether elements on the diagonal are allowed.)
recordlookupIf nullptr, the output consists of tuples, a RecordArray indexed by "0", "1", "2", etc. If recordlookup is a std::vector<std::string>, the output consists of records, a RecordArray indexed by names (strings). The length of recordlookup must be equal to n.
parametersParameters assigned to the new RecordArray. This can be used to set "__record_" = "\"record_name\"" to give the records a custom behavior in Python.
axisThe nesting depth at which this operation is applied. At axis = 0, combinations are formed from elements of the whole array (see example above); at axis = 1, combinations are formed independently for each element. At a deeper axis, outer levels are left untouched. Negative axis counts backward from the deepest levels (-1 is the last valid axis).
depthThe current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis).

Note that axis = 0 is qualitatively different from any other axis because a dataset is typically much larger than any one of its elements. As such, axis = 0 is lazily generated with an IndexedArray, while any other axis is eagerly generated by duplicating elements (with carry).

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ combinations_axis0()

const ContentPtr combinations_axis0 ( int64_t  n,
bool  replacement,
const util::RecordLookupPtr recordlookup,
const util::Parameters parameters 
) const

Internal function to handle the axis = 0 case of combinations.

The axis = 0 case does not depend on array node type, so it is defined universally in the Content class.

Parameters
nThe number of items in each tuple/record.
replacementIf true, the tuples/records are allowed to include the same item more than once, such as (a, a, a) and (a, a, b). In the above examples, replacement = false. (If the output of this function is thought of as the "upper triangle" elements of the Cartesian product of the input with itself n times, the replacement parameter determines whether elements on the diagonal are allowed.)
recordlookupIf nullptr, the output consists of tuples, a RecordArray indexed by "0", "1", "2", etc. If recordlookup is a std::vector<std::string>, the output consists of records, a RecordArray indexed by names (strings). The length of recordlookup must be equal to n.
parametersParameters assigned to the new RecordArray. This can be used to set "__record_" = "\"record_name\"" to give the records a custom behavior in Python.

◆ copy_to()

virtual const ContentPtr copy_to ( kernel::lib  ptr_lib) const
pure virtual

Recursively copies components of the array from main memory to a GPU (if ptr_lib == kernel::lib::cuda) or to main memory (if ptr_lib == kernel::lib::cpu) if those components are not already there.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, RecordArray, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, ByteMaskedArray, VirtualArray, RegularArray, UnmaskedArray, Record, EmptyArray, and None.

◆ deep_copy()

virtual const ContentPtr deep_copy ( bool  copyarrays,
bool  copyindexes,
bool  copyidentities 
) const
pure virtual

Copies this node and all nodes hierarchically nested within it, optionally copying the associated arrays, indexes, and identities, too.

See also shallow_copy.

Parameters
copyarraysIf true, copy the associated array buffers (in NumpyArray and RawArray), not just the lightweight objects that point to them.
copyindexesIf true, copy the Index objects and their buffers as well.
copyidentitiesIf true, copy the Identities objects and their buffers as well.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ dimension_optiontype()

virtual bool dimension_optiontype ( ) const
virtual

Returns true if this dimension has option-type; false otherwise.

Reimplemented in Record.

◆ fieldindex()

virtual int64_t fieldindex ( const std::string &  key) const
pure virtual

◆ fillna()

virtual const ContentPtr fillna ( const ContentPtr value) const
pure virtual

Returns this array with None values replaced by a given value.

Parameters
valueAn array of exactly one element, which need not have the same type as the missing values it's replacing.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ form()

virtual const FormPtr form ( bool  materialize) const
pure virtual

Low-level Form describing all the features of this array except the actual data buffers (Index, Identities, NumpyArray::ptr, and RawArray::ptr).

Parameters
materializeIf true, materialize any VirtualArrays encountered along the way.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ getitem()

virtual const ContentPtr getitem ( const Slice where) const
virtual

Entry point for general slicing: Slice represents a tuple of SliceItem nodes applying to each level of nested lists.

Reimplemented in NumpyArray, VirtualArray, and Record.

◆ getitem_at()

virtual const ContentPtr getitem_at ( int64_t  at) const
pure virtual

Returns the element at a given position in the array, handling negative indexing and bounds-checking like Python.

The first item in the array is at 0, the second at 1, the last at -1, the penultimate at -2, etc.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ getitem_at_nowrap()

virtual const ContentPtr getitem_at_nowrap ( int64_t  at) const
pure virtual

Returns the element at a given position in the array, without handling negative indexing or bounds-checking.

If the array has Identities, the identity bounds are checked.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ getitem_field() [1/2]

virtual const ContentPtr getitem_field ( const std::string &  key) const
pure virtual

◆ getitem_field() [2/2]

virtual const ContentPtr getitem_field ( const std::string &  key,
const Slice only_fields 
) const
pure virtual

◆ getitem_fields() [1/2]

virtual const ContentPtr getitem_fields ( const std::vector< std::string > &  keys) const
pure virtual

◆ getitem_fields() [2/2]

virtual const ContentPtr getitem_fields ( const std::vector< std::string > &  keys,
const Slice only_fields 
) const
pure virtual

◆ getitem_next() [1/10]

virtual const ContentPtr getitem_next ( const SliceItemPtr head,
const Slice tail,
const Index64 advanced 
) const
virtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

Parameters
headFirst element of the Slice tuple.
tailThe rest of the elements of the Slice tuple.
advancedIf empty, no array slices (integer or boolean) have been encountered yet; otherwise, positions in any subsequent array slices to select.

In the NumPy documentation, advanced indexes are described as iterating "as one," which requires an Index to be propagated when implemented recursively.

Reimplemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, UnionArrayOf< T, I >, RecordArray, ByteMaskedArray, VirtualArray, and UnmaskedArray.

◆ getitem_next() [2/10]

virtual const ContentPtr getitem_next ( const SliceAt at,
const Slice tail,
const Index64 advanced 
) const
pure virtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

See generic getitem_next for details.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, RecordArray, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, ByteMaskedArray, VirtualArray, RegularArray, UnmaskedArray, EmptyArray, Record, and None.

◆ getitem_next() [3/10]

virtual const ContentPtr getitem_next ( const SliceRange range,
const Slice tail,
const Index64 advanced 
) const
pure virtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

See generic getitem_next for details.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, RecordArray, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, ByteMaskedArray, VirtualArray, RegularArray, UnmaskedArray, EmptyArray, Record, and None.

◆ getitem_next() [4/10]

virtual const ContentPtr getitem_next ( const SliceEllipsis ellipsis,
const Slice tail,
const Index64 advanced 
) const
virtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

See generic getitem_next for details.

◆ getitem_next() [5/10]

virtual const ContentPtr getitem_next ( const SliceNewAxis newaxis,
const Slice tail,
const Index64 advanced 
) const
virtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

See generic getitem_next for details.

◆ getitem_next() [6/10]

virtual const ContentPtr getitem_next ( const SliceArray64 array,
const Slice tail,
const Index64 advanced 
) const
pure virtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

See generic getitem_next for details.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, RecordArray, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, ByteMaskedArray, VirtualArray, RegularArray, UnmaskedArray, EmptyArray, Record, and None.

◆ getitem_next() [7/10]

virtual const ContentPtr getitem_next ( const SliceField field,
const Slice tail,
const Index64 advanced 
) const
virtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

See generic getitem_next for details.

Reimplemented in NumpyArray, RecordArray, VirtualArray, EmptyArray, Record, and None.

◆ getitem_next() [8/10]

virtual const ContentPtr getitem_next ( const SliceFields fields,
const Slice tail,
const Index64 advanced 
) const
virtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

See generic getitem_next for details.

Reimplemented in NumpyArray, RecordArray, VirtualArray, Record, EmptyArray, and None.

◆ getitem_next() [9/10]

virtual const ContentPtr getitem_next ( const SliceMissing64 missing,
const Slice tail,
const Index64 advanced 
) const
virtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

See generic getitem_next for details.

◆ getitem_next() [10/10]

virtual const ContentPtr getitem_next ( const SliceJagged64 jagged,
const Slice tail,
const Index64 advanced 
) const
pure virtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

See generic getitem_next for details.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, RecordArray, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, ByteMaskedArray, VirtualArray, RegularArray, UnmaskedArray, Record, EmptyArray, and None.

◆ getitem_next_array_wrap()

const ContentPtr getitem_next_array_wrap ( const ContentPtr outcontent,
const std::vector< int64_t > &  shape 
) const
protected

Internal function to wrap putative getitem output with enough RegularArray nodes to satisfy a given shape.

The shape is intended to match a SliceArray::shape.

◆ getitem_next_jagged() [1/4]

virtual const ContentPtr getitem_next_jagged ( const Index64 slicestarts,
const Index64 slicestops,
const SliceItemPtr slicecontent,
const Slice tail 
) const
virtual

Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis.

Parameters
slicestartsEffective starts (similar to ListArray's starts) of the jagged slice.
slicestopsEffective stops of the jagged slice.
slicecontentNested content within the jagged slice.
tailSubsequent SliceItem elements beyond the jagged array hierarchy.

Reimplemented in ListOffsetArrayOf< T >, and VirtualArray.

◆ getitem_next_jagged() [2/4]

virtual const ContentPtr getitem_next_jagged ( const Index64 slicestarts,
const Index64 slicestops,
const SliceArray64 slicecontent,
const Slice tail 
) const
pure virtual

Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis.

See generic getitem_next_jagged for details.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, RecordArray, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, ByteMaskedArray, VirtualArray, RegularArray, UnmaskedArray, Record, EmptyArray, and None.

◆ getitem_next_jagged() [3/4]

virtual const ContentPtr getitem_next_jagged ( const Index64 slicestarts,
const Index64 slicestops,
const SliceMissing64 slicecontent,
const Slice tail 
) const
pure virtual

Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis.

See generic getitem_next_jagged for details.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, RecordArray, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, ByteMaskedArray, VirtualArray, RegularArray, UnmaskedArray, Record, EmptyArray, and None.

◆ getitem_next_jagged() [4/4]

virtual const ContentPtr getitem_next_jagged ( const Index64 slicestarts,
const Index64 slicestops,
const SliceJagged64 slicecontent,
const Slice tail 
) const
pure virtual

Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis.

See generic getitem_next_jagged for details.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, RecordArray, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, ByteMaskedArray, VirtualArray, RegularArray, UnmaskedArray, Record, EmptyArray, and None.

◆ getitem_nothing()

virtual const ContentPtr getitem_nothing ( ) const
pure virtual

◆ getitem_range()

virtual const ContentPtr getitem_range ( int64_t  start,
int64_t  stop 
) const
pure virtual

Subinterval of this array, handling negative indexing and bounds-checking like Python.

The first item in the array is at 0, the second at 1, the last at -1, the penultimate at -2, etc.

Ranges beyond the array are not an error; they are trimmed to start = 0 on the left and stop = length() - 1 on the right.

This operation only affects the node metadata; its calculation time does not scale with the size of the array.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ getitem_range_nowrap()

virtual const ContentPtr getitem_range_nowrap ( int64_t  start,
int64_t  stop 
) const
pure virtual

Subinterval of this array, without handling negative indexing or bounds-checking.

If the array has Identities, the identity bounds are checked.

This operation only affects the node metadata; its calculation time does not scale with the size of the array.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ haskey()

virtual bool haskey ( const std::string &  key) const
pure virtual

◆ identities()

virtual const IdentitiesPtr identities ( ) const
virtual

Optional Identities for each element of the array (may be nullptr).

Reimplemented in Record.

◆ is_subrange_equal()

virtual bool is_subrange_equal ( const Index64 start,
const Index64 stop 
) const
pure virtual

◆ is_unique()

virtual bool is_unique ( ) const
pure virtual

◆ isscalar()

virtual bool isscalar ( ) const
virtual

Returns true if the data represented by this node is scalar, not a true array.

Scalars are not valid inputs to many operations, but they are instances of Content so that they can be outputs of some operations.

Formally, scalars and arrays should probably be distinct C++ types to eliminate this ambiguity, but that would introduce excessive type-testing and casting in all the algorithms that chain operations (output of one is input to another). It is more convenient to test for scalars at runtime with this method.

The only scalars are:

  • NumpyArray with shape.empty() (zero-dimensional array), which represents scalar numbers and booleans. (Immediately converted to a number object in Python.)
  • None, which represents missing values. (Immediately converted to None in Python.)
  • Record, which represents one item from a RecordArray.

Reimplemented in NumpyArray, Record, and None.

◆ kernels()

virtual kernel::lib kernels ( ) const
pure virtual

Returns the kernel library enum for all nested ptr_lib within the array's tree structure. If different libraries are associated with nodes of the same tree, this returns kernel::lib::size to indicate that it is "mixed".

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ kernels_compare()

kernel::lib kernels_compare ( kernel::lib  from_index,
const ContentPtr content 
) const
protected

Internal function to support most of the logic in kernels.

◆ key()

virtual const std::string key ( int64_t  fieldindex) const
pure virtual

The record name associated with a given field index or the tuple index as a string (e.g. "0", "1", "2") if a tuple.

Raises an error if the array does not contain a RecordArray.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ keys()

virtual const std::vector<std::string> keys ( ) const
pure virtual

◆ length()

virtual int64_t length ( ) const
pure virtual

◆ localindex()

virtual const ContentPtr localindex ( int64_t  axis,
int64_t  depth 
) const
pure virtual

A (possibly nested) array of integers indicating the positions of elements within each nested list.

Parameters
axisThe nesting depth at which this operation is applied. If axis = 0, the output is simply an array of integers from 0 to length(). If axis = 1, the output has one level of nesting containing integers from 0 to the length of the nested list. Higher values of axis leave outer layers of the structure untouched. Negative axis counts backward from the deepest levels (-1 is the last valid axis).
depthThe current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis).

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ localindex_axis0()

const ContentPtr localindex_axis0 ( ) const

Internal function to handle the axis = 0 case of localindex.

The axis = 0 case does not depend on array node type, so it is defined universally in the Content class.

◆ merge()

const ContentPtr merge ( const ContentPtr other) const

An array with this and the other concatenated (this first, other last).

◆ merge_as_union()

const ContentPtr merge_as_union ( const ContentPtr other) const

Concatenates this array with other by creating a UnionArray instead of actually merging the data. This does not call UnionArray::simplify_uniontype.

◆ mergeable()

virtual bool mergeable ( const ContentPtr other,
bool  mergebool 
) const
pure virtual

Returns true if this array can be merged with the other; false otherwise.

The merge method will complete without errors if this function returns true.

Parameters
otherThe other array to merge with.
mergeboolIf true, consider boolean types to be equivalent to integers.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ mergemany()

virtual const ContentPtr mergemany ( const ContentPtrVec others) const
pure virtual

Returns an array with this and the others concatenated (in order, this first, others last).

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ merging_strategy()

virtual const std::pair<ContentPtrVec, ContentPtrVec> merging_strategy ( const ContentPtrVec others) const
virtual

Partitions this array plus a list of others into a head sequence and a tail sequence:

[this] + others == head + tail

The order is preserved and no arrays are lost. The others must have at least one array, and this will be added to head, though tail might be empty.

(If tail is empty, head must contain at least two arrays!)

The position of the split between head and tail is such that the first array in tail must be merged with reverse_merge (usually because it is an option-type or union-type array, though this rule depends on whether this is option-type, union-type, or neither).

The purpose of this function is to prepare a head of arrays that can all be merged "normally," followed by a tail whose first array must be "reverse merged" and subsequent arrays recurse.

This is the first step in merge.

Reimplemented in IndexedArrayOf< T, ISOPTION >, and UnionArrayOf< T, I >.

◆ minmax_depth()

virtual const std::pair<int64_t, int64_t> minmax_depth ( ) const
pure virtual

Returns (a) the minimum list-depth and (b) the maximum list-depth of the array, which can differ if this array "branches" (differs when followed through different fields of a RecordArray or UnionArray).

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ nbytes()

int64_t nbytes ( ) const

The number of bytes contained in all array buffers, Index buffers, and Identities buffers, not including the lightweight node objects themselves.

◆ nbytes_part()

virtual void nbytes_part ( std::map< size_t, int64_t > &  largest) const
pure virtual

Internal function used to calculate nbytes.

Parameters
largestThe largest range of bytes used in each reference-counted pointer (size_t).
Note
This method of accounting for overlapping buffers is insufficient: two nodes could use parts of the same buffer in which one doesn't completely overlap the other. It's not likely, but currently the calculation of nbytes is an underestimate.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ num()

virtual const ContentPtr num ( int64_t  axis,
int64_t  depth 
) const
pure virtual

The length of this array (as a NumpyArray scalar) if axis = 0 or the lengths of subarrays (as an array or nested array) if axis != 0.

Parameters
axisThe axis whose length or lengths to quantify. Negative axis counts backward from the deepest levels (-1 is the last valid axis).
depthThe current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis).

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ numbers_to_type()

virtual const ContentPtr numbers_to_type ( const std::string &  name) const
pure virtual

◆ numfields()

virtual int64_t numfields ( ) const
pure virtual

The number of fields in the first nested tuple or records or -1 if this array does not contain a RecordArray.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ offsets_and_flattened()

virtual const std::pair<Index64, ContentPtr> offsets_and_flattened ( int64_t  axis,
int64_t  depth 
) const
pure virtual

Returns (a) an offsets Index and (b) a flattened version of the array at some axis depth.

If axis > 1 (or its negative equivalent), the offsets is empty.

Parameters
axisThe axis to eliminate by flattening. axis = 0 is invalid. Negative axis counts backward from the deepest levels (-1 is the last valid axis).
depthThe current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis).

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ parameter()

const std::string parameter ( const std::string &  key) const

Get one parameter from this array node.

If the key does not exist, this function returns "null".

Keys are simple strings, but values are JSON-encoded strings. For this reason, values that represent single strings are double-quoted: e.g. "\"actual_value"".

◆ parameter_asstring()

const std::string parameter_asstring ( const std::string &  key) const

Returns the parameter associated with key as a string if parameter_isstring; raises an error otherwise.

◆ parameter_equals()

bool parameter_equals ( const std::string &  key,
const std::string &  value 
) const

Returns true if the parameter associated with key exists and is equal to value; false otherwise.

Keys are simple strings, but values are JSON-encoded strings. For this reason, values that represent single strings are double-quoted: e.g. "\"actual_value"".

Equality is checked at the level of JSON DOMs. The value does not need to be exactly the same string; it needs to have equivalent JSON value.

◆ parameter_isname()

bool parameter_isname ( const std::string &  key) const

Returns true if the parameter associated with key is a string that matches [A-Za-z_][A-Za-z_0-9]*; false otherwise.

◆ parameter_isstring()

bool parameter_isstring ( const std::string &  key) const

Returns true if the parameter associated with key is a string; false otherwise.

◆ parameters()

const util::Parameters parameters ( ) const

String-to-JSON map that augments the meaning of this array.

Keys are simple strings, but values are JSON-encoded strings. For this reason, values that represent single strings are double-quoted: e.g. "\"actual_value"".

◆ parameters_equal()

bool parameters_equal ( const util::Parameters other,
bool  check_all 
) const

Returns true if all parameters of this array node are equal to the other parameters.

Keys are simple strings, but values are JSON-encoded strings. For this reason, values that represent single strings are double-quoted: e.g. "\"actual_value"".

Equality is checked at the level of JSON DOMs. The value does not need to be exactly the same string; it needs to have equivalent JSON value.

If check_all, every parameter is checked; otherwise, only "__array__" and "__record__" are checked.

◆ parameters_tostring()

const std::string parameters_tostring ( const std::string &  indent,
const std::string &  pre,
const std::string &  post 
) const
protected

Internal function to convert parameters into a string fragment for tostring.

Parameters
indentIndentation depth as a string of spaces.
prePrefix string, usually an opening XML tag.
postPostfix string, usually a closing XML tag and carriage return.

◆ purelist_depth()

virtual int64_t purelist_depth ( ) const
pure virtual

The list-depth of this array, not counting any contained within a RecordArray.

The purelist_depth of a Record is 0, and a RecordArray is 1 (regardless of what its fields contain).

If this array contains a UnionArray with different depths, the return value is -1.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ purelist_isregular()

bool purelist_isregular ( ) const

Returns true if all nested lists down to the first RecordArray are RegularArray nodes; false otherwise.

◆ purelist_parameter()

virtual const std::string purelist_parameter ( const std::string &  key) const
virtual

The parameter associated with key at the first level that has a non-null value, descending only as deep as the first RecordArray.

◆ reduce()

const ContentPtr reduce ( const Reducer reducer,
int64_t  axis,
bool  mask,
bool  keepdims 
) const

This array with one axis removed by applying a Reducer (e.g. "sum", "max", "any", "all).

This operation is implemented on each node through reduce_next.

Parameters
reducerThe choice of Reducer algorithm.
axisThe axis to remove by reduction. Negative axis counts backward from the deepest levels (-1 is the last valid axis).
maskIf true, the Reducer's identity values will be covered by None using a ByteMaskedArray. This is desirable for ReducerMin, ReducerMax, ReducerArgmin, and ReducerArgmax to indicate that empty lists have no minimum or maximum.
keepdimsIf true, the reduced values will be wrapped by a singleton RegularArray to maintain the same number of dimensions in the output.

◆ reduce_next()

virtual const ContentPtr reduce_next ( const Reducer reducer,
int64_t  negaxis,
const Index64 starts,
const Index64 shifts,
const Index64 parents,
int64_t  outlength,
bool  mask,
bool  keepdims 
) const
pure virtual

This array with one axis removed by applying a Reducer (e.g. "sum", "max", "any", "all).

The user's entry point for this operation is reduce.

Parameters
reducerThe choice of Reducer algorithm.
negaxisThe negative axis: -axis. That is, negaxis = 1 means the deepest axis level.
startsStaring positions of each group to combine as an Index. These are downward pointers from an outer structure into this structure with the same meaning as in ListArray. Only used by a reducer that returns_positions (currently only argmin and argmax).
shiftsPer-element adjustments for any reducer that returns_positions. Allows for variable-length lists with axis != -1 and for missing values (None) in argmin and argmax.
parentsGroups to combine as an Index of upward pointers from this structure to the outer structure to reduce.
outlengthThe length of the reduced array, after the operation completes.
maskIf true, the Reducer's identity values will be covered by None using a ByteMaskedArray. This is desirable for ReducerMin, ReducerMax, ReducerArgmin, and ReducerArgmax to indicate that empty lists have no minimum or maximum.
keepdimsIf true, the reduced values will be wrapped by a singleton RegularArray to maintain the same number of dimensions in the output.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ referentially_equal()

virtual bool referentially_equal ( const ContentPtr other) const
pure virtual

Returns true if this array has all the same buffers and parameters as other; false otherwise.

Parameters
otherThe array to compare this with.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ reverse_merge()

virtual const ContentPtr reverse_merge ( const ContentPtr other) const
virtual

Merges a single other with this array in reverse order: other first, this last.

Only arrays that need to be reversible have this function: option-type and union-type arrays. Others raise a runtime error.

Reimplemented in IndexedArrayOf< T, ISOPTION >, BitMaskedArray, UnionArrayOf< T, I >, ByteMaskedArray, and UnmaskedArray.

◆ rpad()

virtual const ContentPtr rpad ( int64_t  target,
int64_t  axis,
int64_t  depth 
) const
pure virtual

If axis = 0, returns a view of this array padded on the right with None values to have a minimum length; otherwise, returns an array with nested lists all padded to the minimum length.

Parameters
targetThe intended length. The output may be longer than this target length, but not shorter (using ListArray).
axisThe axis at which to apply padding. Negative axis counts backward from the deepest levels (-1 is the last valid axis).
depthThe current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis).

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ rpad_and_clip()

virtual const ContentPtr rpad_and_clip ( int64_t  target,
int64_t  axis,
int64_t  depth 
) const
pure virtual

If axis = 0, returns a view of this array padded on the right.

with None values to have exactly the specified length; otherwise, returns an array with nested lists all padded to the specified length.

Parameters
targetThe intended length; the output has exactly this target length (using RegularArray).
axisThe axis at which to apply padding. Negative axis counts backward from the deepest levels (-1 is the last valid axis).
depthThe current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis).

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ rpad_axis0()

const ContentPtr rpad_axis0 ( int64_t  target,
bool  clip 
) const

Internal function to handle the axis = 0 case of rpad and rpad_and_clip.

The axis = 0 case does not depend on array node type, so it is defined universally in the Content class.

Parameters
targetThe intended length. The output may be longer than this target length, but not shorter (using ListArray).
clipIf true, do rpad_and_clip; if false, do rpad.

◆ setidentities() [1/2]

virtual void setidentities ( )
pure virtual

Assign a surrogate index of Identities to this array (in-place).

This also assigns and possibly replaces Identities in nested arrays.

Note
This mutability is temporary: scikit-hep/awkward-1.0#116 Eventually, this interface will be deprecated and all Content instances will be immutable.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ setidentities() [2/2]

virtual void setidentities ( const IdentitiesPtr identities)
pure virtual

Assign a specified set of Identities to this array (in-place).

This also assigns and possibly replaces Identities in nested arrays.

Note
This mutability is temporary: scikit-hep/awkward-1.0#116 Eventually, this interface will be deprecated and all Content instances will be immutable.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ setparameter()

void setparameter ( const std::string &  key,
const std::string &  value 
)

Assign one parameter for this array node (in-place).

Keys are simple strings, but values are JSON-encoded strings. For this reason, values that represent single strings are double-quoted: e.g. "\"actual_value"".

Note
This mutability is temporary: scikit-hep/awkward-1.0#117 Eventually, this interface will be deprecated and all Content instances will be immutable.

◆ setparameters()

void setparameters ( const util::Parameters parameters)

Assign all the parameters for this array node (in-place).

Keys are simple strings, but values are JSON-encoded strings. For this reason, values that represent single strings are double-quoted: e.g. "\"actual_value"".

Note
This mutability is temporary: scikit-hep/awkward-1.0#117 Eventually, this interface will be deprecated and all Content instances will be immutable.

◆ shallow_copy()

virtual const ContentPtr shallow_copy ( ) const
pure virtual

Copies this node without copying any nodes hierarchically nested within it or any array/index/identity buffers.

See also deep_copy.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ shallow_simplify()

virtual const ContentPtr shallow_simplify ( ) const
pure virtual

Returns an equivalent array simplified at one level only using simplify_optiontype if an option-type array and simplify_uniontype if a union-type array.

For all other types of arrays, this operation is a pass-through.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ sort()

const ContentPtr sort ( int64_t  axis,
bool  ascending,
bool  stable 
) const

This array with one axis sorted by applying a sorting algorithm.

This operation is implemented on each node through sort_next.

Parameters
axisThe axis to sort. Negative axis counts backward from the deepest levels (-1 is the last valid axis).
ascendingIf true, the values will be sorted in an ascending order.
stableIf true, the values will be sorted by a stable sort algorithm to maintain the relative order of records with equal keys (i.e. values).

◆ sort_next()

virtual const ContentPtr sort_next ( int64_t  negaxis,
const Index64 starts,
const Index64 parents,
int64_t  outlength,
bool  ascending,
bool  stable 
) const
pure virtual

This array sorted.

The user's entry point for this operation is sort.

Parameters
negaxisThe negative axis: -axis. That is, negaxis = 1 means the deepest axis level.
startsStaring positions of each group to combine as an Index. These are downward pointers from an outer structure into this structure with the same meaning as in ListArray.
parentsGroups to combine as an Index of upward pointers from this structure to the outer structure to sort.
outlengthThe length of the array, after the operation completes.
ascendingIf true, the values will be sorted in an ascending order.
stableIf true, the values will be sorted by a stable sort algorithm to maintain the relative order of records with equal keys (i.e. values).

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, UnionArrayOf< T, I >, ListArrayOf< T >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ tojson() [1/2]

const std::string tojson ( bool  pretty,
int64_t  maxdecimals,
const char *  nan_string = nullptr,
const char *  infinity_string = nullptr,
const char *  minus_infinity_string = nullptr,
const char *  complex_real_string = nullptr,
const char *  complex_imag_string = nullptr 
) const

Returns a JSON representation of this array.

Parameters
prettyIf true, add spacing to make the JSON human-readable. If false, return a compact representation.
maxdecimalsMaximum number of decimals for floating-point numbers or -1 for no limit.
nan_stringuser-defined string for a not-a-number (NaN) value representation in JSON format
infinity_stringuser-defined string for a positive infinity representation in JSON format
minus_infinity_stringuser-defined string for a negative infinity representation in JSON format

Although the JSON output contains all of the data from the array (and therefore might not be safe to print to the screen), it does not preserve the type information of an array. In particular, the distinction between ListType and RegularType is lost.

◆ tojson() [2/2]

void tojson ( FILE *  destination,
bool  pretty,
int64_t  maxdecimals,
int64_t  buffersize,
const char *  nan_string = nullptr,
const char *  infinity_string = nullptr,
const char *  minus_infinity_string = nullptr,
const char *  complex_real_string = nullptr,
const char *  complex_imag_string = nullptr 
) const

Writes a JSON representation of this array to a destination file.

Parameters
destinationThe file to write into.
prettyIf true, add spacing to make the JSON human-readable. If false, return a compact representation.
maxdecimalsMaximum number of decimals for floating-point numbers or -1 for no limit.
buffersizeSize of a temporary buffer in bytes.
nan_stringuser-defined string for a not-a-number (NaN) value representation in JSON format
infinity_stringuser-defined string for a positive infinity representation in JSON format
minus_infinity_stringuser-defined string for a negative infinity representation in JSON format

Although the JSON output contains all of the data from the array (and therefore might not be safe to print to the screen), it does not preserve the type information of an array. In particular, the distinction between ListType and RegularType is lost.

◆ tojson_part()

virtual void tojson_part ( ToJson builder,
bool  include_beginendlist 
) const
pure virtual

◆ tostring()

const std::string tostring ( ) const

Returns a string representation of this array (multi-line XML).

Although this XML string has detail about every node in the tree, it does not show all elements in the array buffers and therefore does not scale with the size of the dataset (i.e. it is safe to print to the screen).

Thus, it's also not a storage format: see tojson.

◆ tostring_part()

virtual const std::string tostring_part ( const std::string &  indent,
const std::string &  pre,
const std::string &  post 
) const
pure virtual

Internal function to build an output string for tostring.

Parameters
indentIndentation depth as a string of spaces.
prePrefix string, usually an opening XML tag.
postPostfix string, usually a closing XML tag and carriage return.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ type()

virtual const TypePtr type ( const util::TypeStrs typestrs) const
pure virtual

High-level Type describing this array.

Parameters
typestrsA mapping from "__record__" parameters to string representations of those types, to override the derived strings.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, RecordArray, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ unique()

◆ validityerror()

virtual const std::string validityerror ( const std::string &  path) const
pure virtual

Returns an error message if this array is invalid; otherwise, returns an empty string.

Implemented in NumpyArray, IndexedArrayOf< T, ISOPTION >, BitMaskedArray, ListArrayOf< T >, UnionArrayOf< T, I >, ListOffsetArrayOf< T >, RecordArray, ByteMaskedArray, RegularArray, VirtualArray, UnmaskedArray, EmptyArray, Record, and None.

◆ validityerror_parameters()

const std::string validityerror_parameters ( const std::string &  path) const
protected

Internal function to determine if a particular set of parameters are allowed on a particular node (centralized by parameter, not by class).

Member Data Documentation

◆ identities_

IdentitiesPtr identities_
protected

See identities.

◆ parameters_

util::Parameters parameters_
protected

See parameters.


The documentation for this class was generated from the following file: