VirtualArray Class Reference

Represents an array that can be generated on demand. More...

#include <VirtualArray.h>

Inheritance diagram for VirtualArray:
Content

Public Member Functions

 VirtualArray (const IdentitiesPtr &identities, const util::Parameters &parameters, const ArrayGeneratorPtr &generator, const ArrayCachePtr &cache, const std::string &cache_key, const kernel::lib ptr_lib=kernel::lib::cpu)
 Creates a VirtualArray from a full set of parameters. More...
 
 VirtualArray (const IdentitiesPtr &identities, const util::Parameters &parameters, const ArrayGeneratorPtr &generator, const ArrayCachePtr &cache, const kernel::lib ptr_lib=kernel::lib::cpu)
 Creates a VirtualArray with an automatically assigned cache_key (unique per process). More...
 
const ArrayGeneratorPtr generator () const
 Function that materializes the array and possibly checks it against an expected Form. More...
 
const ArrayCachePtr cache () const
 Temporary storage for materialized arrays to avoid calling the generator more than necessary. May be nullptr. More...
 
const kernel::lib ptr_lib () const
 
const ContentPtr peek_array () const
 Returns the array if it exists in the cache; nullptr otherwise. More...
 
const ContentPtr array () const
 Ensures that the array is generated and returns it. More...
 
const std::string cache_key () const
 The key this VirtualArray will use when filling a cache. More...
 
const std::string classname () const override
 User-friendly name of this class: "VirtualArray". More...
 
void setidentities () override
 Assign a surrogate index of Identities to this array (in-place). More...
 
void setidentities (const IdentitiesPtr &identities) override
 Assign a specified set of Identities to this array (in-place). More...
 
const TypePtr type (const util::TypeStrs &typestrs) const override
 High-level Type describing this array. More...
 
const FormPtr form (bool materialize) const override
 Low-level Form describing all the features of this array except the actual data buffers (Index, Identities, NumpyArray::ptr, and RawArray::ptr). More...
 
kernel::lib kernels () const override
 Returns the kernel library enum for all nested ptr_lib within the array's tree structure. If different libraries are associated with nodes of the same tree, this returns kernel::lib::size to indicate that it is "mixed". More...
 
void caches (std::vector< ArrayCachePtr > &out) const override
 Accumulates all the unique #ArrayCache objects from nested VirtualArray nodes. (Uniqueness is determined by pointer value.) More...
 
const std::string tostring_part (const std::string &indent, const std::string &pre, const std::string &post) const override
 Internal function to build an output string for tostring. More...
 
void tojson_part (ToJson &builder, bool include_beginendlist) const override
 Internal function to produce a JSON representation one node at a time. More...
 
void nbytes_part (std::map< size_t, int64_t > &largest) const override
 Internal function used to calculate nbytes. More...
 
int64_t length () const override
 The number of elements in the array. More...
 
const ContentPtr shallow_copy () const override
 Copies this node without copying any nodes hierarchically nested within it or any array/index/identity buffers. More...
 
const ContentPtr deep_copy (bool copyarrays, bool copyindexes, bool copyidentities) const override
 Copies this node and all nodes hierarchically nested within it, optionally copying the associated arrays, indexes, and identities, too. More...
 
void check_for_iteration () const override
 Performs up-front validity checks on an array so that they don't have to be checked in getitem_at_nowrap for each item. More...
 
const ContentPtr getitem_nothing () const override
 Internal function to get an empty slice (with the correct type). More...
 
const ContentPtr getitem_at (int64_t at) const override
 Returns the element at a given position in the array, handling negative indexing and bounds-checking like Python. More...
 
const ContentPtr getitem_at_nowrap (int64_t at) const override
 Returns the element at a given position in the array, without handling negative indexing or bounds-checking. More...
 
const ContentPtr getitem_range (int64_t start, int64_t stop) const override
 Subinterval of this array, handling negative indexing and bounds-checking like Python. More...
 
const ContentPtr getitem_range_nowrap (int64_t start, int64_t stop) const override
 Subinterval of this array, without handling negative indexing or bounds-checking. More...
 
const ContentPtr getitem_field (const std::string &key) const override
 This array with the first nested RecordArray replaced by the field at key. More...
 
const ContentPtr getitem_field (const std::string &key, const Slice &only_fields) const override
 
const ContentPtr getitem_fields (const std::vector< std::string > &keys) const override
 This array with the first nested RecordArray replaced by a RecordArray of a given subset of keys. More...
 
const ContentPtr getitem_fields (const std::vector< std::string > &keys, const Slice &only_fields) const override
 
const ContentPtr getitem_next (const SliceItemPtr &head, const Slice &tail, const Index64 &advanced) const override
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
const ContentPtr getitem_next_jagged (const Index64 &slicestarts, const Index64 &slicestops, const SliceItemPtr &slicecontent, const Slice &tail) const override
 Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis. More...
 
const ContentPtr carry (const Index64 &carry, bool allow_lazy) const override
 Returns an array of the same type with elements filtered, rearranged, and possibly duplicated by the carry array of integers. More...
 
int64_t purelist_depth () const override
 The list-depth of this array, not counting any contained within a RecordArray. More...
 
const std::pair< int64_t, int64_t > minmax_depth () const override
 Returns (a) the minimum list-depth and (b) the maximum list-depth of the array, which can differ if this array "branches" (differs when followed through different fields of a RecordArray or UnionArray). More...
 
const std::pair< bool, int64_t > branch_depth () const override
 Returns (a) whether the list-depth of this array "branches," or differs when followed through different fields of a RecordArray or UnionArray and (b) the minimum list-depth. More...
 
int64_t numfields () const override
 The number of fields in the first nested tuple or records or -1 if this array does not contain a RecordArray. More...
 
int64_t fieldindex (const std::string &key) const override
 The position of a tuple or record key name if this array contains a RecordArray. More...
 
const std::string key (int64_t fieldindex) const override
 The record name associated with a given field index or the tuple index as a string (e.g. "0", "1", "2") if a tuple. More...
 
bool haskey (const std::string &key) const override
 Returns true if the array contains a RecordArray with the specified key; false otherwise. More...
 
const std::vector< std::string > keys () const override
 A list of RecordArray keys or an empty list if this array does not contain a RecordArray. More...
 
const std::string validityerror (const std::string &path) const override
 Returns an error message if this array is invalid; otherwise, returns an empty string. More...
 
const ContentPtr shallow_simplify () const override
 Returns an equivalent array simplified at one level only using simplify_optiontype if an option-type array and simplify_uniontype if a union-type array. More...
 
const ContentPtr num (int64_t axis, int64_t depth) const override
 The length of this array (as a NumpyArray scalar) if axis = 0 or the lengths of subarrays (as an array or nested array) if axis != 0. More...
 
const std::pair< Index64, ContentPtroffsets_and_flattened (int64_t axis, int64_t depth) const override
 Returns (a) an offsets Index and (b) a flattened version of the array at some axis depth. More...
 
bool mergeable (const ContentPtr &other, bool mergebool) const override
 Returns true if this array can be merged with the other; false otherwise. More...
 
bool referentially_equal (const ContentPtr &other) const override
 Returns true if this array has all the same buffers and parameters as other; false otherwise. More...
 
const ContentPtr mergemany (const ContentPtrVec &others) const override
 Returns an array with this and the others concatenated (in order, this first, others last). More...
 
const SliceItemPtr asslice () const override
 Converts this array into a SliceItem that can be used in getitem. More...
 
const ContentPtr fillna (const ContentPtr &value) const override
 Returns this array with None values replaced by a given value. More...
 
const ContentPtr rpad (int64_t target, int64_t axis, int64_t depth) const override
 If axis = 0, returns a view of this array padded on the right with None values to have a minimum length; otherwise, returns an array with nested lists all padded to the minimum length. More...
 
const ContentPtr rpad_and_clip (int64_t target, int64_t axis, int64_t depth) const override
 If axis = 0, returns a view of this array padded on the right. More...
 
const ContentPtr reduce_next (const Reducer &reducer, int64_t negaxis, const Index64 &starts, const Index64 &shifts, const Index64 &parents, int64_t outlength, bool mask, bool keepdims) const override
 This array with one axis removed by applying a Reducer (e.g. "sum", "max", "any", "all). More...
 
const ContentPtr sort_next (int64_t negaxis, const Index64 &starts, const Index64 &parents, int64_t outlength, bool ascending, bool stable) const override
 This array sorted. More...
 
const ContentPtr argsort_next (int64_t negaxis, const Index64 &starts, const Index64 &shifts, const Index64 &parents, int64_t outlength, bool ascending, bool stable) const override
 This array sorted indices. More...
 
const ContentPtr localindex (int64_t axis, int64_t depth) const override
 A (possibly nested) array of integers indicating the positions of elements within each nested list. More...
 
const ContentPtr combinations (int64_t n, bool replacement, const util::RecordLookupPtr &recordlookup, const util::Parameters &parameters, int64_t axis, int64_t depth) const override
 Tuples or records of all n-tuple combinations of list items at some axis depth. More...
 
const ContentPtr getitem (const Slice &where) const override
 Entry point for general slicing: Slice represents a tuple of SliceItem nodes applying to each level of nested lists. More...
 
const ContentPtr getitem_next (const SliceAt &at, const Slice &tail, const Index64 &advanced) const override
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
const ContentPtr getitem_next (const SliceRange &range, const Slice &tail, const Index64 &advanced) const override
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
const ContentPtr getitem_next (const SliceArray64 &array, const Slice &tail, const Index64 &advanced) const override
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
const ContentPtr getitem_next (const SliceField &field, const Slice &tail, const Index64 &advanced) const override
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
const ContentPtr getitem_next (const SliceFields &fields, const Slice &tail, const Index64 &advanced) const override
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
const ContentPtr getitem_next (const SliceJagged64 &jagged, const Slice &tail, const Index64 &advanced) const override
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
const ContentPtr getitem_next_jagged (const Index64 &slicestarts, const Index64 &slicestops, const SliceArray64 &slicecontent, const Slice &tail) const override
 Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis. More...
 
const ContentPtr getitem_next_jagged (const Index64 &slicestarts, const Index64 &slicestops, const SliceMissing64 &slicecontent, const Slice &tail) const override
 Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis. More...
 
const ContentPtr getitem_next_jagged (const Index64 &slicestarts, const Index64 &slicestops, const SliceJagged64 &slicecontent, const Slice &tail) const override
 Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis. More...
 
const ContentPtr copy_to (kernel::lib ptr_lib) const override
 Recursively copies components of the array from main memory to a GPU (if ptr_lib == kernel::lib::cuda) or to main memory (if ptr_lib == kernel::lib::cpu) if those components are not already there. More...
 
const ContentPtr numbers_to_type (const std::string &name) const override
 Change the leaf types to 'totype'. More...
 
bool is_unique () const override
 Returns 'true' if all components of the array are unique. More...
 
const ContentPtr unique () const override
 Returns an array where all components are unique. More...
 
bool is_subrange_equal (const Index64 &start, const Index64 &stop) const override
 Returns 'true' if subranges are equal. More...
 
- Public Member Functions inherited from Content
 Content (const IdentitiesPtr &identities, const util::Parameters &parameters)
 Called by all subclass constructors; assigns identities and parameters upon construction. More...
 
virtual ~Content ()
 Empty destructor; required for some C++ reason. More...
 
virtual bool isscalar () const
 Returns true if the data represented by this node is scalar, not a true array. More...
 
virtual const IdentitiesPtr identities () const
 Optional Identities for each element of the array (may be nullptr). More...
 
virtual const std::string purelist_parameter (const std::string &key) const
 The parameter associated with key at the first level that has a non-null value, descending only as deep as the first RecordArray. More...
 
bool purelist_isregular () const
 Returns true if all nested lists down to the first RecordArray are RegularArray nodes; false otherwise. More...
 
virtual bool dimension_optiontype () const
 Returns true if this dimension has option-type; false otherwise. More...
 
virtual const std::pair< ContentPtrVec, ContentPtrVecmerging_strategy (const ContentPtrVec &others) const
 Partitions this array plus a list of others into a head sequence and a tail sequence: More...
 
virtual const ContentPtr reverse_merge (const ContentPtr &other) const
 Merges a single other with this array in reverse order: other first, this last. More...
 
const ContentPtr merge (const ContentPtr &other) const
 An array with this and the other concatenated (this first, other last). More...
 
const std::string tostring () const
 Returns a string representation of this array (multi-line XML). More...
 
const std::string tojson (bool pretty, int64_t maxdecimals, const char *nan_string=nullptr, const char *infinity_string=nullptr, const char *minus_infinity_string=nullptr, const char *complex_real_string=nullptr, const char *complex_imag_string=nullptr) const
 Returns a JSON representation of this array. More...
 
void tojson (FILE *destination, bool pretty, int64_t maxdecimals, int64_t buffersize, const char *nan_string=nullptr, const char *infinity_string=nullptr, const char *minus_infinity_string=nullptr, const char *complex_real_string=nullptr, const char *complex_imag_string=nullptr) const
 Writes a JSON representation of this array to a destination file. More...
 
int64_t nbytes () const
 The number of bytes contained in all array buffers, Index buffers, and Identities buffers, not including the lightweight node objects themselves. More...
 
const ContentPtr reduce (const Reducer &reducer, int64_t axis, bool mask, bool keepdims) const
 This array with one axis removed by applying a Reducer (e.g. "sum", "max", "any", "all). More...
 
const ContentPtr sort (int64_t axis, bool ascending, bool stable) const
 This array with one axis sorted by applying a sorting algorithm. More...
 
const ContentPtr argsort (int64_t axis, bool ascending, bool stable) const
 This array indices with one axis sorted by applying a sorting algorithm. More...
 
const util::Parameters parameters () const
 String-to-JSON map that augments the meaning of this array. More...
 
void setparameters (const util::Parameters &parameters)
 Assign all the parameters for this array node (in-place). More...
 
const std::string parameter (const std::string &key) const
 Get one parameter from this array node. More...
 
void setparameter (const std::string &key, const std::string &value)
 Assign one parameter for this array node (in-place). More...
 
bool parameter_equals (const std::string &key, const std::string &value) const
 Returns true if the parameter associated with key exists and is equal to value; false otherwise. More...
 
bool parameters_equal (const util::Parameters &other, bool check_all) const
 Returns true if all parameters of this array node are equal to the other parameters. More...
 
bool parameter_isstring (const std::string &key) const
 Returns true if the parameter associated with key is a string; false otherwise. More...
 
bool parameter_isname (const std::string &key) const
 Returns true if the parameter associated with key is a string that matches [A-Za-z_][A-Za-z_0-9]*; false otherwise. More...
 
const std::string parameter_asstring (const std::string &key) const
 Returns the parameter associated with key as a string if parameter_isstring; raises an error otherwise. More...
 
const ContentPtr merge_as_union (const ContentPtr &other) const
 Concatenates this array with other by creating a UnionArray instead of actually merging the data. This does not call UnionArray::simplify_uniontype. More...
 
const ContentPtr rpad_axis0 (int64_t target, bool clip) const
 Internal function to handle the axis = 0 case of rpad and rpad_and_clip. More...
 
const ContentPtr localindex_axis0 () const
 Internal function to handle the axis = 0 case of localindex. More...
 
const ContentPtr combinations_axis0 (int64_t n, bool replacement, const util::RecordLookupPtr &recordlookup, const util::Parameters &parameters) const
 Internal function to handle the axis = 0 case of combinations. More...
 
virtual const ContentPtr getitem_next (const SliceEllipsis &ellipsis, const Slice &tail, const Index64 &advanced) const
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
virtual const ContentPtr getitem_next (const SliceNewAxis &newaxis, const Slice &tail, const Index64 &advanced) const
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
virtual const ContentPtr getitem_next (const SliceMissing64 &missing, const Slice &tail, const Index64 &advanced) const
 Internal function that propagates a generic getitem request through one axis (including advanced indexing). More...
 
const int64_t axis_wrap_if_negative (int64_t axis) const
 Internal function defining the negative axis handling for many operations. More...
 

Additional Inherited Members

- Protected Member Functions inherited from Content
const std::string validityerror_parameters (const std::string &path) const
 Internal function to determine if a particular set of parameters are allowed on a particular node (centralized by parameter, not by class). More...
 
const ContentPtr getitem_next_array_wrap (const ContentPtr &outcontent, const std::vector< int64_t > &shape) const
 Internal function to wrap putative getitem output with enough RegularArray nodes to satisfy a given shape. More...
 
const std::string parameters_tostring (const std::string &indent, const std::string &pre, const std::string &post) const
 Internal function to convert parameters into a string fragment for tostring. More...
 
kernel::lib kernels_compare (kernel::lib from_index, const ContentPtr &content) const
 Internal function to support most of the logic in kernels. More...
 
- Protected Attributes inherited from Content
IdentitiesPtr identities_
 See identities. More...
 
util::Parameters parameters_
 See parameters. More...
 

Detailed Description

Represents an array that can be generated on demand.

See VirtualArray for the meaning of each parameter.

Constructor & Destructor Documentation

◆ VirtualArray() [1/2]

VirtualArray ( const IdentitiesPtr identities,
const util::Parameters parameters,
const ArrayGeneratorPtr generator,
const ArrayCachePtr cache,
const std::string &  cache_key,
const kernel::lib  ptr_lib = kernel::lib::cpu 
)

Creates a VirtualArray from a full set of parameters.

Parameters
identitiesOptional Identities for each element of the array (may be nullptr).
parametersString-to-JSON map that augments the meaning of this array.
generatorFunction that materializes the array and possibly checks it against an expected Form.
cacheTemporary storage for materialized arrays to avoid calling the generator more than necessary. May be nullptr.
cache_keyThe key this VirtualArray will use when filling a cache.

◆ VirtualArray() [2/2]

VirtualArray ( const IdentitiesPtr identities,
const util::Parameters parameters,
const ArrayGeneratorPtr generator,
const ArrayCachePtr cache,
const kernel::lib  ptr_lib = kernel::lib::cpu 
)

Creates a VirtualArray with an automatically assigned cache_key (unique per process).

Member Function Documentation

◆ argsort_next()

const ContentPtr argsort_next ( int64_t  negaxis,
const Index64 starts,
const Index64 shifts,
const Index64 parents,
int64_t  outlength,
bool  ascending,
bool  stable 
) const
overridevirtual

This array sorted indices.

The user's entry point for this operation is argsort.

Parameters
negaxisThe negative axis: -axis. That is, negaxis = 1 means the deepest axis level.
startsStaring positions of each group to combine as an Index. These are downward pointers from an outer structure into this structure with the same meaning as in ListArray.
shiftsPer-element adjustments that allows for variable-length lists with axis != -1 and for missing values (None).
parentsGroups to combine as an Index of upward pointers from this structure to the outer structure to sort.
outlengthThe length of the array, after the operation completes.
ascendingIf true, the values will be sorted in an ascending order.
stableIf true, the values will be sorted by a stable sort algorithm to maintain the relative order of records with equal keys (i.e. values).

Implements Content.

◆ array()

const ContentPtr array ( ) const

Ensures that the array is generated and returns it.

This method does not return nullptr.

◆ asslice()

const SliceItemPtr asslice ( ) const
overridevirtual

Converts this array into a SliceItem that can be used in getitem.

Implements Content.

◆ branch_depth()

const std::pair<bool, int64_t> branch_depth ( ) const
overridevirtual

Returns (a) whether the list-depth of this array "branches," or differs when followed through different fields of a RecordArray or UnionArray and (b) the minimum list-depth.

If the array does not contain any records or heterogeneous data, the first element is always true and the second is simply the depth.

Implements Content.

◆ cache()

const ArrayCachePtr cache ( ) const

Temporary storage for materialized arrays to avoid calling the generator more than necessary. May be nullptr.

◆ cache_key()

const std::string cache_key ( ) const

The key this VirtualArray will use when filling a cache.

◆ caches()

void caches ( std::vector< ArrayCachePtr > &  out) const
overridevirtual

Accumulates all the unique #ArrayCache objects from nested VirtualArray nodes. (Uniqueness is determined by pointer value.)

Implements Content.

◆ carry()

const ContentPtr carry ( const Index64 carry,
bool  allow_lazy 
) const
overridevirtual

Returns an array of the same type with elements filtered, rearranged, and possibly duplicated by the carry array of integers.

The output has the same length as the carry index, not the array that is being manipulated. For each item i in carry, the output is array[index[i]].

This operation is called take in NumPy and Arrow, although this carry is a low-level function that does not handle negative indexes and is not exposed to the Python layer. It is used by many operations to pass filters/rearrangements/duplications from one typed array node to another without knowing the latter's type.

Taking getitem_at_nowrap as a function from integers to the array's item type, A: [0, len(a)) → T, and the carry array's Index64::getitem_at_nowrap as a function C: [0, len(c)) → [0, len(a)), this method represents function composition, A ∘ C: [0, len(c)) → T.

Note
If the carry array has duplicate elements, the array is only duplicated one level deep. For instance, on a ListArray, only the starts and stops get duplicated, not the content (and similarly for all other node types).

Implements Content.

◆ check_for_iteration()

void check_for_iteration ( ) const
overridevirtual

Performs up-front validity checks on an array so that they don't have to be checked in getitem_at_nowrap for each item.

Implements Content.

◆ classname()

const std::string classname ( ) const
overridevirtual

User-friendly name of this class: "VirtualArray".

Implements Content.

◆ combinations()

const ContentPtr combinations ( int64_t  n,
bool  replacement,
const util::RecordLookupPtr recordlookup,
const util::Parameters parameters,
int64_t  axis,
int64_t  depth 
) const
overridevirtual

Tuples or records of all n-tuple combinations of list items at some axis depth.

For example, the n = 2 combinations at axis = 0 of

[a, b, c, d, e]

would be

[[(a, b), (a, c), (a, d), (a, e)],
[(b, c), (b, d), (b, e)],
[(c, d), (c, e)],
[(d, e)]]

and the n = 3 combinations at axis = 1 of

[[a, b, c, d], [], [e, f], [g, h, i]]

would be

[[[(a, b, c), (a, b, d), (a, c, d)], [(b, c, d)]],
[],
[],
[[(g, h, i)]]]
Parameters
nThe number of items in each tuple/record.
replacementIf true, the tuples/records are allowed to include the same item more than once, such as (a, a, a) and (a, a, b). In the above examples, replacement = false. (If the output of this function is thought of as the "upper triangle" elements of the Cartesian product of the input with itself n times, the replacement parameter determines whether elements on the diagonal are allowed.)
recordlookupIf nullptr, the output consists of tuples, a RecordArray indexed by "0", "1", "2", etc. If recordlookup is a std::vector<std::string>, the output consists of records, a RecordArray indexed by names (strings). The length of recordlookup must be equal to n.
parametersParameters assigned to the new RecordArray. This can be used to set "__record_" = "\"record_name\"" to give the records a custom behavior in Python.
axisThe nesting depth at which this operation is applied. At axis = 0, combinations are formed from elements of the whole array (see example above); at axis = 1, combinations are formed independently for each element. At a deeper axis, outer levels are left untouched. Negative axis counts backward from the deepest levels (-1 is the last valid axis).
depthThe current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis).

Note that axis = 0 is qualitatively different from any other axis because a dataset is typically much larger than any one of its elements. As such, axis = 0 is lazily generated with an IndexedArray, while any other axis is eagerly generated by duplicating elements (with carry).

Implements Content.

◆ copy_to()

const ContentPtr copy_to ( kernel::lib  ptr_lib) const
overridevirtual

Recursively copies components of the array from main memory to a GPU (if ptr_lib == kernel::lib::cuda) or to main memory (if ptr_lib == kernel::lib::cpu) if those components are not already there.

Implements Content.

◆ deep_copy()

const ContentPtr deep_copy ( bool  copyarrays,
bool  copyindexes,
bool  copyidentities 
) const
overridevirtual

Copies this node and all nodes hierarchically nested within it, optionally copying the associated arrays, indexes, and identities, too.

See also shallow_copy.

Parameters
copyarraysIf true, copy the associated array buffers (in NumpyArray and RawArray), not just the lightweight objects that point to them.
copyindexesIf true, copy the Index objects and their buffers as well.
copyidentitiesIf true, copy the Identities objects and their buffers as well.

Implements Content.

◆ fieldindex()

int64_t fieldindex ( const std::string &  key) const
overridevirtual

The position of a tuple or record key name if this array contains a RecordArray.

Implements Content.

◆ fillna()

const ContentPtr fillna ( const ContentPtr value) const
overridevirtual

Returns this array with None values replaced by a given value.

Parameters
valueAn array of exactly one element, which need not have the same type as the missing values it's replacing.

Implements Content.

◆ form()

const FormPtr form ( bool  materialize) const
overridevirtual

Low-level Form describing all the features of this array except the actual data buffers (Index, Identities, NumpyArray::ptr, and RawArray::ptr).

Parameters
materializeIf true, materialize any VirtualArrays encountered along the way.

Implements Content.

◆ generator()

const ArrayGeneratorPtr generator ( ) const

Function that materializes the array and possibly checks it against an expected Form.

◆ getitem()

const ContentPtr getitem ( const Slice where) const
overridevirtual

Entry point for general slicing: Slice represents a tuple of SliceItem nodes applying to each level of nested lists.

Reimplemented from Content.

◆ getitem_at()

const ContentPtr getitem_at ( int64_t  at) const
overridevirtual

Returns the element at a given position in the array, handling negative indexing and bounds-checking like Python.

The first item in the array is at 0, the second at 1, the last at -1, the penultimate at -2, etc.

Implements Content.

◆ getitem_at_nowrap()

const ContentPtr getitem_at_nowrap ( int64_t  at) const
overridevirtual

Returns the element at a given position in the array, without handling negative indexing or bounds-checking.

If the array has Identities, the identity bounds are checked.

Implements Content.

◆ getitem_field() [1/2]

const ContentPtr getitem_field ( const std::string &  key) const
overridevirtual

This array with the first nested RecordArray replaced by the field at key.

Implements Content.

◆ getitem_field() [2/2]

const ContentPtr getitem_field ( const std::string &  key,
const Slice only_fields 
) const
overridevirtual

Implements Content.

◆ getitem_fields() [1/2]

const ContentPtr getitem_fields ( const std::vector< std::string > &  keys) const
overridevirtual

This array with the first nested RecordArray replaced by a RecordArray of a given subset of keys.

Implements Content.

◆ getitem_fields() [2/2]

const ContentPtr getitem_fields ( const std::vector< std::string > &  keys,
const Slice only_fields 
) const
overridevirtual

Implements Content.

◆ getitem_next() [1/7]

const ContentPtr getitem_next ( const SliceItemPtr head,
const Slice tail,
const Index64 advanced 
) const
overridevirtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

Parameters
headFirst element of the Slice tuple.
tailThe rest of the elements of the Slice tuple.
advancedIf empty, no array slices (integer or boolean) have been encountered yet; otherwise, positions in any subsequent array slices to select.

In the NumPy documentation, advanced indexes are described as iterating "as one," which requires an Index to be propagated when implemented recursively.

Reimplemented from Content.

◆ getitem_next() [2/7]

const ContentPtr getitem_next ( const SliceAt at,
const Slice tail,
const Index64 advanced 
) const
overridevirtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

See generic getitem_next for details.

Implements Content.

◆ getitem_next() [3/7]

const ContentPtr getitem_next ( const SliceRange range,
const Slice tail,
const Index64 advanced 
) const
overridevirtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

See generic getitem_next for details.

Implements Content.

◆ getitem_next() [4/7]

const ContentPtr getitem_next ( const SliceArray64 array,
const Slice tail,
const Index64 advanced 
) const
overridevirtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

See generic getitem_next for details.

Implements Content.

◆ getitem_next() [5/7]

const ContentPtr getitem_next ( const SliceField field,
const Slice tail,
const Index64 advanced 
) const
overridevirtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

See generic getitem_next for details.

Reimplemented from Content.

◆ getitem_next() [6/7]

const ContentPtr getitem_next ( const SliceFields fields,
const Slice tail,
const Index64 advanced 
) const
overridevirtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

See generic getitem_next for details.

Reimplemented from Content.

◆ getitem_next() [7/7]

const ContentPtr getitem_next ( const SliceJagged64 jagged,
const Slice tail,
const Index64 advanced 
) const
overridevirtual

Internal function that propagates a generic getitem request through one axis (including advanced indexing).

See generic getitem_next for details.

Implements Content.

◆ getitem_next_jagged() [1/4]

const ContentPtr getitem_next_jagged ( const Index64 slicestarts,
const Index64 slicestops,
const SliceItemPtr slicecontent,
const Slice tail 
) const
overridevirtual

Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis.

Parameters
slicestartsEffective starts (similar to ListArray's starts) of the jagged slice.
slicestopsEffective stops of the jagged slice.
slicecontentNested content within the jagged slice.
tailSubsequent SliceItem elements beyond the jagged array hierarchy.

Reimplemented from Content.

◆ getitem_next_jagged() [2/4]

const ContentPtr getitem_next_jagged ( const Index64 slicestarts,
const Index64 slicestops,
const SliceArray64 slicecontent,
const Slice tail 
) const
overridevirtual

Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis.

See generic getitem_next_jagged for details.

Implements Content.

◆ getitem_next_jagged() [3/4]

const ContentPtr getitem_next_jagged ( const Index64 slicestarts,
const Index64 slicestops,
const SliceMissing64 slicecontent,
const Slice tail 
) const
overridevirtual

Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis.

See generic getitem_next_jagged for details.

Implements Content.

◆ getitem_next_jagged() [4/4]

const ContentPtr getitem_next_jagged ( const Index64 slicestarts,
const Index64 slicestops,
const SliceJagged64 slicecontent,
const Slice tail 
) const
overridevirtual

Internal function that propagates a jagged array (array with irregular-length dimensions) slice through one axis.

See generic getitem_next_jagged for details.

Implements Content.

◆ getitem_nothing()

const ContentPtr getitem_nothing ( ) const
overridevirtual

Internal function to get an empty slice (with the correct type).

Implements Content.

◆ getitem_range()

const ContentPtr getitem_range ( int64_t  start,
int64_t  stop 
) const
overridevirtual

Subinterval of this array, handling negative indexing and bounds-checking like Python.

The first item in the array is at 0, the second at 1, the last at -1, the penultimate at -2, etc.

Ranges beyond the array are not an error; they are trimmed to start = 0 on the left and stop = length() - 1 on the right.

This operation only affects the node metadata; its calculation time does not scale with the size of the array.

Implements Content.

◆ getitem_range_nowrap()

const ContentPtr getitem_range_nowrap ( int64_t  start,
int64_t  stop 
) const
overridevirtual

Subinterval of this array, without handling negative indexing or bounds-checking.

If the array has Identities, the identity bounds are checked.

This operation only affects the node metadata; its calculation time does not scale with the size of the array.

Implements Content.

◆ haskey()

bool haskey ( const std::string &  key) const
overridevirtual

Returns true if the array contains a RecordArray with the specified key; false otherwise.

Implements Content.

◆ is_subrange_equal()

bool is_subrange_equal ( const Index64 start,
const Index64 stop 
) const
overridevirtual

Returns 'true' if subranges are equal.

Implements Content.

◆ is_unique()

bool is_unique ( ) const
overridevirtual

Returns 'true' if all components of the array are unique.

Implements Content.

◆ kernels()

kernel::lib kernels ( ) const
overridevirtual

Returns the kernel library enum for all nested ptr_lib within the array's tree structure. If different libraries are associated with nodes of the same tree, this returns kernel::lib::size to indicate that it is "mixed".

Implements Content.

◆ key()

const std::string key ( int64_t  fieldindex) const
overridevirtual

The record name associated with a given field index or the tuple index as a string (e.g. "0", "1", "2") if a tuple.

Raises an error if the array does not contain a RecordArray.

Implements Content.

◆ keys()

const std::vector<std::string> keys ( ) const
overridevirtual

A list of RecordArray keys or an empty list if this array does not contain a RecordArray.

Implements Content.

◆ length()

int64_t length ( ) const
overridevirtual

The number of elements in the array.

Implements Content.

◆ localindex()

const ContentPtr localindex ( int64_t  axis,
int64_t  depth 
) const
overridevirtual

A (possibly nested) array of integers indicating the positions of elements within each nested list.

Parameters
axisThe nesting depth at which this operation is applied. If axis = 0, the output is simply an array of integers from 0 to length(). If axis = 1, the output has one level of nesting containing integers from 0 to the length of the nested list. Higher values of axis leave outer layers of the structure untouched. Negative axis counts backward from the deepest levels (-1 is the last valid axis).
depthThe current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis).

Implements Content.

◆ mergeable()

bool mergeable ( const ContentPtr other,
bool  mergebool 
) const
overridevirtual

Returns true if this array can be merged with the other; false otherwise.

The merge method will complete without errors if this function returns true.

Parameters
otherThe other array to merge with.
mergeboolIf true, consider boolean types to be equivalent to integers.

Implements Content.

◆ mergemany()

const ContentPtr mergemany ( const ContentPtrVec others) const
overridevirtual

Returns an array with this and the others concatenated (in order, this first, others last).

Implements Content.

◆ minmax_depth()

const std::pair<int64_t, int64_t> minmax_depth ( ) const
overridevirtual

Returns (a) the minimum list-depth and (b) the maximum list-depth of the array, which can differ if this array "branches" (differs when followed through different fields of a RecordArray or UnionArray).

Implements Content.

◆ nbytes_part()

void nbytes_part ( std::map< size_t, int64_t > &  largest) const
overridevirtual

Internal function used to calculate nbytes.

Parameters
largestThe largest range of bytes used in each reference-counted pointer (size_t).
Note
This method of accounting for overlapping buffers is insufficient: two nodes could use parts of the same buffer in which one doesn't completely overlap the other. It's not likely, but currently the calculation of nbytes is an underestimate.

The bytes of materialized arrays are not counted, so the nbytes of a VirtualArray is always zero.

Implements Content.

◆ num()

const ContentPtr num ( int64_t  axis,
int64_t  depth 
) const
overridevirtual

The length of this array (as a NumpyArray scalar) if axis = 0 or the lengths of subarrays (as an array or nested array) if axis != 0.

Parameters
axisThe axis whose length or lengths to quantify. Negative axis counts backward from the deepest levels (-1 is the last valid axis).
depthThe current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis).

Implements Content.

◆ numbers_to_type()

const ContentPtr numbers_to_type ( const std::string &  name) const
overridevirtual

Change the leaf types to 'totype'.

Implements Content.

◆ numfields()

int64_t numfields ( ) const
overridevirtual

The number of fields in the first nested tuple or records or -1 if this array does not contain a RecordArray.

Implements Content.

◆ offsets_and_flattened()

const std::pair<Index64, ContentPtr> offsets_and_flattened ( int64_t  axis,
int64_t  depth 
) const
overridevirtual

Returns (a) an offsets Index and (b) a flattened version of the array at some axis depth.

If axis > 1 (or its negative equivalent), the offsets is empty.

Parameters
axisThe axis to eliminate by flattening. axis = 0 is invalid. Negative axis counts backward from the deepest levels (-1 is the last valid axis).
depthThe current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis).

Implements Content.

◆ peek_array()

const ContentPtr peek_array ( ) const

Returns the array if it exists in the cache; nullptr otherwise.

This method does not cause the array to be materialized if it is not.

◆ ptr_lib()

const kernel::lib ptr_lib ( ) const

◆ purelist_depth()

int64_t purelist_depth ( ) const
overridevirtual

The list-depth of this array, not counting any contained within a RecordArray.

The purelist_depth of a Record is 0, and a RecordArray is 1 (regardless of what its fields contain).

If this array contains a UnionArray with different depths, the return value is -1.

Implements Content.

◆ reduce_next()

const ContentPtr reduce_next ( const Reducer reducer,
int64_t  negaxis,
const Index64 starts,
const Index64 shifts,
const Index64 parents,
int64_t  outlength,
bool  mask,
bool  keepdims 
) const
overridevirtual

This array with one axis removed by applying a Reducer (e.g. "sum", "max", "any", "all).

The user's entry point for this operation is reduce.

Parameters
reducerThe choice of Reducer algorithm.
negaxisThe negative axis: -axis. That is, negaxis = 1 means the deepest axis level.
startsStaring positions of each group to combine as an Index. These are downward pointers from an outer structure into this structure with the same meaning as in ListArray. Only used by a reducer that returns_positions (currently only argmin and argmax).
shiftsPer-element adjustments for any reducer that returns_positions. Allows for variable-length lists with axis != -1 and for missing values (None) in argmin and argmax.
parentsGroups to combine as an Index of upward pointers from this structure to the outer structure to reduce.
outlengthThe length of the reduced array, after the operation completes.
maskIf true, the Reducer's identity values will be covered by None using a ByteMaskedArray. This is desirable for ReducerMin, ReducerMax, ReducerArgmin, and ReducerArgmax to indicate that empty lists have no minimum or maximum.
keepdimsIf true, the reduced values will be wrapped by a singleton RegularArray to maintain the same number of dimensions in the output.

Implements Content.

◆ referentially_equal()

bool referentially_equal ( const ContentPtr other) const
overridevirtual

Returns true if this array has all the same buffers and parameters as other; false otherwise.

Parameters
otherThe array to compare this with.

Implements Content.

◆ rpad()

const ContentPtr rpad ( int64_t  target,
int64_t  axis,
int64_t  depth 
) const
overridevirtual

If axis = 0, returns a view of this array padded on the right with None values to have a minimum length; otherwise, returns an array with nested lists all padded to the minimum length.

Parameters
targetThe intended length. The output may be longer than this target length, but not shorter (using ListArray).
axisThe axis at which to apply padding. Negative axis counts backward from the deepest levels (-1 is the last valid axis).
depthThe current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis).

Implements Content.

◆ rpad_and_clip()

const ContentPtr rpad_and_clip ( int64_t  target,
int64_t  axis,
int64_t  depth 
) const
overridevirtual

If axis = 0, returns a view of this array padded on the right.

with None values to have exactly the specified length; otherwise, returns an array with nested lists all padded to the specified length.

Parameters
targetThe intended length; the output has exactly this target length (using RegularArray).
axisThe axis at which to apply padding. Negative axis counts backward from the deepest levels (-1 is the last valid axis).
depthThe current depth while stepping into the array: this value is set to 0 on the array node where the user starts the process and is increased at each level of list-depth (instead of decreasing the user-specified axis).

Implements Content.

◆ setidentities() [1/2]

void setidentities ( )
overridevirtual

Assign a surrogate index of Identities to this array (in-place).

This also assigns and possibly replaces Identities in nested arrays.

Note
This mutability is temporary: scikit-hep/awkward-1.0#116 Eventually, this interface will be deprecated and all Content instances will be immutable.

Implements Content.

◆ setidentities() [2/2]

void setidentities ( const IdentitiesPtr identities)
overridevirtual

Assign a specified set of Identities to this array (in-place).

This also assigns and possibly replaces Identities in nested arrays.

Note
This mutability is temporary: scikit-hep/awkward-1.0#116 Eventually, this interface will be deprecated and all Content instances will be immutable.

Implements Content.

◆ shallow_copy()

const ContentPtr shallow_copy ( ) const
overridevirtual

Copies this node without copying any nodes hierarchically nested within it or any array/index/identity buffers.

See also deep_copy.

Implements Content.

◆ shallow_simplify()

const ContentPtr shallow_simplify ( ) const
overridevirtual

Returns an equivalent array simplified at one level only using simplify_optiontype if an option-type array and simplify_uniontype if a union-type array.

For all other types of arrays, this operation is a pass-through.

Implements Content.

◆ sort_next()

const ContentPtr sort_next ( int64_t  negaxis,
const Index64 starts,
const Index64 parents,
int64_t  outlength,
bool  ascending,
bool  stable 
) const
overridevirtual

This array sorted.

The user's entry point for this operation is sort.

Parameters
negaxisThe negative axis: -axis. That is, negaxis = 1 means the deepest axis level.
startsStaring positions of each group to combine as an Index. These are downward pointers from an outer structure into this structure with the same meaning as in ListArray.
parentsGroups to combine as an Index of upward pointers from this structure to the outer structure to sort.
outlengthThe length of the array, after the operation completes.
ascendingIf true, the values will be sorted in an ascending order.
stableIf true, the values will be sorted by a stable sort algorithm to maintain the relative order of records with equal keys (i.e. values).

Implements Content.

◆ tojson_part()

void tojson_part ( ToJson builder,
bool  include_beginendlist 
) const
overridevirtual

Internal function to produce a JSON representation one node at a time.

Implements Content.

◆ tostring_part()

const std::string tostring_part ( const std::string &  indent,
const std::string &  pre,
const std::string &  post 
) const
overridevirtual

Internal function to build an output string for tostring.

Parameters
indentIndentation depth as a string of spaces.
prePrefix string, usually an opening XML tag.
postPostfix string, usually a closing XML tag and carriage return.

Implements Content.

◆ type()

const TypePtr type ( const util::TypeStrs typestrs) const
overridevirtual

High-level Type describing this array.

Parameters
typestrsA mapping from "__record__" parameters to string representations of those types, to override the derived strings.

Implements Content.

◆ unique()

const ContentPtr unique ( ) const
overridevirtual

Returns an array where all components are unique.

Implements Content.

◆ validityerror()

const std::string validityerror ( const std::string &  path) const
overridevirtual

Returns an error message if this array is invalid; otherwise, returns an empty string.

Implements Content.


The documentation for this class was generated from the following file: