morpheus.pipeline.preallocator_mixin.PreallocatorMixin

class PreallocatorMixin

Bases: ABC

Mixin intented to be added to stages, typically source stages, which are emitting newly constructed DataFrame or MessageMeta instances into the segment. During segment build, if the _needed_columns addtribute is not empty an additional node will be inserted into the graph after the derived class’ node which will perform the allocation.

The exceptions would be non-source stages like DFP’s DFPFileToDataFrameStage which are not sources but are constructing new Dataframe instances, and LinearBoundaryIngressStage which is potentially emitting other message types such as ControlMessage but it would still be the first stage in the given segment emitting the message.

Methods

set_needed_columns(needed_columns)

Sets the columns needed to perform preallocation.

set_needed_columns(needed_columns)

Sets the columns needed to perform preallocation. This should only be called by the Pipeline at build time. The needed_columns shoudl contain the entire set of columns needed by any other stage in this segment.