import { DataTypeFieldConfig , Maybe , SortOrder , ReadonlyDataTypeFields , } from '@terascope/types' ; import { isPrimitiveValue , getHashCodeFrom , } from '@terascope/utils' ; import { inspect } from 'util' ; import { ReadableData , WritableData } from '../core' ; import { DataBuckets , SerializeOptions , VectorType } from './interfaces' ; /**
 * An immutable typed Array class with a constrained API.
*/ export abstract class Vector < T = unknown > { /**
     * Make an instance of a Vector from a config
    */ static make < R > ( data , options ) { throw new Error ( `This is overridden in the index file, ${ options } ${ data }` ) ; } /**
     * Sort the values in a Vector and return
     * an array with the updated indices.
    */ static getSortedIndices ( sortBy ) { const notSortable = sortBy . find ( ( { vector } ) => ! vector . sortable ) ; if ( notSortable ) { throw new Error ( `Sorting is not supported for ${ notSortable . vector ?. constructor . name }` ) ; } const len = sortBy . reduce ( ( size , { vector } ) => Math . max ( size , vector . size ) , 0 ) ; const indices = Array ( len ) ; const original = Array ( len ) ; for ( let i = 0 ; i < len ; i ++ ) { original [ i ] = [ i , sortBy . map ( ( { vector } ) => vector . get ( i ) ) ] ; } original . sort ( ( [ , a ] , [ , b ] ) => ( sortBy . reduce ( ( acc , { vector , direction : d } , i ) => { const res = vector . compare ( a [ i ] , b [ i ] ) ; return acc + ( d === 'asc' ? res : - res ) ; } , 0 ) ) ) . forEach ( ( [ i ] , newPosition ) => { indices [ i ] = newPosition ; } ) ; return indices ; } /**
     * The name of field, if specified this will just be used for metadata
    */ readonly name ? : string ; /**
     * The type of Vector, this should only be set the specific Vector type classes.
    */ readonly type : VectorType ; /**
     * The field type configuration
    */ readonly config : Readonly < DataTypeFieldConfig > ; /**
     * When Vector is an object type, this will be the data type fields
     * for the object
    */ readonly childConfig ? : ReadonlyDataTypeFields ; note internal readonly data : readonly ReadableData < T > [ ] ; /**
     * If set to false, the Vector is not sortable
    */ sortable = true ; /**
     * The cached size of the vector
    */ private __size : number | undefined ; /**
     * The cached consistent size of the vector
    */ private __consistentSize : number | undefined ; constructor ( /**
     * This will be set automatically by specific Vector classes
     */ type , data , options ) { this . type = type ; const res = getDataBuckets ( data ) ; this . data = res [ 0 ] ; this . __size = res [ 1 ] ; this . __consistentSize = res [ 2 ] ; this . name = options . name ; this . config = options . config ; this . childConfig = options . childConfig ; } /**
     * A function for converting an in-memory representation of
     * a value to an JSON spec compatible format.
    */ abstract toJSONCompatibleValue ? ( value , options ? ) ; /**
    * A function for converting an in-memory representation of
    * a value to an JSON spec compatible format.
    */ abstract getComparableValue ? ( value ) ; /**
     * Returns the number items in the Vector
    */ get size ( ) : number { if ( this . __size != null ) return this . __size ; this . _setSizeAndConsistentSize ( ) ; return this . __size ! ; } /**
     * Sometimes when appending data buckets the size
     * is consistent, when that happens we can be
     * much smarter about find the data bucket for a
     * given row, which speeds up some operations
    */ private get _consistentSize ( ) : number { if ( this . __consistentSize != null ) return this . __consistentSize ; this . _setSizeAndConsistentSize ( ) ; return this . __consistentSize ! ; } private _setSizeAndConsistentSize ( ) { this . __size = 0 ; let consistentSize ; for ( const bucket of this . data ) { if ( consistentSize == null ) { consistentSize = bucket . size ; } else if ( consistentSize !== bucket . size ) { consistentSize = - 1 ; } this . __size += bucket . size ; } this . __consistentSize = consistentSize ?? - 1 ; } * [ Symbol . iterator ] ( ) { for ( const data of this . data ) { yield * data ; } } /**
     * Check to see there are any nil values stored in the Vector
    */ hasNilValues ( ) { return this . data . some ( ( data ) => data . hasNilValues ( ) ) ; } /**
     * Get the number of non-nil values in the Vector
    */ countValues ( ) { return this . data . reduce ( ( acc , data ) => acc + data . countValues ( ) , 0 ) ; } /**
     * Returns true if there are no non-nil values
    */ isEmpty ( ) { if ( this . size === 0 ) return true ; return this . data . every ( ( data ) => data . isEmpty ( ) ) ; } /**
     * Iterate over the values and skip the nil ones,
     * returns tuples a with index and value
    */ * values ( ) { let offset = 0 ; for ( const data of this . data ) { for ( const [ i , v ] of data . entries ( ) ) { yield [ offset + i , v ] ; } offset += data . size ; } } note countUnique ( ) { let count = 0 ; for ( const v of this . unique ( ) ) { count ++ ; } return count ; } /**
     * Get the unique values with the index in a tuple form.
     * This is useful for reconstructing an Vector
     * with only the unique values
    */ * unique ( ) { if ( ! this . data . length ) return ; const hashes = new Set < any > ( ) ; const getHash = this . data [ 0 ] . isPrimitive ? ( v ) => v : getHashCodeFrom ; let offset = 0 ; for ( const data of this . data ) { for ( const [ index , value ] of data . entries ( ) ) { const hash = getHash ( value ) ; if ( ! hashes . has ( hash ) ) { hashes . add ( hash ) ; yield [ offset + index , value ] ; } } offset += data . size ; } } /**
    * Get the unique values, excluding nil values.
    * Useful for getting a list of unique values.
    */ * uniqueValues ( ) { if ( ! this . data . length ) return ; const hashes = new Set < any > ( ) ; const getHash = this . data [ 0 ] . isPrimitive ? ( v ) => v : getHashCodeFrom ; for ( const data of this . data ) { for ( const value of data . values ( ) ) { const hash = getHash ( value ) ; if ( ! hashes . has ( hash ) ) { hashes . add ( hash ) ; yield value ; } } } } /**
     * Add ReadableData to a end of the data buckets
    */ append ( data ) { if ( Array . isArray ( data ) ) { if ( ! data . length ) return this ; return this . fork ( this . data . concat ( data ) ) ; } const _singleData = data as ReadableData < T > ; if ( _singleData . size === 0 ) return this ; return this . fork ( this . data . concat ( [ data as ReadableData < T > ] ) ) ; } /**
    *  Add ReadableData to a beginning of the data buckets
    */ prepend ( data ) { const preData = ( Array . isArray ( data ) ? data : [ data as ReadableData < T > ] ) ; if ( preData . length === 0 ) return this ; return this . fork ( preData . concat ( this . data ) ) ; } /**
     * Get value by index
    */ get ( index , json ? , options ? ) { const nilValue = options ?. useNullForUndefined ? null : undefined ; const found = this . findDataWithIndex ( index ) ; if ( ! found ) return nilValue ; const val = found [ 0 ] . get ( found [ 1 ] ) ; if ( val == null ) return val ?? nilValue ; if ( ! json || ! this . toJSONCompatibleValue ) { return val ; } return this . toJSONCompatibleValue ( val as T , options ) ; } /**
     * Returns true if the value for that index is not nil
    */ has ( index ) { const found = this . findDataWithIndex ( index ) ; if ( ! found ) return false ; return found [ 0 ] . has ( found [ 1 ] ) ; } returns findDataWithIndex ( index ) { if ( index < 0 || this . data . length === 0 ) return ; if ( this . data . length === 1 ) { if ( index + 1 > this . size ) return ; return [ this . data [ 0 ] , index ] ; } if ( this . _consistentSize !== - 1 ) { return this . _findConsistentDataIndex ( index ) ; } if ( index > ( this . size / 2 ) ) { return this . _reverseFindDataWithIndex ( index ) ; } return this . _forwardFindDataWithIndex ( index ) ; } /**
     * When data buckets all of have the same size we
     * can use an O(1) optimization to find the correct
     * bucket. This helps when there are 1000s of buckets
    */ private _findConsistentDataIndex ( index ) { const bucketIndex = Math . floor ( index / this . _consistentSize ) ; const bucket = this . data [ bucketIndex ] ; if ( bucket == null ) { throw new Error ( `Unable to find bucket for ${ inspect ( { consistentSize : this . _consistentSize , bucketIndex , index } ) }` ) ; } return [ bucket , index % this . _consistentSize ] ; } /**
     * Find the Data bucket that holds the value for that
     * bucket
    */ private _forwardFindDataWithIndex ( index ) { let offset = 0 ; for ( const data of this . data ) { if ( index < ( data . size + offset ) ) { return [ data , index - offset ] ; } offset += data . size ; } } /**
     * Find the Data bucket that holds the value for that
     * bucket this works the in the reverse direction, this
     * will perform better when there are lots of buckets
    */ private _reverseFindDataWithIndex ( index ) { let offset = this . size ; const start = this . data . length - 1 ; for ( let i = start ; i >= 0 ; i -- ) { const data = this . data [ i ] ; offset -= data . size ; if ( index >= offset ) { return [ data , index - offset ] ; } } } /**
     * Create a new Vector with the same metadata but with different data
    */ fork ( data ) { const Constructor = this . constructor as { new ( data , config ) : Vector < T > ; } ; return new Constructor ( data , { childConfig : this . childConfig , config : this . config , name : this . name , } ) as this ; } /**
     * Create a new Vector with the range of values
    */ slice ( start = 0 , end = this . size ) { const startIndex = start < 0 ? this . size + start : start ; if ( startIndex < 0 || startIndex > this . size ) { throw new RangeError ( `Starting offset of ${ start } is out-of-bounds, must be >=0 OR <=${ this . size }` ) ; } const endIndex = end < 0 ? this . size + end : end ; if ( endIndex < 0 || endIndex > this . size ) { throw new RangeError ( `Ending offset of ${ end } is out-of-bounds, must be >=0 OR <=${ this . size }` ) ; } const returnSize = endIndex - startIndex ; let bucketIndex = 0 ; let totalProcessed = 0 ; let offset = 0 ; let hasStarted = false ; const buckets = [ ] ; while ( totalProcessed < returnSize ) { const bucket = this . data [ bucketIndex ] ; if ( bucket == null ) break ; const startIndexInBucket = hasStarted ? 0 : startIndex - offset ; const endIndexInBucket = Math . min ( endIndex - offset , bucket . size ) ; const totalFromBucket = endIndexInBucket - startIndexInBucket ; if ( startIndexInBucket >= 0 && totalFromBucket > 0 ) { const slicedBucket = new ReadableData ( bucket . slice ( startIndexInBucket , endIndexInBucket ) ) ; buckets . push ( slicedBucket ) ; hasStarted = true ; totalProcessed += slicedBucket . size ; } offset += bucket . size ; bucketIndex ++ ; } return this . fork ( buckets ) ; } /**
     * Compare two different values on the Vector type.
     * This can be used for equality or sorted.
    */ compare ( a , b ) { const aVal = this . _getComparableValue ( a ) ; const bVal = this . _getComparableValue ( b ) ; if ( aVal < bVal ) return - 1 ; if ( aVal > bVal ) return 1 ; return 0 ; } private _getComparableValue ( value ) { if ( value == null ) return null ; if ( this . getComparableValue ) { return this . getComparableValue ( value ) ; } if ( isPrimitiveValue ( value ) ) { return value ; } throw new Error ( 'Unable to convert value to number for comparison' ) ; } /**
     * Convert the Vector an array of values (the output is JSON compatible)
    */ toJSON ( options ? ) { const res = Array ( this . size ) ; for ( let i = 0 ; i < this . size ; i ++ ) { res [ i ] = this . get ( i , true , options ) ; } return res ; } note toArray ( ) { const res = Array ( this . size ) ; for ( let i = 0 ; i < this . size ; i ++ ) { res [ i ] = this . get ( i , false ) as T ; } return res ; } param size toWritable ( size ? ) { return WritableData . make < T > ( size ?? this . size , ( index ) => this . get ( index ) as Maybe < T > ) ; } } /**
 * This will get the data buckets with several optimizations when they
 * are either small or there are lots of data buckets
*/ function getDataBuckets < T > ( data ) { const len = data . length ; if ( len >= 1000 ) { let size = 0 ; for ( let i = 0 ; i < len ; i ++ ) { size += data [ i ] . size ; } const writable = new WritableData < T > ( size ) ; let writeIndex = 0 ; for ( let bucketIndex = 0 ; bucketIndex < len ; bucketIndex ++ ) { for ( const [ index , value ] of data [ bucketIndex ] . entries ( ) ) { writable . set ( writeIndex + index , value ) ; } writeIndex += data [ bucketIndex ] . size ; } return [ [ new ReadableData < T > ( writable ) ] , size , - 1 ] ; } if ( len === 1 ) { return [ data , data [ 0 ] . size , data [ 0 ] . size ] ; } if ( len <= 10 ) { let size = 0 ; for ( let i = 0 ; i < len ; i ++ ) { size += data [ i ] . size ; } return [ data , size , size === len ? 1 : undefined ] ; } return [ data , undefined , undefined ] ; } /**
 * Returns true if the input is a Vector
 */ export function isVector < T > ( input ) { return input instanceof Vector ; } /**
 * A list of Vector Options
 */ export interface VectorOptions { /**
    * The field config
    */ config : DataTypeFieldConfig | Readonly < DataTypeFieldConfig > ; /**
     * The type config for any nested fields (currently only works for objects)
    */ childConfig ? : ReadonlyDataTypeFields ; /**
     * The name of field, if specified this will just be used for metadata
    */ name ? : string ; }