Class CompressedMatrixBlock
- java.lang.Object
-
- org.apache.sysds.runtime.matrix.data.MatrixValue
-
- org.apache.sysds.runtime.matrix.data.MatrixBlock
-
- org.apache.sysds.runtime.compress.CompressedMatrixBlock
-
- All Implemented Interfaces:
Externalizable,Serializable,Comparable,org.apache.hadoop.io.Writable,org.apache.hadoop.io.WritableComparable,CacheBlock
public class CompressedMatrixBlock extends MatrixBlock
- See Also:
- Serialized Form
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.sysds.runtime.matrix.data.MatrixBlock
MatrixBlock.SparsityEstimate
-
Nested classes/interfaces inherited from class org.apache.sysds.runtime.matrix.data.MatrixValue
MatrixValue.CellIndex
-
-
Field Summary
-
Fields inherited from class org.apache.sysds.runtime.matrix.data.MatrixBlock
CONVERT_MCSR_TO_CSR_ON_DEEP_SERIALIZE, DEFAULT_INPLACE_SPARSEBLOCK, DEFAULT_SPARSEBLOCK, HEADER_SIZE, MAX_SHALLOW_SERIALIZE_OVERHEAD, SPARSITY_TURN_POINT, ULTRA_SPARSE_BLOCK_NNZ, ULTRA_SPARSITY_TURN_POINT, ULTRA_SPARSITY_TURN_POINT2
-
-
Constructor Summary
Constructors Constructor Description CompressedMatrixBlock()CompressedMatrixBlock(int rl, int cl)Main constructor for building a block from scratch.CompressedMatrixBlock(CompressedMatrixBlock that)Copy constructor taking that CompressedMatrixBlock and populate this new compressedMatrixBlock with pointers to the same columnGroups.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description MatrixBlockaggregateBinaryOperations(MatrixBlock m1, MatrixBlock m2, MatrixBlock ret, AggregateBinaryOperator op)MatrixBlockaggregateBinaryOperations(MatrixBlock m1, MatrixBlock m2, MatrixBlock ret, AggregateBinaryOperator op, boolean transposeLeft, boolean transposeRight)MatrixBlockaggregateTernaryOperations(MatrixBlock m1, MatrixBlock m2, MatrixBlock m3, MatrixBlock ret, AggregateTernaryOperator op, boolean inCP)MatrixBlockaggregateUnaryOperations(AggregateUnaryOperator op, MatrixValue result, int blen, MatrixIndexes indexesIn, boolean inCP)voidallocateAndResetSparseBlock(boolean clearNNZ, SparseBlock.Type stype)Future<MatrixBlock>allocateBlockAsync()voidallocateColGroup(AColGroup cg)Allocate the given column group and remove all references to old column groups.voidallocateColGroupList(List<AColGroup> colGroups)Replace the column groups in this CompressedMatrixBlock with the given column groupsbooleanallocateDenseBlock(boolean clearNNZ)booleanallocateSparseRowsBlock(boolean clearNNZ)MatrixBlockappend(MatrixBlock[] that, MatrixBlock ret, boolean cbind)voidappend(MatrixValue v2, ArrayList<IndexedMatrixValue> outlist, int blen, boolean cbind, boolean m2IsLast, int nextNCol)voidappendRow(int r, SparseRow row, boolean deep)voidappendRowToSparse(SparseBlock dest, MatrixBlock src, int i, int rowoffset, int coloffset, boolean deep)voidappendValue(int r, int c, double v)Append value is only used when values are appended at the end of each row for the sparse representation This can only be called, when the caller knows the access pattern of the blockvoidappendValuePlain(int r, int c, double v)static longbaseSizeInMemory()MatrixBlockbinaryOperations(BinaryOperator op, MatrixValue thatValue, MatrixValue result)MatrixBlockbinaryOperationsInPlace(BinaryOperator op, MatrixValue thatValue)MatrixBlockbinaryOperationsLeft(BinaryOperator op, MatrixValue thatValue, MatrixValue result)MatrixBlockchainMatrixMultOperations(MatrixBlock v, MatrixBlock w, MatrixBlock out, MapMultChain.ChainType ctype, int k)voidcheckNaN()Checks for existing NaN values in the matrix block.voidclearSoftReferenceToDecompressed()CM_COV_ObjectcmOperations(CMOperator op)CM_COV_ObjectcmOperations(CMOperator op, MatrixBlock weights)MatrixBlockcolSum()Wrapper method for single threaded reduceall-colSum of a matrix.voidcompactEmptyBlock()Free unnecessarily allocated empty block.booleancontainsValue(double pattern)voidcopy(int rl, int ru, int cl, int cu, MatrixBlock src, boolean awareDestNZ)In-place copy of matrix src into the index range of the existing current matrix.voidcopy(MatrixValue thatValue)Copy that MatrixValue into this MatrixValue.voidcopy(MatrixValue thatValue, boolean sp)Copy that MatrixValue into this MatrixValue.MatrixBlockcopyShallow(MatrixBlock that)CM_COV_ObjectcovOperations(COVOperator op, MatrixBlock that)CM_COV_ObjectcovOperations(COVOperator op, MatrixBlock that, MatrixBlock weights)voidctableOperations(Operator op, double scalar, double scalar2, CTableMap resultMap, MatrixBlock resultBlock)D = ctable(A,v2,w) this <- A; scalar_that <- v2; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (w) from scalar_input3 (scalarThat2)voidctableOperations(Operator op, double scalar, MatrixValue that, CTableMap resultMap, MatrixBlock resultBlock)D = ctable(A,v2,W) this <- A; scalarThat <- v2; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (i3,j3,w) from input3 (that2)voidctableOperations(Operator op, MatrixIndexes ix1, double scalar, boolean left, int brlen, CTableMap resultMap, MatrixBlock resultBlock)Specific ctable case of ctable(seq(...),X), where X is the only matrix input.voidctableOperations(Operator op, MatrixValue that, double scalar, boolean ignoreZeros, CTableMap resultMap, MatrixBlock resultBlock)D = ctable(A,B,w) this <- A; that <- B; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2) NOTE: This method supports both vectors and matrices.voidctableOperations(Operator op, MatrixValue that, MatrixValue that2, CTableMap resultMap, MatrixBlock resultBlock)MatrixBlockctableSeqOperations(MatrixValue thatMatrix, double thatScalar, MatrixBlock resultBlock, boolean updateClen)MatrixBlockdecompress()Decompress block into a MatrixBlockMatrixBlockdecompress(int k)Decompress block into a MatrixBlockvoiddropLastRowsOrColumns(Types.CorrectionLocationType correctionLocation)longestimateCompressedSizeInMemory()Obtain an upper bound on the memory used to store the compressed block.longestimateSizeInMemory()longestimateSizeOnDisk()booleanevalSparseFormatInMemory()Evaluates if this matrix block should be in sparse format in memory.booleanevalSparseFormatOnDisk()Evaluates if this matrix block should be in sparse format on disk.voidexamSparsity(boolean allowCSR)Evaluates if this matrix block should be in sparse format in memory.MatrixBlockextractTriangular(MatrixBlock ret, boolean lower, boolean diag, boolean values)MatrixBlockgetCachedDecompressed()Get the cached decompressed matrix (if it exists otherwise null).List<AColGroup>getColGroups()Get the column groups of this CompressedMatrixBlockDenseBlockgetDenseBlock()double[]getDenseBlockValues()longgetExactSizeOnDisk()NOTE: The used estimates must be kept consistent with the respective write functions.longgetInMemorySize()Get the in-memory size in bytes of the cache block.SoftReference<MatrixBlock>getSoftReferenceToDecompressed()SparseBlockgetSparseBlock()Iterator<IJV>getSparseBlockIterator()Iterator<IJV>getSparseBlockIterator(int rl, int ru)MatrixBlockgetUncompressed()MatrixBlockgetUncompressed(String operation)MatrixBlockgetUncompressed(String operation, int k)static MatrixBlockgetUncompressed(MatrixValue mVal)static MatrixBlockgetUncompressed(MatrixValue mVal, String message)doublegetValueDenseUnsafe(int r, int c)MatrixBlockgroupedAggOperations(MatrixValue tgt, MatrixValue wghts, MatrixValue ret, int ngroups, Operator op, int k)voidincrementalAggregate(AggregateOperator aggOp, MatrixValue newWithCorrection)voidincrementalAggregate(AggregateOperator aggOp, MatrixValue correction, MatrixValue newWithCorrection, boolean deep)voidinit(double[][] arr, int r, int c)NOTE: This method is designed only for dense representation.voidinit(double[] arr, int r, int c)NOTE: This method is designed only for dense representation.doubleinterQuartileMean()booleanisAllocated()booleanisEmptyBlock(boolean safe)booleanisInSparseFormat()Returns the current representation (true for sparse).booleanisOverlapping()booleanisShallowSerialize(boolean inclConvert)Indicates if the cache block is subject to shallow serialized, which is generally true if in-memory size and serialized size are almost identical allowing to avoid unnecessary deep serialize.booleanisSparsePermutationMatrix()booleanisThreadSafe()Indicates if concurrent modifications of disjoint rows are thread-safe.booleanisUltraSparse()booleanisUltraSparse(boolean checkNnz)MatrixBlockleftIndexingOperations(ScalarObject scalar, int rl, int cl, MatrixBlock ret, MatrixObject.UpdateType update)Explicitly allow left indexing for scalars.MatrixBlockleftIndexingOperations(MatrixBlock rhsMatrix, int rl, int ru, int cl, int cu, MatrixBlock ret, MatrixObject.UpdateType update)doublemax()Wrapper method for reduceall-max of a matrix.doublemean()Wrapper method for reduceall-mean of a matrix.voidmerge(MatrixBlock that, boolean appendOnly, boolean par, boolean deep)doublemin()Wrapper method for reduceall-min of a matrix.doubleminNonZero()Utility function for computing the min non-zero value.voidpermutationMatrixMultOperations(MatrixValue m2Val, MatrixValue out1Val, MatrixValue out2Val, int k)doublepickValue(double quantile, boolean average)MatrixBlockpickValues(MatrixValue quantiles, MatrixValue ret)doubleprod()Wrapper method for reduceall-product of a matrix.voidputInto(MatrixBlock target, int rowOffset, int colOffset, boolean sparseCopyShallow)Method for copying this matrix into a target matrix.MatrixBlockquaternaryOperations(QuaternaryOperator qop, MatrixBlock um, MatrixBlock vm, MatrixBlock wm, MatrixBlock out, int k)doublequickGetValue(int r, int c)doublequickGetValueThreadSafe(int r, int c)voidquickSetValue(int r, int c, double v)MatrixBlockrandOperationsInPlace(RandomMatrixGenerator rgen, org.apache.commons.math3.random.Well1024a bigrand, long bSeed)Function to generate a matrix of random numbers.MatrixBlockrandOperationsInPlace(RandomMatrixGenerator rgen, org.apache.commons.math3.random.Well1024a bigrand, long bSeed, int k)Function to generate a matrix of random numbers.voidreadExternal(ObjectInput is)Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd de-serialization.voidreadFields(DataInput in)longrecomputeNonZeros()Recomputes and materializes the number of non-zero values of the entire matrix block.longrecomputeNonZeros(int rl, int ru)longrecomputeNonZeros(int rl, int ru, int cl, int cu)Recomputes the number of non-zero values of a specified range of the matrix block.MatrixBlockremoveEmptyOperations(MatrixBlock ret, boolean rows, boolean emptyReturn, MatrixBlock select)MatrixBlockreorgOperations(ReorgOperator op, MatrixValue ret, int startRow, int startColumn, int length)MatrixBlockreplaceOperations(MatrixValue result, double pattern, double replacement)voidreset(int rl, int cl, boolean sp, long estnnz, double val)Internal canonical reset of dense and sparse matrix blocks.MatrixBlockrexpandOperations(MatrixBlock ret, double max, boolean rows, boolean cast, boolean ignore, int k)MatrixBlockscalarOperations(ScalarOperator sop, MatrixValue result)MatrixBlockseqOperationsInPlace(double from, double to, double incr)voidsetDenseBlock(DenseBlock dblock)voidsetOverlapping(boolean overlapping)voidsetSparseBlock(SparseBlock sblock)MatrixBlockslice(int rl, int ru, int cl, int cu, boolean deep, CacheBlock ret)Method to perform rightIndex operation for a given lower and upper bounds in row and column dimensions.voidslice(ArrayList<IndexedMatrixValue> outlist, IndexRange range, int rowCut, int colCut, int blen, int boundaryRlen, int boundaryClen)Slice out up to 4 matrixBlocks that are separated by the row and col Cuts.MatrixBlocksortOperations(MatrixValue weights, MatrixBlock result)voidsortSparseRows()Sorts all existing sparse rows by column indexes.voidsortSparseRows(int rl, int ru)Sorts all existing sparse rows in range [rl,ru) by column indexes.voidsparseToDense()CompressedMatrixBlocksquash(int k)doublesum()Wrapper method for reduceall-sum of a matrix.doublesumSq()Wrapper method for reduceall-sumSq of a matrix.doublesumWeightForQuantile()In a given two column matrix, the second column denotes weights.MatrixBlockternaryOperations(TernaryOperator op, MatrixBlock m2, MatrixBlock m3, MatrixBlock ret)voidtoShallowSerializeBlock()Converts a cache block that is not shallow serializable into a form that is shallow serializable.StringtoString()MatrixBlocktransposeSelfMatrixMultOperations(MatrixBlock out, MMTSJ.MMTSJType tstype, int k)MatrixBlockuaggouterchainOperations(MatrixBlock mbLeft, MatrixBlock mbRight, MatrixBlock mbOut, BinaryOperator bOp, AggregateUnaryOperator uaggOp)MatrixBlockunaryOperations(UnaryOperator op, MatrixValue result)voidwrite(DataOutput out)voidwriteExternal(ObjectOutput os)Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd serialization.MatrixBlockzeroOutOperations(MatrixValue result, IndexRange range, boolean complementary)-
Methods inherited from class org.apache.sysds.runtime.matrix.data.MatrixBlock
aggregateBinaryOperations, aggregateUnaryOperations, allocateBlock, allocateDenseBlock, allocateDenseBlockUnsafe, allocateSparseRowsBlock, append, append, appendRow, appendToSparse, binaryOperations, chainMatrixMultOperations, checkCMOperations, checkDimensionsForAppend, checkNonZeros, checkSparseRows, checkSparseRows, cleanupBlock, colMax, colMin, compareTo, computeIQMCorrection, ctableOperations, ctableSeqOperations, denseToSparse, equals, estimateSizeDenseInMemory, estimateSizeDenseInMemory, estimateSizeInMemory, estimateSizeOnDisk, estimateSizeSparseInMemory, estimateSizeSparseInMemory, estimateSizeSparseInMemory, estimateSizeSparseInMemory, estimateSizeSparseInMemory, evalSparseFormatInMemory, evalSparseFormatInMemory, evalSparseFormatInMemory, evalSparseFormatInMemory, evalSparseFormatOnDisk, examSparsity, getDataCharacteristics, getDouble, getDoubleNaN, getExactSerializedSize, getHeaderSize, getLength, getNonZeros, getNumColumns, getNumRows, getSparsity, getString, getValue, groupedAggOperations, hashCode, isEmpty, isEmptyBlock, isShallowSerialize, isThreadSafe, isVector, leftIndexingOperations, median, merge, merge, merge, naryOperations, permutationMatrixMultOperations, pickValue, prepareAggregateUnaryOutput, quaternaryOperations, randOperations, randOperations, randOperations, randOperations, removeEmptyOperations, reset, reset, reset, reset, reset, reset, rowSum, rowSum, sampleOperations, seqOperations, setAllNonZeros, setNonZeros, setNumColumns, setNumRows, setValue, slice, slice, slice, slice, slice, slice, sortOperations, sortOperations, sortOperations, transposeSelfMatrixMultOperations
-
-
-
-
Constructor Detail
-
CompressedMatrixBlock
public CompressedMatrixBlock()
-
CompressedMatrixBlock
public CompressedMatrixBlock(int rl, int cl)Main constructor for building a block from scratch. Use with caution, since it constructs an empty matrix block with nothing inside.- Parameters:
rl- number of rows in the blockcl- number of columns
-
CompressedMatrixBlock
public CompressedMatrixBlock(CompressedMatrixBlock that)
Copy constructor taking that CompressedMatrixBlock and populate this new compressedMatrixBlock with pointers to the same columnGroups.- Parameters:
that- CompressedMatrixBlock to copy values from
-
-
Method Detail
-
reset
public void reset(int rl, int cl, boolean sp, long estnnz, double val)Description copied from class:MatrixBlockInternal canonical reset of dense and sparse matrix blocks.- Overrides:
resetin classMatrixBlock- Parameters:
rl- number of rowscl- number of columnssp- sparse representationestnnz- estimated number of non-zerosval- initialization value
-
allocateColGroup
public void allocateColGroup(AColGroup cg)
Allocate the given column group and remove all references to old column groups. This is done by simply allocating a ned _colGroups list and adding the given column group- Parameters:
cg- The column group to use after.
-
allocateColGroupList
public void allocateColGroupList(List<AColGroup> colGroups)
Replace the column groups in this CompressedMatrixBlock with the given column groups- Parameters:
colGroups- new ColGroups in the MatrixBlock
-
getColGroups
public List<AColGroup> getColGroups()
Get the column groups of this CompressedMatrixBlock- Returns:
- the column groups
-
decompress
public MatrixBlock decompress()
Decompress block into a MatrixBlock- Returns:
- a new uncompressed matrix block containing the contents of this block
-
decompress
public MatrixBlock decompress(int k)
Decompress block into a MatrixBlock- Parameters:
k- degree of parallelism- Returns:
- a new uncompressed matrix block containing the contents of this block
-
putInto
public void putInto(MatrixBlock target, int rowOffset, int colOffset, boolean sparseCopyShallow)
Description copied from class:MatrixBlockMethod for copying this matrix into a target matrix. Note that this method does not maintain number of non zero values. The method should output into the allocated block type of the target, therefore before any calls an appropriate block must be allocated. CSR sparse format is not supported. If allocating into a sparse matrix MCSR block the rows have to be sorted afterwards with a call to target.sortSparseRows()- Overrides:
putIntoin classMatrixBlock- Parameters:
target- Target MatrixBlock, that can be allocated dense or sparserowOffset- The Row offset to allocate into.colOffset- The column offset to allocate into.sparseCopyShallow- If the output is sparse, and shallow copy of rows is allowed from this block
-
getCachedDecompressed
public MatrixBlock getCachedDecompressed()
Get the cached decompressed matrix (if it exists otherwise null). This in practice means that if some other instruction have materialized the decompressed version it can be accessed though this method with a guarantee that it did not go through the entire decompression phase.- Returns:
- The cached decompressed matrix, if it does not exist return null
-
squash
public CompressedMatrixBlock squash(int k)
-
recomputeNonZeros
public long recomputeNonZeros()
Description copied from class:MatrixBlockRecomputes and materializes the number of non-zero values of the entire matrix block.- Overrides:
recomputeNonZerosin classMatrixBlock- Returns:
- number of non-zeros
-
recomputeNonZeros
public long recomputeNonZeros(int rl, int ru)- Overrides:
recomputeNonZerosin classMatrixBlock
-
recomputeNonZeros
public long recomputeNonZeros(int rl, int ru, int cl, int cu)Description copied from class:MatrixBlockRecomputes the number of non-zero values of a specified range of the matrix block. NOTE: This call does not materialize the compute result in any form.- Overrides:
recomputeNonZerosin classMatrixBlock- Parameters:
rl- row lower index, 0-based, inclusiveru- row upper index, 0-based, inclusivecl- column lower index, 0-based, inclusivecu- column upper index, 0-based, inclusive- Returns:
- the number of non-zero values
-
getInMemorySize
public long getInMemorySize()
Description copied from interface:CacheBlockGet the in-memory size in bytes of the cache block.- Specified by:
getInMemorySizein interfaceCacheBlock- Overrides:
getInMemorySizein classMatrixBlock- Returns:
- in-memory size in bytes of cache block
-
estimateSizeInMemory
public long estimateSizeInMemory()
- Overrides:
estimateSizeInMemoryin classMatrixBlock
-
estimateCompressedSizeInMemory
public long estimateCompressedSizeInMemory()
Obtain an upper bound on the memory used to store the compressed block.- Returns:
- an upper bound on the memory used to store this compressed block considering class overhead.
-
baseSizeInMemory
public static long baseSizeInMemory()
-
quickGetValue
public double quickGetValue(int r, int c)- Overrides:
quickGetValuein classMatrixBlock
-
getExactSizeOnDisk
public long getExactSizeOnDisk()
Description copied from class:MatrixBlockNOTE: The used estimates must be kept consistent with the respective write functions.- Overrides:
getExactSizeOnDiskin classMatrixBlock- Returns:
- exact size on disk
-
estimateSizeOnDisk
public long estimateSizeOnDisk()
- Overrides:
estimateSizeOnDiskin classMatrixBlock
-
readFields
public void readFields(DataInput in) throws IOException
- Specified by:
readFieldsin interfaceorg.apache.hadoop.io.Writable- Overrides:
readFieldsin classMatrixBlock- Throws:
IOException
-
write
public void write(DataOutput out) throws IOException
- Specified by:
writein interfaceorg.apache.hadoop.io.Writable- Overrides:
writein classMatrixBlock- Throws:
IOException
-
readExternal
public void readExternal(ObjectInput is) throws IOException
Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd de-serialization.- Specified by:
readExternalin interfaceExternalizable- Overrides:
readExternalin classMatrixBlock- Parameters:
is- object input- Throws:
IOException- if IOException occurs
-
writeExternal
public void writeExternal(ObjectOutput os) throws IOException
Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd serialization.- Specified by:
writeExternalin interfaceExternalizable- Overrides:
writeExternalin classMatrixBlock- Parameters:
os- object output- Throws:
IOException- if IOException occurs
-
scalarOperations
public MatrixBlock scalarOperations(ScalarOperator sop, MatrixValue result)
- Overrides:
scalarOperationsin classMatrixBlock
-
binaryOperations
public MatrixBlock binaryOperations(BinaryOperator op, MatrixValue thatValue, MatrixValue result)
- Overrides:
binaryOperationsin classMatrixBlock
-
binaryOperationsLeft
public MatrixBlock binaryOperationsLeft(BinaryOperator op, MatrixValue thatValue, MatrixValue result)
-
append
public MatrixBlock append(MatrixBlock[] that, MatrixBlock ret, boolean cbind)
- Overrides:
appendin classMatrixBlock
-
append
public void append(MatrixValue v2, ArrayList<IndexedMatrixValue> outlist, int blen, boolean cbind, boolean m2IsLast, int nextNCol)
- Overrides:
appendin classMatrixBlock
-
chainMatrixMultOperations
public MatrixBlock chainMatrixMultOperations(MatrixBlock v, MatrixBlock w, MatrixBlock out, MapMultChain.ChainType ctype, int k)
- Overrides:
chainMatrixMultOperationsin classMatrixBlock
-
aggregateBinaryOperations
public MatrixBlock aggregateBinaryOperations(MatrixBlock m1, MatrixBlock m2, MatrixBlock ret, AggregateBinaryOperator op)
- Overrides:
aggregateBinaryOperationsin classMatrixBlock
-
aggregateBinaryOperations
public MatrixBlock aggregateBinaryOperations(MatrixBlock m1, MatrixBlock m2, MatrixBlock ret, AggregateBinaryOperator op, boolean transposeLeft, boolean transposeRight)
-
aggregateUnaryOperations
public MatrixBlock aggregateUnaryOperations(AggregateUnaryOperator op, MatrixValue result, int blen, MatrixIndexes indexesIn, boolean inCP)
- Overrides:
aggregateUnaryOperationsin classMatrixBlock
-
transposeSelfMatrixMultOperations
public MatrixBlock transposeSelfMatrixMultOperations(MatrixBlock out, MMTSJ.MMTSJType tstype, int k)
- Overrides:
transposeSelfMatrixMultOperationsin classMatrixBlock
-
replaceOperations
public MatrixBlock replaceOperations(MatrixValue result, double pattern, double replacement)
- Overrides:
replaceOperationsin classMatrixBlock
-
reorgOperations
public MatrixBlock reorgOperations(ReorgOperator op, MatrixValue ret, int startRow, int startColumn, int length)
- Overrides:
reorgOperationsin classMatrixBlock
-
toString
public String toString()
- Overrides:
toStringin classMatrixBlock
-
isOverlapping
public boolean isOverlapping()
-
setOverlapping
public void setOverlapping(boolean overlapping)
-
slice
public MatrixBlock slice(int rl, int ru, int cl, int cu, boolean deep, CacheBlock ret)
Description copied from class:MatrixBlockMethod to perform rightIndex operation for a given lower and upper bounds in row and column dimensions. Extracted submatrix is returned as "result". Note: This operation is now 0-based. This means that if you call with rl == ru then you get 1 row output.- Specified by:
slicein interfaceCacheBlock- Overrides:
slicein classMatrixBlock- Parameters:
rl- row lower if this value is below 0 or above the number of rows contained in the matrix an exception is thrownru- row upper if this value is below rl or above the number of rows contained in the matrix an exception is throwncl- column lower if this value us below 0 or above the number of columns contained in the matrix an exception is throwncu- column upper if this value us below cl or above the number of columns contained in the matrix an exception is throwndeep- should perform deep copy, this is relevant in cases where the matrix is in sparse format, or the entire matrix is sliced outret- output sliced out matrix block- Returns:
- matrix block output matrix block
-
slice
public void slice(ArrayList<IndexedMatrixValue> outlist, IndexRange range, int rowCut, int colCut, int blen, int boundaryRlen, int boundaryClen)
Description copied from class:MatrixValueSlice out up to 4 matrixBlocks that are separated by the row and col Cuts. This is used in the context of spark execution to distributed sliced out matrix blocks of correct block size.- Overrides:
slicein classMatrixBlock- Parameters:
outlist- The output matrix blocks that is extracted from the matrixrange- An index range containing overlapping information.rowCut- The row to cut and split the matrix.colCut- The column to cut ans split the matrix.blen- The Block size of the output matrices.boundaryRlen- The row length of the edge case matrix block, used for the final blocks that does not have enough rows to construct a full block.boundaryClen- The col length of the edge case matrix block, used for the final blocks that does not have enough cols to construct a full block.
-
unaryOperations
public MatrixBlock unaryOperations(UnaryOperator op, MatrixValue result)
- Overrides:
unaryOperationsin classMatrixBlock
-
containsValue
public boolean containsValue(double pattern)
- Overrides:
containsValuein classMatrixBlock
-
max
public double max()
Description copied from class:MatrixBlockWrapper method for reduceall-max of a matrix.- Overrides:
maxin classMatrixBlock- Returns:
- the maximum value of all values in the matrix
-
min
public double min()
Description copied from class:MatrixBlockWrapper method for reduceall-min of a matrix.- Overrides:
minin classMatrixBlock- Returns:
- the minimum value of all values in the matrix
-
sum
public double sum()
Description copied from class:MatrixBlockWrapper method for reduceall-sum of a matrix.- Overrides:
sumin classMatrixBlock- Returns:
- Sum of the values in the matrix.
-
colSum
public MatrixBlock colSum()
Description copied from class:MatrixBlockWrapper method for single threaded reduceall-colSum of a matrix.- Overrides:
colSumin classMatrixBlock- Returns:
- A new MatrixBlock containing the column sums of this matrix.
-
sumSq
public double sumSq()
Description copied from class:MatrixBlockWrapper method for reduceall-sumSq of a matrix.- Overrides:
sumSqin classMatrixBlock- Returns:
- Sum of the squared values in the matrix.
-
prod
public double prod()
Description copied from class:MatrixBlockWrapper method for reduceall-product of a matrix.- Overrides:
prodin classMatrixBlock- Returns:
- the product sum of the matrix content
-
mean
public double mean()
Description copied from class:MatrixBlockWrapper method for reduceall-mean of a matrix.- Overrides:
meanin classMatrixBlock- Returns:
- the mean value of all values in the matrix
-
rexpandOperations
public MatrixBlock rexpandOperations(MatrixBlock ret, double max, boolean rows, boolean cast, boolean ignore, int k)
- Overrides:
rexpandOperationsin classMatrixBlock
-
isEmptyBlock
public boolean isEmptyBlock(boolean safe)
- Overrides:
isEmptyBlockin classMatrixBlock
-
binaryOperationsInPlace
public MatrixBlock binaryOperationsInPlace(BinaryOperator op, MatrixValue thatValue)
- Overrides:
binaryOperationsInPlacein classMatrixBlock
-
incrementalAggregate
public void incrementalAggregate(AggregateOperator aggOp, MatrixValue correction, MatrixValue newWithCorrection, boolean deep)
- Overrides:
incrementalAggregatein classMatrixBlock
-
incrementalAggregate
public void incrementalAggregate(AggregateOperator aggOp, MatrixValue newWithCorrection)
- Overrides:
incrementalAggregatein classMatrixBlock
-
permutationMatrixMultOperations
public void permutationMatrixMultOperations(MatrixValue m2Val, MatrixValue out1Val, MatrixValue out2Val, int k)
- Overrides:
permutationMatrixMultOperationsin classMatrixBlock
-
leftIndexingOperations
public MatrixBlock leftIndexingOperations(MatrixBlock rhsMatrix, int rl, int ru, int cl, int cu, MatrixBlock ret, MatrixObject.UpdateType update)
- Overrides:
leftIndexingOperationsin classMatrixBlock
-
leftIndexingOperations
public MatrixBlock leftIndexingOperations(ScalarObject scalar, int rl, int cl, MatrixBlock ret, MatrixObject.UpdateType update)
Description copied from class:MatrixBlockExplicitly allow left indexing for scalars. Note: This operation is now 0-based. * Operations to be performed: 1) result=this; 2) result[row,column] = scalar.getDoubleValue();- Overrides:
leftIndexingOperationsin classMatrixBlock- Parameters:
scalar- scalar objectrl- row lowercl- column lowerret- ?update- ?- Returns:
- matrix block
-
zeroOutOperations
public MatrixBlock zeroOutOperations(MatrixValue result, IndexRange range, boolean complementary)
- Overrides:
zeroOutOperationsin classMatrixBlock
-
cmOperations
public CM_COV_Object cmOperations(CMOperator op)
- Overrides:
cmOperationsin classMatrixBlock
-
cmOperations
public CM_COV_Object cmOperations(CMOperator op, MatrixBlock weights)
- Overrides:
cmOperationsin classMatrixBlock
-
covOperations
public CM_COV_Object covOperations(COVOperator op, MatrixBlock that)
- Overrides:
covOperationsin classMatrixBlock
-
covOperations
public CM_COV_Object covOperations(COVOperator op, MatrixBlock that, MatrixBlock weights)
- Overrides:
covOperationsin classMatrixBlock
-
sortOperations
public MatrixBlock sortOperations(MatrixValue weights, MatrixBlock result)
- Overrides:
sortOperationsin classMatrixBlock
-
aggregateTernaryOperations
public MatrixBlock aggregateTernaryOperations(MatrixBlock m1, MatrixBlock m2, MatrixBlock m3, MatrixBlock ret, AggregateTernaryOperator op, boolean inCP)
- Overrides:
aggregateTernaryOperationsin classMatrixBlock
-
uaggouterchainOperations
public MatrixBlock uaggouterchainOperations(MatrixBlock mbLeft, MatrixBlock mbRight, MatrixBlock mbOut, BinaryOperator bOp, AggregateUnaryOperator uaggOp)
- Overrides:
uaggouterchainOperationsin classMatrixBlock
-
groupedAggOperations
public MatrixBlock groupedAggOperations(MatrixValue tgt, MatrixValue wghts, MatrixValue ret, int ngroups, Operator op, int k)
- Overrides:
groupedAggOperationsin classMatrixBlock
-
removeEmptyOperations
public MatrixBlock removeEmptyOperations(MatrixBlock ret, boolean rows, boolean emptyReturn, MatrixBlock select)
- Overrides:
removeEmptyOperationsin classMatrixBlock
-
ctableOperations
public void ctableOperations(Operator op, double scalar, MatrixValue that, CTableMap resultMap, MatrixBlock resultBlock)
Description copied from class:MatrixBlockD = ctable(A,v2,W) this <- A; scalarThat <- v2; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (i3,j3,w) from input3 (that2)- Overrides:
ctableOperationsin classMatrixBlock
-
ctableOperations
public void ctableOperations(Operator op, double scalar, double scalar2, CTableMap resultMap, MatrixBlock resultBlock)
Description copied from class:MatrixBlockD = ctable(A,v2,w) this <- A; scalar_that <- v2; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (w) from scalar_input3 (scalarThat2)- Overrides:
ctableOperationsin classMatrixBlock
-
ctableOperations
public void ctableOperations(Operator op, MatrixIndexes ix1, double scalar, boolean left, int brlen, CTableMap resultMap, MatrixBlock resultBlock)
Description copied from class:MatrixBlockSpecific ctable case of ctable(seq(...),X), where X is the only matrix input. The 'left' input parameter specifies if the seq appeared on the left, otherwise it appeared on the right.- Overrides:
ctableOperationsin classMatrixBlock
-
ctableOperations
public void ctableOperations(Operator op, MatrixValue that, double scalar, boolean ignoreZeros, CTableMap resultMap, MatrixBlock resultBlock)
Description copied from class:MatrixBlockD = ctable(A,B,w) this <- A; that <- B; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2) NOTE: This method supports both vectors and matrices. In case of matrices and ignoreZeros=true we can also use a sparse-safe implementation- Overrides:
ctableOperationsin classMatrixBlock
-
ctableSeqOperations
public MatrixBlock ctableSeqOperations(MatrixValue thatMatrix, double thatScalar, MatrixBlock resultBlock, boolean updateClen)
- Overrides:
ctableSeqOperationsin classMatrixBlock- Parameters:
thatMatrix- matrix valuethatScalar- scalar doubleresultBlock- result matrix blockupdateClen- when this matrix already has the desired number of columns updateClen can be set to false- Returns:
- result matrix block
-
ctableOperations
public void ctableOperations(Operator op, MatrixValue that, MatrixValue that2, CTableMap resultMap, MatrixBlock resultBlock)
- Overrides:
ctableOperationsin classMatrixBlock
-
ternaryOperations
public MatrixBlock ternaryOperations(TernaryOperator op, MatrixBlock m2, MatrixBlock m3, MatrixBlock ret)
- Overrides:
ternaryOperationsin classMatrixBlock
-
quaternaryOperations
public MatrixBlock quaternaryOperations(QuaternaryOperator qop, MatrixBlock um, MatrixBlock vm, MatrixBlock wm, MatrixBlock out, int k)
- Overrides:
quaternaryOperationsin classMatrixBlock
-
randOperationsInPlace
public MatrixBlock randOperationsInPlace(RandomMatrixGenerator rgen, org.apache.commons.math3.random.Well1024a bigrand, long bSeed)
Description copied from class:MatrixBlockFunction to generate a matrix of random numbers. This is invoked both from CP as well as from MR. In case of CP, it generates an entire matrix block-by-block. Abigrandis passed so that block-level seeds are generated internally. In case of MR, it generates a single block for given block-level seedbSeed. When pdf="uniform", cell values are drawn from uniform distribution in range[min,max]. When pdf="normal", cell values are drawn from standard normal distribution N(0,1). The range of generated values will always be (-Inf,+Inf).- Overrides:
randOperationsInPlacein classMatrixBlock- Parameters:
rgen- random matrix generatorbigrand- ?bSeed- seed value- Returns:
- matrix block
-
randOperationsInPlace
public MatrixBlock randOperationsInPlace(RandomMatrixGenerator rgen, org.apache.commons.math3.random.Well1024a bigrand, long bSeed, int k)
Description copied from class:MatrixBlockFunction to generate a matrix of random numbers. This is invoked both from CP as well as from MR. In case of CP, it generates an entire matrix block-by-block. Abigrandis passed so that block-level seeds are generated internally. In case of MR, it generates a single block for given block-level seedbSeed. When pdf="uniform", cell values are drawn from uniform distribution in range[min,max]. When pdf="normal", cell values are drawn from standard normal distribution N(0,1). The range of generated values will always be (-Inf,+Inf).- Overrides:
randOperationsInPlacein classMatrixBlock- Parameters:
rgen- random matrix generatorbigrand- ?bSeed- seed valuek- ?- Returns:
- matrix block
-
seqOperationsInPlace
public MatrixBlock seqOperationsInPlace(double from, double to, double incr)
- Overrides:
seqOperationsInPlacein classMatrixBlock
-
getUncompressed
public static MatrixBlock getUncompressed(MatrixValue mVal)
-
getUncompressed
public static MatrixBlock getUncompressed(MatrixValue mVal, String message)
-
getUncompressed
public MatrixBlock getUncompressed()
-
getUncompressed
public MatrixBlock getUncompressed(String operation)
-
getUncompressed
public MatrixBlock getUncompressed(String operation, int k)
-
isShallowSerialize
public boolean isShallowSerialize(boolean inclConvert)
Description copied from interface:CacheBlockIndicates if the cache block is subject to shallow serialized, which is generally true if in-memory size and serialized size are almost identical allowing to avoid unnecessary deep serialize.- Specified by:
isShallowSerializein interfaceCacheBlock- Overrides:
isShallowSerializein classMatrixBlock- Parameters:
inclConvert- if true report blocks as shallow serialize that are currently not amenable but can be brought into an amenable form viatoShallowSerializeBlock.- Returns:
- true if shallow serialized
-
toShallowSerializeBlock
public void toShallowSerializeBlock()
Description copied from interface:CacheBlockConverts a cache block that is not shallow serializable into a form that is shallow serializable. This methods has no affect if the given cache block is not amenable.- Specified by:
toShallowSerializeBlockin interfaceCacheBlock- Overrides:
toShallowSerializeBlockin classMatrixBlock
-
copy
public void copy(MatrixValue thatValue)
Description copied from class:MatrixValueCopy that MatrixValue into this MatrixValue. If the MatrixValue is a MatrixBlock evaluate the sparsity of the original matrix, and copy into either a sparse or a dense matrix.- Overrides:
copyin classMatrixBlock- Parameters:
thatValue- object to copy the values into.
-
copy
public void copy(MatrixValue thatValue, boolean sp)
Description copied from class:MatrixValueCopy that MatrixValue into this MatrixValue. But select sparse destination block depending on boolean parameter.- Overrides:
copyin classMatrixBlock- Parameters:
thatValue- object to copy the values into.sp- boolean specifying if output should be forced sparse or dense. (only applicable if the 'that' is a MatrixBlock)
-
copyShallow
public MatrixBlock copyShallow(MatrixBlock that)
- Overrides:
copyShallowin classMatrixBlock
-
copy
public void copy(int rl, int ru, int cl, int cu, MatrixBlock src, boolean awareDestNZ)Description copied from class:MatrixBlockIn-place copy of matrix src into the index range of the existing current matrix. Note that removal of existing nnz in the index range and nnz maintenance is only done if 'awareDestNZ=true',- Overrides:
copyin classMatrixBlock- Parameters:
rl- row lower index, 0-basedru- row upper index, 0-based, inclusivecl- column lower index, 0-basedcu- column upper index, 0-based, inclusivesrc- matrix blockawareDestNZ- true, forces (1) to remove existing non-zeros in the index range of the destination if not present in src and (2) to internally maintain nnz false, assume empty index range in destination and do not maintain nnz (the invoker is responsible to recompute nnz after all copies are done)
-
getSoftReferenceToDecompressed
public SoftReference<MatrixBlock> getSoftReferenceToDecompressed()
-
clearSoftReferenceToDecompressed
public void clearSoftReferenceToDecompressed()
-
getDenseBlock
public DenseBlock getDenseBlock()
- Overrides:
getDenseBlockin classMatrixBlock
-
setDenseBlock
public void setDenseBlock(DenseBlock dblock)
- Overrides:
setDenseBlockin classMatrixBlock
-
getDenseBlockValues
public double[] getDenseBlockValues()
- Overrides:
getDenseBlockValuesin classMatrixBlock
-
getSparseBlock
public SparseBlock getSparseBlock()
- Overrides:
getSparseBlockin classMatrixBlock
-
setSparseBlock
public void setSparseBlock(SparseBlock sblock)
- Overrides:
setSparseBlockin classMatrixBlock
-
getSparseBlockIterator
public Iterator<IJV> getSparseBlockIterator()
- Overrides:
getSparseBlockIteratorin classMatrixBlock
-
getSparseBlockIterator
public Iterator<IJV> getSparseBlockIterator(int rl, int ru)
- Overrides:
getSparseBlockIteratorin classMatrixBlock
-
quickSetValue
public void quickSetValue(int r, int c, double v)- Overrides:
quickSetValuein classMatrixBlock
-
quickGetValueThreadSafe
public double quickGetValueThreadSafe(int r, int c)- Overrides:
quickGetValueThreadSafein classMatrixBlock
-
getValueDenseUnsafe
public double getValueDenseUnsafe(int r, int c)- Overrides:
getValueDenseUnsafein classMatrixBlock
-
appendValue
public void appendValue(int r, int c, double v)Description copied from class:MatrixBlockAppend value is only used when values are appended at the end of each row for the sparse representation This can only be called, when the caller knows the access pattern of the block- Overrides:
appendValuein classMatrixBlock- Parameters:
r- rowc- columnv- value
-
appendValuePlain
public void appendValuePlain(int r, int c, double v)- Overrides:
appendValuePlainin classMatrixBlock
-
appendRow
public void appendRow(int r, SparseRow row, boolean deep)- Overrides:
appendRowin classMatrixBlock
-
appendRowToSparse
public void appendRowToSparse(SparseBlock dest, MatrixBlock src, int i, int rowoffset, int coloffset, boolean deep)
- Overrides:
appendRowToSparsein classMatrixBlock
-
sortSparseRows
public void sortSparseRows()
Description copied from class:MatrixBlockSorts all existing sparse rows by column indexes.- Overrides:
sortSparseRowsin classMatrixBlock
-
sortSparseRows
public void sortSparseRows(int rl, int ru)Description copied from class:MatrixBlockSorts all existing sparse rows in range [rl,ru) by column indexes.- Overrides:
sortSparseRowsin classMatrixBlock- Parameters:
rl- row lower bound, inclusiveru- row upper bound, exclusive
-
minNonZero
public double minNonZero()
Description copied from class:MatrixBlockUtility function for computing the min non-zero value.- Overrides:
minNonZeroin classMatrixBlock- Returns:
- minimum non-zero value
-
isInSparseFormat
public boolean isInSparseFormat()
Description copied from class:MatrixBlockReturns the current representation (true for sparse).- Overrides:
isInSparseFormatin classMatrixBlock- Returns:
- true if sparse
-
isUltraSparse
public boolean isUltraSparse()
- Overrides:
isUltraSparsein classMatrixBlock
-
isUltraSparse
public boolean isUltraSparse(boolean checkNnz)
- Overrides:
isUltraSparsein classMatrixBlock
-
isSparsePermutationMatrix
public boolean isSparsePermutationMatrix()
- Overrides:
isSparsePermutationMatrixin classMatrixBlock
-
evalSparseFormatInMemory
public boolean evalSparseFormatInMemory()
Description copied from class:MatrixBlockEvaluates if this matrix block should be in sparse format in memory. Note that this call does not change the representation - for this please call examSparsity.- Overrides:
evalSparseFormatInMemoryin classMatrixBlock- Returns:
- true if matrix block should be in sparse format in memory
-
evalSparseFormatOnDisk
public boolean evalSparseFormatOnDisk()
Description copied from class:MatrixBlockEvaluates if this matrix block should be in sparse format on disk. This applies to any serialized matrix representation, i.e., when writing to in-memory buffer pool pages or writing to local fs or hdfs.- Overrides:
evalSparseFormatOnDiskin classMatrixBlock- Returns:
- true if matrix block should be in sparse format on disk
-
examSparsity
public void examSparsity(boolean allowCSR)
Description copied from class:MatrixBlockEvaluates if this matrix block should be in sparse format in memory. Depending on the current representation, the state of the matrix block is changed to the right representation if necessary. Note that this consumes for the time of execution memory for both representations.- Overrides:
examSparsityin classMatrixBlock- Parameters:
allowCSR- allow CSR format on dense to sparse conversion
-
sparseToDense
public void sparseToDense()
- Overrides:
sparseToDensein classMatrixBlock
-
merge
public void merge(MatrixBlock that, boolean appendOnly, boolean par, boolean deep)
- Overrides:
mergein classMatrixBlock
-
compactEmptyBlock
public void compactEmptyBlock()
Description copied from interface:CacheBlockFree unnecessarily allocated empty block.- Specified by:
compactEmptyBlockin interfaceCacheBlock- Overrides:
compactEmptyBlockin classMatrixBlock
-
dropLastRowsOrColumns
public void dropLastRowsOrColumns(Types.CorrectionLocationType correctionLocation)
- Overrides:
dropLastRowsOrColumnsin classMatrixBlock
-
interQuartileMean
public double interQuartileMean()
- Overrides:
interQuartileMeanin classMatrixBlock
-
pickValues
public MatrixBlock pickValues(MatrixValue quantiles, MatrixValue ret)
- Overrides:
pickValuesin classMatrixBlock
-
pickValue
public double pickValue(double quantile, boolean average)- Overrides:
pickValuein classMatrixBlock
-
sumWeightForQuantile
public double sumWeightForQuantile()
Description copied from class:MatrixBlockIn a given two column matrix, the second column denotes weights. This function computes the total weight- Overrides:
sumWeightForQuantilein classMatrixBlock- Returns:
- sum weight for quantile
-
extractTriangular
public MatrixBlock extractTriangular(MatrixBlock ret, boolean lower, boolean diag, boolean values)
- Overrides:
extractTriangularin classMatrixBlock
-
isThreadSafe
public boolean isThreadSafe()
Description copied from class:MatrixBlockIndicates if concurrent modifications of disjoint rows are thread-safe.- Overrides:
isThreadSafein classMatrixBlock- Returns:
- true if thread-safe
-
checkNaN
public void checkNaN()
Description copied from class:MatrixBlockChecks for existing NaN values in the matrix block.- Overrides:
checkNaNin classMatrixBlock
-
init
public void init(double[][] arr, int r, int c)Description copied from class:MatrixBlockNOTE: This method is designed only for dense representation.- Overrides:
initin classMatrixBlock- Parameters:
arr- 2d double array matrixr- number of rowsc- number of columns
-
init
public void init(double[] arr, int r, int c)Description copied from class:MatrixBlockNOTE: This method is designed only for dense representation.- Overrides:
initin classMatrixBlock- Parameters:
arr- double array matrixr- number of rowsc- number of columns
-
isAllocated
public boolean isAllocated()
- Overrides:
isAllocatedin classMatrixBlock
-
allocateBlockAsync
public Future<MatrixBlock> allocateBlockAsync()
- Overrides:
allocateBlockAsyncin classMatrixBlock
-
allocateDenseBlock
public boolean allocateDenseBlock(boolean clearNNZ)
- Overrides:
allocateDenseBlockin classMatrixBlock
-
allocateSparseRowsBlock
public boolean allocateSparseRowsBlock(boolean clearNNZ)
- Overrides:
allocateSparseRowsBlockin classMatrixBlock
-
allocateAndResetSparseBlock
public void allocateAndResetSparseBlock(boolean clearNNZ, SparseBlock.Type stype)- Overrides:
allocateAndResetSparseBlockin classMatrixBlock
-
-