Class RemoteParForUtils
- java.lang.Object
-
- org.apache.sysds.runtime.controlprogram.parfor.RemoteParForUtils
-
public class RemoteParForUtils extends Object
Common functionalities for parfor workers in MR jobs. Used by worker wrappers in mappers (base RemoteParFor) and reducers (fused data partitioning and parfor)
-
-
Constructor Summary
Constructors Constructor Description RemoteParForUtils()
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description static voidcleanupWorkingDirectories()Cleanup all temporary files created by this SystemDS process.static voidcleanupWorkingDirectoriesOnShutdown()Cleanup all temporary files created by this SystemDS process, on shutdown via exit or interrupt.static voidexportLineageItems(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars, Lineage lineage)Export lineage for remote Spark parfor workers.static ArrayList<String>exportResultVariables(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars)For remote Spark parfor workers.static Lineage[]getLineages(LocalVariableMap[] results)static LocalVariableMap[]getResults(List<scala.Tuple2<Long,String>> out, org.apache.commons.logging.Log LOG)static voidincrementParForMRCounters(org.apache.hadoop.mapred.Reporter reporter, long deltaTasks, long deltaIterations)static voidsetupBufferPool(long workerID, boolean isLocal)Init and register-cleanup of buffer pool
-
-
-
Method Detail
-
incrementParForMRCounters
public static void incrementParForMRCounters(org.apache.hadoop.mapred.Reporter reporter, long deltaTasks, long deltaIterations)
-
exportResultVariables
public static ArrayList<String> exportResultVariables(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars) throws IOException
For remote Spark parfor workers. This is a simplified version compared to MR.- Parameters:
workerID- worker idvars- local variable mapresultVars- list of result variables- Returns:
- list of result variables
- Throws:
IOException- if IOException occurs
-
exportLineageItems
public static void exportLineageItems(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars, Lineage lineage) throws IOExceptionExport lineage for remote Spark parfor workers.- Parameters:
workerID- worker idvars- local variable mapresultVars- list of result variableslineage- lineage object- Throws:
IOException- if IOException occurs
-
cleanupWorkingDirectories
public static void cleanupWorkingDirectories()
Cleanup all temporary files created by this SystemDS process.
-
cleanupWorkingDirectoriesOnShutdown
public static void cleanupWorkingDirectoriesOnShutdown()
Cleanup all temporary files created by this SystemDS process, on shutdown via exit or interrupt.
-
getLineages
public static Lineage[] getLineages(LocalVariableMap[] results)
-
getResults
public static LocalVariableMap[] getResults(List<scala.Tuple2<Long,String>> out, org.apache.commons.logging.Log LOG)
-
setupBufferPool
public static void setupBufferPool(long workerID, boolean isLocal) throws IOExceptionInit and register-cleanup of buffer pool- Parameters:
workerID- worker idisLocal- in local spark mode (single JVM)- Throws:
IOException- exception
-
-