Class InfrastructureAnalyzer
- java.lang.Object
-
- org.apache.sysds.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer
-
public class InfrastructureAnalyzer extends Object
Central place for analyzing and obtaining static infrastructure properties such as memory and number of logical processors.
-
-
Field Summary
Fields Modifier and Type Field Description static longDEFAULT_JVM_SIZEstatic StringMR_FRAMEWORK_NAMEstatic StringMR_JOBTRACKER_ADDRESS
-
Constructor Summary
Constructors Constructor Description InfrastructureAnalyzer()
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description static longextractMaxMemoryOpt(String javaOpts)static intgetCkMaxCP()Gets the maximum local parallelism constraint.static intgetCkMaxMR()Gets the maximum remote parallelism constraintstatic doublegetClusterUtilization(boolean mapOnly)Gets the fraction of running map/reduce tasks to existing map/reduce task slots.static longgetCmMax()Gets the maximum memory constraint [in bytes].static longgetHDFSBlockSize()Gets the HDFS blocksize of the used cluster in bytes.static longgetLocalMaxMemory()Gets the maximum memory [in bytes] of the current JVM.static doublegetLocalMaxMemoryFraction()static intgetLocalParallelism()Gets the number of logical processors of the current node, including hyper-threading if enabled.static intgetRemoteParallelMapTasks()Gets the number of remote parallel map slots.static intgetRemoteParallelNodes()Gets the number of cluster nodes (number of tasktrackers).static intgetRemoteParallelReduceTasks()Gets the total number of remote parallel reduce slots.static booleanisLocalMode()static booleanisLocalMode(org.apache.hadoop.mapred.JobConf job)static voidsetLocalMaxMemory(long localMem)static voidsetMaxMemoryOpt(org.apache.hadoop.mapred.JobConf job, String key, long bytes)static voidsetRemoteParallelMapTasks(int pmap)static voidsetRemoteParallelReduceTasks(int preduce)
-
-
-
Field Detail
-
DEFAULT_JVM_SIZE
public static final long DEFAULT_JVM_SIZE
- See Also:
- Constant Field Values
-
MR_FRAMEWORK_NAME
public static final String MR_FRAMEWORK_NAME
- See Also:
- Constant Field Values
-
MR_JOBTRACKER_ADDRESS
public static final String MR_JOBTRACKER_ADDRESS
- See Also:
- Constant Field Values
-
-
Method Detail
-
getLocalParallelism
public static int getLocalParallelism()
Gets the number of logical processors of the current node, including hyper-threading if enabled.- Returns:
- number of local processors of the current node
-
getRemoteParallelNodes
public static int getRemoteParallelNodes()
Gets the number of cluster nodes (number of tasktrackers). If multiple tasktracker are started per node, each tasktracker is viewed as individual node.- Returns:
- number of cluster nodes
-
getRemoteParallelMapTasks
public static int getRemoteParallelMapTasks()
Gets the number of remote parallel map slots.- Returns:
- number of remote parallel map tasks
-
setRemoteParallelMapTasks
public static void setRemoteParallelMapTasks(int pmap)
-
getRemoteParallelReduceTasks
public static int getRemoteParallelReduceTasks()
Gets the total number of remote parallel reduce slots.- Returns:
- number of remote parallel reduce tasks
-
setRemoteParallelReduceTasks
public static void setRemoteParallelReduceTasks(int preduce)
-
getLocalMaxMemory
public static long getLocalMaxMemory()
Gets the maximum memory [in bytes] of the current JVM.- Returns:
- maximum memory of the current JVM
-
setLocalMaxMemory
public static void setLocalMaxMemory(long localMem)
-
getLocalMaxMemoryFraction
public static double getLocalMaxMemoryFraction()
-
isLocalMode
public static boolean isLocalMode()
-
isLocalMode
public static boolean isLocalMode(org.apache.hadoop.mapred.JobConf job)
-
getCkMaxCP
public static int getCkMaxCP()
Gets the maximum local parallelism constraint.- Returns:
- maximum local parallelism constraint
-
getCkMaxMR
public static int getCkMaxMR()
Gets the maximum remote parallelism constraint- Returns:
- maximum remote parallelism constraint
-
getCmMax
public static long getCmMax()
Gets the maximum memory constraint [in bytes].- Returns:
- maximum memory constraint
-
getHDFSBlockSize
public static long getHDFSBlockSize()
Gets the HDFS blocksize of the used cluster in bytes.- Returns:
- HDFS block size
-
extractMaxMemoryOpt
public static long extractMaxMemoryOpt(String javaOpts)
-
setMaxMemoryOpt
public static void setMaxMemoryOpt(org.apache.hadoop.mapred.JobConf job, String key, long bytes)
-
getClusterUtilization
public static double getClusterUtilization(boolean mapOnly) throws IOExceptionGets the fraction of running map/reduce tasks to existing map/reduce task slots. NOTE: on YARN the number of slots is a spurious indicator because containers are purely scheduled based on memory.- Parameters:
mapOnly- if true, only look at map tasks- Returns:
- cluster utilization (current / capacity)
- Throws:
IOException- if IOException occurs
-
-