Difference between revisions of "Hadoop notes"

From Simson Garfinkel
Jump to navigationJump to search
m
m
 
(2 intermediate revisions by the same user not shown)
Line 1: Line 1:
Run a command with Apache Yarn:
Run a command with Apache Yarn as the **yarn** user:


     export DSJAR=/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.8.3-amzn-0.jar
     export DSJAR=/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.8.3-amzn-0.jar
     yarn jar $DSJAR -jar $DSJAR org.apache.hadoop.yarn.applications.distributedshell.Client -master_memory 1000 -container_memory 1000 -shell_command 'hostname'
     yarn jar $DSJAR -jar $DSJAR org.apache.hadoop.yarn.applications.distributedshell.Client -master_memory 1000 -container_memory 1000 -shell_command 'hostname'
   
    yarn jar $DSJAR -jar $DSJAR org.apache.hadoop.yarn.applications.distributedshell.Client -num_containers 32 -master_memory 1000 -container_memory 1000 -shell_command 'hostname'


Output:
Output:
Line 9: Line 13:
     18/05/03 21:35:32 INFO distributedshell.Client: Got application report from ASM for, appId=7, clientToAMToken=null, appDiagnostics=, appMasterHost=host/127.0.0.1, appQueue=default, appMasterRpcPort=-1, appStartTime=1525383325669, yarnAppState=FINISHED, distributedFinalState=SUCCEEDED, appTrackingUrl=http://host:20888/proxy/application_1525380541355_0007/, appUser=hadoop
     18/05/03 21:35:32 INFO distributedshell.Client: Got application report from ASM for, appId=7, clientToAMToken=null, appDiagnostics=, appMasterHost=host/127.0.0.1, appQueue=default, appMasterRpcPort=-1, appStartTime=1525383325669, yarnAppState=FINISHED, distributedFinalState=SUCCEEDED, appTrackingUrl=http://host:20888/proxy/application_1525380541355_0007/, appUser=hadoop
     ....
     ....
See the apps that ran with:
    yarn application  -list -appStates ALL





Latest revision as of 11:45, 2 August 2018

Run a command with Apache Yarn as the **yarn** user:

   export DSJAR=/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.8.3-amzn-0.jar
   yarn jar $DSJAR -jar $DSJAR org.apache.hadoop.yarn.applications.distributedshell.Client -master_memory 1000 -container_memory 1000 -shell_command 'hostname'
   
   yarn jar $DSJAR -jar $DSJAR org.apache.hadoop.yarn.applications.distributedshell.Client -num_containers 32 -master_memory 1000 -container_memory 1000 -shell_command 'hostname'


Output:

   ....
   18/05/03 21:35:32 INFO distributedshell.Client: Got application report from ASM for, appId=7, clientToAMToken=null, appDiagnostics=, appMasterHost=host/127.0.0.1, appQueue=default, appMasterRpcPort=-1, appStartTime=1525383325669, yarnAppState=FINISHED, distributedFinalState=SUCCEEDED, appTrackingUrl=http://host:20888/proxy/application_1525380541355_0007/, appUser=hadoop
   ....

See the apps that ran with:

   yarn application  -list -appStates ALL


view the logs with:

   yarn logs -applicationId application_1525380541355_0007