시작하세요 하둡프로그래밍 4장

read a book

write a book

/home/hadoop/hadoop-1.2.1 > ./bin/hadoop fs -put input2.txt input2.txt

/home/hadoop/hadoop-1.2.1 > ./bin/hadoop jar hadoop-beginner-examples-1.0.jar wikibooks.hadoop.chapter04.WordCount input2.txt wordcount_output2

//정상로그

[hadoop@wikibooks01 hadoop]$ ./bin/hadoop jar hadoop-beginner-examples-1.0.jar wikibooks.hadoop.chapter04.WordCount input2.txt wordcount_output2

16/12/06 17:28:50 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

16/12/06 17:28:51 INFO input.FileInputFormat: Total input paths to process : 1

16/12/06 17:28:51 INFO util.NativeCodeLoader: Loaded the native-hadoop library

16/12/06 17:28:51 WARN snappy.LoadSnappy: Snappy native library not loaded

16/12/06 17:28:51 INFO mapred.JobClient: Running job: job_201612061722_0002

16/12/06 17:28:52 INFO mapred.JobClient:  map 0% reduce 0%

16/12/06 17:28:59 INFO mapred.JobClient:  map 100% reduce 0%

16/12/06 17:29:07 INFO mapred.JobClient:  map 100% reduce 33%

16/12/06 17:29:08 INFO mapred.JobClient:  map 100% reduce 100%

16/12/06 17:29:10 INFO mapred.JobClient: Job complete: job_201612061722_0002

16/12/06 17:29:11 INFO mapred.JobClient: Counters: 29

16/12/06 17:29:11 INFO mapred.JobClient:   Job Counters

16/12/06 17:29:11 INFO mapred.JobClient:     Launched reduce tasks=1

16/12/06 17:29:11 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=6438

16/12/06 17:29:11 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0

16/12/06 17:29:11 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0

16/12/06 17:29:11 INFO mapred.JobClient:     Launched map tasks=1

16/12/06 17:29:11 INFO mapred.JobClient:     Data-local map tasks=1

16/12/06 17:29:11 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=9382

16/12/06 17:29:11 INFO mapred.JobClient:   File Output Format Counters

16/12/06 17:29:11 INFO mapred.JobClient:     Bytes Written=26

16/12/06 17:29:11 INFO mapred.JobClient:   FileSystemCounters

16/12/06 17:29:11 INFO mapred.JobClient:     FILE_BYTES_READ=67

16/12/06 17:29:11 INFO mapred.JobClient:     HDFS_BYTES_READ=136

16/12/06 17:29:11 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=115173

16/12/06 17:29:11 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=26

16/12/06 17:29:11 INFO mapred.JobClient:   File Input Format Counters

16/12/06 17:29:11 INFO mapred.JobClient:     Bytes Read=25

16/12/06 17:29:11 INFO mapred.JobClient:   Map-Reduce Framework

16/12/06 17:29:11 INFO mapred.JobClient:     Map output materialized bytes=67

16/12/06 17:29:11 INFO mapred.JobClient:     Map input records=2

16/12/06 17:29:11 INFO mapred.JobClient:     Reduce shuffle bytes=67

16/12/06 17:29:11 INFO mapred.JobClient:     Spilled Records=12

16/12/06 17:29:11 INFO mapred.JobClient:     Map output bytes=49

16/12/06 17:29:11 INFO mapred.JobClient:     Total committed heap usage (bytes)=159387648

16/12/06 17:29:11 INFO mapred.JobClient:     CPU time spent (ms)=1150

16/12/06 17:29:11 INFO mapred.JobClient:     Combine input records=0

16/12/06 17:29:11 INFO mapred.JobClient:     SPLIT_RAW_BYTES=111

16/12/06 17:29:11 INFO mapred.JobClient:     Reduce input records=6

16/12/06 17:29:11 INFO mapred.JobClient:     Reduce input groups=4

16/12/06 17:29:11 INFO mapred.JobClient:     Combine output records=0

16/12/06 17:29:11 INFO mapred.JobClient:     Physical memory (bytes) snapshot=259866624

16/12/06 17:29:11 INFO mapred.JobClient:     Reduce output records=4

16/12/06 17:29:11 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=1453490176

16/12/06 17:29:11 INFO mapred.JobClient:     Map output records=6

확인

http://wikibooks01:50030/jobtracker.jsp

질문? 

혹시나 자바 코드에서 memory leak 이 발생해서 심각한 속도 저하가 생긴다면, 어떻게 찾아낼 수 있을까?

못찾으면 하둡 쓰나 마나???