Posts tagged ‘hadoop’

今天运行hadoop的php脚本时,总是出现一个很奇怪的现象:

10/05/20 15:33:41 INFO streaming.StreamJob: map 100% reduce 100%
10/05/20 15:33:41 INFO streaming.StreamJob: To kill this job, run:
10/05/20 15:33:41 INFO streaming.StreamJob: /usr/local/hadoop-0.20.2/bin/../bin/hadoop job -Dmapred.job.tracker=tj1cschsvc0001:9001 -kill job_201004301826_0595
10/05/20 15:33:41 INFO streaming.StreamJob: Tracking URL: http://tj1cschsvc0001:50030/jobdetails.jsp?jobid=job_201004301826_0595
10/05/20 15:33:41 ERROR streaming.StreamJob: Job not Successful!
10/05/20 15:33:41 INFO streaming.StreamJob: killJob…
Streaming Job Failed!

mapreducer明明已经100%完成了,但是却报Job not Successful。
查看log,发现syslog中有如下错误:
2010-05-20 15:32:38,494 INFO org.apache.hadoop.streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]
2010-05-20 15:32:38,495 INFO org.apache.hadoop.streaming.PipeMapRed: R/W/S=10/0/0 in:NA [rec/s] out:NA [rec/s]
2010-05-20 15:32:38,496 INFO org.apache.hadoop.streaming.PipeMapRed: R/W/S=100/0/0 in:NA [rec/s] out:NA [rec/s]
2010-05-20 15:32:38,508 INFO org.apache.hadoop.streaming.PipeMapRed: R/W/S=1000/0/0 in:NA [rec/s] out:NA [rec/s]
2010-05-20 15:32:38,558 INFO org.apache.hadoop.streaming.PipeMapRed: MROutputThread done
2010-05-20 15:32:38,558 INFO org.apache.hadoop.streaming.PipeMapRed: MRErrorThread done
2010-05-20 15:32:38,562 INFO org.apache.hadoop.streaming.PipeMapRed: mapRedFinished
2010-05-20 15:32:38,567 INFO org.apache.hadoop.streaming.PipeMapRed: mapRedFinished
2010-05-20 15:32:38,581 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
java.io.IOException: subprocess still running
R/W/S=4851/0/0 in:NA [rec/s] out:NA [rec/s]
minRecWrittenToEnableSkip_=9223372036854775807 LOGNAME=null
HOST=null
USER=ms
HADOOP_USER=null
last Hadoop input: |null|
last tool output: |null|
Date: Thu May 20 15:32:38 CST 2010
Broken pipe
at org.apache.hadoop.streaming.PipeReducer.reduce(PipeReducer.java:131)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:463)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
2010-05-20 15:32:38,586 INFO org.apache.hadoop.mapred.TaskRunner: Runnning cleanup for the task

而且并不是每个task都出错,运行几次之后,发现4个reducer进程中,只有一个出现这个问题。
首先排查了reducer脚本代码,各个服务器代码都一致。其实从task的重试也可以看到,它在两台不同的服务器都尝试过,而其中的一台上的其他task是执行成功的。
更改reducer代码,仅读取输入,然后echo出来,还是一样的现象。
初步判断不是reducer脚本的问题。那么会同输入相关吗?将reducer改为/bin/cat,发现4个part文件中,有一个开头有一个空行,而且其他的都没有。

猜测罪魁祸首有可能就是这个空行!删除它之后,再次执行mapreducer,成功!

但是原因是什么呢?使用我的recuder脚本时,有空行就出错;而使用cat作为reducer,就正常运行。那么还是因为reducer脚本对于空行的处理不足!但是奇怪的是,直接运行reducer脚本,以echo的几个空行作为输入,也没有错误啊!

ps:这个空行的原因是,想要清空某个文件,用了 echo > filename。以后还是直接rm吧!

Hadoop的分布式文件系统HDFS为java提供了原生的接口,可以像访问本地文件一样的,对HDFS中的文件进行增删改查等操作。

对于其他非java语言的支持,hadoop使用了Thrift

对于该方式,hadoop里针对thrift的hadoop-0.20.2/src/contrib/thriftfs/readme是这样说的:

Thrift is a software framework for scalable cross-language services
development. It combines a powerful software stack with a code generation
engine to build services that work efficiently and seamlessly
between C++, Java, Python, PHP, and Ruby.
This project exposes HDFS APIs using the Thrift software stack. This
allows applciations written in a myriad of languages to access
HDFS elegantly.
The Application Programming Interface (API)
===========================================
The HDFS API that is exposed through Thrift can be found in if/hadoopfs.thrift.
Compilation
===========
The compilation process creates a server org.apache.hadoop.thriftfs.HadooopThriftServer
that implements the Thrift interface defined in if/hadoopfs.thrift.
Th thrift compiler is used to generate API stubs in python, php, ruby,
cocoa, etc. The generated code is checked into the directories gen-*.
The generated java API is checked into lib/hadoopthriftapi.jar.
There is a sample python script hdfs.py in the scripts directory. This python
script, when invoked, creates a HadoopThriftServer in the background, and then
communicates wth HDFS using the API. This script is for demonstration purposes
only.

由于它说的过于简单,而我又对java世界了解甚少,导致走了不少弯路,现记录如下:

1、下载thrift源码,安装(./bootstrap.sh ;./configure –prefix=/usr/local/thrift; make;sudo make install)

2、将一些必须的文件cp到thrift安装目录下:

cp /path/to/thrift-0.2.0/lib/php/ /usr/local/thrift/lib/ -r

mkdir /usr/local/thrift/lib/php/src/packages/

cp /path/to/hadoop-0.20.2/src/contrib/thriftfs/gen-php/ /usr/local/thrift/lib/php/src/packages/hadoopfs/ -r

3、安装thrift的php扩展(针对php而言)

cd /path/to/thrift-0.2.0/lib/php/src/ext/thrift_protocol;phpize; ./configure;make ;make install;

修改php.ini,添加extension=thrift_protocol.so

4、编译hadoop

cd /path/to/hadoop-0.20.2; ant compile (ant -projecthelp可以查看项目信息;compile 是编译core和contrib目录)

5、启动hadoop的thrift代理

cd /path/to/hadoop-0.20.2/src/contrib/thriftfs/scripts/; ./start_thrift_server.sh [your-port] (如果不输入port,则随机一个port)

6、执行php测试代码

<?php

error_reporting(E_ALL);

ini_set(‘display_errors’, ‘on’);

$GLOBALS[‘THRIFT_ROOT’] = ‘/usr/local/thrift/lib/php/src‘;

define(‘ETCC_THRIFT_ROOT’, $GLOBALS[‘THRIFT_ROOT’]);

require_once(ETCC_THRIFT_ROOT.’/Thrift.php’ ); require_once(ETCC_THRIFT_ROOT.’/transport/TSocket.php’ ); require_once(ETCC_THRIFT_  ROOT.’/transport/TBufferedTransport.php’ ); require_once(ETCC_THRIFT_ROOT.’/protocol/TBinaryProtocol.php’ );

$socket = new TSocket(your-host, your-port);

$socket->setSendTimeout(10000);

$socket->setRecvTimeout(20000);

$transport = new TBufferedTransport($socket);

$protocol = new TBinaryProtocol($transport);

require_once(ETCC_THRIFT_ROOT.’/packages/hadoopfs/ThriftHadoopFileSystem.php’);

$client = new ThriftHadoopFileSystemClient($protocol);

$transport->open();

try

{

$pathname = new Pathname(array(‘pathname’ => ‘your-hdfs-file-name‘));

$fp = $client->open($pathname);

var_dump($client->stat($pathname));

var_dump($client->read($fp, 0, 1024));

exit;

} catch(Exception $e)

{

print_r($e);

}

$transport->close();

?>

可能出现的问题:

1、可以创建目录或者文件,但是读取不到文件内容

这时可以打开hadoop thrift的log4j配置(如果你用的是log4j记录日志的话),在/path/to/hadoop/conf/log4j.properties 里,修改:

hadoop.root.logger=ALL,console

这时,没执行一条访问HDFS的操作,都会把debug信息打印出来。

我这里看到的是file的id很奇怪,怀疑是在32位机上溢出了,尝试修改未果。之后迁移到64位机,运行正常!

问题代码在:/usr/local/thrift/lib/php/src/protocol/TBinaryProtocol.php (由于我代码里用的TBinaryProtocol类)的readI64函数里。

2、启动start_thrift_server.s失败

Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/thriftfs/HadoopThriftServer

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.thriftfs.HadoopThriftServer

at java.net.URLClassLoader$1.run(URLClassLoader.java:217)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:205)

at java.lang.ClassLoader.loadClass(ClassLoader.java:323)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)

at java.lang.ClassLoader.loadClass(ClassLoader.java:268)

at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:336)

Could not find the main class: org.apache.hadoop.thriftfs.HadoopThriftServer. Program will exit.

可以查看classpath,是否正确。我是添加了以下语句才正常的:

CLASSPATH=$CLASSPATH:$TOP/build/contrib/thriftfs/classes/:$TOP/build/classes/:$TOP/conf/

3、安装thrift出现问题

运行./bootstrap.sh出错: 查看boost是否安装,或者版本过旧

make出错:需要jdk 1.6以上

make出错:ImportError: No module named java_config_2 可能因为python升级,导致java-config不可用,重新安装java-config即可

line 832: X–tag=CXX: command not found : 把thrif/libtool文件内所有的“$echo”改为”$ECHO”即可(应该是libtool版本问题)

如果过于频繁的出现问题,且现象说明是大部分软件版本过旧,那可以考虑emerge –sync更新全部软件。

比较惨的时候,emerge总是被masked,那就手动安装一些依赖库吧,比如boost:

首先,下载boost包 http://www.boost.org/users/download/ ;并解压到/usr/local下(想安装的地方)

然后,建软连接到/usr/include下:ln -s /usr/local/boost-version/boost /usr/include/boost,这样就安装完了不需要编译的部分

如果还要继续安装需要编译的部分,那么进入到boost目录,运行bootstrap.sh脚本,生成bjam,运行./bjam install即可

4、ant安装报找不到jdk

但是我已经在/etc/profile,每个用户的.bash_profile里都把JAVA_HOME指向jdk目录了,而且echo $JAVA_HOME的结果也是jdk目录。

查看ant脚本,在第一行echo $JAVA_HOME,发现为空。无奈,只能手工把javahome添加到ant脚本里。

5、hadoop版本统一以后,可以正常启动datanode了,但是tasktracker还是起不来,在log里找到:

2010-04-30 18:07:31,975 ERROR org.apache.hadoop.mapred.TaskTracker: Shutting down. Incompatible buildVersion.

JobTracker’s: 0.20.3-dev from  by ms on Thu Apr 29 17:44:22 CST 2010

TaskTracker’s: 0.20.3-dev from  by root on Fri Apr 30 17:48:14 CST 2010

解决方法:

把master的hadoop目录,整个覆盖过去!

(尝试过用ms账号重新ant hadoop,无效;即版本一致了,但是安装时间不一致也不行。。。。)

1、hadoop常见问题汇总

http://www.cs.brandeis.edu/~cs147a/lab/hadoop-troubleshooting/

借助这个网址,解决了下面的问题:

$bin/hadoop fs -copyFromLocal /ms/wwwroot/cyrecommand/v1/hadoopTest/urlRec/buildData/smallLog smalllog

Error: File /user/ms/smalllog could only be replicated to 0 nodes, instead of 1

对应于以下问题:

Your DataNode is started and you can create directories with bin/hadoop dfs -mkdir, but you get an error message when you try to put files into the HDFS (e.g., when you run a command like bin/hadoop dfs -put).

我在namenode的web界面:http://tj1cantispam002.cnmsprod.local:50070/dfshealth.jsp  查看了live node数目是正常的(>0);这时去hadoop-username下看了看,硬盘满了。

清理了磁盘,释放了空间,就可以了。

2、网址

http://hadoop.apache.org/common/docs/r0.18.2/cn/hdfs_user_guide.html 中文的帮助手册

http://hadoop.apache.org/common/docs/current/commands_manual.html 英文的帮助手册

http://hadoop.apache.org/common/docs/current/hadoop-default.html 默认配置文件

http://hadoop.apache.org/common/docs/current/ 当前版本的hadoop文档

http://wiki.apache.org/hadoop/首页 hadoop的中文wiki

http://hadoop-server:50070/dfshealth.jsp 查看namenode、datanode的状态(端口号根据配置变化)

http://hadoop-server:50030/jobtracker.jsp 查看jobtracker的状态(端口号根据配置变化)

3、自带资源

Hadoop configuration is driven by two types of important configuration files:

  1. Read-only default configuration – src/core/core-default.xmlsrc/hdfs/hdfs-default.xml and src/mapred/mapred-default.xml.
  2. Site-specific configuration – conf/core-site.xmlconf/hdfs-site.xml and conf/mapred-site.xml.

To learn more about how the Hadoop framework is controlled by these configuration files, look here.

Additionally, you can control the Hadoop scripts found in the bin/ directory of the distribution, by setting site-specific values via the conf/hadoop-env.sh.

介绍:

hadoop是apache基金会下的开源软件,援引其官网的说法,它提供了可靠的、可扩展的、分布式的计算功能。

The Apache Hadoop project develops open-source software for reliable, scalable, distributed computing.

hadoop是一个总称,其下包括了多款子项目:

  • Hadoop Common: The common utilities that support the other Hadoop subprojects.
  • Avro: A data serialization system that provides dynamic integration with scripting languages.
  • Chukwa: A data collection system for managing large distributed systems.
  • HBase: A scalable, distributed database that supports structured data storage for large tables.
  • HDFS: A distributed file system that provides high throughput access to application data.
  • Hive: A data warehouse infrastructure that provides data summarization and ad hoc querying.
  • MapReduce: A software framework for distributed processing of large data sets on compute clusters.
  • Pig: A high-level data-flow language and execution framework for parallel computation.
  • ZooKeeper: A high-performance coordination service for distributed applications.

我是一个初学者,所以首先安装了一个单机版的,用来学习hadoop运行的原理。

1、安装java

首先从http://java.com下载1.6以上的java安装包,有两种可选的:rpm格式和自解压文件(bin)。我选择了bin文件下载。

例如,本地下载到~/software/jdk-6u18-linux-i586-rpm.bin

cd ~/software; chmod u+x jdk-6u18-linux-i586-rpm.bin # 使其可以运行

./jdk-6u18-linux-i586-rpm.bin #开始安装jdk、jre,默认会安装在/usr/java/jdkversion/下(geetoo默认安装在当前目录下)

修改配置文件/etc/profile,在末尾添加:

JAVA_HOME=/usr/java/jdk1.6.0_18
JAVA_BIN=$JAVA_HOME/bin
PATH=$PATH:$JAVA_BIN

将/usr/bin/java做软连接:/usr/bin/java -> /usr/java/jdk1.6.0_18/bin/java

这时,查看java版本号是否正确:

[flykobe@localhost software]$ java -version
java version “1.6.0_18″
Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
Java HotSpot(TM) Client VM (build 16.0-b13, mixed mode, sharing)

这样java就安装完毕了!

2、ssh公钥免密码登陆本机

$ssh-keygen ; cd ~/.ssh; cat id_rsa.pub >> authorized_keys

3、安装hadoop

其实hadoop是不用安装的,直接下载源代码,解压就可以运行了。

http://hadoop.apache.org/common/docs/current/quickstart.html#Local

安装上面网址中,单机版hadoop的例子:

$ mkdir input
$ cp conf/*.xml input
$ bin/hadoop jar hadoop-*-examples.jar grep input output ‘dfs[a-z.]+’
$ cat output/*

运行完以上的例子,可以在output中看到输出。

那么,这例子是什么意思呢?等同于以下代码:

$ grep -E ‘dfs[a-z.]+’ * -oh|sort | uniq -c
1 dfsadmin
3 dfs.class
1 dfs.file
1 dfsmetrics.log
2 dfs.period
1 dfs.servers

至此,hadoop单机版安装完毕。

4、单机多线程版hadoop

修改hadoop的环境配置:

# The java implementation to use.  Required.
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
JAVA_HOME=/usr/local/java/jre1.6.0_19
JAVA_BIN=$JAVA_HOME/bin
PATH=$PATH:$JAVA_BIN

安装指南上的命令依次执行:

524  bin/hadoop namenode -format

525  bin/start-all.sh

526  bin/hadoop fs -put conf input

527  bin/hadoop jar hadoop-*-examples.jar grep input output ‘dfs[a-z.]+’

531  bin/hadoop fs -cat output/*

532  bin/stop-all.sh

这里的input和outpput,其实都不是当前目录下的,可以通过以下命令查看:

ms@tj1cantispam002 /usr/local/hadoop-0.20.2 $ bin/hadoop fs -ls input output

Found 13 items

-rw-r–r–   1 ms supergroup       3936 2010-04-11 14:32 /user/ms/input/capacity-scheduler.xml

-rw-r–r–   1 ms supergroup        535 2010-04-11 14:32 /user/ms/input/configuration.xsl

-rw-r–r–   1 ms supergroup        276 2010-04-11 14:32 /user/ms/input/core-site.xml

-rw-r–r–   1 ms supergroup       2320 2010-04-11 14:32 /user/ms/input/hadoop-env.sh

-rw-r–r–   1 ms supergroup       1245 2010-04-11 14:32 /user/ms/input/hadoop-metrics.properties

-rw-r–r–   1 ms supergroup       4190 2010-04-11 14:32 /user/ms/input/hadoop-policy.xml

-rw-r–r–   1 ms supergroup        257 2010-04-11 14:32 /user/ms/input/hdfs-site.xml

-rw-r–r–   1 ms supergroup       2815 2010-04-11 14:32 /user/ms/input/log4j.properties

-rw-r–r–   1 ms supergroup        272 2010-04-11 14:32 /user/ms/input/mapred-site.xml

-rw-r–r–   1 ms supergroup         10 2010-04-11 14:32 /user/ms/input/masters

-rw-r–r–   1 ms supergroup         10 2010-04-11 14:32 /user/ms/input/slaves

-rw-r–r–   1 ms supergroup       1243 2010-04-11 14:32 /user/ms/input/ssl-client.xml.example

-rw-r–r–   1 ms supergroup       1195 2010-04-11 14:32 /user/ms/input/ssl-server.xml.example

Found 2 items

drwxr-xr-x   – ms supergroup          0 2010-04-11 14:32 /user/ms/output/_logs

-rw-r–r–   1 ms supergroup         96 2010-04-11 14:33 /user/ms/output/part-00000