luzhuomi 发表于 2013-6-8 20:27:47

cubieboard 上架设Hadoop cluster 和 HBase

本帖最后由 luzhuomi 于 2013-11-21 17:11 编辑

更新：HBase cubieboard集群架设成功，详情请看12楼。

http://player.youku.com/player.php/sid/XNTU1MDEwOTg4/v.swf


小弟在家倒腾3块小CB A10.弄出个Hadoop Cluster。
家里三台CB的IP
192.168.1.8
192.168.1.12
192.168.1.214





#注:Cluster里所有的成员机都需要以下设定

# 安装 java

sudo apt-get update

sudo apt-get -y install openjdk-6-jdk


# 安装 hadoop:

# 改成相应的IP地址
MASTER="192.168.1.8"
SLAVES="192.168.1.8 192.168.1.12 192.168.1.214"
wget http://mirror.nus.edu.sg/apache/ ... hadoop-1.1.2.tar.gz

mkdir -p /opt/

tar zxvf hadoop-1.1.2.tar.gz
sudo mv hadoop-1.1.2 /opt/
sudo addgroup hadoop
sudo chown -R linaro:hadoop /opt/hadoop-1.1.2

cp /opt/hadoop-1.1.2/conf/hadoop-env.sh/opt/hadoop-1.1.2/conf/hadoop-env.sh.orig
sed -i "s/# export JAVA_HOME=.*/export\ JAVA_HOME=\/usr\/lib\/jvm\/java-6-openjdk-amd64/g" /opt/hadoop-1.1.2/conf/hadoop-env.sh


sudo mkdir -p /mnt/hadoop/datanode/hadoop-${USER}

sudo chown -R linaro:hadoop /mnt/hadoop

cp /opt/hadoop-1.1.2/conf/core-site.xml /opt/hadoop-1.1.2/conf/core-site.xml.orig

echo -e "<?xml version=\"1.0\"?>
<?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?>
<\x21-- Put site-specific property overrides in this file. -->
<configuration>
<property>
    <name>fs.default.name</name>
    <value>hdfs://${MASTER}:9000</value>
</property>
<property>
    <name>hadoop.tmp.dir</name>
    <value>/mnt/hadoop/datanode/hadoop-\${user.name}</value>
</property>
</configuration>
" > /opt/hadoop-1.1.2/conf/core-site.xml


cp /opt/hadoop-1.1.2/conf/hdfs-site.xml/opt/hadoop-1.1.2/conf/hdfs-site.xml.orig

echo -e "<?xml version=\"1.0\"?>
<?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?>

<\x21-- Put site-specific property overrides in this file. -->

<configuration>
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>
</configuration>
" > /opt/hadoop-1.1.2/conf/hdfs-site.xml


cp /opt/hadoop-1.1.2/conf/mapred-site.xml/opt/hadoop-1.1.2/conf/mapred-site.xml.orig
echo -e "<?xml version=\"1.0\"?>
<?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?>

<\x21-- Put site-specific property overrides in this file. -->

<configuration>
<property>
    <name>mapred.job.tracker</name>
    <value>${MASTER}:9001</value>
</property>
</configuration>
" > /opt/hadoop-1.1.2/conf/mapred-site.xml

cp /opt/hadoop-1.1.2/conf/masters /opt/hadoop-1.1.2/conf/masters.bak
cp /opt/hadoop-1.1.2/conf/slaves /opt/hadoop-1.1.2/conf/slaves.bak
echo -e "${MASTER}" > /opt/hadoop-1.1.2/conf/masters
cat /dev/null > /opt/hadoop-1.1.2/conf/slaves
for ip in ${SLAVES};do echo -e "${ip}" >> /opt/hadoop-1.1.2/conf/slaves ; done

# 在master上执行

/opt/hadoop-1.1.2/bin/hadoop namenode -format
/opt/hadoop-1.1.2/bin/start-all.sh

#大功告成!


hipboi 发表于 2013-6-10 09:44:22

不错，会火。有什么应用可以在上面跑的。

luzhuomi 发表于 2013-6-11 01:45:08

基本上大部分hadoop相关的都能跑。 我在调试HBase 和 Scalding。好了再继续放上来。:)

elboble 发表于 2013-6-13 12:57:52

乐高的盒子，不错，盒子怎么做的？

luzhuomi 发表于 2013-6-13 17:25:15

乐高盒子请参考之前的旧帖子
http://forum.cubietech.com/forum.php?mod=viewthread&tid=161&page=1#pid704

topking 发表于 2013-6-13 21:11:50

乐高盒子太有才了。

aaron 发表于 2013-7-23 14:58:42

CB里面跑的是什么系统？？

luzhuomi 发表于 2013-7-27 13:02:36

linaro-ubuntu

matson 发表于 2013-7-28 14:18:32

cb1的lubuntu固件(http://cubiebook.org/index.php?title=Cubieboard/Ubuntu_Nand_Images)已经集成了jdk 1.8 armhf，计划下周发布lubuntu_a20的固件，也集成jdk，喜欢玩hadoop的很方便搭建集群了

luzhuomi 发表于 2013-8-3 22:57:31

好也。会有SD的Image吗？
页: [1] 2
查看完整版本: cubieboard 上架设Hadoop cluster 和 HBase