Home | 簡體中文 | 繁體中文 | 雜文 | 打賞(Donations) | ITEYE 博客 | OSChina 博客 | Facebook | Linkedin | 知乎專欄 | Search | About

第 38 章 Hypertable

目錄

38.1. Hypertable 安裝
38.1.1. Hypertable standalone 單機安裝
38.1.2. Hypertable on HDFS(hadoop) 安裝
38.1.3. MapR
38.1.4. Ceph
38.1.5. 檢驗安裝
38.2. Code examples
38.2.1. PHP
38.3. HQL
38.3.1. namespace 命名空間管理
38.3.2. Table 表
38.4. FAQ
38.4.1. 切換 DFS Broker

http://hypertable.org/

38.1. Hypertable 安裝

38.1.1. Hypertable standalone 單機安裝

過程 38.1. Hypertable standalone 安裝過程

  1. 安裝 Hypertable 軟件包

    # cd /usr/local/src/
    # wget http://cdn.hypertable.com/packages/0.9.7.0/hypertable-0.9.7.0-linux-x86_64.rpm
    					
  2. 安裝 Hypertable, 我個人比較喜歡用yum localinstall他會解決軟件之間的依賴關係

    # yum localinstall hypertable-0.9.7.0-linux-x86_64.rpm
    					

    相關的軟件會自動安裝

    Dependencies Resolved
    
    ====================================================================================================================
     Package                       Arch          Version                  Repository                               Size
    ====================================================================================================================
    Installing:
     hypertable                    x86_64        0.9.7.0-1                /hypertable-0.9.7.0-linux-x86_64        288 M
    Installing for dependencies:
     mailcap                       noarch        2.1.31-2.el6             base                                     27 k
     perl-Bit-Vector               x86_64        7.1-2.el6                base                                    169 k
     perl-Carp-Clan                noarch        6.03-2.el6               base                                     25 k
     perl-Compress-Raw-Zlib        x86_64        1:2.020-127.el6          base                                     68 k
     perl-Compress-Zlib            x86_64        2.020-127.el6            base                                     43 k
     perl-HTML-Parser              x86_64        3.64-2.el6               base                                    109 k
     perl-HTML-Tagset              noarch        3.20-4.el6               base                                     17 k
     perl-IO-Compress-Base         x86_64        2.020-127.el6            base                                     67 k
     perl-IO-Compress-Zlib         x86_64        2.020-127.el6            base                                    134 k
     perl-IO-String                noarch        1.08-9.el6               base                                     15 k
     perl-URI                      noarch        1.40-2.el6               base                                    117 k
     perl-libwww-perl              noarch        5.833-2.el6              base                                    387 k
    
    Transaction Summary
    ====================================================================================================================
    Install      13 Package(s)
    					
  3. Hypertable 預設安裝在 /opt/hypertable/0.9.7.0

    備份配置檔案,

    # cd /opt/hypertable/0.9.7.0/conf
    # cp hypertable.cfg hypertable.cfg.original
    					
  4. FHS-IZE 安裝

    # bin/fhsize.sh
    Setting up /var/opt/hypertable
    Setting up /etc/opt/hypertable
    fshize /opt/hypertable/0.9.7.0:  success
    					
  5. 設計 "CURRENT" 連接

    # cd /opt/hypertable
    # ln -s 0.9.7.0 current
    					
  6. 安裝 notification-hook.sh 腳本.

    # cp conf/notification-hook.tmpl conf/notification-hook.sh
    # chmod o+x conf/notification-hook.sh
    					

    測試 notification-hook.sh腳本 .

    /opt/hypertable/current/conf/notification-hook.sh "Test Message" "This is a test."
    					
  7. 啟動 hypertable

    # /opt/hypertable/current/bin/start-all-servers.sh local
    DFS broker: available file descriptors: 1024
    Started DFS Broker (local)
    Started Hyperspace
    Started Hypertable.Master
    /proc/sys/vm/swappiness = 60
    Started Hypertable.RangeServer
    Started ThriftBroker
    					
    					
    # /opt/hypertable/current/bin/ht shell
    
    Welcome to the hypertable command interpreter.
    For information about Hypertable, visit http://hypertable.com
    
    Type 'help' for a list of commands, or 'help shell' for a
    list of shell meta commands.
    
    hypertable>
    					
    					
  8. 測試安裝是否有效

    					
    # /opt/hypertable/current/bin/ht shell
    
    Welcome to the hypertable command interpreter.
    For information about Hypertable, visit http://hypertable.com
    
    Type 'help' for a list of commands, or 'help shell' for a
    list of shell meta commands.
    
    hypertable> help
    
    USE ................ Sets the current namespace
    CREATE NAMESPACE ... Creates a new namespace
    DROP NAMESPACE ..... Removes a namespace
    EXISTS TABLE ....... Check if table exists
    CREATE TABLE ....... Creates a table
    DELETE ............. Deletes all or part of a row from a table
    DESCRIBE TABLE ..... Displays a table's schema
    DROP TABLE ......... Removes a table
    RENAME TABLE ....... Renames a table
    DUMP TABLE ......... Create efficient backup file
    ALTER TABLE ........ Add/remove column family from existing table
    INSERT ............. Inserts data into a table
    LOAD DATA INFILE ... Loads data from a TSV input file into a table
    SELECT ............. Selects (and display) cells from a table
    SHOW CREATE TABLE .. Displays CREATE TABLE command used to create table
    SHOW TABLES ........ Displays only the list of tables in the current namespace
    GET LISTING ........ Displays the list of tables and namespace in the current namespace
    
    Statements must be terminated with ';'.  For more information on
    a specific statement, type 'help <statement>', where <statement> is from
    the preceding list.
    
    hypertable>quit
    					
    					
  9. 停止 hypertable

    運行下列命令停止 Hypertable

    $ /opt/hypertable/current/bin/stop-servers.sh
    					

38.1.2. Hypertable on HDFS(hadoop) 安裝

Hadoop - HDFS 安裝指南

過程 38.2. Hypertable on HDFS

  1. 創建工作目錄

    $ hadoop fs -mkdir /hypertable
    $ hadoop fs -chmod 777 /hypertable
    					
  2. 安裝 Java 運行環境

    yum install java-1.7.0-openjdk
    yum localinstall http://ftp.cuhk.edu.hk/pub/packages/apache.org/hadoop/common/hadoop-1.1.2/hadoop-1.1.2-1.x86_64.rpm
    					
  3. 修改 jrun bug

    cp /opt/hypertable/current/bin/jrun /opt/hypertable/current/bin/jrun.old
    
    vim /opt/hypertable/current/bin/jrun
    #HT_JAR=`ls -1 /opt/hypertable/doug/current/lib/java/*.jar | grep "hypertable-[^-]*.jar" | awk 'BEGIN {FS="/"} {print $NF}'`
    HT_JAR=`ls -1 /opt/hypertable/current/lib/java/*.jar | grep "hypertable-[^-]*.jar" | awk 'BEGIN {FS="/"} {print $NF}'`
    					
     export JAVA_HOME=/usr
     export HADOOP_HOME=/usr
     export HYPERTABLE_HOME=/opt/hypertable/current
    					
  4. hypertable.cfg

    # cat conf/hypertable.cfg
    #
    # hypertable.cfg
    #
    
    # HDFS Broker
    #HdfsBroker.Hadoop.ConfDir=/etc/hadoop/conf
    HdfsBroker.Hadoop.ConfDir=/etc/hadoop
    
    # Ceph Broker
    CephBroker.MonAddr=192.168.6.2:6789
    
    # Local Broker
    DfsBroker.Local.Root=fs/local
    
    # DFS Broker - for clients
    DfsBroker.Port=38030
    
    # Hyperspace
    Hyperspace.Replica.Host=localhost
    Hyperspace.Replica.Port=38040
    Hyperspace.Replica.Dir=hyperspace
    
    # Hypertable.Master
    #Hypertable.Master.Host=localhost
    Hypertable.Master.Port=38050
    
    # Hypertable.RangeServer
    Hypertable.RangeServer.Port=38060
    
    Hyperspace.KeepAlive.Interval=30000
    Hyperspace.Lease.Interval=1000000
    Hyperspace.GracePeriod=200000
    
    # ThriftBroker
    ThriftBroker.Port=38080
    					

    Hadoop 配置檔案 /etc/hadoop/core-site.xml

    					
    # cat /etc/hadoop/core-site.xml
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    
    <!-- Put site-specific property overrides in this file. -->
    
    <configuration>
        <property>
             <name>fs.default.name</name>
             <value>hdfs://namenode.example.com:9000</value>
        </property>
        <property>
             <name>hadoop.tmp.dir</name>
             <value>/var/tmp/hadoop</value>
        </property>
    </configuration>
    					
    					

    Hadoop 配置檔案 /etc/hadoop/hdfs-site.xml

    					
    # cat /etc/hadoop/hdfs-site.xml
    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    
    <!-- Put site-specific property overrides in this file. -->
    
    <configuration>
        <property>
            <name>dfs.name.dir</name>
            <value>/var/hadoop/name1</value>
            <description>  </description>
        </property>
        <property>
            <name>dfs.data.dir</name>
            <value>/var/hadoop/hdfs/data1</value>
            <description> </description>
        </property>
        <property>
            <name>dfs.replication</name>
            <value>2</value>
        </property>
    </configuration>
    					
    					
  5. 啟動 dfsbroker

    # /opt/hypertable/current/bin/set-hadoop-distro.sh cdh4
    Hypertable successfully configured for Hadoop cdh4
    					
    # /opt/hypertable/current/bin/start-dfsbroker.sh hadoop
    DFS broker: available file descriptors: 1024
    Started DFS Broker (hadoop)
    					

    查看啟動日誌

    # tail -f /opt/hypertable/current/log/DfsBroker.hadoop.log
    log4j:WARN No appenders could be found for logger (org.apache.hadoop.conf.Configuration).
    log4j:WARN Please initialize the log4j system properly.
    HdfsBroker.dfs.client.read.shortcircuit=false
    HdfsBroker.dfs.replication=2
    HdfsBroker.Server.fs.default.name=hdfs://namenode.example.com:9000
    Apr 23, 2013 6:43:18 PM org.hypertable.AsyncComm.IOHandler DeliverEvent
    INFO: [/192.168.6.25:53556 ; Tue Apr 23 18:43:18 HKT 2013] Connection Established
    Apr 23, 2013 6:43:18 PM org.hypertable.DfsBroker.hadoop.ConnectionHandler handle
    INFO: [/192.168.6.25:53556 ; Tue Apr 23 18:43:18 HKT 2013] Disconnect - COMM broken connection : Closing all open handles from /192.168.6.25:53556
    Closed 0 input streams and 0 output streams for client connection /192.168.6.25:53556
    					

38.1.3. MapR

38.1.4. Ceph

修改 CephBroker.MonAddr 對應的伺服器與連接埠號即可

# cat hypertable.cfg
#
# hypertable.cfg
#

# HDFS Broker
HdfsBroker.Hadoop.ConfDir=/etc/hadoop/conf

# Ceph Broker
CephBroker.MonAddr=192.168.6.2:6789

# Local Broker
DfsBroker.Local.Root=fs/local

# DFS Broker - for clients
DfsBroker.Port=38030

# Hyperspace
Hyperspace.Replica.Host=localhost
Hyperspace.Replica.Port=38040
Hyperspace.Replica.Dir=hyperspace

# Hypertable.Master
Hypertable.Master.Port=38050

# Hypertable.RangeServer
Hypertable.RangeServer.Port=38060

Hyperspace.KeepAlive.Interval=30000
Hyperspace.Lease.Interval=1000000
Hyperspace.GracePeriod=200000

# ThriftBroker
ThriftBroker.Port=38080
			

啟動 dfsbroker

# /opt/hypertable/current/bin/start-dfsbroker.sh ceph
			

38.1.5. 檢驗安裝

創建一個表

			
# echo "USE '/'; CREATE TABLE foo ( c1, c2 ); GET LISTING;" \
>     | /opt/hypertable/current/bin/ht shell --batch
foo
sys	(namespace)
tmp	(namespace)
			
			

插入一些數據

			
# echo "USE '/'; INSERT INTO foo VALUES('001', 'c1', 'very'), \
>     ('000', 'c1', 'Hypertable'), ('001', 'c2', 'easy'), ('000', 'c2', 'is');" \
>     | /opt/hypertable/current/bin/ht shell --batch
			
			

查詢數據

			
# echo "USE '/'; SELECT * FROM foo;" \
>     | /opt/hypertable/current/bin/ht shell --batch
000	c1	Hypertable
000	c2	is
001	c1	very
001	c2	easy
			
			

如果你想清楚所有表運行下面命令

$ /opt/hypertable/current/bin/stop-servers.sh
$ /opt/hypertable/current/bin/clean-database.sh