Skip to main content
Version: Next(1.7.0)

HBase engine usage documentation

This article mainly introduces the installation, use and configuration of the hbase engine plugin in Linkis.

1. Preliminary work

1.1 Engine Verification

It is strongly recommended that you check these environment variables for the executing user before executing hbase tasks. The specific way is

sudo su - ${username}
echo ${JAVA_HOME}

2. Engine plugin installation

2.1 Engine plugin preparation non-default engine

Compile the engine plug-in separately (requires a maven environment)

# compile
cd ${linkis_code_dir}/linkis-engineconn-plugins/hbase/
mvn clean install -DskipTests -Dhbase.profile=1.2|1.4|2.2|2.5

# hbase.profile 1.2|1.4|2.2|2.5, The corresponding HBase version is: 1.2.0, 1.4.3, 2.2.6, 2.5.3
# default hbase.profile=2.5, Compile plug-ins for different HBase versions on demand
# The compiled engine plug-in package is located in the following directory
${linkis_code_dir}/linkis-engineconn-plugins/hbase/target/out/

EngineConnPlugin engine plugin installation

2.2 Upload and load engine plugins

Upload the engine plug-in package in 2.1 to the engine directory of the server

${LINKIS_HOME}/lib/linkis-engineplugins

The directory structure after uploading is as follows

linkis-engineconn-plugins/
├── hbase
│   ├── dist
│   │   └── 2.5.3
│   │   ├── conf
│   │   └── lib
│   └── plugin
│   └── 2.5.3

2.3 Engine refresh

2.3.1 Restart and refresh

Refresh the engine by restarting the linkis-cg-linkismanager service

cd ${LINKIS_HOME}/sbin
sh linkis-daemon.sh restart cg-linkismanager

2.3.2 Check if the engine is refreshed successfully

You can check whether the last_update_time of this table in the linkis_engine_conn_plugin_bml_resources in the database is the time when the refresh is triggered.

#Login to the linkis database
select * from linkis_cg_engine_conn_plugin_bml_resources;

3. Use of HBase engine

The HBase engine of Linkis runs a Jruby environment in the background to execute instructions in hbase-shell, and is fully compatible with hbase-shell instructions.

3.1 Submit tasks through Linkis-cli

 sh bin/linkis-cli -engineType hbase-2.5.3 -code "list" \
-codeType shell -runtimeMap linkis.hbase.zookeeper.quorum=zk1,zk2,zk3

More Linkis-Cli command parameter reference: Linkis-Cli usage

HBase engine supports connection parameters:

Configuration nameRemarks and default valueIs it necessary
linkis.hbase.zookeeper.quorumZooKeeper address for connecting to HBase, e.g. zk1,zk2,zk3, default value: localhostRequired
linkis.hbase.zookeeper.property.clientPortZooKeeper port for connecting to HBase, default value: 2181Optional, usually the default value
linkis.zookeeper.znode.parentZooKeeper znode path for connecting to HBase, default value: /hbaseOptional, usually the default value
linkis.hbase.rootdirHDFS root directory for the HBase cluster, default value: /hbaseOptional, usually the default value
linkis.hbase.security.authenticationAuthentication method for connecting to the HBase cluster: simple or kerberos, default value: simpleOptional, depending on the cluster configuration
linkis.hbase.kerberos.principalHBase Kerberos authentication principal, e.g. hbase@HADOOP.LINKIS.COMRequired for Kerberos authentication
linkis.hbase.keytab.filePath to the keytab file required for HBase Kerberos authentication, e.g. /tmp/hbase.keytabRequired for Kerberos authentication
linkis.hbase.kerberos.proxy.userKerberos proxy user, ensure that the Kerberos authenticated user has the privilege to proxy as a regular userOptional, can be omitted
linkis.hbase.regionserver.kerberos.principalHBase RegionServer's Kerberos authentication principal, e.g. hbase/_HOST@HADOOP.LINKIS.COM (obtain from the cluster configuration file)Required for Kerberos authentication
linkis.hbase.master.kerberos.principalHBase Master's Kerberos authentication principal, e.g. hbase/_HOST@HADOOP.LINKIS.COM (obtain from the cluster configuration file)Required for Kerberos authentication