Skip to main content

2 posts tagged with "hadoop3.1.1"

View All Tags

· 8 min read
livi12138

Overview

The team needs to use SQL and Python syntax to analyze the data at the same time on the page. During the investigation, I found that Linkis can meet the needs. As a Huawei MRS is used, it is different from the open source software. It also carried out secondary development and adaptation. This article will share the experience, hoping to help students in need.

environment and version

  • JDK-1.8.0_112, Maven-3.5.2
  • Hadoop-3.1.1, spark-3.1.1, hive-3.1.0, zookerper-3.5.9 (Huawei MRS version)
  • Linkis-1.3.0
  • Scriptis-Web 1.1.0

dependence adjustment and packaging

First download the source code of 1.3.0 from the Linkis official website, and then adjust the dependent version

Linkis outermost adjustment pom file

<hadoop.version>3.1.1</hadoop.version>
<zookerper.version>3.5.9</zookerper.version>
<curaor.version>4.2.0</curaor.version>
<guava.version>30.0-jre</guava.version>
<json4s.version>3.7.0-M5</json4s.version>
<scala.version>2.12.15</scala.version>
<scala.binary.version>2.12</scala.binary.version>

linkis-engineplugin-hive的pom文件

<hive.version>3.1.2</hive.version>

linkis-engineplugin-spark的pom文件

<spark.version>3.1.1</spark.version>

linkis-hadoop-common的pom文件

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId> <!-Just replace the line and replace it with <Arttifactid> Hadoop-HDFS-Client </Artifactid>->->
<version>${hadoop.version}</version>
</dependency>
Modify the Hadoop-HDFS to:
<dependency>
<cepid> org.apache.hadoop </groupid>
<Artifactid> Hadoop-HDFS-Client </Artifactid>
<Version> $ {Hadoop.Version} </version>
</dependency>

Linkis-Label-Common

org.apache.linkis.manager.label.conf.labelcommonconfig Modify the default version, which is convenient for subsequent self -compiling scheduling components

    Public Static Final Commonvars <string> Spark_engine_Version =
Commonvars.apply ("wds.linkis.spark.engine.version", "3.1.1");

Public Static Final Commonvars <string> Hive_engine_Version =
Commonvars.apply ("wds.linkis.hive.engine.version", "3.1.2");

Linkis-computation-Governance-Common

org.apache.linkis.governance.Common.conf.governanceCommonConf Modify the default version, which is convenient for subsequent self -compiling scheduling components

  Val spark_engine_version = Commonvars ("wds.linkis.spark.engine.version", "3.1.1")

VAL HIVE_ENGINE_VERSION = Commonvars ("wds.linkis.hive.engine.version", "3.1.2")

Compilation

After the above configuration is adjusted, you can start compiling full amount, and execute the following commands in turn

    cd linkis-x.x.x
MVN -N Install
MVN CLEAN Install -DSKIPTESTS

Compile Error

-If when you compile it, there is an error, try to enter a module alone to compile, see if there are errors, and adjust according to specific errors -Since the SCALA language is used in Linkis for code writing, it is recommended that you can configure the SCALA environment first to facilitate reading the source code -Aar package conflict is the most common problem, especially after upgrading Hadoop, please adjust the dependent version patiently

DatasphereStudio pom file

As we upgrade the version of Scala, the error will be reported when deploying. Conn to BML Now Exit Java.net.socketexception: Connection Reset. Here you need to modify the SCALA version and re -compile.

  1. Delete the low version of the DSS-Gateway-SUPPPPORT JAR package,
  2. Modify the scala version in DSS 1.1.0 to 2.12, compile it, get the new DSS-Gateway-SUPPPORT -.1.0.JAR, replace the linkis_installhome/lib/linkis-spaint-service/linkis-mg-gateway The original jar package of the Central Plains
<!-The SCALA environment is consistent->
<scala.version> 2.12.15 </scala.version>

According to the adjustment of the dependent version above, you can solve most of the problems. If you have any problems, you need to carefully adjust the corresponding log. If a complete package can be compiled, it represents the full compilation of Linkis and can be deployed.

deployment

  • In in order to allow the engine node to have sufficient resource execution script, we have adopted multiple server deployments, and the general deployment structure is as follows. -SLB 1 load balancing is rotary inquiry -E ECS-WEB 2 Nginx, static resource deployment, background agent forwarding -ECS-APP 2 micro-service governance, computing governance, public enhancement and other node deployment -ECS-APP 4 Engineconnmanager node deployment

linkis deployment

  • At the use of multiple node deployments, we did not peel the code, or put the full amount on the server, but just modified the startup script to make it only start the service required

Refer to the official website single machine deployment example: https: //linkis.apache.org/zh-docs/1.3.0/dePlayment/dePlay-qick

Linkis Deployment Points Attention Point

    1. Deployment user: The startup user of the core process of Linkis. At the same time, this user will default as an administrator permissions. During the deployment process, the corresponding administrator login password, located in the linkis support specified in CONF/LINKIS-MG-Gateway.properties file file Submitted and executed users. The main process service of Linkis will be switched to the corresponding user through the SUDO -U $ {linkis-user}, and then executes the corresponding engine startup command, so the user of the engine linkis -ngine processes is the executor of the task. -The user defaults to the submission and executor of the task, if you want to change to the login user, you need to modify org.apache.linkis.entRance.Restful.entRANCERESTFAPI class json.put (taskConstant.execute_user, moduleuseuserutills.GetOperationUser (REQ)); json.put (taskConstant.submit_user, SecurityFilter.getLoginusername (REQ)); Change the above settings to the user and execute user to the Scriptis page to log in to the user
    1. Sudo -U $ {linkis -user} Switch to the corresponding user. If you use the login user, this command may fail, and you need to modify the command here.
  • org.apache.linkis.ecm.server.operator.EngineConnYarnLogOperator.sudoCommands
private def sudoCommands(creator: String, command: String): Array[String] = {
Array(
"/bin/bash",
"-c",
"sudo su " + creator + " -c \"source ~/.bashrc 2>/dev/null; " + command + "\""
)
} change into
private def sudoCommands(creator: String, command: String): Array[String] = {
Array(
"/bin/bash",
"-c",
"\"source ~/.bashrc 2>/dev/null; " + command + "\""
)
}
    1. Mysql's driver package must be Copy to/lib/linkis-commons/public-module/and/lib/linkis-spring-cloud-services/linkis-mg-gateway/
    1. The default is to use static users and passwords. Static users are deploying users. Static passwords will generate a password string in execution deployment, stored at $ {linkis_home} /conf/linkis-mg-gateway.properties
    1. database script execution, linkis itself needs to use the database, but when we execute the script of the inserted data of Linkis 1.3.0, we found an error. We directly deleted the data of the error part at that time.
    1. Yarn's certification. When performing the spark task, the task will be submitted to the queue. The resource information of the queue will be obtained first to determine whether there is a resource to submit. For certification, if the file authentication is enabled, the file needs to be placed in the corresponding directory of the server, and the information is updated in the linkis_cg_rm_extRNAL_Resource_Provider library table.

Install web front end

  • WEB side uses nginx as a static resource server, download the front -end installation package and decompress it, and place it on the directory corresponding to the Nginx server

scriptis tool installation

  • Scriptis is a pure front -end project. As a component integrates in the web code component of DSS, we only need to compile the DSSWEB project for separate Scriptis modules, upload the compiled static resources to Visit, note: Linkis stand -by -machine deployment defaults to use session for verification. You need to log in to the Linkis management desk first, and then log in to Scriptis to use.

Nginx deployment for example

nginx.conf

upstream linkisServer{
server ip:port;
server ip:port;
}
Server {
Listen 8088;# Access port
Server_name localhost;
#Charset Koi8-R;
#access_log /var/log/nginx/host.access.log main;
#Scriptis static resources
local /scriptis {
# Modify to your own front path
alias/home/nginx/scriptis-weight; # static file directory
#Root/Home/Hadoop/DSS/Web/DSS/Linkis;
index index.html index.html;
}
#The default resource path points to the static resource of the front end of the management platform
location / {
# Modify to your own front path
root/Home/Nginx/Linkis-Web/DIST; # r r r r
#Root/Home/Hadoop/DSS/Web/DSS/Linkis;
index index.html index.html;
}

local /ws {
Proxy_pass http:// linkisserver/api #back -end linkis address
proxy_http_version 1.1;
proxy_set_header upgrade $ http_upgrade;
proxy_set_header connection upgrade;
}

location /api {
proxy_pass http:// linkisserver/api; #The address of the back end linkis
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header x_real_ipP $remote_addr;
proxy_set_header remote_addr $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_connect_timeout 4s;
proxy_read_timeout 600s;
proxy_send_timeout 12s;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection upgrade;
}

#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}

How to check the question

    1. There are more than 100 modules in Linkis, and the final service has 7 services, which are linkis-cg -ngineconnmanager, linkis-cg -ngineplugin, linkis-cg-entrance, linkis-cg-linkisManager, Linkis-Mg-Gateway, Linkis-Mg-Eureka, Linkis-PS-PublicService, each module has this different features. Among them, Linkis-CG-ENGINECONNMANAGER is responsible for managing the start-engine service, which will generate the corresponding engine script to pull up the engine. Services, so our team launched the Linkis-CG-ENGINECONNMANAGER alone on the server for sufficient resources to execute on the server.
    1. The execution of engines like JDBC, Spark.hedu and other engines require some JAR package support. When the linkis species is called material, these jar packs will be hit in the linkis-cg-oblmphin engine when packaging , Conf and LIB directory will appear. When starting this service, two packages will be uploaded to the configuration directory, which will generate two ZIP files. We use OSS to store these material information. Download it to the Linkis-CG-ENGINECONNMANAGER service, and then configure the following two configurations of wds.linkis.enginecoon.public.dir and wds.linkis.enginecoon.root.dir, then the bag will be pulled to WDS. Linkis.engineCoon.public.dir is the directory of wds.linkis.enginecoon.root.dir. .dir.
    1. If you want to check the engine log, you can see the directory under wds.linkis.enginecoon.root.dir configuration. Of course, the log information will be displayed on the log of the scriptis page. Just paste it to find it.

· 14 min read
ruY9527

Environment and Version

  • jdk-8 , maven-3.6.3
  • node-14.15.0(Compiling the front end requires)
  • Gradle-4.6(Compile qualitis quality service)
  • hadoop-3.1.1,Spark-3.0.1,Hive-3.1.2,Flink-1.13.2,Sqoop-1.4.7 (Apache version)
  • linkis-1.1.1
  • DataSphereStudio-1.1.0
  • Schudulis-0.7.0
  • Qualitis-0.9.2
  • Visualis-1.0.0
  • Streamis-0.2.0
  • Exchangis-1.0.0
  • Chrome recommends versions below 100

Scenarios and versions of each component

System nameVersionscene
linkis1.1.1Engine orchestration, running and executing hive, spark, flinksql, shell, python, etc., unified data source management, etc
DataSphereStudio1.1.0Implement DAG scheduling of tasks, integrate the specifications of other systems and provide unified access, and provide sparksql based service API
Schudulis0.7.0Task scheduling, as well as scheduling details and rerouting, and provide trap data based on the selected time
Qualitis0.9.2Provide built-in SQL version and other functions, check common data quality and customizable SQL, verify some data that does not conform to the rules, and write it to the corresponding library
Exchangis1.0.0Hive to MySQL, data exchange between MySQL and hive
Streamis0.2.0Streaming development and Application Center
Visualis1.0.0Visual report display, can share external links

Deployment sequence

You can select and adjust the sequence after serial number 3 However, one thing to pay attention to when deploying exchangis is to copy the sqoop engine plug-in of exchangis to the engine plug-in package under lib of linkis Schedulis, qualitis, exchangis, streamis, visualis and other systems are integrated with DSS through their respective appconn. Note that after integrating the component appconn, restart the service module corresponding to DSS or restart DSS

  1. linkis
  2. DataSphereStudio
  3. Schedulis
  4. Qualitis
  5. Exchangis
  6. Streamis
  7. Visualis

image.png

If you integrate skywalking, you can see the service status and connection status in the extended topology diagram, as shown in the following figure: image.png At the same time, you can also clearly see the call link in the trace, as shown in the following figure, which is also convenient for you to locate the error log file of the specific service image.png

Dependency adjustment and packaging

linkis

Since spark uses version 3. X, Scala also needs to be upgraded to version 12 Original project code address Adaptation modification code reference address

The pom file of linkis

<hadoop.version>3.1.1</hadoop.version>
<scala.version>2.12.10</scala.version>
<scala.binary.version>2.12</scala.binary.version>

<!-- hadoop-hdfs replace with hadoop-hdfs-client -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs-client</artifactId>
<version>${hadoop.version}</version>

The pom file of linkis-hadoop-common

       <!-- Notice here <version>${hadoop.version}</version> , adjust according to whether you have encountered any errors --> 
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs-client</artifactId>
<version>${hadoop.version}</version>
</dependency>

The pom file of linkis-engineplugin-hive

<hive.version>3.1.2</hive.version>

The pom file of linkis-engineplugin-spark

<spark.version>3.0.1</spark.version>

The getfield method in sparkscalaexecutor needs to adjust the following code

protected def getField(obj: Object, name: String): Object = {
// val field = obj.getClass.getField(name)
val field = obj.getClass.getDeclaredField("in0")
field.setAccessible(true)
field.get(obj)
}
<flink.version>1.13.2</flink.version>

Due to the adjustment of some classes in Flink 1.12.2 and 1.13.2, we refer to the temporary "violence" method given by the community students: copy the classes in part 1.12.2 to 1.13.2, adjust the scala version to 12, and compile them by ourselves It involves the specific modules of flink: flink-sql-client_${scala.binary.version}

-- Note that the following classes are copied from 1.12.2 to 1.13.2
org.apache.flink.table.client.config.entries.DeploymentEntry
org.apache.flink.table.client.config.entries.ExecutionEntry
org.apache.flink.table.client.gateway.local.CollectBatchTableSink
org.apache.flink.table.client.gateway.local.CollectStreamTableSink

image.pngimage.png

linkis-engineplugin-python

Reference pr If resource / Python's python In the PY file, there is import pandas as PD. If you do not want to install pandas, you need to remove it

linkis-label-common

org.apache.linkis.manager.label.conf.LabelCommonConfig Modify the default version to facilitate the use of subsequent self compilation scheduling components

    public static final CommonVars<String> SPARK_ENGINE_VERSION =
CommonVars.apply("wds.linkis.spark.engine.version", "3.0.1");

public static final CommonVars<String> HIVE_ENGINE_VERSION =
CommonVars.apply("wds.linkis.hive.engine.version", "3.1.2");

linkis-computation-governance-common

org.apache.linkis.governance.common.conf.GovernanceCommonConf Modify the default version to facilitate the use of subsequent self compilation scheduling components

  val SPARK_ENGINE_VERSION = CommonVars("wds.linkis.spark.engine.version", "3.0.1")

val HIVE_ENGINE_VERSION = CommonVars("wds.linkis.hive.engine.version", "3.1.2")

Compile

Ensure that the above modifications and environments are available and implemented in sequence

    cd linkis-x.x.x
mvn -N install
mvn clean install -DskipTests

Compilation error troubleshooting

  • If there is an error when you compile, try to enter a module to compile separately to see if there is an error and adjust it according to the specific error
  • For example, the following example (the py4j version does not adapt when the group Friends adapt to the lower version of CDH): if you encounter this problem, you can adjust the version with the corresponding method to determine whether to adapt

image.png

DataSphereStudio

Original project code address Adaptation modification code reference address

The pom file of DataSphereStudio

Since DSS relies on linkis, all compilers should compile linkis before compiling DSS

<!-- scala consistent environment -->
<scala.version>2.12.10</scala.version>

dss-dolphinschuduler-token

DolphinSchedulerTokenRestfulApi: Remove type conversion

responseRef.getValue("expireTime")

web tuning

Front end compilation address Reference pr Overwrite the contents of the following directories from the master branch, or build the web based on the master branch image.png

Compile

    cd DataSphereStudio
mvn -N install
mvn clean install -DskipTests

Schedulis

Original project code address Adaptation modification code reference address

The pom file of Schedulis

       <hadoop.version>3.1.1</hadoop.version>
<hive.version>3.1.2</hive.version>
<spark.version>3.0.1</spark.version>

azkaban-jobtype

Download the jobtype file of the corresponding version (note the corresponding version): Download address: After downloading, put the entire jobtypes under jobtypes image.png

Qualitis

Original project code address

Forgerock package download

release地址 of release-0.9.1,after decompression, put it under. m2\repository\org

Compile

Gradle version 4.6

cd Qualitis
gradle clean distZip

After compiling, there will be a qualitis-0.9.2.zip file under qualitis image.png

dss-qualitis-appconn compile

Copy the appconn to the appconns under datasphere studio (create the DSS quality appconn folder), as shown in the following figure: Compile the DSS qualitis appconn. The qualitis under out is the package of integrating qualitis with DSS image.png

Exchangis

Original project code address Adaptation modification code reference address

The pom file of Exchangis

<!-- scala Consistent version -->
<scala.version>2.12.10</scala.version>

Back end compilation

Official compiled documents In the target package of the assembly package, wedatasphere-exchangis-1.0.0.tar.gz is its own service package Linkis engineplug sqoop needs to be put into linkis (lib/linkis enginecon plugins) Exchangis-appconn.zip needs to be put into DSS (DSS appconns)

mvn clean install 

image.png

Front end compilation

If you deploy the front-end using nginx yourself, you need to pay attention to the dist folder under dist image.png

Visualis

Original project code address Adaptation modification code reference address

The pom file of Visualis

<scala.version>2.12.10</scala.version>

Compile

Official compiled documents In the target under assembly, visuis server zip is the package of its own service The target of visualis appconn is visualis.zip, which is the package required by DSS (DSS appconns) Build is the package printed by the front end

cd Visualis
mvn -N install
mvn clean package -DskipTests=true

image.png

Streamis

Original project code address Adaptation modification code reference address

The pom file of Streamis

<scala.version>2.12.10</scala.version>

The pom file of streamis-project-server

       <!-- If you are 1.0.1 here, adjust it to ${dss.version} -->
<dependency>
<groupId>com.webank.wedatasphere.dss</groupId>
<artifactId>dss-sso-integration-standard</artifactId>
<version>${dss.version}</version>
<scope>compile</scope>
</dependency>

Compile

Official compiled documents Under assembly, the target package wedatasphere-streams-0.2.0-dist.tar.gz is the package of its own back-end service The stream.zip package of target under stream appconn is required by DSS (DSS appconns) dist under dist is the front-end package

cd ${STREAMIS_CODE_HOME}
mvn -N install
mvn clean install

image.png

Installation deployment

Official deployment address Common error address

Path unification

It is recommended to deploy the relevant components in the same path (for example, I unzip them all in /home/hadoop/application) image.png

Notes on linkis deployment

Deploy config folder

db.sh, the address of the links connection configured by mysql, and the metadata connection address of hive linkis-env.sh

-- The path to save the script script. Next time, there will be a folder with the user's name, and the script of the corresponding user will be stored in this folder
WORKSPACE_USER_ROOT_PATH=file:///home/hadoop/logs/linkis
-- Log files for storing materials and engine execution
HDFS_USER_ROOT_PATH=hdfs:///tmp/linkis
-- Log of each execution of the engine and information related to starting engineconnexec.sh
ENGINECONN_ROOT_PATH=/home/hadoop/logs/linkis/apps
-- Access address of yarn master node (active resource manager)
YARN_RESTFUL_URL
-- Conf address of Hadoop / hive / spark
HADOOP_CONF_DIR
HIVE_CONF_DIR
SPARK_CONF_DIR
-- Specify the corresponding version
SPARK_VERSION=3.0.1
HIVE_VERSION=3.1.2
-- Specify the path after the installation of linkis. For example, I agree to specify the path under the corresponding component here
LINKIS_HOME=/home/hadoop/application/linkis/linkis-home

If you use Flink, you can try importing it from flink-engine.sql into the database of linkis

Need to modify @Flink_LABEL version is the corresponding version, and the queue of yarn is default by default

At the same time, in this version, if you encounter the error of "1g" converting digital types, try to remove the 1g unit and the regular check rules. Refer to the following:

flink3.png

lzo

If your hive uses LZO, copy the corresponding LZO jar package to the hive path. For example, the following path:

lib/linkis-engineconn-plugins/hive/dist/v3.1.2/lib

Frequently asked questions and precautions

  • The MySQL driver package must be copied to /lib/linkis-commons/public-module/ and /lib/linkis-spring-cloud-services/linkis-mg-gateway/
  • Initialization password in conf/linkis-mg-gateway.properties -> wds.linkis.admin.password
  • ps-cs in the startup script,there may be failures, if any,use sh linkis-daemon.sh ps-cs , start it separately
  • At present, if there is time to back up the log, sometimes if the previous error log cannot be found, it may be backed up to the folder of the corresponding date
  • At present lib/linkis-engineconn-plugins have only spark/shell/python/hive,If you want appconn, flink and sqoop, go to DSS, linkis and exchangis to get them
  • Configuration file version check
linkis.properties,flink see if it is used
wds.linkis.spark.engine.version=3.0.1
wds.linkis.hive.engine.version=3.1.2
wds.linkis.flink.engine.version=1.13.2

image.png image.png

Error record

  1. Incompatible versions. If you encounter the following error, it is whether the scala version is not completely consistent. Check and compile it

1905943989d7782456c356b6ce0d72b.png

  1. Yarn configures the active node address. If the standby address is configured, the following error will appear:

1ca32f79d940016d72bf1393e4bccc8.jpg

Considerations for DSS deployment

Official installation document

config folder

db.sh: configure the database of DSS config.sh

-- The installation path of DSS, for example, is defined in the folder under DSS
DSS_INSTALL_HOME=/home/hadoop/application/dss/dss

conf folder

dss.properties

# Mainly check whether spark / hive and other versions are available. If not, add
wds.linkis.spark.engine.version=3.0.1
wds.linkis.hive.engine.version=3.1.2
wds.linkis.flink.engine.version=1.13.2

dss-flow-execution-server.properties

# Mainly check whether spark / hive and other versions are available. If not, add
wds.linkis.spark.engine.version=3.0.1
wds.linkis.hive.engine.version=3.1.2
wds.linkis.flink.engine.version=1.13.2

If you want to use dolphin scheduler for scheduling, please add the corresponding spark / hive version to this pr Reference pr

dss-appconns

Exchangis, qualitis, streamis and visualis should be obtained from the projects of exchangis, qualitis, streamis and visualis respectively

Frequently asked questions and precautions

  • Since we integrate scheduleis, qualitis, exchangis and other components into DSS, all the interfaces of these components will be called synchronously when creating a project, so we ensure that dss_appconn_instance configuration paths in the instance are correct and accessible
  • The Chrome browser recommends that the kernel use version 100 or below. Otherwise, there will be a problem that you can separate scdulis, qaulitis and other components, but you cannot log in successfully through DSS
  • Hostname and IP. If IP access is used, make sure it is IP when executing appconn-install.sh installation Otherwise, when accessing other components, you will be prompted that you do not have login or permission

ec4989a817646f785c59f6802d0fab2.jpg

Schedulis deployment considerations

Official deployment document

conf folder

azkaban.properties

# azkaban.jobtype.plugin.dir and executor.global.properties. It's better to change the absolute path here
# Azkaban JobTypes Plugins
azkaban.jobtype.plugin.dir=/home/hadoop/application/schedulis/apps/schedulis_0.7.0_exec/plugins/jobtypes

# Loader for projects
executor.global.properties=/home/hadoop/application/schedulis/apps/schedulis_0.7.0_exec/conf/global.properties

# Engine version
wds.linkis.spark.engine.version=3.0.1
wds.linkis.hive.engine.version=3.1.2
wds.linkis.flink.engine.version=1.13.2

web modular

plugins/viewer/system/conf: Here, you need to configure the database connection address to be consistent with scheduleis azkaban.properties: Configuration of user parameters and system management

viewer.plugins=system
viewer.plugin.dir=/home/hadoop/application/schedulis/apps/schedulis_0.7.0_web/plugins/viewer

Frequently asked questions and precautions

If there are resources or there are no static files such as CSS in the web interface, change the relevant path to an absolute path If the configuration file cannot be loaded, you can also change the path to an absolute path For example:

### web module
web.resource.dir=/home/hadoop/application/schedulis/apps/schedulis_0.7.0_web/web/
viewer.plugin.dir=/home/hadoop/application/schedulis/apps/schedulis_0.7.0_web/plugins/viewer

### exec module
azkaban.jobtype.plugin.dir=/home/hadoop/application/schedulis/apps/schedulis_0.7.0_exec/plugins/jobtypes
executor.global.properties=/home/hadoop/application/schedulis/apps/schedulis_0.7.0_exec/conf/global.properties

Considerations for qualitis deployment

Official deployment document

conf folder

application-dev.yml

  # The correct spark version is configured here
spark:
application:
name: IDE
reparation: 50
engine:
name: spark
version: 3.0.1

Exchange deployment considerations

Official deployment document

Frequently asked questions and precautions

If you click the data source and there is an error that has not been published, you can try to add linkisps_dm_datasource -> published_version_id Modify the published_version_id value to 1 (if it is null)

Visualis

Official deployment document

Frequently asked questions and precautions

If the preview view is inconsistent, please check whether the bin / phantomjs file is uploaded completely If you can see the following results, the upload is complete

./phantomjs -v
2.1.1

Streamis

Official deployment document

dss-appconn

Qualitis, exchangis, streams and visualis are compiled from various modules, copied to DSS appconns under DSS, and then executed appconn-install.sh under bin to install their components If you find the following SQL script errors during integration, please check whether there are comments around the wrong SQL. If so, delete the comments and try appconn install again 903ceec2f69fc1c7a2be5f309f69726.png For example, for qualitis, the following IP and host ports are determined according to their specific use

qualitis
127.0.0.1
8090

Nginx deployment example

linkis.conf: dss/linkis/visualis front end exchangis.conf: exchangis front end streamis.conf: streamis front end Scheduling and Qaulitis are in their own projects Linkis / Visualis needs to change the dist or build packaged from the front end to the name of the corresponding component here image.png image.png image.png

linkis.conf

server {
listen 8089;# Access port:
server_name localhost;
#charset koi8-r;
#access_log /var/log/nginx/host.access.log main;

location /dss/visualis {
# Modify to your own front-end path
root /home/hadoop/application/webs; # Static file directory
autoindex on;
}

location /dss/linkis {
# Modify to your own front-end path
root /home/hadoop/application/webs; # linkis Static file directory of management console
autoindex on;
}

location / {
# Modify to your own front-end path
root /home/hadoop/application/webs/dist; # Static file directory
#root /home/hadoop/dss/web/dss/linkis;
index index.html index.html;
}

location /ws {
proxy_pass http://127.0.0.1:9001;#Address of back-end linkis
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection upgrade;
}

location /api {
proxy_pass http://127.0.0.1:9001; #Address of back-end linkis
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header x_real_ipP $remote_addr;
proxy_set_header remote_addr $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_connect_timeout 4s;
proxy_read_timeout 600s;
proxy_send_timeout 12s;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection upgrade;
}

#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}

exchangis.conf

server {
listen 9800; # Access port: if the port is occupied, it needs to be modified
server_name localhost;
#charset koi8-r;
#access_log /var/log/nginx/host.access.log main;
location / {
# Modify to own path
root /home/hadoop/application/webs/exchangis/dist/dist; #Modify to your own path
autoindex on;
}

location /api {
proxy_pass http://127.0.0.1:9001; # The address of the backend link needs to be modified
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header x_real_ipP $remote_addr;
proxy_set_header remote_addr $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_connect_timeout 4s;
proxy_read_timeout 600s;
proxy_send_timeout 12s;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection upgrade;
}

#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}

streamis.conf

server {
listen 9088;# Access port: if the port is occupied, it needs to be modified
server_name localhost;
location / {
# Modify to your own path
root /home/hadoop/application/webs/streamis/dist/dist; #Modify to your own path
index index.html index.html;
}
location /api {
proxy_pass http://127.0.0.1:9001; # The address of the backend link needs to be modified
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header x_real_ipP $remote_addr;
proxy_set_header remote_addr $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_connect_timeout 4s;
proxy_read_timeout 600s;
proxy_send_timeout 12s;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection upgrade;
}

#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}