Skip to main content
Version: 1.1.1

Data Source Management Service Architecture

Background

Exchangis0.x and Linkis0.x in earlier versions both have integrated data source modules. In order to manage the ability to reuse data sources, Linkis reconstructs the data source module based on linkis-datasource (refer to related documents), and converts the data source Management unpacks into data source management services and metadata management services。

This article mainly involves the DataSource Manager Server data source management service, which provides the following functions:

1)、Linkis unified management service startup and deployment, does not increase operation and maintenance costs, reuse Linkis service capabilities;

2)、Provide management services of graphical interface through Linkis Web. The interface provides management services such as new data source, data source query, data source update, connectivity test and so on;

3)、 the service is stateless, multi-instance deployment, so that the service is highly available. When the system is deployed, multiple instances can be deployed. Each instance provides services independently to the outside world without interfering with each other. All information is stored in the database for sharing.

4)、Provide full life cycle management of data sources, including new, query, update, test, and expiration management.

5)、Multi-version data source management, historical data sources will be saved in the database, and data source expiration management is provided.

6)、The Restful interface provides functions, a detailed list: data source type query, data source detailed information query, data source information query based on version, data source version query, get data source parameter list, multi-dimensional data source search, get data source environment query and Update, add data source, data source parameter configuration, data source expiration setting, data source connectivity test.

Architecture Diagram

datasource Architecture diagram

Architecture Description

1、The service is registered in the Linkis-Eureak-Service service and managed in a unified manner with other Linkis microservices. The client can obtain the data source management service by connecting the Linkis-GateWay-Service service and the service name data-source-manager.

2、The interface layer provides other applications through the Restful interface, providing additions, deletions, and changes to data sources and data source environments, data source link and dual link tests, data source version management and expiration operations;

3、The Service layer is mainly for the service management of the database and the material library, and permanently retains the relevant information of the data source;

4、The link test of the data source is done through the linkis metastore server service, which now provides the mysql\es\kafka\hive service

Core Process

1、To create a new data source, firstly, the user of the new data source will be obtained from the request to determine whether the user is valid. The next step will be to verify the relevant field information of the data source. The data source name and data source type cannot be empty. The data source name is used to confirm whether the data source exists. If it does not exist, it will be inserted in the database, and the data source ID number will be returned.

2、 To update the data source, firstly, the user of the new data source will be obtained from the request to determine whether the user is valid. The next step will be to verify the relevant field information of the new data source. The data source name and data source type cannot be empty. It will confirm whether the data source exists according to the data source ID number. If it does not exist, an exception will be returned. If it exists, it will be further judged whether the user has update permission for the data source. The user is the administrator or the owner of the data source. Only have permission to update. If you have permission, the data source will be updated and the data source ID will be returned.

3、 To update the data source parameters, firstly, the user of the new data source will be obtained from the request to determine whether the user is valid, and the detailed data source information will be obtained according to the passed parameter data source ID, and then it will be determined whether the user is the owner of the changed data source or not. For the administrator, if there is any, the modified parameters will be further verified, and the parameters will be updated after passing, and the versionId will be returned.

Entity Object

Class NameDescribe
DataSourceTypeIndicates the type of data source
DataSourceParamKeyDefinitionDeclare data source property configuration definitions
DataSourceData source object entity class, including permission tags and attribute configuration definitions
DataSourceEnvData source environment object entity class, which also contains attribute configuration definitions
DataSourceParameterData source specific parameter configuration
DatasourceVersionData source version details

Database Design

Database Diagram:

Data Table Definition:

Table:linkis_ps_dm_datatsource <-->Object:DataSource

Serial NumberColumnDescribe
1idData source ID
2datasource_nameData source name
3datasource_descData source detailed description
4datasource_type_idData source type ID
5create_identifycreate identify
6create_systemSystem for creating data sources
7parameterData source parameters
8create_timeData source creation time
9modify_timeData source modification time
10create_userData source create user
11modify_userData source modify user
12labelsData source label
13version_idData source version ID
14expireWhether the data source is out of date
15published_version_idData source release version number

Table Name:linkis_ps_dm_datasource_type <-->Object:DataSourceType

Serial NumberColumnDescribe
1idData source type ID
2nameData source type name
3descriptionData source type description
4optionType of data source
5classifierData source type classifier
6iconData source image display path
7layersData source type hierarchy

Table:linkis_ps_dm_datasource_env <-->Object:DataSourceEnv

Serial NumberColumnDescribe
1idData source environment ID
2env_nameData source environment name
3env_descData source environment description
4datasource_type_idData source type ID
5parameterData source environment parameters
6create_timeData source environment creation time
7create_userData source environment create user
8modify_timeData source modification time
9modify_userData source modify user

Table:linkis_ps_dm_datasource_type_key <-->Object:DataSourceParamKeyDefinition

Serial NumberColumnDescribe
1idKey-value type ID
2data_source_type_idData source type ID
3keyData source parameter key value
4nameData source parameter name
5default_valueData source parameter default value
6value_typeData source parameter type
7scopeData source parameter range
8requireIs the data source parameter required?
9descriptionData source parameter description
10value_regexRegular data source parameters
11ref_idData source parameter association ID
12ref_valueData source parameter associated value
13data_sourceData source
14update_timeupdate time
15create_timeCreate Time

Table:linkis_ps_dm_datasource_version <-->Object:DatasourceVersion

Serial NumberColumnDescribe
1version_idData source version ID
2datasource_idData source ID
3parameterThe version parameter of the data source
4commentcomment
5create_timeCreate Time
6create_userCreate User