Skip to main content
Version: 1.1.3

EngineConnPlugin (ECP) architecture design

The engine connector plug-in is an implementation that can dynamically load the engine connector and reduce the occurrence of version conflicts. It has the characteristics of convenient expansion, fast refresh, and selective loading. In order to allow developers to freely extend Linkis's Engine engine, and dynamically load engine dependencies to avoid version conflicts, the EngineConnPlugin was designed and developed, allowing new engines to be introduced into the execution life cycle of computing middleware by implementing established plug-in interfaces. The plug-in interface disassembles the definition of the engine, including parameter initialization, allocation of engine resources, construction of engine connections, and setting of engine default tags.

一、ECP architecture diagram

Introduction to the second-level module:

EngineConn-Plugin-Server

The engine connector plug-in service is an entrance service that provides external registration plug-ins, management plug-ins, and plug-in resource construction. The engine plug-in that is successfully registered and loaded will contain the logic of resource allocation and startup parameter configuration. During the engine initialization process, EngineConn Other services such as Manager call the logic of the corresponding plug-in in Plugin Server through RPC requests.

Core ClassCore Function
EngineConnLaunchServiceResponsible for building the engine connector launch request
EngineConnResourceFactoryServiceResponsible for generating engine resources
EngineConnResourceServiceResponsible for downloading the resource files used by the engine connector from BML

EngineConn-Plugin-Loader Engine Connector Plugin Loader

The engine connector plug-in loader is a loader used to dynamically load the engine connector plug-ins according to request parameters, and has the characteristics of caching. The specific loading process is mainly composed of two parts: 1) Plug-in resources such as the main program package and program dependency packages are loaded locally (not open). 2) Plug-in resources are dynamically loaded from the local into the service process environment, for example, loaded into the JVM virtual machine through a class loader.

Core ClassCore Function
EngineConnPluginsResourceLoaderLoad engine connector plug-in resources
EngineConnPluginsLoaderLoad the engine connector plug-in instance, or load an existing one from the cache
EngineConnPluginClassLoaderDynamically instantiate engine connector instance from jar

EngineConn-Plugin-Cache engine plug-in cache module

Engine connector plug-in cache is a cache service specially used to cache loaded engine connectors, and supports the ability to read, update, and remove. The plug-in that has been loaded into the service process will be cached together with its class loader to prevent multiple loading from affecting efficiency; at the same time, the cache module will periodically notify the loader to update the plug-in resources. If changes are found, it will be reloaded and refreshed automatically Cache.

Core ClassCore Function
EngineConnPluginCacheCache loaded engine connector instance
RefreshPluginCacheContainerEngine connector that refreshes the cache regularly

EngineConn-Plugin-Core: Engine connector plug-in core module

The engine connector plug-in core module is the core module of the engine connector plug-in. Contains the implementation of the basic functions of the engine plug-in, such as the construction of the engine connector start command, the construction of the engine resource factory and the implementation of the core interface of the engine connector plug-in.

Core ClassCore Function
EngineConnLaunchBuilderBuild Engine Connector Launch Request
EngineConnFactoryCreate Engine Connector
EngineConnPluginThe engine connector plug-in implements the interface, including resources, commands, and instance construction methods.
EngineResourceFactoryEngine Resource Creation Factory

EngineConn-Plugins: Engine connection plugin collection

The engine connection plug-in collection is used to place the default engine connector plug-in library that has been implemented based on the plug-in interface defined by us. Provides the default engine connector implementation, such as jdbc, spark, python, shell, etc. Users can refer to the implemented cases based on their own needs to implement more engine connectors.

Core ClassCore Function
engineplugin-jdbcjdbc engine connector
engineplugin-shellShell engine connector
engineplugin-sparkspark engine connector
engineplugin-pythonpython engine connector