Speaker
Description
With the increasing of data storage and data access in EAST experiment, standardized data access and user behavior monitoring become even more important. Therefore, a perfect EAST MDSplus logging system is required, and a mature log data management system is built on this basis. The EAST MDSplus log data management system architecture includes four parts: the log data collection layer, the data storage layer, the data analysis layer and the data application layer. (1) The data collection layer uses MDSplus hook function to improve the log system, and adopts log rotate mechanism for log rotation cutting, which effectively avoids the problem of excessive log storage. (2) The data storage layer uses HDFS for distributed offline data storage to support data multi-copy backup, which improves data security performance and non-volatileness. Moreover, Kafka is also used as the stream data storage middleware to ensure the release and subscription of log messages with high throughput. (3) The data analysis layer uses MapReduce to clean the offline data and establish the log data warehouse belonging to EAST. In addition, Spark Streaming is used to calculate log stream data to ensure the quasi-real-time nature of log data. (4) The data application layer is designed to visualize analysis layer data and intuitively reflect the entire EAST log data access status. Each layer of the log management system framework provides a corresponding interface, which reduces the coupling degree of the system. The EAST MDSplus log data management system aims at providing a standardized management solution for the whole EAST log data and it will be implemented in the next campaign of EAST experiment.
