Since 18 of December 2019 uses Nucleus credentials. Visit our help pages for information on how to Register and Sign-in using Nucleus.
13-17 May 2019
Daejeon, Republic of Korea
Europe/Vienna timezone
Meeting Material is now available and accessible from the left-menu

EAST MDSplus Log Data Management System

15 May 2019, 15:25
2h 35m
Daejeon, Republic of Korea

Daejeon, Republic of Korea

Board: P/3-9
Poster Database Techniques for Information Storage and Retrieval Poster


Feng WANG (Institute of Plasma Physics, Chinese Academy of Sciences)


With the increasing of data storage and data access in EAST experiment, standardized data access and user behavior monitoring become even more important. Therefore, a perfect EAST MDSplus logging system is required, and a mature log data management system is built on this basis. The EAST MDSplus log data management system architecture includes four parts: the log data collection layer, the data storage layer, the data analysis layer and the data application layer. (1) The data collection layer uses MDSplus hook function to improve the log system, and adopts log rotate mechanism for log rotation cutting, which effectively avoids the problem of excessive log storage. (2) The data storage layer uses HDFS for distributed offline data storage to support data multi-copy backup, which improves data security performance and non-volatileness. Moreover, Kafka is also used as the stream data storage middleware to ensure the release and subscription of log messages with high throughput. (3) The data analysis layer uses MapReduce to clean the offline data and establish the log data warehouse belonging to EAST. In addition, Spark Streaming is used to calculate log stream data to ensure the quasi-real-time nature of log data. (4) The data application layer is designed to visualize analysis layer data and intuitively reflect the entire EAST log data access status. Each layer of the log management system framework provides a corresponding interface, which reduces the coupling degree of the system. The EAST MDSplus log data management system aims at providing a standardized management solution for the whole EAST log data and it will be implemented in the next campaign of EAST experiment.

Primary author

Feng WANG (Institute of Plasma Physics, Chinese Academy of Sciences)

Presentation Materials