一种基于多源数据清洗与融合的高质量海洋观测廓线数据集构建方法OA
A High-Quality Ocean Observation Profile Datasets Construction Scheme Based on Multi-Source Data Cleaning and Fusion
[背景]随着海洋观测技术的发展,各类海洋设备、计划应运而生,海洋科学领域的研究进入以大数据为代表的"数据密集型"科研阶段.[目的]为了结合不同来源的异构海洋观测数据,形成一套大而完整的海洋观测数据集,全面提升对海洋科学问题的科研水平,本文提出一种对多源异构海洋原位观测廓线数据的标准化、标注、清洗,构建高质量海洋观测廓线数据集的方案.[方法]具体为:从若干海洋数据中心/观测机构获取多源异构原始海洋观测廓线数据以及数据描述信息;根据原始海洋观测廓线数据和描述信息确定的唯一标识符,依次对原始海洋观测廓线数据进行黑名单设备数据清洗、多版本数据清洗,以及基于时空联合特征的高频数据清洗,得到目标海洋观测廓线数据;将目标海洋观测廓线数据进行标准化处理后,进行数据质量标注和误差修订,构建高质量的廓线数据集.[结论]该方案将促进多源异构廓线数据融合应用,提升数据的一致性、准确性与可用性.
[Background]With the development of ocean observation technologies,various marine equip-ment and programs have emerged,propelling research in marine science into a"data-intensive"stage character-ized by big data.[Objective]To integrate heterogeneous ocean observation data from diverse sources into a com-prehensive and unified dataset,thereby enhancing holistic scientific capabilities in addressing marine research questions,this paper proposes a scheme for standardizing,annotating,and cleaning multi-source heterogeneous in situ ocean observation profile data to construct a high-quality ocean observation profile dataset.[Methods]Specifically,the scheme involves acquiring multi-source in-situ ocean observation profile data and corresponding metadata from several ocean data centers/agencies;applying a unique identifier derived from the raw data and de-scriptors to sequentially execute greylist filtering,multi-version filtering,and high-frequency observations filter-ing based on spatiotemporal characteristics,yielding refined ocean observation profile data;standardizing the pro-cessed data,followed by quality control and bias correction to construct a high-quality profile dataset.[Conclu-sions]This scheme promotes the application of multi-source heterogeneous profile data,improving data consis-tency,accuracy,and usability.
原惠峰;朱雨静;潘玉莹;张荣望;金钟
中国科学院计算机网络信息中心,北京 100083||中国科学院大学,北京 100190中国科学院大气物理研究所,北京 100029||中国科学院大学,北京 100190中国科学院大气物理研究所,北京 100029中国科学院南海海洋研究所,广东 广州 510301中国科学院计算机网络信息中心,北京 100083||中国科学院大学,北京 100190
海洋大数据数据清洗海洋观测数据集
ocean big datadata cleanocean observationdatasets
《数据与计算发展前沿》 2026 (3)
68-80,13
亚洲合作资金项目(102173250600000000010)国家重点研发计划(2023YFB3001900)
评论