Invention Grant
- Patent Title: Meta-data driven data ingestion using MapReduce framework
- Patent Title (中): 使用MapReduce框架进行元数据驱动的数据采集
-
Application No.: US13466981Application Date: 2012-05-08
-
Publication No.: US08949175B2Publication Date: 2015-02-03
- Inventor: Mingxi Wu , Songting Chen
- Applicant: Mingxi Wu , Songting Chen
- Applicant Address: US CA Redwood City
- Assignee: Turn Inc.
- Current Assignee: Turn Inc.
- Current Assignee Address: US CA Redwood City
- Agency: Kwan & Olynick LLP
- Main IPC: G06F17/30
- IPC: G06F17/30

Abstract:
A generic approach for automatically ingesting data into an HDFS (Hadoop File System) based data warehouse includes a datahub server, a generic pipelined data loading framework, and a meta-data model that, together, address data loading efficiency, data source heterogeneities, and data warehouse schema evolvement. The loading efficiency is achieved via the MapReduce scale-out solution. The meta-data model is comprised of configuration files and a catalog. The configuration file is setup per ingestion task. The catalog manages the data warehouse schema. When a scheduled data loading task is executed, the configuration files and the catalog collaboratively drive the datahub server to load the heterogeneous data to their destination schemas automatically.
Public/Granted literature
- US20130275363A1 META-DATA DRIVEN DATA INGESTION USING MAPREDUCE FRAMEWORK Public/Granted day:2013-10-17
Information query