Time Sensitive Business Intelligence - Big Data Processing Methodology for Frequently Changing Business Dimensions

Background: In the competitive data driven business world, business Intelligence (BI) team converts the raw operational data to information for decision making. Operational system captures the day-to-day operations and BI database refreshes operational data periodically. Methods: A component to create the metadata repository which maintains the current BI database summary by logical data partitioning using range based partition for frequently changing parameter which are critical to business. During different time frequency, using metadata repository component identifies the latest data victim between BI vs operational data and refreshes the modified victim to BI database. Findings: In traditional data loading approach from Operational system to BI database, huge volume of data gets refreshed periodically irrespective of modifications, which leads to higher processing time and cost. To overcome this limitation, this proposed methodology helps to identify the latest data victims present in operational systems instead of bulk data replacement which can minimize the processing time and enables faster data transformation to achieve "Time to Decision" and "Quick to Market" implementation for business enhancements. Also component can be scheduled for data refresh with different time frequency for multiple critical to business as well as frequently changing parameters. Applications: In financial, traffic, weather, e-business, Logistics&stock management transactions, data changes frequently and process big data periodically to gain real-time knowledge discovery for time sensitive decision making.