Data Pipelines Case Study - India Mutual Fund Data

InferEdge has built a scalable system for consuming structured and unstructured data from various sources. This system enables agentic retrieval, indexed and standardised storage and data analytics.

This document uses the example of India Mutual Fund industry data pipeline to explain how the system is built and how we ensure that the data it produces is correct. The principle we work to is this: whatever a human analyst would check before trusting a number, the pipeline should check first.

Architecture

Our Data Pipeline broadly contains 4 components:

• Document Extraction - fetching raw data from various sources
• Transform to Structured Data - mapping unstructured data to structured fields
• Validations - checks around completeness, timeliness and accuracy of data
• Alerts and Resolution - timely action on validations to maintain data quality

India Mutual Fund Data Overview

Process Overview

Document Extraction

We run end-of-day jobs to pull NAV, Expense Ratio, AUM and benchmark data. We expect to be able to extract data for each fund/shareclass every day.

We also have a pipeline for fetching holdings disclosure of each fund. The pipeline fetches the holdings disclosure (normally an Excel sheet) for each fund from the respective AMC’s website.

Transform to Structured Data

Every AMC discloses its holdings in its own format. While all the formats are fairly similar, there are small differences in how they group their holdings (by asset class, hedged positions, etc).

We first read the sheet to extract its raw contents. The system then identifies the header row and column labels, maps each column to a canonical field, and normalises the sheet into a single standard schema.

For example, some AMCs list futures contracts as part of total holdings, while others disclose them outside the total.

Validations

We have validation checks in place to ensure all the data is complete, accurate and available in time. In all, we have 15 validations in place - across reference-data, time-series and portfolio-holdings.

Completeness checks:

• Inception dates, category, benchmarks
• Ensure 100% of active funds have NAV, AUM and Expense Ratio

Timeliness checks:

• Ensure NAV, AUM and Expense Ratio are available on all trading days
• Ensure 100% of active funds have holdings data published after the 7th of every month

Accuracy checks:

• NAV does not deviate by more than 20% day over day
• AUM does not deviate by more than 20% day over day
• Expense Ratio does not deviate and is within plausible limits
• Expense Ratio equals the sum of Base Expense ratios and other expenses
• Holdings sum up to AUM with a tolerance of 5%

These validation checks run on a daily basis, and the flagged records are sent to the internal data quality dashboard. Alerts are raised to the data team

Metrics from the system in action

Fund Metrics Data Alerts Stats for Aug-2026

Holding Data Alerts Stats for Aug-2026 (monthly average)

Holdings data is published by funds once every month for equity funds and twice for fixed income funds. This data usually has a lot of internal inconsistencies and requires robust validations.