Platform Methodology & Data Pipeline

01 / Ingestion Sources

Alethman operates exclusively on data released to the public domain or under open-data terms. Our pipelines systematically poll and ingest records from three primary nodes:

• Central Public Procurement Portal (CPPP): Ingestion of active tenders, technical evaluation summaries, financial evaluation matrices, and final award details.
• Government e-Marketplace (GeM): Ingestion of contract logs, direct buy agreements, catalog lists, and purchase order records.
• Public Financial Management System (PFMS): Ingestion of payment vouchers, scheme release registers, and bank disbursement transaction IDs.

Portals are polled at scheduled intervals ranging from 24 to 72 hours, depending on system latency and publication schedules.

02 / Data Normalisation

Raw source data exhibits high levels of structural variance. The normalisation layer cleanses and maps raw strings into structured transactional schemas:

• Date Harmonisation: Mapped to ISO 8601 formatting (YYYY-MM-DD) across inconsistent entry forms.
• Currency & Values: Standardized numerical values, accounting for Crores, Lakhs, and Rupees declarations into single float decimals.
• Corporate Identifiers: Entity names undergo token sanitisation (e.g., removing "Ltd", "PVT", "LLP" variations, normalizing whitespace, and resolving character encodings) to establish base strings.

03 / Deterministic Matching

Deterministic matches occur when shared, verified identifiers exist across systems.

For example, if a GeM Contract explicitly registers a CPPP Tender ID, our database joins the records with 100% confidence. This stage forms the backbone of the reconciled graph and is executed first before probabilistic heuristics are applied.

04 / Probabilistic Matching

When unique shared identifiers are missing, malformed, or blank, Alethman utilizes probabilistic entity resolution.

We compare multi-dimensional weights across:

• Vendor Name Similarity: Calculated using Jaro-Winkler and Levenshtein distance metrics on cleansed corporate names.
• Buyer Entity Mapping: Hierarchical matching to identify parent department variations.
• Transactional Value Windows: Value correlations within an acceptable variance boundary (accounting for taxes or processing fees).
• Date Constraints: Matching events occurring within logical administrative windows (e.g., a PFMS disbursement occurring 5 to 45 days post GeM contract award date).

05 / Confidence Scoring

Every joined record receives an explicit confidence score represented as a percentage.

• >95% Confidence: Automatic join. Deterministic identification confirmed or highly dense probabilistic correlation on multiple dimensions.
• 75% - 95% Confidence: Auto-flagged with "Needs Review". The join is displayed but highlights candidate matches and mismatched fields.
• <75% Confidence: Match rejected. Records are kept separate to prevent false correlations.

06 / Conflict Handling

Data sources can contradict one another. For example, a CPPP award record may declare a vendor payout value that differs from the corresponding PFMS bank clearance ledger value.

Alethman does not silently overwrite conflict values. We retain both source inputs, display them side-by-side in the interface, and apply a conflict flag, leaving final interpretation to the analytical user.

07 / Corrections & Feedback

As public source systems modify records or issue corrections, Alethman updates its datasets. We maintain a formal process for corrections:

If a government department or vendor identifies an incorrect join or mapping, they can submit an official correction request. We verify the request against primary source records and document revisions in our system changelogs. We do not edit historical raw data fields—we only adjust matching associations.

08 / Source Traceability

Reconciled datasets never exist as floating entities. Every data cell retains an unbroken link to its origin:

1. Originating system tag (e.g., CPPP Portal)
2. Primary source identifier (e.g., Tender ID)
3. Capture date and ingest run ID
4. Unmodified JSON raw payload cache

This ensures that any analytical output is completely auditable and verifiable.

09 / Methodology Changelog

Reconciliation heuristics and parser rules are continually updated. We log version modifications below to maintain institutional accountability.