A leading trade services platform needed assistance in fixing and future-proofing their data pipeline, which we took from 2.2 million conflicts down to just 15,000. During this process, we introduced real-time synchronisation, removed manual overheads, and ensured scalability and stability far into the future.
Challenges & Limitations
The client faced several critical challenges:
-
WebOffice (SQL-based) serving as the primary data source, causing data inconsistencies and synchronisation issues
-
Lack of real-time data updates between WebOffice and Salesforce, leading to delays and mismatches
-
High number of data conflicts (2.2 million), making reconciliation inefficient
-
No automated monitoring, making it hard to detect and prevent data integrity issues
Solutions & Improvements
To address these challenges, our team took the following approach:
-
Transitioned the primary data source from WebOffice to Salesforce, leveraging an event-driven framework on GCP
-
Implemented gRPC synchronisation with Salesforce platform events, enabling real-time data updates
-
Identified and resolved root causes of data inconsistencies, reducing conflicts from 2.2 million to just 15,958
-
Developed an automated daily monitoring system, generating Excel-based reports to track and prevent data discrepancies
Achievements & Impact
The successful execution of these initiatives delivered significant business and technical benefits:
-
Data Integrity: Reduced data conflicts from 2.2 million to 15,958, significantly improving system reliability
-
Real-Time Synchronisation: Implemented gRPC-based real-time updates, eliminating manual reconciliation efforts
-
Operational Efficiency: Introduced automated monitoring & reporting, ensuring data integrity and preventing future discrepancies
-
Platform Stability: Enhanced system architecture and error-handling protocols, reducing system failures and improving scalability
-
New Foundational Service: Leading development of a trade profile management service, designing resilient APIs and error-handling mechanisms for long-term system stability
