Ingestion

Efficient Data Ingestion for Accurate Reconciliation

The Ingestion screen is a key part of our Reconciliation Feature, designed to make data management easier by letting businesses upload and manage multiple data sources at once. This tool is perfect for businesses that deal with large volumes of transactions across different platforms, ensuring that all necessary data is collected and ready for reconciliation.

With the ability to upload multiple files at the same time, track upload status, view detailed metadata, and apply filters, our Ingestion tool gives full control and visibility over the data ingestion process. This ensures that your reconciliation is based on accurate and up-to-date data, reducing the risk of errors and mismatches.

Data Ingestion Walkthrough

This guide offers a straightforward, step-by-step process for ingesting data for reconciliation on the Portone platform.

In this walkthrough, users will:

• Upload and Verify Data: Learn how to efficiently upload data for reconciliation.
• Access the Data Ingestion Screen: Navigate to the appropriate section for managing data sources.
• Select time range: Choose relevant time frames to filter the data ingestion list.
• Check Upload Status: Monitor the status of uploaded files to ensure successful ingestion.
• View Detailed Summaries: Access summaries of ingested records and track reconciliations related to data sources.

This process is essential for maintaining accurate and timely reconciliation, helping users streamline their operations.

Data Ingestion Sequence Diagram

The Reconciliation Ingestion Flow in PortOne is designed to handle data uploads seamlessly and provide visibility into the status of each ingestion.

The key steps in this process include:

1. Select Data Source and Upload File:
   Users begin by selecting the relevant data source from the list of configured sources.
   Files can be uploaded manually or ingested through alternative channels like:
   • SFTP: Securely transfer files via SFTP for automated ingestion.
   • Email Auto-forwarding: Set up email forwarding to automatically process files received via email.
   • API Integration: If APIs are available, data can be fetched directly from the source, allowing for real-time ingestion.
2. Create Ingestion Record:
   Once the file is uploaded, the system generates an ingestion record that logs key metadata such as the file name, source, upload time, and user details.
3. Display List of Ingestions:
   The ingestion screen displays a list of all ingested files with the following information:
   • Ingestion Status: Indicates the current processing stage:
     • Initiated: The system has initiated data processing.
     • Validation Failed: Issues identified during the validation step, requiring user intervention.
     • Ingestion Failed: Issues occurred during ingestion, PortOne team will be notified and the issue will be resolved on priority.
     • Success: Data is successfully ingested and ready for reconciliation.
   • Metadata: Shows relevant details like file size, upload date, data source, and number of records.
4. Validate and Ingest Data:
   The system performs data validation checks to ensure the uploaded files meet the required format and standards.
   If validation is successful, the data is ingested into the reconciliation system. Errors during validation are highlighted for user review and correction.
5. Update Ingestion Status:
   The status of each ingestion record is updated in real time to reflect the current state.

Key Features

• Bulk File Uploads: Easily upload multiple files from different sources at once. This is especially useful for businesses needing to reconcile data from various platforms like payment gateways, marketplaces, and internal systems.
• Real-Time Status Tracking: Track the progress of each file upload in real-time, allowing you to quickly identify and fix any issues during the ingestion process.
• Detailed Metadata: Get detailed information for each file, including the source, upload status, and any filters applied. This ensures you know the context of each data source, helping with reconciliation.
• Error Handling and Resolution: Detect and fix errors like missing data or formatting issues during ingestion. Our tool provides notifications and guides you on how to resolve these issues, ensuring your data is ready for reconciliation.

Data Ingestion Functionalities

The Data Ingestion screen is key for managing and preparing data for reconciliation. It helps businesses handle large amounts of different data sources by allowing bulk uploads, tracking progress in real-time, and providing detailed insights for each file. This ensures that all transaction data from payment gateways, e-commerce platforms, or internal records is accurately collected and ready for reconciliation.

Bulk Uploads from Multiple Sources

1. Upload Multiple Files: Quickly upload several files from different sources like PSPs, marketplaces, and accounting systems, perfect for businesses that deal with many data files each day.
2. Support for Various File Types: Accepts different file formats, making it easy to gather data from diverse platforms.

Real-Time Tracking

1. Live Status Updates: Monitor the progress of each upload in real-time, with clear indicators showing if the upload is Pending, Successful, or Failed.
2. Detailed Progress Reports: See real-time updates on total files, processed records, successful uploads, and any failures, helping you handle issues proactively.

Detailed Metadata Information

1. File Metadata Display: Each upload shows important details like the upload date, source, and validation status for transparency and context.
2. Automatic File Labels: Categorizes metadata to help you quickly identify the source of each file, distinguishing between settlement files and transaction records.

Error Handling and Validation

1. Automatic Error Detection: The tool checks for formatting errors, missing information, or unsupported file types.
2. Validation Feedback: Files are validated to meet reconciliation standards. If errors are found, users receive detailed feedback and suggestions to fix them before reconciliation.

Date Range and Source Filters

1. Custom Filters: Filter data by date range, source, or validation status for a focused view of specific transactions or periods.
2. Search Functionality: Quickly find specific files or data sources to speed up data verification and management.

Flexible Ingestion Options

1. Multiple Ingestion Options: Upload files directly or fetch them from secure FTP sources, API integrations, cloud drive support, via email etc, allowing for different workflows.
2. Scheduled Uploads: Set up regular upload schedules for recurring data sources, making it easier to manage frequent reconciliations.

Benefits of the Data Ingestion Screen

1. Centralized Data Management: Manage large amounts of data from various sources with clear visibility and control.
2. Reduced Errors: Proactive error handling lowers the chances of discrepancies during reconciliation, improving accuracy.
3. Increased Efficiency: Bulk uploads, real-time tracking, and custom filters save time and enhance productivity for teams handling reconciliation.

The Data Ingestion screen provides the control, flexibility, and accuracy needed to ensure that reconciliation data is complete and timely, forming a solid foundation for financial reliability.

Practical Applications

• Multi-Platform Businesses: Reconcile data from multiple platforms like payment gateways, e-commerce sites, and internal systems.
• Large Volume Transactions: Efficiently manage and reconcile large volumes of transactions without the risk of overload or missed data.
• Custom Filters: Apply filters to focus on specific data sets, such as transactions from certain dates, sources, or types.

Conclusion

Effective data ingestion is the backbone of successful reconciliation. Our Ingestion tool gives you the power to handle large amounts of data from various sources, ensuring all the important information is captured, organized, and ready for accurate reconciliation. With real-time tracking, detailed metadata, and strong error handling, you can be confident that your reconciliation process uses the most accurate and complete data.