Overview
The LogiSense Billing application has a built in data archival tool which removes old data from the system to prevent its database from growing too large and prevent certain tables from becoming bloated.
Data that is archived is sent to an S3 bucket and organized by day and target (targets are explained below).
Archived Data and Targets
The table below details data archive targets and recommended settings for them. These settings can be modified per target if required. The general default is to store data in a production system for 3 months.
Note: There are certain targets that cannot have their archive type changed (for example, API Logs currently can’t be archived, they must be purged).
Release | Target | Description | Archive Type and Duration | Reason |
|---|---|---|---|---|
10.4.0 | Usage | Data relating to Usage Records (Rated, Mediated, Extensions, Aggregates). | Archive & Purge - 3 Months | Usage Records that were processed could be audited at a later date, by auditors or customer disputes. |
10.4.0 | Unprocessed Usage Exceptions | The usage exceptions that have not been resolved. | Purge - 3 Months | If the exceptions haven’t been dealt with after 3 months, they probably aren’t going to be resolved or charged. |
10.4.0 | Processed Usage Exceptions | Usage Exception records that have been sent for Rerating. | Purge - 1 Day | These would normally get deleted once sent for reprocessing. |
10.4.0 | Rerated Usage Records | Original Usage Records that have been sent in for Rerating, making the original record obsolete. | Purge - 1 Day | These would normally get deleted once sent for reprocessing. |
10.4.0 | API Logs | The request/response API logs from every API call made to the system. | Purge - 5 days | There are many API calls made every day in the system and this table fills up quickly. This data also is rarely referenced again. |
10.4.0 | Tax Logs | The Tax Logs from every tax calculation made in the system. | Archive & Purge - 3 Months | Tax calculations are audited annually and could be challenged on any invoice. |
10.4.0 | Webhook Logs | The response data from every webhook sent from the system. | Purge - 3 Months | These generally are not required once the change has been made or the action has completed. |
10.5.0 | Data Import Tracking and Batch details | Usage API and Data Import Tracking details | Purge - 3 Months | Once imported, this data is used by the system from the UDRMediated and UsageRated tables. The import details are no longer used by the system. |
Downloading Archived Data from S3
Monitoring Data Archival Activity
Within the LogiSense Billing AdminPortal data archiving activity can be tracked by loading the Operations / Monitor / Data Archive screen.
All data archive operations are by default configured to run for a maximum of six hours. If an operation takes more than six hours it will be cancelled and listed as Cancelled on this screen.