Archiving Data

Overview

The LogiSense Billing application has a built in data archival tool which removes old data from the system to prevent its database from growing too large and prevent certain tables from becoming bloated.

Data that is archived is sent to an S3 bucket and organized by day and target (targets are explained below).

Archived Data and Targets

The table below details data archive targets and recommended settings for them. These settings can be modified per target if required. The general default is to store data in a production system for 3 months.

Note: There are certain targets that cannot have their archive type changed (for example, API Logs currently can’t be archived, they must be purged).

Release	Target	Description	Archive Type and Duration	Reason

Release	Target	Description	Archive Type and Duration	Reason
10.4.0	Usage	Data relating to Usage Records (Rated, Mediated, Extensions, Aggregates).	Archive & Purge - 3 Months	Usage Records that were processed could be audited at a later date, by auditors or customer disputes.
10.4.0	Unprocessed Usage Exceptions	The usage exceptions that have not been resolved.	Purge - 3 Months	If the exceptions haven’t been dealt with after 3 months, they probably aren’t going to be resolved or charged.
10.4.0	Processed Usage Exceptions	Usage Exception records that have been sent for Rerating.	Purge - 1 Day	These would normally get deleted once sent for reprocessing.
10.4.0	Rerated Usage Records	Original Usage Records that have been sent in for Rerating, making the original record obsolete.	Purge - 1 Day	These would normally get deleted once sent for reprocessing.
10.4.0	API Logs	The request/response API logs from every API call made to the system.	Purge - 5 days	There are many API calls made every day in the system and this table fills up quickly. This data also is rarely referenced again.
10.4.0	Tax Logs	The Tax Logs from every tax calculation made in the system.	Archive & Purge - 3 Months	Tax calculations are audited annually and could be challenged on any invoice.
10.4.0	Webhook Logs	The response data from every webhook sent from the system.	Purge - 3 Months	These generally are not required once the change has been made or the action has completed.
10.5.0	Data Import Tracking and Batch details	Usage API and Data Import Tracking details	Purge - 3 Months	Once imported, this data is used by the system from the UDRMediated and UsageRated tables. The import details are no longer used by the system.

Monitoring Data Archival Activity

Within the LogiSense Billing AdminPortal data archiving activity can be tracked by loading the Operations / Monitor / Data Archive screen.

All data archive operations are by default configured to run for a maximum of six hours. If an operation takes more than six hours it will be cancelled and listed as Cancelled on this screen.

Downloading Archived Data from S3

When data is successfully archived to an S3 bucket it will be listed as Completed on the Data Archive screen and the archived files will be listed under the Files tab. From this tab you can download the archived files if you are required to keep them longer than they will be stored in the S3 bucket.