Usage Data

EngageIP contains core features for ingesting data that details the consumption of measurable resources. Usage data records (UDR) can enter the EngageIP system in two ways:

Real Time API
Feed files

This section will discuss the feed files and how to ingest the data within these feeds. Rating of data from feeds is not done in real time but is instead performed at regular intervals (typically every 10 seconds).

A feed is usage data of a specific type that is contained within one or more files on the local EngageIP file system. The format of the feed files can be based on the EngageIP standard for a UDR record or any format the customer chooses. Most customers will have a format that is not based on the EngageIP UDR record.

The start of the feed workflow is the EngageIPRatingService. EngageIPRatingService is a Windows Service by the name "EngageIP Rating Service" which is installed and configured as part of deploying a customer instance of EngageIP. When EngageIPRatingService runs it invokes the UDR Rating Engine to perform the majority of the business logic.

Within the UDR Rating Engine Rate() method the following sequence of steps happens:

Rerating will pick up reconfigurations in buckets that might impact this rating cycle due to UserPackage additions, UserPackage cancellations, or rate changes
Iterate through all configured and enabled feeds
1. Setup each feed as a data source. A data source is a custom code implementation of the IUDRDataSource C# Interface
2. Initialize the data source using the custom code implementation of the Init method in the IUDRDataSource C# Interface
3. Each line from the data source feed file is retrieved by calling the data source implementation of the IUDRDataSource Read method
4. The data source Mediate() method is executed to transform the feed data into EngageIP UDR records. The Mediate method is a custom code implementation of the IUDRDataSource C# Interface
5. Additional mediation activities might need to be performed
  1. The UDR Rating Engine iterates through all configured preprocessors and executes the Process() method on them. Each preprocessor is a custom code implementation of the C# IUDRProcessor Interface
  2. A final data linkage is needed to match up the incoming usage data record with the information previously configured in the EngageIP database. The database table UDRUserIdentifierHistory (UUIH) is queried and added to the UDR record. The UUIH data is added to the database prior to the usage data ingestion using business logic in a custom code implementation derivation of the C# UUIHWorkflowBase (a derived implementation of the C# IAction Interface). This custom code module frequently implements other C# Interfaces as well. For example, IOverrideuserService, IUseUserServiceAttributeProfileAnswer, and IUseUserAttributeProfileAnswer
  3. The next custom code integration point is the post processors. Similar to the preprocessors above UDR Rating Engine iterates through all configured post processors and executes them. Each Post processor is a custom code implementation of the C# IUDRProcessor Interface
  4. Later in the workflow the UDR Rating Engine iterates through each configured custom code implementations of the C# IUDRProcessorInUDRWriter Interface and executes the Process() method on them

UDR records can be extended with additional columns to meet a customer’s needs. In the method Logisense.Boss.Logic.CustomCodeInstallerHelper.EnsureCustomFieldExists() there is a call to Logisense.Boss.Logic.Core.UDRCustomField.Create(). This Create method executes the UDRCustomField base class Create() method which adds records to the UDRCustomField table while the derived class Create() method adds the corresponding changes to the UDR table:

Add a column to UDR
Add an index on the new column in UDR

This logic is commonly inside the Install() method in the custom code which is implementing the C# ICustomCodeInstaller Interface.

Interfaces

The following interfaces are available to be used:

IUDRDataSource - Used to implement a mediation script for rating
IUDRProcessor - Used to insert custom code into the rating workflow
IUDRProcessorInUDRWriter - Used to insert custom code into the rating workflow within the UDRWriter process
IUDRReratingProcessor - Used to insert custom code into the re-rating workflow

public interface IUDRDataSource {
	string Name { get; }

	/// <summary>
	/// Called when the rating engine is starting up. 
	/// </summary>
	/// <param name="details">Passed from UDRFeed.DataSourceDetails</param>
	/// <param name="udrclasses">Maps UDRClass names to their database ID</param>
	void Init(UDRFeed feed, Dictionary<string, int> udrclasses);

	/// <summary>
	/// Reads a single UDR from the data source
	/// </summary>
	/// <returns>A string array that contains the raw field data. This will be passed to the Mediate function later. If there are no more records an empty array should be returned.</returns>
	string[] Read();

	/// <summary>
	/// Called by the rating engine to mediate a single record.
	/// </summary>
	/// <param name="rawFields">The raw fields of the record</param>
	/// <returns>A UDR class with the appropriate fields filled in</returns>
	Logisense.Boss.Logic.Core.UDR Mediate(string[] rawFields, ref Dictionary<string, string> ratingContext);

	/// <summary>
	/// Called when the current rating run is complete. Any cleanup should be done here (closing files, etc)
	/// </summary>
	void Close();
}

public interface IUDRProcessor {
	string Name { get; }
	string Description { get; }
	/// <summary>
	/// Indicates whether this processor should be run pre- or post- rating.
	/// </summary>
	UDRProcessorType Type { get; }

	/// <summary>
	/// Process a UDR Record
	/// </summary>
	/// <param name="udr">The UDR record</param>
	/// <returns>False to halt processing of the UDR (ie drop it), true otherwise</returns>
	bool Process(Logisense.Boss.Logic.Core.UDR udr, ref Dictionary<string, string> ratingContext);
}

/*
* WriteUDR must be called within a transaction for the following assumption to hold
*  otherwise UDRBiller.Amount would suffice.
* 1. WriteUDR calls are currently wrapped in a transaction
* 2. WriteUDR updates the UDRBiller.Amount
* 3. WriteUDR calls IUDRProcessorInUDRWriter
* 4. The transaction is not finished when WriteUDR exits so the updated UDRBiller has not been persisted
* 5. Thus to get the most up to date usage amount the UDRBiller.Amount and UDR.Amount must be added together
*/
public interface IUDRProcessorInUDRWriter
{
	string Name { get; }
	string Description { get; }
	/// <summary>
	/// Indicates that this process is run InUDRWriter.
	/// </summary>
	UDRProcessorType Type { get; }

	/// <summary>
	/// Process a UDR Record After Bucketing
	/// </summary>
	/// <param name="udr">The UDR record</param>
	/// <param name="udrWriterProcessorParameters">Parameters provided from rating</param>
	/// <returns>False to halt processing of the UDR (ie drop it), true otherwise</returns>
	bool Process(Logisense.Boss.Logic.Core.UDR udr, UDRWriterProcessorParameters udrWriterProcessorParameters);

	void BulkUDRWriteComplete(bool success);
	void IndividualUDRWriteComplete(bool success);
}

/// <summary>
/// Triggers actions that need to happen when a UDRBiller is to be re-rated
/// </summary>
public interface IUDRReratingProcessor
{
	string Name { get; }
	string Description { get; }
	void Process(ReratingProcessorData data);
}

Examples

Mediation and Rating

The following is an example implementation of the usage and rating customization touch points. In the first class, Installer, we can see the implementation of the ICustomCodeInstaller interface. It is doing some integrity checks as part of its initialization upon install. The second class, GenericOccurenceDataSource, is derived from the AbstractFileParsingDataSource class. This class does the core business logic inside the Mediate method. Mediate is responsible for mapping and modify fields from the input data rows into database UDR table rows. In this example there is very little modification needed to map the fields from the input to the UDR table. The key field that is needed later in the rating cycle is the UDR.BillingIdentifier. This is highlighted in bold in the code snipit below. A UDR.Billingidentifier value is used by the rating engine to pull a matching record in the Identifier column on the database UDRUserIdentifierHistory table to perform its rating calculations of this specific usage record. In this example the value in BillingIdentifier is the customer's phone number. All usage records for this customer will have their phone number in the usage data being ingested by EngageIP.

public class Installer : Logisense.Boss.Logic.Core.ICustomCodeInstaller {
    public void Install(int ownerID) {
        Owner owner = Owner.GetByID(ownerID);

        EnsureClassExists("Occurrences-Geo", "GeoTree", owner, UDRUnitTypeID.Occurrences);
        EnsureClassExists("Volume-Geo", "GeoTree", owner, UDRUnitTypeID.Data_KiB_MiB_GiB);
        EnsureClassExists("Occurrences-Class", "Class", owner, UDRUnitTypeID.Occurrences);
        EnsureClassExists("Volume-Class", "Class", owner, UDRUnitTypeID.Data_KiB_MiB_GiB);
    }

    private static void EnsureClassExists(string className, string rater, Owner owner, int unitTypeID) {
        if (owner.SearchUDRClassByName(className).Length == 0) {
            UDRClass ld = UDRClass.GetNew();
            ld.Name = className;
            ld.OwnerID = owner.ID;
            ld.UDRRater = rater;
            ld.UDRUnitTypeID = unitTypeID;
            ld.Create();
        }
    }
}

public class GenericOccurenceDataSource : AbstractFileParsingDataSource, IUDRDataSource {
    protected SHA1CryptoServiceProvider _cryptoTransformSHA1 = new SHA1CryptoServiceProvider();
    protected Dictionary<string, string> _countryCache = null;

    public GenericOccurenceDataSource() : base(new string[] { "transactionId" }, new string[] { "," }) {
    }

    public override string Name {
        get { return "Generic Occurrence-Based"; }
    }

    public override Logic.Core.UDR Mediate(string[] rawFields, ref Dictionary<string, string> ratingContext) {
        var urep = System.Runtime.Remoting.Messaging.CallContext.GetData("UDRRatingEngineProcess") as UDRRatingEngineProcess;

	//The follow is an example of the input format being mapped
        //0,555-123-4567,2,3,555-111-2222,2012-12-14 23:59:59.000 <Date>,IMSI-3027 <Identifier>,MT <UDR Class>,1 <Volume>,NULL <Geo Location>
        var udr = UDR.GetNew() as Logic.Core.UDR;
        udr.Date = DateTime.Parse(rawFields[5]);
        udr.ImportedValue = Convert.ToInt64(Convert.ToDouble(rawFields[3]));
        udr.Value = udr.ImportedValue;
        udr.ValueActual = udr.ImportedValue;
        udr.Cost = 0;
        udr.UDRClassID = urep.GetCachedUDRClass(rawFields[2]).ID;
        udr.OriginatingIdentifier = rawFields[1];
        udr.TerminatingIdentifier = rawFields[4];
        udr.BillingIdentifier = rawFields[1];
        udr.UniquenessIdentifier = BitConverter.ToString(_cryptoTransformSHA1.ComputeHash(Encoding.UTF8.GetBytes(String.Join(",", rawFields)))).Replace("-", "");

        return udr;
    }
}

With incoming data being mediated correctly into EngageIP UDR records the next step is to see how it gets used during the rating process. The Rating Engine uses the UDR.BillingIdentifier to look up information in UDRUserIdentifierHistory table based on the values in the Identifier column. A matching row will contain several useful pieces of information:

Identifier - the value of the matching UDR.BillingIdentifier
UserID - the EngageIP user ID associated with this row
Start and end dates for the usage record
Foreign keys to associated records in other tables for Packages, Services and Rate Plans

In this example the UDRUserIdentifierHistory.Identifier is "555-123-4567" which matches the UDR.BillingIdentifier value in the commented out sample line in the Mediate method above. This value was configured in EngageIP using a required Profile Question when the user account was created and assigned a Package. When an account is created and assigned a Package that contains a Service with a Profile Question a value must be entered. In this example the Profile Question would be something like this, "What is the Phone Number for this account?". This value gets save in the ProfileAnswer database table and is referenced by UserServiceAttributeProfileAnswer.ProfileAnswerID. From this table the Service and Package can be retrieved and the appropriate Rate Plan determined.

The value in UDRUserIdentifierHistory.Identifier is not set by EngageIP core code. It is set using a custom code implementation of a C# IAction Interface. A comprehensive solution would include and IAction.EventCode list to detect new account creations, profile answer creations, rate plan changes and anything else that might impact the database records for UDRUserIdentifierHistory. Our simple example is subscribed to events based on the creation or updating of Profile Answers. The EventCode string is "ProfileAnswer.Create,ProfileAnswer.Update".

The following is a sample custom code file that takes a profile answer create or update and creates/updates the UDRUserIdentifierHistory table for the account. Inside the ScriptContext instance will be a set of keys. The value for each key can be used to operate on the specific record that has changed in the database. In this example it is the phone number value stored in the ProfileAnswer table row.

using System.Data;
using Logisense.Boss.Logic;
using Logisense.Boss.Logic.DomainModel;
using System;
using System.Collections.Generic;

namespace Logisense.Boss.CustomScripts.Examples
{
    public class SimpleUUIH : IAction
    {
        public string Name { get { return "UDRUserIdentifierHistory Update Workflow"; } }
        public string Description { get { return "UDRUserIdentifierHistory Update Workflow"; } }
        public string EventCode { get { return "ProfileAnswer.Create,ProfileAnswer.Update"; } }
        public int OwnerID
        {
            get { return 1; }
            set { }
        }

        private const string PROFILEANSWERID = "ProfileAnswerID";  //example of a key in a ScriptContext instance

        public void Run(ScriptContext context)
        {
            if (context.ContainsKey(PROFILEANSWERID))
            {
                ProfileAnswer profileAnswer = ProfileAnswer.GetByID(Int32.Parse(context[PROFILEANSWERID].ToString()));

                IdentifierDetails identifierDetails = null;
                if (GetIdentifierAndUserForUserServiceAttributeProfileAnswer(ref identifierDetails, profileAnswer))
                {
                    UserService userService = identifierDetails.IdentifierUserService;
                    User user = User.GetByID(userService.UserID);
                    RatePlanDetails ratePlan = GetRatePlanForUUIHUserServiceEntry(userService);

                    UDRUserIdentifierHistory udrUsageIdentifierHistory = UDRUserIdentifierHistory.GetNew();
                    udrUsageIdentifierHistory.UserID = user.ID;
                    udrUsageIdentifierHistory.Identifier = identifierDetails.Identifier;
                    udrUsageIdentifierHistory.OwnerID = User.GetByID(user.ID).OwnerID;
                    udrUsageIdentifierHistory.End = DateTime.MaxValue;
                    udrUsageIdentifierHistory.Parent_UserID = user.GetUserParent().Parent_UserID;
                    udrUsageIdentifierHistory.UserServiceID = userService.ID;
                    udrUsageIdentifierHistory.UDRRatePlanID = ratePlan.CurrentUDRRatePlan.ID;
                    udrUsageIdentifierHistory.RatePlan_ServiceID = ratePlan.ServiceID;
                    udrUsageIdentifierHistory.RatePlan_UserPackageID = ratePlan.UserPackageID;
                    udrUsageIdentifierHistory.IdentifierSource = identifierDetails.IdentifierSource;
                    udrUsageIdentifierHistory.Start = DateTime.Now;;
                    udrUsageIdentifierHistory.Create();
                }
            }
        }

        private bool GetIdentifierAndUserForUserServiceAttributeProfileAnswer(ref IdentifierDetails identifier,
            ProfileAnswer profileAnswer)
        {
            UserServiceAttributeProfileAnswer userServiceAttributeProfileAnswer =
                GetUserServiceAttributeProfileAnswer(profileAnswer);
            if (userServiceAttributeProfileAnswer == null)
            {
                return false;
            }

            identifier = new IdentifierDetails(profileAnswer.Value, profileAnswer.GetProfileQuestion().Name,
                userServiceAttributeProfileAnswer.GetUserService());

            return true;
        }

        private UserServiceAttributeProfileAnswer GetUserServiceAttributeProfileAnswer(ProfileAnswer pa)
        {
            UserServiceAttributeProfileAnswer[] attributeProfileAnswers =
                pa.GetUserServiceAttributeProfileAnswerCollection();

            if (attributeProfileAnswers == null || attributeProfileAnswers.Length == 0)
                return null; //Not connected to a Userservice must not be one we are looking for

            return (Logic.Core.UserServiceAttributeProfileAnswer) attributeProfileAnswers[0];
        }

        private RatePlanDetails GetRatePlanForUUIHUserServiceEntry(UserService userService)
        {
            UserPackage userPackage = UserPackage.GetByID(userService.UserPackageID);
            UserPackageAttributeUDRRatePlan[] userPackageAttributeUDRRatePlans =
                userPackage.GetUserPackageAttributeUDRRatePlanCollection();

            //Should never be more than one record but in case there are multiple records choose the first one
            if (userPackageAttributeUDRRatePlans.Length > 0) 
            {
                UserPackageAttributeUDRRatePlan userPackageAttributeUDRRatePlan = userPackageAttributeUDRRatePlans[0];
                return new RatePlanDetails(UDRRatePlan.GetByID(userPackageAttributeUDRRatePlan.UDRRatePlanID),
                    userPackageAttributeUDRRatePlan.ServiceID, userService.UserPackageID);
            }

            Package package = Package.GetByID(userPackage.PackageID);
            PackageAttributeUDRRatePlan[] packageAttributeUDRRatePlans =
                package.GetPackageAttributeUDRRatePlanCollection();
            foreach (PackageAttributeUDRRatePlan packageAttributeUDRRatePlan in packageAttributeUDRRatePlans)
            {
                if (PackageServiceConnector.GetByID(packageAttributeUDRRatePlan.PackageServiceConnectorID).ServiceID ==
                    userService.ServiceID)
                {
                    return new RatePlanDetails(UDRRatePlan.GetByID(packageAttributeUDRRatePlan.UDRRatePlanID),
                        userService.ServiceID, userService.UserPackageID);
                }
            }

            //Should never be more than one record but in case there are multiple records choose the first one
            if (packageAttributeUDRRatePlans.Length > 0) 
            {
                PackageAttributeUDRRatePlan packageAttributeUDRRatePlan = packageAttributeUDRRatePlans[0];
                return new RatePlanDetails(UDRRatePlan.GetByID(packageAttributeUDRRatePlan.UDRRatePlanID),
                    PackageServiceConnector.GetByID(packageAttributeUDRRatePlan.PackageServiceConnectorID).ServiceID,
                    userService.UserPackageID);
            }

            return null;
        }
    }

    public class IdentifierDetails
    {
        public readonly string Identifier;
        public readonly string IdentifierSource;
        public readonly UserService IdentifierUserService;

        public IdentifierDetails(string identifier, string identifierSource, UserService identifierUserService)
        {
            Identifier = identifier;
            IdentifierSource = identifierSource;
            IdentifierUserService = identifierUserService;
        }
    }

    public class RatePlanDetails
    {
        public UDRRatePlan CurrentUDRRatePlan;
        public int ServiceID;
        public int UserPackageID;

        public RatePlanDetails(UDRRatePlan udrRatePlan, int serviceID, int userPackageID)
        {
            CurrentUDRRatePlan = udrRatePlan;
            ServiceID = serviceID;
            UserPackageID = userPackageID;
        }
    }
}

v9 Documentation

Usage Data

Interfaces

Examples

Mediation and Rating

Related pages