xDR Import Configuration

Overview

Checklist

Obtain Vendor CDR Samples

Usage Scenario

Add xDR Mediator Instances on the Configuration Server

Add the CDR Extraction Instance

Add the CDR Rating Instance

Add the Interim Data Storage

Mandatory Attributes for xDR Import

Initial Configuration of PortaSwitch®

Create a Node

Create a Vendor Tariff

Enter Rates

Create a Vendor

Define Connections

Create Customer Tariff

Create a Bundled Product

Test the xDR Import

Upload CDR Source File

Verify the xDR Import Process

Overview

The xDR Mediation utility allows CDRs from various external sources to be imported into PortaBilling®, which bills them as if they were regular calls or messages being sent by a remote gateway.

 

If source files already contain charged amounts (e.g. calculated by the carrier or some outsource billing agency), simply import them to PortaBilling® so that these charges will be included in customer invoices.

 

This utility is useful for service providers who operate in mobile networks (2.5 / 3G) and for legacy equipment owners whose gateways do not support RADIUS and can therefore only save xDRs to files.

 

The xDR Mediation utility consists of three main components:

·       CDR Extraction, which parses input data from source files, converts it and then creates CDR collections,

·       Elasticsearch as the interim data storage for CDR collections, and

·       CDR Rating, which communicates with the billing engine via RADIUS. It sends CDR collection data for processing and receives notifications about the import status.

 

Though the xDR Mediation utility is typically used to import CDR records for such common services as voice calls, SMS and data transfer, you can also use it to import and bill customers for IPTV or other quantity-based services (e.g. pizza-delivery orders).

 

The xDR Mediation utility can process the following source data formats:

·       .csv

·       FixedWidth

·       .TAP3

·       ASN.Huawei3000

 

This document provides instructions on how to configure the import of xDRs from the .csv format. For detailed description how to import xDRs in other formats, please refer to the How to handbook.

Checklist

Print the following page and use it to check off the operations you have completed while performing the system setup according to the instructions in this chapter. Please be sure to perform all of the operations in the order designated (all of the boxes should be checked), otherwise the service will not work.

 

 

Operation

 

Done

General Configuration

 

Obtain vendor CDR samples

[     ]

Submit a request to PortaOne Support for the xDR mediator module implementation

[     ]

Network configuration

 

Configure the CDR extraction instance

[     ]

Configure the CDR rating instance

[     ]

Add the interim data storage

[     ]

Create a node to represent the xDR Mediation utility

[     ]

Rating configuration (Vendor)

 

Create a tariff (referred to as tariff A later on) that describes your termination costs and routing for off-net calls (make sure it is not the Routing type!)

[     ]

Upload the rates provided by the mobile carrier in tariff A 

[     ]

Create a vendor

[     ]

Create a connection for this vendor using tariff A

[     ]

Rating configuration (Customer)

 

Create a tariff (referred to as tariff B later on) that will be applied to users for making outgoing calls

[     ]

Enter rates in tariff B for the destinations that your customers will call (both on-net and off-net calls)

[     ]

Create a bundled product

[     ]

Create a rating entry for this product for the Voice Calls service and tariff B. Leave the Node and Access Code fields empty

[     ]

Account provisioning

 

Create an invoice template

[     ]

Create a customer class for your customers

[     ]

Create a residential customer

[     ]

Create an account for this customer

[     ]

Testing

 

Upload the CDR source file

[     ]

Verify the xDR import process in PortaBilling®

[     ]

Obtain Vendor CDR Samples

Prior to configuring the xDR Mediation utility, obtain the file that contains examples of xDRs to be imported to PortaBilling® from your carrier. Carriers can present CDRs in their own manner: storing CLI / CLD in the local format, having a different column order, providing several records applicable to the same session, etc.

 

Therefore, CDR samples are analyzed and, if necessary, the xDR mediator module is implemented to correctly parse and transform the data for further processing.

 

Implementing the xDR mediator module requires specialized programming and testing efforts. You can implement it on your own or request assistance from the PortaOne Support team. For assistance, please contact support@portaone.com with a request. Please note that such assistance is not included in PortaCare and will be billed separately.

Usage Scenario

Let’s say you are a service provider operating on the CDMA network. You provide SIP on net calls to your customers via PortaSIP® while off net calls are terminated by your carrier TATA Telecom. TATA Telecom sends you CDRs that you need to import to PortaBilling®.

 

The .csv file from TATA Telecom has the following structure:

 

Source CDRs

 

To automate the CDR retrieval process, you configure the xDR mediator to automatically download the source CDR files from TATA Telecom via the FTP protocol.

 

Note that you require the credentials to access TATA telecom system and the path to the directory from where to download CDR files.

Add xDR Mediator Instances on the Configuration Server

Add the CDR Extraction Instance

1.     Go to the Configurations tab on the Configuration server.

2.     Clone the existing configuration.

3.     From the Configuration tree, select Auxiliaries ->CDR Mediation -> CDR Extraction and click the Wor1F12 Instance Create icon.

 

Add CDR Extraction instance

 

4.     Fill in the Instance create form:

·       Server – Select one of your servers. The one recommended is the web server;

·       Service IP – Select one of the web server’s IP addresses. You may utilize a private IP address for better security (e.g. to avoid possible brute force attacks). Please note that the CDR Extraction instance cannot have the same IP address as the RADIUS instance.

·       Click Save.

 

Add CDR Extraction

 

5.     In the CDRSource group specify the following parameters:

·       type – Select the CDR source file type. Leave .CSV here.

·       csv_field_separator – This is the separator to use when splitting .CSV file lines into columns. 

·       column_list – Specify the header names for the .CSV file. Type in the following string: User-Name,Calling-Station-Id,Called-Station-Id,Acct-Session-Time,h323-connect-time,h323-disconnect-time,PortaOne-Service-Type,h323-remote-id,NAS-IP-Address. Please refer to the Mandatory Attributes for xDR Import section for the list of attributes required for a particular service.

 

_Hint_ExclamationMark The header order is important. Make sure the names you define directly correspond to the column order in the source .CSV file.

NOTE: You must use the actual separator here. For example, if your field separator is a “|” (pipe) and not a comma, the value will appear as follows:

User-Name|Calling-Station-Id|Called-Station-Id|Acct-Session-Time|h323-connect-time|h323-disconnect-time| PortaOne-Service-Type|h323-remote-id|NAS-IP-Address

·       file_pattern – Specify the file name pattern. It may contain the wildcard (e.g. abc*.csv). The xDR mediator will import only those files with names that fit this pattern.

·       files_input_directory – Specify the path to the folder from where the xDR mediator will extract source files (e.g. /porta_var/upload).

·       time_zone – Select the time zone in which the xDRs will be imported. However, if a particular time zone is specified in the source file, it will override the selected time zone.

 

CDR Extraction instance

 

6.     Define the parameters for automatic CDR download in the CDRSourceFTPClient group:

·       enable – Set Yes.

·       files_directory – Specify the path to the directory with the source CDR files on the remote host.

·       host – Specify the IP address or domain name of your vendor’s host. The xDR mediator will connect to it to download CDR files.

·       login, password – Specify the credentials obtained from your vendor to access their host.

·       protocol – Select the protocol for file download: FTP or SFTP.

 

CDR download configuration

 

7.     In the Global group specify how many parallel processes will convert source CDRs and send CDR collections to interim data storage. Leave the default value here for now.

8.     In the Output group specify the following:

·      cdr_export_delay – Specify the time interval to import CDRs from several source files in chronological order (e.g. 43200 seconds meaning 12 hours). This defines how long an extractor waits before adding CDRs to the collection for rating.

 

The period starts when the first CDR is parsed. When the timer fires, the extractor aggregates the ready-to-import CDRs and adds them to the collection. Then a new period starts. The default value is 0 seconds meaning that processed CDRs from a source file are immediately added to the collection.

NOTEThe delay period is applied to complete CDRs only. If you configure the Extractor to merge several CDRs into one, the delay period is skipped and the value from the max_wait_incomplete_cdr is considered

·       max_wait_incomplete_cdr – This defines how long an extractor waits before adding incomplete CDRs to the collection. The default value is 3600 seconds. This value is applied if you configure the Extractor to merge several CDRs into a single xDR. Please find details about this in the merge several CDRs into a single xDR record section of the How to handbook.

·       min_freq – This defines how often the ready-to-import CDRs must be added to the collection when processing a large source file.

NOTE:  When configured, the value from the delay period overrides this value.

·       name_pattern – This specifies the pattern for the collection’s name. If left empty, the system names the collection based on the date and sequential number (e.g. cdr2018-07-22 or cdr20170801_0745_1662100001621_0001).

·       separate_collections – This selects how the Extractor creates a new collection:

·      per file – a separate collection is created for each file in the input folder.

·      per day – a single collection is created for files that appear in the input folder during a current day.

·      per import – a separate collection is created for all the files in the input folder.

 

TipTIP: You may define data transformation rules that will be applied to the parsed data from the source CDR files. To do this, create your custom Perl module and define its name and the path to it in the DataTransformation group. Please refer to the Data Transformation Module for xDR Import handbook for details.

Add the CDR Rating Instance

1.     From the Configuration tree, select Auxiliaries ->CDR Mediation -> CDR Rating and click the Wor1F12 Instance Create icon.

 

Add CDR Rating instance

 

2.     Fill in the Instance create form:

·       Server – Select one of your servers. You can use the same server where the CDR-Extraction instance is located;

·       Service IP – Select one of the server’s IP addresses. Please note that the CDR Rating instance cannot have the same IP address as the RADIUS instance.

·       Click Save.

 

Add CDR rating instance

 

3.     In the Global group define how many parallel processes will run to process the CDR collections. Leave the default value here.

Add the Interim Data Storage

1.     From the Configuration tree select DB->Elasticsearch-><elasticsearch_instance>

2.     Click Add and select the interim-data value for the with_indices option.

 

Elasticsearch DB configuration

 

3.     Click the Save button and then the check_apply Verify button.

4.     Verify the new configuration and click the check_apply Check / Apply button.

Mandatory Attributes for xDR Import

Global attributes

These attributes must be present when importing CDRs for any type of services:

 

Attribute Name

Type

Description

User-Name

String

This determines whom to charge for the session.

PortaOne-Service-Type

Voice, Msg, Netaccess, Quantity, Iptv.

This is the type of service used within a session. Default – Voice.

Voice Calls

Define these attributes in addition to the global ones when importing CDRs for voice call service:

 

Attribute Name

Type

Description

h323-connect-time

Unix timestamp (1379466655), ISO (2013-09-16 17:34:00) or RADIUS (02:32:17.000 UTC Mon Nov 26 2012

The connecting time for a session.

h323-disconnect-time

Unix timestamp (1379466655), ISO (2013-09-16 17:34:00) or RADIUS (02:32:17.000 UTC Mon Nov 26 2012

Session disconnect time. Either h323-disconnect-time or Acct-Session-Time must be provided.

Acct-Session-Time

Int

Either h323-disconnect-time or Acct-Session-Time must be provided.

Calling-Station-Id

String

The originator of the call.

Called-Station-Id

String

The destination number.

h323-remote-address

IP

The IP address of the vendor gateway.

h323-remote-Id

string

The ID of the remote gateway.

Messaging services

Define these attributes in addition to the global ones when importing CDRs for messaging service:

 

Attribute Name

Type

Description

Calling-Station-Id

String

The sender of the message.

Called-Station-Id

String

The destination number.

Event-Timestamp

Unix timestamp (1379466655), ISO (2013-09-16 17:34:00) or RADIUS (02:32:17.000 UTC Mon May 26 2017

The timestamp for the message being sent.

PortaOne-Used-Resource

int

The number of messages transferred. Default - 1

Internet Access

Define these attributes in addition to the global ones when importing CDRs for Internet Access service:

 

Attribute Name

Type

Description

Event-Timestamp

Unix timestamp (1379466655), ISO (2013-09-16 17:34:00) or RADIUS (02:32:17.000 UTC Mon May 26 2017

The timestamp for the session.

Acct-Output-Octets

int

Indicates the number of bytes sent to the user during a session.

Acct-Input-Octets

int

Indicates the number of bytes received from the user during a session.

Quantity-based services

Define these attributes in addition to the global ones when importing CDRs for quantity-based service:

 

Attribute Name

Type

Description

Event-Timestamp

Unix timestamp (1379466655), ISO (2013-09-16 17:34:00) or RADIUS (02:32:17.000 UTC Mon May 26 2017

The timestamp for the session.

PortaOne-Used-Resource

int

The number of units transferred. Default – 1.

IPTV

Define these attributes when importing CDRs for the IPTV service:

 

Attribute Name

Type

Description

PortaOne-Service-Type

Iptv

The type of service used within a session.

User-Name

String

Determines whom to charge for the session.

PortaOne-Used-Resource

int

The amount of units transferred. Default - 1

Event-Timestamp

Int

The timestamp for the session.

Initial Configuration of PortaSwitch®

If you have just installed the PortaSwitch® software or just dedicated a new billing environment to configure the services described in this handbook, make sure to first perform the initial configuration of PortaSwitch®. To do this, use the PortaSwitch® Initial Configuration handbook.

Create a Node

Now create a node that represents the xDR mediator.

 

Create a node

 

1.     On the navigation menu, select Infrastructure, then Billing data sources, and click Nodes.

2.     On the Create node panel, fill in the node details:

·       Name – Type a short descriptive name for your SIP server (that will be used in the lists).

·       Node ID – Enter the IP address assigned to the CDR rating instance on the Configuration server web interface.

·       IP – This is the IP address assigned to the CDR rating instance on the Configuration server web interface.

·       Manufacturer – Select PortaOne.

·       Type – Select Generic.

 

3.     Click Save.

4.     On the Communication with billing panel move the Enable communication with billing slider.

5.     Configure the communication with the billing engine for this node:

·       Client protocol – Select RADIUS, since the server (node) must be able to communicate with PortaBilling® via RADIUS.

·       Radius key – Specify the authentication key used for communication with PortaBilling® via RADIUS. Specify SecretKey here.

NOTE: The Radius Key value can be found in the /home/porta-cdrimport/etc/cdr_importer.<xdr-import_instance_name>.conf file in the [Radius] section.

Communication with billing

 

6.     Click Save.

Create a Vendor Tariff

The tariff is a single price list of your calling services or your termination costs. PortaBilling® requires that every call be accounted for, both on the revenue side (customers) and on the cost side (vendors). Therefore, the tariff is used to calculate your costs.

 

Since xDRs are imported into PortaBilling® for sessions already completed and the vendor must not participate in call routing, the Routing option must be disabled for the vendor tariff.

 

 

1.     On the navigation menu to the left, select Service catalog and click Tariffs.

2.     On the Create tariff panel, fill in the tariff details:

o   Name – Type a short name for the tariff object; this is the name you will see in the select menus (for example, GlobalNet Termination).

o   Currency – Choose the currency in which the vendor charges you.

NOTE: The currency for the tariff may be chosen only once, and cannot be changed later.

o   Service – Choose Voice Calls here.

o   Applied to – Choose Vendor in the Applied to list.

o   Routing – Clear the Routing option.

 

3.     Click Save.

Enter Rates

Rates are the per-destination prices. Please refer to the Call Billing Parameters section of the PortaBilling Administrator Guide for more information on billing parameters.

 

Depending on whether or not you control the vendor costs in PortaBilling® by creating vendor xDRs, specify the rates for the following destinations in the tariff:

·       Enter the symbolic destination “|” (pipe) and specify a zero price for it if you do not control vendor costs, or

·       Upload the rates for geographical destinations according to the tariff sheet provided by the carrier.

 

Please consult the Rate Import section for more details.

Create a Vendor

Vendors are your termination partners. PortaBilling® requires that every call be accounted for on both the revenue side (customers) and the cost side (vendors). In the case of calls going via the mobile carrier’s network, no vendor (in the traditional sense) is involved – so you still need to create a vendor that will be used to keep the xDRs for these calls. This vendor and the connections to the vendor are required in order to properly bill for calls.

 

1.     In the left upper corner click dashboard to open the navigation menu.

2.     On the navigation menu, select Infrastructure, then select Vendors.

3.     On the Create vendor panel, fill in the vendor details:

o   Name – Type a short name for the vendor object; this will be used on the web interface (for example, TATA Telecom).

o   Currency – Choose the currency in which this vendor charges you.

o   Opening balance – This indicates a starting balance for the vendor; the default is zero.

o   Billing period – Split period for vendor statistics.

 

Create a vendor

 

4.     Click Save.

5.     Fill in the information about the vendor as described in the Basic Residential VoIP Service handbook.

Define Connections

A connection represents the point from which calls leave or enter your network and / or are directed to or from vendors when charges are incurred.

 

1.     While on the vendor’s panel, click Connections

2.     On the Create connection panel, fill in the connection details: 

·       Description –Type a descriptive name for this connection. It will be displayed in the list of connections.

·       Service type – Select Voice Calls.

·       Type of connections – Select SIP.

·       Direction – Select To vendor.

·       Tariff – Choose the tariff that defines your termination costs for this connection / vendor.

·       Active – Use the slider to set this connection as inactive.

·       In the Identify gateway by ID option select the Gateway ID and specify the ID of the vendor’s gateway or switch. Type in CDRIMPORTER.

·       Capacity – Define the capacity value for this connection.

 

define connection

 

3.     Click  Save.

Create Customer Tariff

To charge your customers for making calls create the tariffs and define the rates within.

 

Please refer to the Create Customer Tariffs and Enter Rates sections of the Basic Residential VoIP Service handbook for guidelines about tariff creation.

 

Tip Tip: To import CDRs for incoming calls, do the following:

·       Make sure source CDRs are clearly defined as incoming and outgoing ones;

·       Define the PortaBilling_AccessCode attribute in the column_list option for the CDR extraction instance and create an appropriate data transformation rule for it. (e.g. The CDR source file contains the CallType column with an IN value for incoming calls and an OUT value for outgoing calls. Then the data transformation rule will be PortaBilling_Access_Code=<<ACC

  if ($data{CallType} eq 'IN') {

    return 'INCOMING';}

elsif ($data{CallType} eq 'OUT') {

    return 'OUTGOING';}

else {

    return undef;

  }

ACC)

·       Create a tariff to charge users for incoming calls and include it into the product rating list.

Create a Bundled Product

End users will use accounts issued for specific products to access the services you provide. Products are powerful tools that define different ways to bill an account for one or several included services. Product definition is always realized through these steps: product definition, service definition and configuration and the creation of a rating list.

 

include a service

 

1.     On the navigation menu, select Service catalog and click Products.

2.     On the Create product panel, fill in the product details:

o   Name – Type an internal product name that will be shown on the administrator interface.

o   Name visible to end users – Type a name of the product that will be shown to end users on their self-care interfaces.

o   Product type – Select Main product here.

o   Currency – Choose a currency the product will be priced in.

o   Managed by – Select Administrator only here, since we are setting up a service without the involvement of resellers.

o   Account default ACL – Choose an Access Control List (ACL) for accounts with this product assigned. ACLs control which objects end users can access to and which actions they can perform.

o   Account role – Select Mobile from the list since this product is intended to be used by mobile subscribers.

Included services

Define which service types are included in the product. A service type is a description of the physical service provided to end users.

 

To add a service type:

1.     On your product’s panel, click Services.

2.     On the Services panel, click Add a service.

3.     In the Select services to add dialog box, select Voice calls and click Add.

 

Create a product

Usage charges

The rating list has two functions: it defines the permitted access points (nodes and access numbers) and specifies which tariff must be used for billing each of these points. Now you create the rating entries to handle outgoing calls.

 

1.     On your product’s panel, click Charges, then click Usage charges.

2.     On the Usage charges panel, click Add.

3.     Fill in the required information:

o   Node – Select the node you created for the xDR importer.

o   Access code – Leave this field empty.

o   Service – Choose Voice Calls.

o   Tariff – Select the tariff that will be used to calculate charges for outgoing calls.

o   Click Save.

 

usage charges

 

4.     Click save_close Save & Close.

 

 

Proceed with the customer configuration by performing the Enter SIM Cards into the SIM Card Inventory, Create Invoice Template, Configure Taxation, Create a Customer Class, Create a Customer and Create Accounts steps from the MVNO Service Provisioning handbook.

Test the xDR Import

Upload CDR Source File

1.     Log in to the server with the xDR mediator configured (in our example – the web server) using ssh. (For how to configure the ssh access your user, please refer to the Users section of the PortaSwitch Configuration Server Web Reference guide.)

2.     Upload the source CDR file to the folder you defined for the xDR mediator configuration by using the following command:

sudo scp user@remote.host: /full_path_to_source_file/CDRs.csv /porta_var/upload/CDRs.csv

user@remote.host password:

Verify the xDR Import Process

PortaBilling® enables you to control the xDR import process and adjust the configuration if any issues occur.

 

1.     From the StatisticsStatistics section of the Admin-index, select CDR Mediation.

 

CDR collections page

 

2.     You will see the most recent CDR collections on the CDR Mediation page. View the information in the table:

·       CDR Collection – This shows the name of the CDR collection created after parsing source file data.

·       Status – This is the current status of the CDR collection.

·       Added – This is the date and time when the collection was added to the import queue.

·       Processed – This is the date and time when the collection was processed

·       Total Records, Imported, Skipped, Rejected –This is the information about CDR records’ numbers and status within the collection.

3.     Click the xDRs XDRs icon next to the CDR collection name to view detailed information about the CDR collection.

 

xDR details

 

4.     Verify the xDR status and charges calculated for the customer and the vendor. Use filters from the left-hand side of the screen to search for particular xDRs

5.     Click the Details Details icon to see detailed information about an individual xDR.

6.     Click the Log BE Log icon to check log files for a particular xDR record with the BE Log Viewer.