DataManagementCPUP

This Atom contains all the necessary scripts to join the records provided by Otarie (FTP_Session file) and Astellia on IU-PS interfaces in the format awaiting by CPUP Atom. You will need also the CNX files provided by Otarie to create the correspondance between MSISDN and IMSI and a topology network file containing at least these following fields : lcid (local cell id) , longitude & latitude of the antenna.

As these scripts have been written in PIG language, the raw data (Otarie and astellia files) must be stored in a Big data environment like DIOD platforme (Big Data IMT plateform). These Raw data must have been captured in the same period of time. You have to modify all these scripts by indicating, for each of them, the HDFS folder locations where to find the data source and where to store the results. It is important to launch the scricpts in the following order.

1- _MSISDN_locstart_IMSI.pig

2- _Clean_Astellia.pig

3- _Filter_Astellia_on_cause.pig

4- _Clean_Otarie.pig

5- _Jointure_AST_OTA.pig

6- _filter_on_date.pig

The first script works only on CNX files to create a correspondance table beween IMSI and MSISDN. The second and third script clean and filter Astellia records, the second works on Astellia Raw data and topology files, while the third works only on the result of the second script. The fourth script clean the Otarie raw data and used the result of the first script to bring IMSI to Otarie data and create an ID that will be used to link Otarie and Astellia data. The fifth script join Otarie and Astellia data thanks to an ID made by the concatenation of IMSI, LAC and CI. The last script filter the result of the fifth script to keep only those that have a suspicious signaling message occured during a data usage of a user.

Getting started

Running them then should be as simple as: pig "name of the PIG script"