Challenge context

BNP Paribas Personal Finance is the leader in personal financing in France and in Europe through its consumer credit activity. A 100% subsidiary of the BNP Paribas Group, BNP Paribas Personal Finance gathers more than 20 000 employees and operates in some thirty countries. Under various brands like Cetelem, Cofinoga and Findomestic, BNP Paribas Personal Finance provides its clients with a complete range of consumer credit, available in stores and car dealership or directly through customer relationship centers and via the company’s local websites.

BNP Paribas Personal Finance had developed an active strategy of support for retailers, carmakers and dealers, Web merchants, and various financial institutions (banking and insurance), based on its credit's market experience and its ability to offer services adapted to the activity and the commercial strategy of its business partners. It is also a key player in terms of responsible credit and budgetary awareness.

BNPP Personal Finance is, by nature, exposed to Credit Risk, and relies heavily on quantitative models to manage it. Within BNP Paribas Personal Finance Central Risk Department is responsible for the relevance of risk rating models used within all local entities and for maintaining a high level of expertise with integrating new statistical techniques in new modelling environments.

The Credit Process Optimization team is part of the RISK department of BNPP PF, within Risk Personal Finance Global Credit Decision-making Policies, we contribute to the rationalization and the optimization of risk decision processes through an analytical approach. We support local risk teams to improve the efficiency of credit processes including fraud part, participating to the best balance between profitability, customer journey and risk profiles.

Fraud is a major problem for merchants. Criminals use a wide variety of methods to attack organizations across systems, channels, processes and products. The development of fraud detection methods is thus of crucial importance. Fraud detection is a challenging problem because fraudsters make their best efforts to make their behaviour look legitimate. Another difficulty is that the number of legitimate records is far greater than the number of fraudulent cases.

Challenge goals

The aim of our challenge is to find the best way to process and analyse basket data from one of our retailer partners in order to detect fraud cases.

Using these basket data, fraudulent customers should be detected, to be then refused in the future.

Data description

1. Dataset

This dataset contains a list of financed purchases, in which there is information about the contents of the baskets.

For each line of the dataset there are 147 columns, in which 144 are grouped in 6 categories:

1. item,

2. cash_price,

3. make,

4. model,

5. goods_code,

6. Nbr_of_prod_purchas.

For each of these categories there are 24 instances that will either be filled with relevant information when an item exists in the basket, or null when it does not. For example, if an application has 3 items in the basket there will be information in the columns item1 to item3, cash_price1 to cash_price3, make1 to make3, model1 to model3, goods_code1 to goods_code3 and Nbr_of_prod_purchas1 to Nbr_of_prod_purchas3; but the remaining columns of these categories will be null (e.g. item4 to item24).

The variable Nb_of_items corresponds to the unique number of items in the basket, and to the sum of all of them in such cases where one of the Nbr_of_prod_purchas instances is more than 1.

Finally, the fraud_flag variable indicates if that application is fraudulent or not.

1.1. Input variables (X)

1.2. Output variable (Y)

1.3. Size of the dataset

Shape: 115 988 observations, 147 columns.

Distribution of Y:

1. Fraud (Y=1) : 1 681 observations

2. No Fraud (Y=0) : 114 307 observations

The fraud rate for the whole population is around 1.4%.

2. Sampling

The sampling method applied is a simple random without replacement. 80% of the initial dataset has been used to create the training sample and 20% for the test sample.

2.1. Training sample

Shape: 92 790 observations, 147 columns.

Distribution of Y_train:

1. Fraud (Y=1) : 1 319 observations

2. No Fraud (Y=0) : 91 471 observations

2.2. Test sample

Shape: 23 198 observations, 147 columns.

Distribution of Y_test:

1. Fraud (Y=1) : 362 observations

2. No Fraud (Y=0) : 22 836 observations

Benchmark description

Evaluation metric

The objective is to identify fraudulent operations within a population by predicting a fraud risk/probability. Hence, the metric that will be used is the area under the precision-recall curve, also noted the PR-AUC.

The precision-recall curve is obtained by plotting the precision ($\frac{TP}{TP+FN}$) on the y-axis and the recall ($\frac{TP}{TP+FP}$) on the x-axis for all values of the probability thresholds between 0 and 1.

This metric is very useful for properly evaluating a model’s performance on the minority class in severely imbalanced classification problems.

The higher the PR-AUC, the better the model is at correctly detecting the minority class.

In this challenge, the PR-AUC will be calculated as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight:

$\text{PR-AUC} = \Sigma_{n} (R_n - R_{n-1}) P_n$,

where $P_n$ and $R_n$ are the precision and recall at the nth threshold.

N.B. This implementation corresponds to the average_precision_score metric of sklearn.

Hence, your submission file should have the following format:

You can use the Y_test_random and Y_test_benchmark .csv files as examples to verify the exact format and size of a valid submission file for this challenge.

Benchmarks

1. Benchmark 1: $\text{PR-AUC}_{1} = 0,017$
   This first benchmark is a naive one and is based on a baseline model which randomly predicts probabilities between 0 and 1.

2. Benchmark 2: $\text{PR-AUC}_{2} = 0,14$
   This second benchmark is based on our current solution where several pre-processing steps are applied, and a fine-tuned Machine Learning model is used to predict fraud risk.