Propensity Scoring (Binary Classifier)

Score every customer on how likely they are to convert, churn, or respond, so campaigns reach the people most likely to act.

The Propensity Scoring solution from Treasure AI's AI Signals platform learns from labeled history and scores every customer with the likelihood of a single outcome you define. You supply a table of customer features and one column marking who did and did not do the thing. It returns a score per customer, a yes/no label, and the diagnostics you need to trust the result.

Propensity Scoring is one of the most flexible options in AI Signals because you choose the outcome. Any question of the form will this customer do X? fits, as long as you can label the answer for enough people in the past. Convert, churn, upgrade, open, click, redeem, renew, respond — all use the same model, but with different labels.

Common use cases

• Rank prospects by likelihood to convert so sales works the hottest leads first
• Flag customers likely to lapse and trigger proactive win-back journeys
• Target the customers most likely to buy in the next campaign window
• Find customers primed for a premium tier or an add-on
• Predict who is likely to open, click, or unsubscribe, and tune frequency accordingly
• Feed the score into CLTV, NBA, or lookalike targeting as an input signal

Who benefits most: Marketing analysts, CRM managers, and lifecycle teams who have a clear yes/no outcome to predict and want an explainable score across their whole base without building or maintaining a custom ML pipeline.

Propensity or RFM? RFM describes what a customer has been, using past purchase behavior to produce segments. That ranking is descriptive: it summarizes history rather than predicting the future, and it cannot tell a loyal customer who is simply between purchase cycles from one who has genuinely lapsed. Propensity Scoring answers the narrower question directly, against an outcome you name.

These signals feed each other rather than compete. RFM quartiles and CLTV forecasts make strong input features for a propensity model, and pairing a propensity score with CLTV tells you both how likely a customer is to act and how much that action is worth.

Propensity Scoring runs as two functions on the Treasure AI ML Batch API:

• classifier_train fits a model on your labeled table, tunes it, picks a decision threshold, writes diagnostic tables, and registers the model under model_name.
• classifier_predict loads that registered model and scores new customers.

Your input is a flat table, one row per customer, with a binary target column and any number of feature columns. Column names are yours to choose: point target_column at your label and user_id_column at your identifier. Everything else in the table is treated as a feature.

You only need to set a handful of parameters. input_table, output_table, and target_column are required for training. Add model_name so you can predict with the model later, and user_id_column so scores come back attached to a customer. Everything else has a working default, including model_type, which defaults to xgboost.

Feature engineering happens upstream. The solution expects the aggregation and joins to be done already, one row per customer, before the table reaches it.

# classifier_train.dig
# Step 1: Train, tune, select a threshold, and register the model under `model_name`
+train:
  http>: https://ml-batch-api.treasuredata.com/v1/runs
  method: POST
  headers:
    - authorization: "TD1 ${secret:td.apikey}"
    - X-TD-ML-SESSION-ID: ${session_id}
    - X-TD-ML-ATTEMPT-ID: ${attempt_id}
  store_content: true
  content:
    input_table: your_database.customer_features
    output_table: your_database.propensity_output
    solution_name: classifier_train
    solution_arguments:
      model_name: "lead_scorer_v1"
      model_type: "xgboost"
      target_column: "converted"
      user_id_column: "user_id"
      tuning:
        n_trials: 30
        optuna_metric: "fbeta"
        fbeta_beta: 0.5
      threshold:
        strategy: "precision_optimal"
        min_metric_floor: 0.35

+print_response:
  echo>: "Training job submitted. Response: ${http.last_content}"

# Poll until training completes. The API returns HTTP 408 while the job
# is running; Treasure Workflow retries until it receives HTTP 200.
+train_poll_status:
  http>: https://ml-batch-api.treasuredata.com/v1/runs/${JSON.parse(http.last_content)['id']}/status
  method: GET
  headers:
    - authorization: "TD1 ${secret:td.apikey}"

Expected output: a family of diagnostic tables written into the output_table database, each prefixed classifier_. The headline one is classifier_run_log, which carries test-set metrics and the selected threshold for the run. The trained model is saved to managed storage under model_name. See Model Configuration for the full list and Reading the Diagnostics for how to read them.

# classifier_predict.dig
# Step 2: Predict, load the registered model and write a score per customer
+predict:
  http>: https://ml-batch-api.treasuredata.com/v1/runs
  method: POST
  headers:
    - authorization: "TD1 ${secret:td.apikey}"
    - X-TD-ML-SESSION-ID: ${session_id}
    - X-TD-ML-ATTEMPT-ID: ${attempt_id}
  store_content: true
  content:
    input_table: your_database.customers_to_score
    output_table: your_database.propensity_scores
    solution_name: classifier_predict
    solution_arguments:
      model_name: "lead_scorer_v1"     # Must match the training run
      user_id_column: "user_id"

+print_response:
  echo>: "Prediction job submitted. Response: ${http.last_content}"

+predict_poll_status:
  http>: https://ml-batch-api.treasuredata.com/v1/runs/${JSON.parse(http.last_content)['id']}/status
  method: GET
  headers:
    - authorization: "TD1 ${secret:td.apikey}"

Expected output: one row per customer with a score, a label, the threshold that produced it, and a score bucket for segmentation.

user_id      score    predicted_label  threshold_applied  score_bucket
3105285968   0.9412   1                0.41               0.9-1.0
1850985734   0.6558   1                0.41               0.6-0.7
274382808    0.2231   0                0.41               0.2-0.3
358273144    0.0487   0                0.41               0.0-0.1

There is no separate tuning function. Hyperparameter search happens inside classifier_train, so training and prediction are the only two steps.

To create and schedule these workflows, see Getting Started with Treasure Workflow.

Each function takes two kinds of input: the data table you point it at, and the parameters you pass it. Both are documented under Data Preparation and Workflow Parameters below.

Training parameters arrive in a core block plus four optional nested blocks — tuning, threshold, metric, and one model-specific block named after the algorithm. Omit any nested block to accept its defaults. Model Types lists each algorithm's complete classifier_train search-range parameters, so you can configure an algorithm from one table.

classifier_train fits one model on your labeled table, tunes it, selects a decision threshold, and saves it for reuse.

A flat, customer-level table, one row per observation, with a binary target column and any number of feature columns. Column names are yours to choose.

How features are handled

• Numeric columns are used as-is. Missing values pass through to the tree model, which handles them natively; there is no imputation step.
• Categorical (string) columns are used natively — XGBoost through the pandas category dtype, CatBoost through cat_features — with a "missing" token standing in for nulls and for values not seen during training.
• Correlated numeric features are pruned. Pairs above corr_threshold (default 0.90) have one member dropped, and the decision is recorded in classifier_corr_drop_report alongside correlation and VIF diagnostics.
• session_id is dropped automatically. Add anything else you want excluded to columns_to_drop.

Core parameters

tuning block — controls the embedded hyperparameter search. Applies to both algorithms.

threshold block — controls how the score-to-label cutoff is chosen.

metric block — evaluation and preprocessing controls.

The remaining block is named after the algorithm — xgboost or catboost — and sets the search ranges for that family's hyperparameters. Those, plus the shared boosting ranges, are documented in full under Model Types.

Training writes a family of diagnostic tables into the output_table database. Every table is prefixed classifier_ and stamped with session_id and created_at, so runs accumulate rather than overwrite each other.

The trained model is saved to managed model storage under model_name. It carries the run configuration, the fitted estimator, the feature names, the winning hyperparameters, the selected threshold, and the preprocessing metadata — category vocabularies and the list of dropped correlated features — so prediction reproduces training exactly.

classifier_predict loads a registered model and writes a score per customer.

A customer-level table without the target column, carrying the same feature columns used at training. You do not have to match the training table exactly: missing columns are backfilled, extra columns are dropped, and the remainder is reordered to the trained feature set. Categorical values the model never saw during training map to the "missing" token.

Prediction is never row-capped. max_load_rows bounds training only; every row of the prediction input gets scored.

One row per scored customer.

Using the output. For most marketing work, target by score rather than predicted_label. Sort descending and take the top decile for a high-value campaign, or suppress the bottom half from expensive paid media. predicted_label answers a different question — it applies one fixed cutoff chosen during training, which is the right tool when you need a single yes/no decision per customer and the wrong one when you have a fixed budget and want the best N people you can afford. Join the output to your customer table so it becomes a Master Segment attribute, then activate through Journeys, Triggers, Personalization, or Segment Builder.

Propensity Scoring supports two gradient-boosting algorithms behind one interface. They share the same preprocessing, tuning, evaluation, threshold selection, explainability, and output schema, so switching between them is a one-line change and the results stay comparable.

Recommendation: Start with xgboost. If your feature set is categorical-heavy, train catboost under a second model_name and compare classifier_run_log rows. In both cases, read the SHAP output before activating, to confirm the model is learning sensible signals rather than leaking the answer.

The default. XGBoost builds an ensemble of decision trees, each correcting the errors of the ones before it, using the histogram-based hist tree method. Categorical columns are handled natively through the pandas category dtype rather than one-hot expansion, which keeps wide categorical feature sets manageable. Class imbalance is handled by setting scale_pos_weight automatically when class_balance_strategy is balanced.

Shared boosting search ranges — each is a _min / _max pair inside the xgboost block.

XGBoost-specific search ranges

+train_xgboost:
  http>: https://ml-batch-api.treasuredata.com/v1/runs
  method: POST
  headers:
    - authorization: "TD1 ${secret:td.apikey}"
    - X-TD-ML-SESSION-ID: ${session_id}
    - X-TD-ML-ATTEMPT-ID: ${attempt_id}
  store_content: true
  content:
    input_table: your_database.customer_features
    output_table: your_database.propensity_output
    solution_name: classifier_train
    solution_arguments:
      model_name: "churn_xgb_v1"
      model_type: "xgboost"
      target_column: "churned"
      user_id_column: "user_id"
      columns_to_drop: "email|signup_ts"
      tuning:
        n_trials: 50
        optuna_metric: "auc_pr"
        cv_folds: 5
      threshold:
        strategy: "f1_optimal"
      metric:
        corr_threshold: 0.85
      xgboost:
        max_depth_min: 3
        max_depth_max: 8
        subsample_min: 0.7

CatBoost is a gradient-boosting library built around categorical features. It encodes them using ordered target statistics, computed over a random permutation of the data so that a row's own label never contributes to its own encoding. That is what keeps high-cardinality categories — postal codes, SKUs, campaign IDs — from quietly leaking the answer, which is the usual failure mode when those columns are target-encoded by hand. Class imbalance is handled by CatBoost's Balanced automatic class weights when class_balance_strategy is balanced.

Shared boosting search ranges — each is a _min / _max pair inside the catboost block.

CatBoost-specific search ranges

+train_catboost:
  http>: https://ml-batch-api.treasuredata.com/v1/runs
  method: POST
  headers:
    - authorization: "TD1 ${secret:td.apikey}"
    - X-TD-ML-SESSION-ID: ${session_id}
    - X-TD-ML-ATTEMPT-ID: ${attempt_id}
  store_content: true
  content:
    input_table: your_database.customer_features
    output_table: your_database.propensity_output
    solution_name: classifier_train
    solution_arguments:
      model_name: "churn_cat_v1"
      model_type: "catboost"
      target_column: "churned"
      user_id_column: "user_id"
      tuning:
        n_trials: 50
        optuna_metric: "auc_pr"
      threshold:
        strategy: "f1_optimal"
      catboost:
        border_count_min: 64
        border_count_max: 255
        l2_leaf_reg_min: 2.0

Unlike NBA, Propensity Scoring has no separate tuning function. The search runs inside classifier_train every time, so a single job produces a tuned model.

Every trial is written to classifier_optuna_trials, so you can see the search rather than just its winner. That table is the place to check whether the search converged or whether the best trial was an outlier.

Three levers control cost against quality:

• n_trials is the main one. The default of 30 is a reasonable balance. Raise it to 50–100 when model quality matters more than runtime. Set it to 0 to skip the search entirely and train on built-in defaults, which is useful for a fast first look at a new dataset.
• cv_folds trades stability for time. Three folds is fast; five is steadier on smaller or noisier data.
• max_tuning_rows caps only the search. Lowering it speeds up tuning on a very large table without changing what the final model is fit on.

Choose optuna_metric to match your problem rather than accepting the default reflexively. fbeta with fbeta_beta below 1 biases toward precision, which suits campaigns where contacting the wrong person costs money. auc_pr is the right choice when positives are rare, since it focuses on the positive class in a way auc_roc does not.

The decision threshold is chosen during training and travels with the model, which is why classifier_predict needs no threshold parameter of its own.

It is selected on out-of-fold cross-validation predictions rather than on the model's own training scores. This matters more than it sounds. Boosted trees can nearly memorize their training split, which makes in-sample probabilities look far more separable than they really are and produces a threshold that collapses the moment it meets new data. Fitting the threshold out-of-fold avoids that. The untouched test split is then scored separately, purely for honest reporting.

min_metric_floor is the guardrail that stops an optimizer from producing a technically-optimal but useless operating point — a threshold with perfect precision that flags eleven people. Its direction follows fbeta_beta: when fbeta_beta is below 1 the floor is a minimum recall, and when it is 1 or above the floor is a minimum precision.

For most marketing work you can ignore the label and the threshold entirely, and target by score. The threshold matters when you need one automatic yes/no decision per customer.

Training writes far more than a metrics row. The tables worth reading first, in order:

classifier_run_log against classifier_train_run_log. These carry the same metrics on the test and train splits. Read them side by side. A model that scores far better on train than test is overfitting, and the gap tells you how much. A model that scores similarly on both is generalizing, whatever the absolute numbers say.

classifier_feature_importance and classifier_shap_directional. SHAP importance ranks which features moved the model most. The directional table goes further and says which way each one pushed, with the share of customers it pushed up versus down. Read this before you activate anything. One feature dominating the ranking is the classic signature of label leakage.

classifier_prediction_bins and classifier_roc_curve. The score histogram shows how the population spreads across the 0–1 range. A healthy model separates; a poor one piles everyone into the middle. The ROC curve is the standard picture of the precision-recall trade-off across every possible threshold, useful when you want to argue for a different cutoff than the one selected.

PSI, recorded in the run log, compares the train and test score distributions. It is a drift measure: a large value means the two splits are scoring differently, which usually points to a time-ordered split or a population shift rather than a modeling problem.

Two headline metrics deserve a note. AUC-ROC measures overall ranking quality and is the number most people ask for, but it flatters models on imbalanced data. AUC-PR focuses on the positive class and is the more honest headline when positives are rare — which is why aucpr is the default eval_metric. The run log also records pseudo-R² (McFadden and Cox-Snell) and a chi-square statistic for readers who want a goodness-of-fit view alongside the ranking metrics.

Treat score as a reliable ranking — higher genuinely means more likely — rather than as a literal probability. Validate against real outcomes before reading a score of 0.30 as exactly a 30% chance.

Leakage is the most common way a propensity model looks excellent in testing and fails in production. It happens when a feature is really a stand-in for the outcome: a conversion_date that only exists for converters, a cancellation_reason populated only for churners, an account-status flag updated at the moment the event occurred.

The solution removes some of this risk for you. session_id is dropped automatically, correlated numeric features above corr_threshold are pruned, and the identity column named in user_id_column is excluded from the feature set.

The rest is yours to catch:

• Drop post-outcome columns with columns_to_drop. Anything whose value was written at or after the moment the outcome happened does not belong in the feature set.
• Drop raw identifiers and timestamps. Add Treasure AI's implicit time column to columns_to_drop if it survives into your feature table.
• Read the SHAP output. A single feature accounting for most of the model's importance is a leak until proven otherwise. classifier_target_correlations is the fast version of the same check — a feature correlating near-perfectly with the target is almost never a real signal.

Trained models are saved to managed model storage under model_name, unique per Treasure AI account, and retrieved automatically by classifier_predict.

The stored artifact carries more than the fitted estimator. It also holds the run configuration, the ordered feature names, the winning hyperparameters, the selected threshold, and the preprocessing metadata — category vocabularies and the list of correlated features that were dropped. That is why prediction reproduces training exactly, and why the prediction input does not have to match the training table column for column: the model knows what it expects and reshapes what it receives.

Reusing a model_name overwrites the model saved under it. When comparing algorithms, give each its own name — churn_xgb_v1 and churn_cat_v1 — so one training run does not silently replace another.

What data do I need? A flat table with one row per customer, a binary target column marking the outcome you want to predict, and any number of feature columns describing each customer. You need enough historical examples of both outcomes for the model to learn the pattern. Feature engineering happens upstream; the solution does no aggregation of its own.

Do my columns need specific names? No. target_column and user_id_column point at whatever your table already uses, and every remaining column is treated as a feature. This is different from RFM, which requires three fixed column names.

Which algorithm should I choose? Start with xgboost, the default. If your features are heavily categorical, train catboost under a second model_name and compare the classifier_run_log rows. Both share the same evaluation and output schema, so the comparison is direct.

Is there a separate tuning step like NBA has? No. Hyperparameter search is embedded in classifier_train, so training and prediction are the only two functions.

How should I encode the label? As a column with two distinct classes, integer or boolean. A table with only one class fails the run with a user input error rather than training a model that cannot learn anything.

What happens to missing values? Numeric missing values pass straight through to the tree model, which handles them natively — there is no imputation step. Missing or unseen categorical values map to a "missing" token.

Do I need to one-hot encode my categorical columns? No. Both algorithms handle categorical columns natively, XGBoost through the pandas category dtype and CatBoost through its own categorical handling. Leave them as strings.

Will the model still work if my positive class is very rare? Yes, within limits. Leave class_balance_strategy on balanced and set optuna_metric to auc_pr, which focuses on the positive class. Very rare positives still produce noisier scores, so validate against real outcomes before acting on them at scale.

Why does my model look great in testing but do poorly live? Almost always label leakage — a feature that is really a stand-in for the outcome. Read classifier_feature_importance and classifier_target_correlations to spot a suspiciously dominant feature, then exclude it with columns_to_drop and retrain. See Preventing Label Leakage.

How often should I retrain versus re-score? Retrain when your customer base or business shifts enough to change behavior, typically monthly. Re-score more often, daily or weekly, against the saved model so recently active customers get fresh scores. Splitting the two lets you tune each cadence independently, and prediction is far cheaper than training.

How do I interpret the score? As a ranking of likelihood, from 0.0 to 1.0, where higher means more likely to take the action. For campaigns, sort by score and take the top slice — the top decile, or as many customers as your budget covers — rather than relying on predicted_label. See the note under Reading the Diagnostics on reading scores as absolute probabilities.

How do I change the threshold? Set threshold.strategy at training time. Use recall_optimal when missing a positive is the expensive error, precision_optimal when contacting the wrong person is, or fixed with fixed_threshold when a business rule sets the cutoff. The threshold is chosen during training and travels with the model, so it cannot be changed at prediction time without retraining.

Can I see why the model made a prediction? Yes. Every training run writes SHAP feature importance and a directional breakdown showing which way each feature pushed scores. Use shap_max_samples to control how much data that computation uses.

Can I speed up training? Set tuning.n_trials: 0 to skip the hyperparameter search and train on built-in defaults. Lowering max_tuning_rows or cv_folds are gentler versions of the same trade.

How large a table can I train on? Loading is capped by max_load_rows, which defaults to 10 million rows and can be raised.

How do I score millions of customers? Prediction distributes across workers automatically, sharded by a stable hash of the identity column. Set user_id_column — without it, prediction is limited to a single worker.

Can this predict more than two outcomes? No. This is a binary classifier: one yes/no outcome, one score per customer. Multi-class problems are not yet supported. Model each outcome separately if you need several classes.

Can I get scores in real time? No. Scoring runs as scheduled batch jobs. Frequent workflow runs can get you to a sub-daily cadence, but true real-time serving is not supported.

How do I activate scores in a campaign? The output table carries a score and a score_bucket for every scored customer. Join it to your customer table so it becomes a Master Segment attribute, then build segments from it — a high-intent segment as score >= 0.7, or a top-decile audience — and activate through standard CDP activation workflows.