Forecasting, Causal Inference & Bandits

A lot of the confusion in applied ML starts not with the model, but with the wrong question. `Forecasting` asks: what is most likely to happen next? `Causal inference` asks: what will change if we intervene? `Bandit/policy` asks: what is better to choose right now, while we are still learning?

THREE CLASSES OF QUESTIONS

Forecasting
  "What will be the demand tomorrow?"
  “how much money will the client bring?”

Causal inference
  “Will a discount increase conversion?”
  "what will happen if you change onboarding?"

Bandit/policy
  “Which offer should I show this user now?”
  "Which ranking option should I use in the next impression?"

Similar at the product level, but mathematically these are different problems

Classic supervised ML often assumes that the examples are independent and the distribution does not drift too quickly. In a time series, neighboring points depend on each other in time. In causal, we are not just interested in the observed correlation, but in the effect of the intervention. In bandits, the data collection policy itself affects future data.

time series:
  x_t depends on x_(t-1), x_(t-2), ...

causal: causal
  observed outcome != effect of action

bandits:
  you select action
  and with this you change what you see next

Sometimes the model needs to predict a future value. Sometimes estimate the effect size of an action. Sometimes choose the action with the best expected reward. If you mix these three outputs, you can get a technically neat, but productually useless system.

To speak the same language with researchers and analysts, it is enough to remember a few central terms. They constantly pop up in articles, discussions of experiments, and product reviews.

horizon:
  how many steps forward do we predict?

counterfactual:
  what would happen if the action was different

reward:
  Useful framing of the chosen action

regret:
  how much was lost relative to the best action

policy:
  rule for choosing an action by context

A time series is rarely just a list of numbers. It is typically a mixture of slow baseline changes, repeating seasonal cycles, random noise, and occasional external shocks. A conversation about forecasting often begins with the decomposition of these components.

WHAT IS THE SERIES CONSISTED OF?

y_t =
  trend long rise/decline
  + seasonality day of week, month, hour
  + cycles longer oscillations
  + noise random noise
  + shocks holidays, promotions, accidents, releases

A good model either explicitly or implicitly separates these effects

If today's value is similar to yesterday's or to the value a week ago, then there is auto-dependence in the series. This is why lags and rolling statistics are so important: they give the model direct access to the past state of the system.

lag_1   = y_(t-1)
lag_7   = y_(t-7)
lag_24  = y_(t-24)

roll_mean_7  = mean(y_(t-6) ... y_t)
roll_std_30  = std(last_30_points)

calendar:
  day_of_week, month, holiday_flag

Forecasting one step ahead and forecasting 30 steps ahead are not the same task. The further the horizon, the higher the uncertainty and the more errors accumulate. In multi-step systems, you need to decide in advance whether you will build one general multi-horizon forecast or repeat a single-step forecast along the chain.

TWO MODES

one-step:
  y_(t+1) from past to t

recursive multi-step:
  first predict y_(t+1)
  then use it to predict y_(t+2)
  and so on

direct multi-step:
  separate model/head for each horizon

recursive is simpler, direct is often more stable on the far horizon

The usual random split for time series is almost always wrong. The future should not be included in the training of the model that supposedly predicts this future. Therefore, they use time-based split, rolling origin evaluation and `TimeSeriesSplit`-like schemes, where train is always strictly before validation.

CORRECT SERIES VALIDATION

Fold 1:
  train = [1 ... 100]
  test = [101 ... 120]

Fold 2:
  train = [1 ... 120]
  test = [121 ... 140]

Fold 3:
  train = [1 ... 140]
  test = [141 ... 160]

train grows, test always lies in the future

The most common failure of a forecasting project is not the “wrong model”, but an unnoticeable leak of the future. Any rolling mean, target encoding, cumulative metric or exogenous signal should be considered as if you are actually living at time `t` and do not yet know the future.

bad:
  average demand of the month, if the month has not yet ended
  normalizer, fit on the entire row
  signs from post-event tables

good:
  only information available at t
  feature engineering inside each fold

In forecasting it is very easy to build a complex model that is worse than the simplest baseline. Therefore, they almost always consider at least `last value`, `seasonal naive` or `drift`. If you don't beat these primitives, the project has not yet reached the real ML win.

naive:
  y_hat_(t+1) = y_t

seasonal naive:
  y_hat_(t+h) = y_(t+h-m)

drift:
  continue the last linear trend

basic question: are we really better than this?

In practice, forecasting often follows two paths. First: specialized time-series models like ETS / ARIMA, which explicitly describe the dynamics of the series. Second: feature-based ML, where you build lags, rolling features and exogenous features, and then train gradient boosting or another supervised model.

Very often forecasting is solved not by a separate “magic” model, but by ordinary supervised ML. You create lags, calendar features, prices, stocks, balances, weather data and turn each time point into a table row. Then you can learn boosting, a linear model or a neural network.

row at time t:
  lag_1
  lag_7
  roll_mean_14
  day_of_week
  holiday
  price
  promo_flag
  stock
  ...
target:
  y_(t+h)

Different forecasting metrics penalize the model in different ways. RMSE hits harder for large errors, MAE is more robust, MAPE breaks down around zero, and sMAPE partially corrects this defect. Therefore, one cannot say “the model is better” without specifying by what metric and on what horizon.

The usual predictive model answers the question “who buys more often?” well, but it still does not answer the question “what will happen if we send a promotional code?”. A model can find strong correlations and yet not know the true effect of an action at all. Causal inference is needed exactly where we are interested in the result of the intervention.

KEY DIFFERENCE

Prediction:
  P(Y | X)
  "What kind of users usually buy?"

Causal:
  effect of do(T=1) vs do(T=0)
  "What will change if we actually turn on treatment?"

High predictive accuracy does not guarantee correct causal assessment

The basic causal intuition is simple: the same person or object has at least two potential outcomes - with treatment and without treatment. The problem is that in reality we only see one of them. The second world remains a counterfactual, and that is why causal inference is more difficult than ordinary prediction.

Y(1):
  outcome if treatment was

Y(0):
  outcome if there was no treatment

individual effect:
  τ = Y(1) - Y(0)

ATE:
  E[Y(1) - Y(0)]

If treatment is not obtained by chance, but because the object already differs in important ways, a simple comparison of the treated and untreated groups will be biased. This is confounding. Usually, causal work is largely a fight against it.

EXAMPLE CONFOUNDING

observation:
  people with expensive tariffs buy more

naive output:
  expensive tariff causes purchases

reality:
  perhaps they switch to an expensive tariff more often
  already active and solvent clients

hidden reason:
  user_intent/affluence/engagement

Randomization makes the treatment independent of hidden factors on the sample average. This is why A/B tests are so valuable: they eliminate much of the confounding without complex math. If a good experiment can be done, this is usually the best way to estimate a causal effect.

random assign:
  T ~ Bernoulli(p)

→ treatment does not depend on user traits
→ groups are comparable on average
→ difference in outcomes is closer to causal effect

If an experiment is not possible or too expensive, causal inference attempts to approach it through research design. There is no one universal method: matching, regression adjustment, inverse propensity weighting, difference-in-differences, regression discontinuity and instrumental variables solve different types of bias.

The average effect is useful, but heterogeneity is often more important in products. One segment benefits from treatment, another is neutral, and the third even gets worse. Hence the language of `CATE`, `uplift modeling`, `treatment heterogeneity` and personalized interventions.

ATE:
  average effect over the entire population

CATE:
  effect for subgroup/context

uplift:
  increase in outcome precisely because of treatment

In personalization, one is almost always interested not just in P(Y), but in ΔY from action

Even neat teams regularly make mistakes in causal statements. Most often, this is either an incorrect assumption about parallel trends / ignorability, or post-treatment leakage, or an overly optimistic transfer of the effect from one environment to another.

The Bandit problem occurs when you need to choose an action right now, but you are not yet sure which action is best. If you always choose the current leader, you may miss the best option. If you experiment too much, you lose the reward today. It is this compromise that is called exploration vs exploitation.

CLASSICAL METAPHOR

you have several “hands” of the machine:
  arm A
  arm B
  arm C

each gives reward with unknown distribution

At each step you need to decide:
  play the best known hand?
  or else check the less studied one?

bandit = limited risk online learning

In bandits, it is important not only how much reward you collected, but also how much you lost relative to the ideal oracle, which always knows the best action. This difference is regret. A good bandit policy makes regret grow slowly as experience accumulates.

oracle reward:
  always chooses the best arm

policy reward:
  sometimes makes mistakes while learning

regret =
  oracle cumulative reward
  - policy cumulative reward

Epsilon-greedy works crudely, but is very clear. With probability `1 - ε` you choose the best known arm, and with probability `ε` you randomly try something else. This is a simple baseline that is often used as a starting point before smarter strategies.

if rand() < ε:
  action = random arm
else:
  action = argmax estimated_reward

update estimate after observing reward

Upper Confidence Bound chooses not just the arm with the best average reward, but the arm with the best “optimistic score”. If an option has been little explored, an exploration bonus is added to it. Therefore, UCB automatically balances learning and use without random rolls like epsilon-greedy.

score(arm) =
  empirical_mean(arm)
  + uncertainty_bonus(arm)

if arm is poorly studied:
  bonus big

if arm is studied well:
  bonus small

Thompson Sampling stores the uncertainty distribution over the quality of each arm and samples from this belief at each step. An Arm with a potentially high reward and high uncertainty has a natural chance of being selected. This is one of the most practical bandit approaches.

for each arm:
  sample θ_arm from posterior

action =
  argmax sampled_θ_arm

observe reward
update posterior

In many products there is no one best arm for everyone. It is better to show different options to different users. Then bandit receives the user's context as input and learns a policy of the form `π(action | context)`. This is closer to personalization, advertising, recommendations and adaptive ranking.

FROM MULTI-ARMED TO CONTEXTUAL

simple bandit:
  choose best arm overall

contextual bandit:
  user features x
  ↓
  choose arm, the best one for x

example:
  new user → onboarding A
  power user → onboarding B
  budget user→ offer C

An A/B test is good when you need to reliably measure the effect of fixed options. Bandits are good when you need to adaptively collect rewards along the way, without showing weak options for too long. They are often combined in products: first A/B for pure measurement, then bandit for adaptive serving.

Forecasting can be fairly fairly validated by backtest. Causal - through experimental design and assumptions. Bandits and policy systems often run into the fact that offline logs are collected by the old policy, and the new policy will collect different data. Therefore, online validation is especially important here.

In many real-world systems, the reward does not come immediately, but after hours or days. Plus, the environment is changing: season, competitors, interface, economic situation, new user segments. This complicates forecasting, bandits, and post-launch causal assessment.

delayed feedback:
  action now
  reward tomorrow

non-stationarity:
  P(data) changes over time

policy feedback loop:
  the system itself changes user behavior

When a product problem is formulated vaguely, it is useful to quickly run it through one decision tree. Usually this is enough to avoid launching an erroneous project under the guise of “let’s just make an ML model.”

QUICK SELECTION OF POSITION

Question:
  you need to understand what will happen next,
  if we don't change anything?
    → Forecasting

Need to understand the effect of a change?
    → Causal inference

You need to select action online,
Are we still learning?
    → Bandit / political learning

Do you just need to recognize the class of an object?
    → regular supervised ML

Bandits are often called “RL lite”. The difference is that bandit usually does not model long sequences of states and long-term consequences through state transitions. RL is needed where the action now affects what state you will be in next, and this is repeated along the chain.

bandit:
  context → action → immediate reward

RL:
  state → action → next state → ... → long-term return

If there are no long state dynamics, bandit is often simpler and more practical

There is no one universal “ML model for business”. There are different classes of questions. If you need a forecast, build forecasting and think about the horizon and backtesting. If you want an intervention effect, think causal and assumptions. If you need an online selection of actions, go to bandits and regret. Correct setup saves months of useless optimization.