Two programs, one feature name
A card issuer's fraud model has a feature called card_txn_count_1h, how many times the card was used in the last hour, and two programs compute it. For training, a Spark job over the warehouse counts settled transactions per clock hour. In production, an engineer wrote it again inside the scoring service as a rolling 60-minute count of authorization attempts, declines included. A stolen card that a fraudster tests with ten small declined payments reads 10 in production and close to 0 in the training data, because declined payments never settle.
The model learned what the training numbers meant, and live it catches about half the fraud its evaluation promised. That gap is training-serving skew: a feature computed one way for training and another way at prediction time. Nothing raises an error: unless someone compares the values the two paths produce, it shows up only as a model that performs worse than its evaluation said.
One definition, two stores
A feature is one input value the model reads, computed from raw data and attached to an entity such as a card, a merchant or a customer. A feature store (Feast, Tecton) defines each feature once and serves the same values two ways: a low-latency online store for inference and a historical offline store for training, so the values a model trains on match the values it sees in production. Pipelines, batch or streaming, do the computing; the store holds what they produce and serves it. The match holds only while every pipeline computes the feature from that one definition.
The offline store keeps history: each feature's past values, each with the time it took effect, usually as tables in a data lake or warehouse. It answers "what did we know about card 4410 on 3 March?" for millions of rows at once, in minutes if need be. The online store keeps only the latest value per entity, in a key-value database such as Redis or DynamoDB, and answers "what is true about card 4410 now?", usually in a few milliseconds. It has to: the fraud check runs inside a card authorization that gives it about 100 ms in total. Loading values into the online store is called materialization: a scheduled job loads the latest batch values, and streaming jobs write theirs as they compute them.
Streaming and nightly pipelines compute the features; the feature store keeps their history for training sets and their latest values for the scorer.
A training set that knows the future
The offline store also builds the training sets, and the obvious way to build one leaks. Fraud labels arrive late: a cardholder disputes a payment days or weeks after it happened. Suppose the team joins every labelled payment to the card's features as they are today. For a payment made on 3 March and disputed on 20 March, today's card_chargebacks_90d already counts that very dispute. The model learns that a recent chargeback means fraud, a rule nearly perfect in training and useless at authorization time, when the dispute has not happened yet. That is data leakage: information in the training data that the model could not have had when it made the prediction.
A point-in-time correct training set prevents it. For each labelled event and each feature, the join takes the latest value whose timestamp is at or before the event's own time:
- The training job reads a labelled event: payment 88213, card 4410, 3 March, fraud.
- The offline store returns the card's
card_chargebacks_90dhistory: 0 from 1 March, 1 from 20 March. - The join keeps the last value at or before the payment, which is 0.
- The job does the same for every other feature, then writes the row with its label.
This needs history with timestamps; a table that overwrites each card's row every night has nothing to look back at. The timestamps also have to be honest. If a nightly job writes Tuesday's totals at 03:00 on Wednesday but stamps them with Tuesday's date, a training row for a payment at 18:00 on Tuesday sees totals the online store did not have until the next morning: a smaller leak of the same kind. The safe timestamp for the join is the moment the value could actually have been served, not the day it describes; when the two differ, keep both and join on the first. Many stores also give a feature a maximum age, so training does not use a value the online store would no longer serve.
How fresh is fresh enough
card_txn_count_1h is useless an hour late, once the burst of declined test payments is over. Features like that come from a stream processor, Flink in this design, which updates the count as each payment arrives and as older ones leave the hour, and writes it to the online store within seconds. A customer's 90-day average spend can come from the nightly batch; a day of delay barely moves it. Freshness is chosen per feature. Windows and late events belong to Stream processing: windows, watermarks & state, scheduled batch jobs to ETL/ELT & data orchestration.
A streaming feature can bring skew back by another door. The offline store needs its history too, and if a separate batch job recomputes that history, two pieces of code define one feature again. The usual answers: the streaming job writes each value to the offline store too, or a platform runs one definition both as the streaming job and as the backfill over history.
Whatever the design, check it: log the feature values the scorer actually received with each prediction, recompute a sample offline, and compare. Some teams train directly on those logged values, which removes this skew, but a new feature then has to wait for the log to fill.
One name, one meaning
Skew also hides in missing values. A brand-new card has no history; if training recorded that as missing while the online lookup returns 0 for an unknown key, a first payment looks like a card that has simply been quiet for an hour. So the default belongs in the feature's definition, kept in the feature registry with its name, entity, source and transformation, owner and freshness. The registry makes features reusable: a credit-limit model reads the existing card_txn_count_1h instead of growing a third version with its own idea of an hour. It is also why a feature store is more than a cache: training and inference both depend on it, and changing a definition changes the inputs of every model that reads it. Its data is still derived: it can be recomputed as long as the inputs are kept, here the payment events with their declines, which the settled transactions in the warehouse leave out. Only the features a deployed model reads need to be online; each extra one costs storage for every entity and pipeline work to keep it fresh.
Follow one feature both ways
In the ML Serving & Feature Store challenge, follow one feature from raw data to both of its consumers: the scheduled job that builds training sets from the data lake, and the inference service's lookup at request time, both reading the one Feature Store. Write on the training path which timestamp the join uses, and mark which features would need a streaming path. Real-Time Fraud Detection asks for the same consistency inside a full fraud decision.
Consistent inputs do not keep a model right over time. When fraudsters change tactics, every feature can be computed perfectly and the model still decays; watching for that drift happens where the model is served, in Model serving: inference, versions & safe rollout. Rankers read features from a store the same way, as Ad serving & real-time auctions and Recommendation systems: retrieval, ranking & cold start show.