13. Extended Study Notes and Practice Sets¶

This section turns the 12 lecture topics into deeper study notes. Use it after reading Chapters 1-12 once. Each mini-module gives extra intuition, formal structure, properties, and a practice-style worked example.

13.1 Lecture 1 Deep Dive: Overview¶

Subtopic: decision quality over time¶

Intuition

In static business analysis, we often ask whether a decision is good immediately. In DDM, a decision is good if it leads to good total performance over time. This is why "closest driver" or "cheapest shipment" can be bad. They may damage future flexibility.

Formal definition

For a policy $\pi$, total performance over a finite horizon is:

\[ \operatorname{Total}(\pi) = E\left[\sum_{k=0}^{K}R(S_k,X^\pi(S_k))\right] \]

where:

$X^\pi(S_k)$ means the decision selected by policy $\pi$ in state $S_k$,
$\mathbb{E}[\cdot]$ averages over random future information.

Translation:

Judge the rule by total expected performance, not by one decision.

Properties

Short-term and long-term performance may conflict.
A good policy can make a locally worse decision to improve future options.
The exam often rewards explicit future-consequence language.

Worked example

Policy A always accepts a request if it earns at least $10. Policy B rejects requests in low-demand zones when only one central vehicle remains.

Policy	Current accepted requests	Later missed high-value requests	Total profit
A	10	4	$80
B	7	1	$110

Policy B looks less aggressive now but performs better over time.

Subtopic: business decision support vs automation¶

Intuition

DDM models can support human decisions or automate decisions. Business students should understand that the same model can be used in different ways:

recommend an action,
rank alternatives,
simulate policy performance,
automatically choose actions.

Formal structure

\[ \text{data}\rightarrow S_k\rightarrow \pi(S_k)\rightarrow x \]

Properties

Human-in-the-loop systems need interpretability.
Automated systems need reliability and monitoring.
Policies should be evaluated before deployment.

13.2 Lecture 2 Deep Dive: Information Modeling and Decision Making¶

Subtopic: value of information¶

Intuition

Information is valuable only if it can change a decision. A perfect traffic forecast is useful if route choice depends on traffic. It is less useful if there is only one feasible route.

Formal idea

Expected value with information:

best expected performance after observing information

Expected value without information:

best expected performance before observing information

Value of information:

VOI = performance with information - performance without information

For cost minimization:

VOI = cost without information - cost with information

Translation:

Information is valuable when it improves the decision.

Properties

Better forecasts are not automatically valuable.
Information has operational value only if decisions can react.
Real-time information is more valuable in dynamic systems than in static systems.

Worked example

Without a traffic forecast, expected route cost is:

Route A expected cost = 30
Route B expected cost = 32
choose A

With a traffic forecast, the company chooses A on clear days and B on congested days. Average cost becomes 24.

VOI = 30 - 24 = 6 cost units

Subtopic: modeling granularity¶

Intuition

Granularity means how detailed the model is. A city can be modeled by exact GPS coordinates, street segments, or zones. More detail is not always better.

Formal representation

Fine state:

\[ S_k=(\text{exact vehicle coordinates},\ \text{exact customer coordinates},\ \text{exact routes}) \]

Aggregated state:

\[ S_k=(\text{vehicles by zone},\ \text{requests by zone},\ \text{time period}) \]

Properties

Granularity	Merit	Disadvantage
Fine	accurate detail	huge state space
Aggregated	easier to learn and optimize	loses detail
Hybrid	balances detail and tractability	design effort

Exam phrase

The state should include decision-relevant information, but excessive detail may worsen the curse of dimensionality.

13.3 Lecture 3 Deep Dive: Stochasticity¶

Subtopic: conditional probability for business forecasts¶

Intuition

Uncertainty often depends on context. Demand at 8 AM differs from demand at 11 PM. Travel time in rain differs from travel time in clear weather.

Formal definition

Conditional probability:

P(A | B)

means:

probability that A happens given that B is known

Example:

P(high demand | Friday evening)

Properties

Conditional probabilities make information models state-dependent.
State-dependent uncertainty is central in DDM.
Good states include variables that change future probability distributions.

Worked example

Suppose:

P(high demand) = 0.3
P(high demand | rain) = 0.6

If rain is observed, the model should use 0.6, not 0.3.

Decision effect:

more vehicles should be positioned near high-demand zones when rain is observed

Subtopic: random variables in notation¶

Intuition

A random variable is a name for an uncertain number. It could be demand, travel time, service time, or cost.

Formal definition

D = random demand
T = random travel time
C = random cost

Realization:

D = 12

means demand turned out to be 12.

Properties

Before observation, use uppercase or descriptive random notation.
After observation, use realized value.
In DDM, $\omega_{k+1}$ can contain several random variables at once.

Worked example

\[ \omega_{k+1}=(\text{new orders},\ \text{travel-time realizations},\ \text{driver availability changes}) \]

This single symbol represents all new exogenous information after decision $k$.

13.4 Lecture 4 Deep Dive: Dynamism¶

Subtopic: event-based vs time-based decision points¶

Intuition

Decision points can happen at fixed times or when events occur.

Time-based:

decide every 5 minutes

Event-based:

decide whenever a new customer request arrives

Formal notation

Time-based:

\[ k=0,1,\ldots,K \]

Event-based:

k increments when a decision-relevant event occurs

Properties

Decision timing	Merit	Disadvantage
fixed time intervals	easy to implement	may miss urgent events
event-based	responsive	irregular timing can complicate modeling
hybrid	practical compromise	more design choices

Worked example

Meal delivery may use hybrid decision points:

reoptimize every 5 minutes
also reoptimize immediately if a high-priority order arrives

Subtopic: commitment and flexibility¶

Intuition

Dynamic decisions differ in how much they commit the future. Some decisions are reversible. Others are not.

Formal idea

If decision $x$ changes the feasible set later:

\[ X(S_{k+1})\ \text{depends on }x. \]

Translation:

What you can do tomorrow depends on what you choose today.

Properties

Irreversible decisions need more anticipation.
Flexible resources have high future value.
Policies should preserve flexibility when future uncertainty is important.

Worked example

Shipping a container by truck today removes it from the yard permanently. Holding it preserves the option to consolidate by barge tomorrow.

13.5 Lecture 5 Deep Dive: Stochastic and Dynamic Decision Problems¶

Subtopic: transition functions vs transition probabilities¶

Intuition

There are two ways to describe movement from one state to the next.

Transition function:

Tell me exactly how the next state is computed.

Transition probabilities:

Tell me the probability of each possible next state.

Formal definitions

Transition function:

\[ S_{k+1}=f(S_k,x,\omega_{k+1}) \]

Transition probability:

\[ P(S_{k+1}\mid S_k,x) \]

Properties

Simulators often use transition functions.
Small MDP graphs often use transition probabilities.
Both describe the same conceptual movement from current state to next state.

Worked example

Inventory:

\[ \begin{aligned} S_k &= \text{current inventory},\\ x &= \text{order quantity},\\ D_{k+1} &= \text{random demand},\\ S_{k+1} &= S_k+x-D_{k+1}. \end{aligned} \]

If inventory cannot be negative:

\[ S_{k+1}=\max\{S_k+x-D_{k+1},0\} \]

Subtopic: finite horizon vs infinite horizon¶

Intuition

A finite-horizon problem has a clear end, such as the end of the day or five work days. The DDM course often uses finite horizons.

Formal finite horizon

k = 0, 1, ..., K

Objective:

\[ \min E\left[\sum_{k=0}^{K}R(S_k,x_k)\right] \]

Properties

End-of-horizon costs matter.
Decisions near the end can differ from decisions early in the horizon.
Terminal value should be included if specified.

Worked example

In Trucks & Barges, containers left at the end may have stochastic clean-up costs. Therefore terminal cost is not zero if the exam specifies clean-up costs.

13.6 Lecture 6 Deep Dive: Approximate Dynamic Programming¶

Subtopic: choosing an ADP method¶

Intuition

The best ADP method depends on the business setting. There is no universal winner.

Decision guide

Situation	Good candidate
need fast transparent rule	PFA
strong existing optimization model	RHO
model is myopic in known ways	CFA
future scenarios important	Look-ahead/MSA
repeated simulation data available	VFA

Formal connection

All methods approximate:

\[ \arg\min_{x\in X(S_k)} \left[R(S_k,x)+V(S_k^x)\right] \]

but in different ways.

Worked example: selecting method

Problem:

Same-day delivery with many vehicles and uncertain future orders.

Recommended combined approach:

Use RHO for current routing feasibility.
Add CFA penalty for using scarce central vehicles.
Use sampled look-ahead during peak periods.
Train VFA offline if simulation is available.

Subtopic: approximation error¶

Intuition

ADP methods are useful because they simplify. Simplification creates error.

Formal idea

True value:

\[ V(S_k^x) \]

Approximate value:

\[ \hat{V}(S_k^x) \]

Approximation error:

\[ \text{error}=\hat V(S_k^x)-V(S_k^x) \]

Properties

Error can come from poor features, too little data, or biased simulation.
Approximation error may change the chosen decision.
Validation is needed to detect harmful errors.

13.7 Lecture 7 Deep Dive: Sampling Methods I¶

Subtopic: confidence through repeated simulation¶

Intuition

One simulation run is only one possible future. Many runs give a more reliable estimate.

Formal estimate

For policy $\pi$:

\[ \hat J(\pi)=\frac{1}{N}\sum_{n=1}^{N}J^n(\pi) \]

where $J^n(\pi)$ is total cost in sample path $n$.

Properties

More sample paths improve stability.
Rare events require many samples or targeted rare-event analysis.
Report averages and variability when possible.

Worked example

Policy costs over five simulation days:

100, 120, 110, 90, 130

Average:

\[ \frac{100+120+110+90+130}{5}=110 \]

Range:

90 to 130

Business interpretation:

The expected cost is around 110, but performance varies substantially.

Subtopic: sample path¶

Intuition

A sample path is one simulated timeline of the future.

Formal definition

sample path n = (omega_1^n, omega_2^n, ..., omega_K^n)

Worked example

For delivery:

sample path = all order arrivals and travel-time realizations for one simulated day

13.8 Lecture 8 Deep Dive: Sampling Methods II¶

Subtopic: deterministic vs stochastic look-ahead¶

Intuition

Deterministic look-ahead uses one forecasted future. Stochastic look-ahead uses multiple possible futures.

Formal comparison

Deterministic:

solve using forecast omega_bar

Stochastic:

solve using samples omega^1,...,omega^N

Properties

Method	Merit	Disadvantage
deterministic	fast, simple	ignores variability
stochastic	anticipates uncertainty	more computationally expensive

Worked example

Forecast says 10 orders will arrive. Stochastic scenarios say:

5, 8, 10, 14, 18 orders

If capacity is tight, stochastic look-ahead may keep more reserve capacity than deterministic look-ahead.

Subtopic: first-decision consistency¶

Intuition

In scenario methods, future decisions may differ by scenario because each scenario imagines a different future. But the current decision must be the same in reality because only one current action can be taken now.

Formal idea

x_0 must be chosen before knowing which scenario occurs

Future decisions:

x_1^n may differ by scenario n

Properties

The first decision is binding.
Later scenario decisions are planning aids.
Real future decisions will be reoptimized after information arrives.

13.9 Lecture 9 Deep Dive: Supervised Learning¶

Subtopic: features¶

Intuition

Features are the inputs used for prediction.

Examples:

time of day
weather
zone
day of week
route distance
number of open orders

Formal definition

z = feature vector
f(z) = prediction

Properties

Good features capture important patterns.
Too few features underfit.
Too many weak features can overfit.
Features should be available at decision time.

Worked example

Travel-time prediction:

z = (distance, time of day, rain indicator, traffic level)
f(z) = predicted travel time

Subtopic: train/test split¶

Intuition

A model should be tested on data it did not train on.

Formal split

training data -> fit f
test data -> evaluate f

Properties

Training error can be misleadingly low.
Test error estimates future performance.
In time-dependent settings, test data should often come after training data chronologically.

Worked example

Use January-March delivery data to train a travel-time model. Test it on April data. If April error is high, the model may not generalize.

13.10 Lecture 10 Deep Dive: Reinforcement Learning¶

Subtopic: reward design¶

Intuition

RL and DDM methods learn what you reward. If the reward is badly designed, the policy can learn the wrong behavior.

Formal reward

\[ R(S_k,x)=\text{revenue}-\text{operating cost}-\text{delay penalty}-\text{rejection penalty} \]

Properties

Reward must match business goals.
Missing penalties can create bad behavior.
Multi-objective tradeoffs need weights.

Worked example

If reward includes revenue but no lateness penalty, the policy may accept too many orders and deliver them late.

Subtopic: simulation as a training environment¶

Intuition

Exploring in the real world can be expensive. Simulation allows policies to learn safely.

Formal learning loop

Simulate $S_k$.
Choose $x$.
Observe $R(S_k,x)$ and $S_{k+1}$.
Update the value or policy estimate.
Repeat.

Properties

Safe for experimentation.
Depends on simulator realism.
Useful for tuning PFA, CFA, and VFA parameters.

13.11 Lecture 11 Deep Dive: Combined Methods¶

Subtopic: architecture of a decision-support system¶

Intuition

A real DDM system has layers:

data layer,
prediction layer,
optimization/decision layer,
simulation/evaluation layer,
monitoring layer.

Formal flow

\[ \text{data}\rightarrow S_k\rightarrow \pi(S_k)\rightarrow x \rightarrow S_k^x\rightarrow \omega_{k+1}\rightarrow \text{evaluation} \]

Properties

Each layer can fail.
Monitoring detects model drift.
Evaluation should compare policies under the same sample paths when possible.

Worked example

An e-grocery platform:

predict demand by zone
reserve delivery slots
route vehicles
simulate future orders
update capacity values
monitor lateness and rejections

Subtopic: comparing policies fairly¶

Intuition

To compare policies, test them under the same scenarios. Otherwise differences may come from luck.

Formal paired comparison

For scenario $n$, compare:

difference_n = cost_A^n - cost_B^n

Average difference:

\[ \text{average difference} =\frac{1}{N}\sum_n \text{difference}_n \]

Worked example

Scenario	Cost A	Cost B	Difference A-B
1	100	90	10
2	120	130	-10
3	110	95	15

Average difference:

\[ \frac{10-10+15}{3}=5 \]

Interpretation:

Policy B is 5 cost units better on average because A-B is positive.

13.12 Lecture 12 Deep Dive: Exam Strategy¶

Subtopic: how to answer "compare methods"¶

Use a three-part structure:

1. Define each method.
2. Say what each method approximates.
3. Give one merit and one drawback.

Example answer:

RHO repeatedly solves the current optimization problem and is practical when existing static models are available, but it may be myopic. CFA modifies the cost function or constraints to preserve future flexibility, but its penalties or buffers must be tuned. VFA approximates the future value $V(S_k^x)$, making decisions more anticipatory, but it requires data, features, and validation.

Subtopic: how to answer calculation questions¶

Use this scratch-work pattern:

Decision:
Immediate:
Future:
Total:
Chosen:

Worked format:

Decision	Immediate	Future	Total
A	5	20	25
B	12	8	20

For minimization:

choose B

Subtopic: how to avoid notation mistakes¶

Write symbols with words:

Notation	Meaning
$S_k$	current state before decision
$x\in X(S_k)$	feasible decision
$S_k^x$	state after decision
$\omega_{k+1}$	new random information

This prevents the examiner from thinking you memorized symbols without understanding them.

13.13 Course Add-On: How To Read Slide Examples Like Exam Models¶

The DDM slides often show pictures, cases, and operational stories before giving notation. Translate each story into the same modeling checklist:

Story phrase	DDM translation
"we observe traffic speeds"	information model
"orders arrive during the day"	stochastic information $\omega_{k+1}$
"the dispatcher decides"	decision $x\in X(S_k)$
"drivers become unavailable"	post-decision state $S_k^x$
"new requests appear"	transition to $S_{k+1}$
"minimize total delay"	objective

Example: floating-car data

The slide case begins with observed vehicle positions. Those observations do not directly solve routing. They must be converted into an information model:

map positions to street segments,
estimate speeds,
aggregate by time and location,
validate spatial and temporal patterns,
convert to travel-time matrices,
feed the matrices into a routing decision model.

The managerial lesson is that predictive analytics and prescriptive analytics are connected but distinct. Better forecasts only create value if they improve the decisions made by the decision model.

13.14 Course Add-On: Why Post-Decision States Are So Useful¶

The post-decision state separates two things students often mix together:

what the decision maker controls
what the world reveals afterward

If a driver is assigned to an order, that is controlled. If a new order appears five minutes later, that is random information. The post-decision state records the controlled update before the randomness arrives.

Formal sequence:

\[ S_k \xrightarrow{x} S_k^x \xrightarrow{\omega_{k+1}} S_{k+1}. \]

In words:

\[ \text{state now}\rightarrow\text{decision effect}\rightarrow \text{random future information}\rightarrow\text{next state} \]

This makes Bellman calculations cleaner because the future value can be attached to $S_k^x$.

13.15 Course Add-On: How ADP Methods Fail¶

Good exam answers do not only praise methods. They explain failure modes.

Method	Typical failure mode	Business symptom
PFA	threshold is too simple	rules work on normal days but fail in unusual states
RHO	ignores future value	uses scarce resources too early
CFA	penalty is badly tuned	system becomes too conservative or too aggressive
Look-ahead	poor scenarios or short horizon	future planning looks precise but misses the real risk
VFA	weak features or sparse data	learned values are unstable or misleading
Combined method	too many moving parts	hard to validate which component helps

The safest exam structure is:

Method -> what it approximates -> why useful -> main limitation -> example.

Notation	Meaning
\(S_k\)	current state before decision
\(x\in X(S_k)\)	feasible decision
\(S_k^x\)	state after decision
\(\omega_{k+1}\)	new random information