Dynamic Decision Making Mini-Book¶

For business students with high-school mathematics.

Primary course sources:

DDM Slides.pdf
DDM Tutorials.pdf

Textbook enrichment sources:

Mykel J. Kochenderfer, Tim A. Wheeler, and Kyle H. Wray, Algorithms for Decision Making
Warren B. Powell, Reinforcement Learning and Stochastic Optimization: A Unified Framework for Sequential Decisions

How the sources are used:

The DDM slides and tutorials are the exam authority. Their notation wins when sources differ.
Kochenderfer, Wheeler, and Wray support probability, expected utility, MDPs, approximate values, receding horizon planning, rollout, sampling, Monte Carlo tree search, and validation.
Powell supports the universal modeling framework for sequential decisions, policy classes, cost function approximations, lookup tables, aggregation, stochastic search, and the blend of learning and optimization.
Textbook ideas are paraphrased and simplified. This note does not copy textbook prose.

How To Read This Mini-Book¶

This mini-book is designed for someone who can do high-school algebra but does not feel comfortable with mathematical notation yet. Each topic follows the same rhythm:

Intuition: what the idea means in plain business language.
Formal notation: the smallest useful mathematical version.
Properties: what must be true, what the method assumes, and what it is good or bad at.
Worked example: a small example you can calculate by hand.
Exam framing: sentences and traps that help in the written exam.

The main course notation is:

Object	Notation	Business meaning
Decision point	\(k=0,\ldots,K\)	A time or event where a decision is made
Decision state	\(S_k\in\mathcal{S}\)	All information available before the decision
Feasible decisions	\(x\in X(S_k)\)	Choices allowed in the current state
Reward/cost	\(R(S_k,x)\)	Immediate performance of choosing \(x\)
Post-decision state	\(S_k^x\)	State after the decision, before new randomness
New information	\(\omega_{k+1}\)	Random information revealed after the decision
Transition probability	\(P:\mathcal{S}\times X\times\mathcal{S}\rightarrow[0,1]\)	Probability of moving from one state to another
Value function	\(V(S_k^x)\)	Expected future reward/cost from a post-decision state
Approximate value	\(\hat{V}(S_k^x)\)	Estimated future value when exact value is infeasible
Policy	\(\pi\)	A rule that maps states to decisions

The core loop:

\[ S_k \rightarrow x\in X(S_k) \rightarrow R(S_k,x) \rightarrow S_k^x \rightarrow \omega_{k+1} \rightarrow S_{k+1} \]

flowchart LR
    A["S_k<br/>decision state"] --> B["x in X(S_k)<br/>decision"]
    B --> C["R(S_k,x)<br/>immediate reward/cost"]
    B --> D["S_k^x<br/>post-decision state"]
    D --> E["omega_{k+1}<br/>new information"]
    E --> F["S_{k+1}<br/>next state"]
    F --> A

Table Of Contents¶

Start Reading¶

Use the navigation sidebar to read the book by lecture. The source note remains in each chapter, and the original full note is preserved at DDM/DDM Exam Notes v2.md.

Best reading path

Start with Overview, then read Information Modeling and Decision Making, Stochasticity, and Dynamism before moving into MDPs and ADP methods.