Q-Learning Gridworld
Q-learning is the classic reinforcement-learning algorithm: an agent wanders a gridworld, collects rewards and learns a table Q(s, a) estimating the long-term value of each action in each cell. Paint your own walls, goals and traps — or load the classic maze, cliff-walk and four-rooms layouts — and press Play: episodes stream by while the value heatmap warms from blue to orange, the greedy-policy arrows sort themselves out and the reward-per-episode chart climbs. Tune the learning rate α, the discount γ, the exploration rate ε (with optional decay) and the per-step penalty, and toggle the raw Q numbers and visit counts to see exactly what the agent believes. The cliff-walk layout shows why an exploring agent keeps its distance from the edge.
Runs 100% in your browser — models are trained and computed locally on your device.
Read the full guide to this tool
Notes
- The update rule is Q(s,a) ← Q(s,a) + α[r + γ·max Q(s′,·) − Q(s,a)]; because the target maximises over the next actions regardless of the action actually taken, Q-learning is an off-policy method.
- ε-greedy exploration takes a random action with probability ε; without exploration the agent can lock onto the first rewarding path it finds and never discover a better one.
- The discount γ < 1 makes distant rewards worth less, so learned values fade with distance from the goal — that gradient is exactly what the heatmap shows.
- On the cliff walk Q-learning learns the shortest path along the edge while its on-policy cousin SARSA learns the safer detour — a textbook illustration of the off-policy/on-policy difference.
- Runs 100% in your browser — models are trained and computed locally on your device.
Related tools
AI Learns to Drive (Neuroevolution)
Watch a population of neural-network cars evolve from crashing into corners to lapping a race track.
CartPole Balancing AI
Balance the pole yourself, then watch the cross-entropy method evolve a policy that never drops it.
Multi-Armed Bandit Lab
Race ε-greedy, UCB1 and Thompson sampling on the same bandit and compare their regret.
Backpropagation Step-by-Step
Walk through one forward and backward pass of a tiny neural network, number by number.
Decision Tree Builder
Grow a CART decision tree split by split and watch the partition map and tree diagram build together.
Diffusion Model Toy
Train a tiny denoising diffusion model and watch it turn pure noise back into a 2D shape.