Librería Portfolio

Clear, rigorous, and intuitive, Markov Processes provides a bridge from an undergraduate probability course to a course in stochastic processes and also as a reference for those that want to see detailed proofs of the theorems of Markov processes. It contains copious computational examples that motivate and illustrate the theorems. The text is designed to be understandable to students who have taken an undergraduate probability course without needing an instructor to fill in any gaps.

The book begins with a review of basic probability, then covers the case of finite state, discrete time Markov processes. Building on this, the text deals with the discrete time, infinite state case and provides background for continuous Markov processes with exponential random variables and Poisson processes. It presents continuous Markov processes which include the basic material of Kolmogorov's equations, infinitesimal generators, and explosions. The book concludes with coverage of both discrete and continuous reversible Markov chains.

While Markov processes are touched on in probability courses, this book offers the opportunity to concentrate on the topic when additional study is required. It discusses how Markov processes are applied in a number of fields, including economics, physics, and mathematical biology. The book fills the gap between a calculus based probability course, normally taken as an upper level undergraduate course, and a course in stochastic processes, which is typically a graduate course.



Review of Probability

Short History

Review of Basic Probability Definitions

Some Common Probability Distributions

Properties of a Probability Distribution

Properties of the Expected Value

Expected Value of a Random Variable with Common Distributions

Generating Functions

Moment Generating Functions

Exercises

Discrete-Time, Finite-State Markov Chains

Introduction

Notation

Transition Matrices

Directed Graphs: Examples of Markov Chains

Random Walk with Reflecting Boundaries

Gamblerâ?Ts Ruin

Ehrenfest Model

Central Problem of Markov Chains

Condition to Ensure a Unique Equilibrium State

Finding the Equilibrium State

Transient and Recurrent States

Indicator Functions

Perron-Frobenius Theorem

Absorbing Markov Chains

Mean First Passage Time

Mean Recurrence Time and the Equilibrium State

Fundamental Matrix for Regular Markov Chains

Dividing a Markov Chain into Equivalence Classes

Periodic Markov Chains

Reducible Markov Chains

Summary

Exercises

Discrete-Time, Infinite-State Markov Chains

Renewal Processes

Delayed Renewal Processes

Equilibrium State for Countable Markov Chains

Physical Interpretation of the Equilibrium State

Null Recurrent versus Positive Recurrent States

Difference Equations

Branching Processes

Random Walk in

Exercises

Exponential Distribution and Poisson Process

Continuous Random Variables

Cumulative Distribution Function (Continuous Case)

Exponential Distribution

o(h) Functions

Exponential Distribution as a Model for Arrivals

Memoryless Random Variables

Poisson Process

Poisson Processes with Occurrences of Two Types

Exercises

Continuous-Time Markov Chains

Introduction

Generators of Continuous Markov Chains: The Kolmogorov Forward and Backward Equations

Connection Between the Steady State of a Continuous Markov Chain and the Steady State of the Embedded Matrix

Explosions

Birth and Birth-Death Processes

Birth and Death Processes

Queuing Models

Detailed Balance Equations

Exercises

Reversible Markov Chains

Random Walks on Weighted Graphs

Discrete-Time Birth-Death Process as a Reversible Markov Chain

Continuous-Time Reversible Markov Chains

Exercises

Bibliography