I will discuss the prospects for constructing topologically invariant state sum models of gauge theory or gravity coupled to matter. The first model to consider is a one-dimensional model with fermions. Then I will explain what is known about three-dimensional models.