Machine vision offers opportunities to analyze large numbers of cancer images, to discover novel histological features in pathology images that may have biological significance, and to use those characters for automating classification and prognosis. However, machine vision in pathology has been limited to narrow domains and has yet to offer a real alternative to human assessment. Deep Neural Networks (DNNs) are a class of algorithms that have demonstrated great potential over the past few years, approaching or even exceeding human performance in various visual recognition tasks. But DNNs have yet to impact pathology, in large part because of limitations in the size and variety of current pathology datasets. We are developing novel deep learning architectures to both localize malignancy in images and to relate morphological features to patient outcomes, leading to machine vision systems that can provide accurate diagnosis and prognosis of cancer in tissue slides.