Understanding vision, whether from a neurobiological, psychological or philosophical perspective, represents a daunting challenge that has been pursued for millennia. During at least the last few centuries, natural philosophers, and more recently vision scientists, have recognized that a fundamental problem in biological vision is that the physical sources underlying sensory stimuli are unknowable in any direct sense. In vision, because physical qualities are conflated when the 3-D world is projected onto the 2-D image plane of the retina, the provenance of light reaching the eye at any moment is inevitably uncertain. This quandary is referred to as the inverse optics problem. The relationship of the real world and the information conveyed to the brain by light present a profound problem. Successful behavior in a complex and potentially hostile environment clearly depends on responding appropriately to the sources of visual stimuli rather than to the physical characteristics of the stimuli as such. If the retinal images generated by light cannot specify the underlying reality an observer must deal with, how then does the visual system produce behavior that is generally successful? Perceiving Geometry considers the evidence that, with respect to the perception of geometry, the human visual system solves this problem by incorporating past human experience of what retinal images have typically corresponded to in the real world.