Imagine that an elderly patient complaining of chest pain sees a highly experienced physician. Slowly, the clinician asks the appropriate questions and gets a representation of the patient as determined by the features of that patient's signs and symptoms. The patient says they have a history of high blood pressure but no other cardiac risk factors. The chest pain varies in intensity with the heartbeat (also known as pleuritic chest pain). The patient also reports they just came back to the United States from Europe. They also complain of swelling in the calf muscle. Slowly, the physician combines these lower level pieces of information (the absence of cardiac risk factors, the pleuritic chest pain, the prolonged period of immobility, a positive Homan's sign) and integrates it with memories of previous patients and the physician's own extensive knowledge to build a higher level view of this patient and realizes that the patient is having a pulmonary embolism. The physician orders a V/Q scan and proceeds to save the patient's life.

Such stories happen every day across the globe in medical clinics, hospitals, and emergency departments. Physicians use information from the patient history, exam, and test results to compose higher level understandings of their patients. How do they do it? The answer may lie in neural networks and deep learning.