Pictures of the Future      Spring 2008

Computers Get the Picture

How can a computer learn to interpret images, catalogue them, find them in databases, and recognize similarities? That’s what researchers want to find out in the Medico project—a part of the German Theseus program.

It has been said that life should be thought of in shades of gray rather than just black and white. And yet, computers do the latter. They‘re simple-minded and "think" only in terms of zeros and ones, black and white. The reason is, simply put, that they were designed to process numbers; not texts or images. But this is precisely what researchers want to teach them to do. In the not-too-distant future, computers will be expected to distinguish between nuances and interpret the content of images.

That is the driving force behind the development of the Semantic Web, which is also referred to as Web 3.0. Web 3.0 is the core of Theseus, a German Federal Ministry of Education and Research project led by Siemens that was initiated in 2007. "One of our greatest challenges is to develop computers that can automatically recognize image content and relationships," explains Prof. Hartmut Raffler, head of Information and Communications Technology and coordinator of all Theseus activities at Siemens Corporate Technology (CT).

An important part of Theseus is Medico, a program that is also led by Siemens. Here, experts from Corporate Technology and Siemens’ Industry Sector Divisions are collaborating on the development of Medico’s software core. Other Medico partners include the German Research Center for Artificial Intelligence in Kaiserslautern, the Fraunhofer Institute for Computer Graphics in Darmstadt, and Ludwig Maximilians University in Munich.

In the Medico scenario, medical knowledge will be linked for the first time with new image processing methods, knowledge-based data processing, and machine learning. Alok Gupta, PhD, vice president of the CAD and Knowledge Solutions Group at Siemens Medical Solutions in Malvern, Pennsylvania, and Dr. Jörg Freund of Sector Healthcare in Erlangen, Germany are coordinating the quest to create a universally usable Medico search engine for medical images.

Image Associations. Medico is expected to recognize anatomical structures such as bones, vessels and organs, as well as associated pathological changes, automatically catalogue the data, and collect comparison images and treatment reports from multiple databases.

According to Dr Freund, "The idea is to provide any physician—for example, one who’s studying a new x-ray image—with all the available information ever generated about the object being examined, including clinical summaries, symptoms, diagnoses, and therapies." Medico’s first test run is already planned for late 2009 at the Erlangen University Medical Center.

Intelligent image search is still in its infancy, and experts have no illusions about that. Current databases such as Web-based picture archiving and communication systems (PACS) and radiology information systems (RIS), are still based on a keyword index system in which associations are selected by people rather than being generated automatically by image interpretation systems. Experts are initially focusing on imaging methods such as computed tomography, ­magnetic resonance imaging and ultrasound in order to close the "semantic gap" in a clearly demarcated field of study. Linguists define ­semantics as the study of the meaning of words, but here the much broader definition of semantics as the study of meaning is being ­applied to the comprehension of image contents by a computer program.

"Our key objective in Theseus is to address the inchoate mass of data by developing a generally applicable way of expressing contents that ensures order and hierarchy," explains Dr. Volker Tresp, who is the central contact person at CT for Theseus’ six application scenarios. His aim is to enable computers to assign information to the relevant image, find such information, and interpret it—regardless of whether it’s text, video, or multimedia data. To create this intelligent image search engine, many ­different components must be developed, from new methods of pattern recognition and ontology modeling (which makes background knowledge comprehensible to a computer) to computer-aided recognition systems and clinical decision support systems.

Understanding the Doctor. To describe image contents Tresp and other experts use ontologies developed by physicians, such as RadLex  and the Foundational Model of Anatomy ontology. "Using these tools, we describe image content in a hierarchy based on human anatomy," explains Freund. "This results in quick answers to questions such as whether an organ or growth has changed in size since the last exam."

"With Medico," Freund adds, "the initial focus is on semantic search in medical databases, but the range of potential applications is much wider." He points out, for instance, that once associated ethical issues and data security regulations are resolved, the pharmaceutical industry, for example, may benefit from this technology by automating the analysis of clinical studies, including their image content, to more rapidly and accurately draw conclusions regarding the effects of medical treatments.
                                                                                                                Klaudia Kunze