Pictures of the Future      Spring 2008

Harvest without End

An army of algorithms is being developed. Built on expert knowledge and capable of learning from experience, these systems are pointing out anomalies in radiology exams, providing decision support in a range of fields, and optimizing split-second decision making in high-speed industrial processes. As such systems harvest knowledge, there will be no limit to what we will learn from them.

Patterns previously invisible to machines and humans are today providing insights that make medical treatments increasingly personalized and effective, production more customized and efficient, and intelligence—whether in a security camera or a picture archiving system—more distributed and flexible. Across the board—from health care (Health Care) to energy management (Networks), and from finance (Financial Sector) to security and sales (Safety and Security)—information is being mined from machines, processes and experts, and crystallized into machine knowledge used by algorithms. These algorithms, which range from systems that can interrogate cardiac data for anomalies to the analysis of sales information to predict a customer’s probability of consummating an order, are becoming our invisible assistants.

For SMS, the spot where this avalanche of data converges is a comprehensive knowledge platform for medical decision support called the Remind (Reliable Extraction and Meaningful Inference from Nonstructured Data) platform. The ultimate invisible assistant, "Remind will make it possible to dynamically integrate medical images, in-vitro diagnostic information, and genetic information into a patient’s profile, providing personalized decision support based on analysis of data from large numbers of patients with similar conditions," explains Bharat Rao, PhD, senior director of Knowledge Solutions for Healthcare Providers at Siemens Medical Solutions in Malvern and inventor of the Remind platform.

Remind adds up to a diagnostic crossroads for Siemens’ imaging-related businesses and its more recently acquired in-vitro businesses, now known as Siemens Diagnostics (for more, see Pictures of the Future, Spring 2007, In Vitro Diagnostics). "The vision is to integrate the information from imaging and laboratory tests into a single database, and eventually a single patient record," says Gupta.

On the long road to realizing the Remind vision, Siemens is developing an army of invisible assistants designed to support physicians as "second readers."

The idea is that once a specialist has examined a scan, he or she can run the appropriate assistant to increase the probability that nothing has been missed. Known as knowledge-driven products, these assistants (which plug into Siemens’ syngo user interface) offer computer aided detection of lung nodules, colon polyps, breast lesions, and much more (Health Care).

Other assistants support physicians in accelerating the process of accurately quantifying functions such as cardiac ejection fraction and vessel flow abnormalities, and in providing comparative analysis of images produced at different times and from different imaging modalities.

Among the many assistants heading for commercialization is one that extracts a 4D model (3D over time) of the aortic valve from ultrasound data "that will allow physicians to interrogate it regarding a variety of real-time, quantitative functions," says Helene Houle, a senior sonographer with Siemens Ultrasound in Mountain View, California, who worked closely with Siemens Corporate Research in Princeton on its development. Another assistant now under joint development will create a 3D interactive model of the heart from computer tomography (CT) data. The model, now in prototype, will display the outlines of the beating heart and provide information regarding anomalies in the volume of blood pumped by the atria.

But such assistants are just the beginning. "We are looking at what it would mean to add genetic information to the imaging data in these products," says Gupta. With this in mind, Siemens is working with an expanding group of medical centers in the context of the EU-funded Health-e-Child program (see Pictures of the Future, Spring 2007, Health-e-Child). The program, which is coordinated by SCR and the CAD group, is developing an integrated health-care platform for pediatric information designed to provide seamless integration of traditional sources of biomedical information, as well as emerging sources, such as genetic and proteomic data.

Voice Command. As medical assistants multiply and their underlying databases expand, new systems of addressing this cornucopia of information will be needed. One solution that is approaching market introduction in 2008 is Automatic Localization and Parsing of Human Anatomy (ALPHA). Trained on a huge anatomical database and capable of learning with each exam, ALPHA recognizes landmarks throughout the body, thus opening the door to voice-based interaction. "Questions such as ‘show me the lower left lobe of the patient’s lung and compare it with the previous two exams,’ will become routine," says Arun Krishnan, PhD, head of CAD research and development at SMS in Malvern. "This will accelerate throughput, because it will no longer be necessary to search through image sets to find a desired anatomical slice. The target will appear automatically in response to a voice command," he says. Compatible with hospital picture archiving and communication systems, ALPHA will provide a quantum leap in terms of the rapid accessibility of CT, MR, PET, and other imaging modalities and their content.

Understanding the complex meanings and information locked in images is a topic that is also being examined by Theseus, a German Federal Ministry of Education and Research project led by Siemens. "A big part of the Theseus vision is to automatically recognize image data in order to transform it from an unstructured to a structured state, so that it can be used in the semantic Web for retrieval," say Dr. Hartmut Raffler, coordinator of Theseus and head of the Information and Communications Division of Siemens Corporate Technology (CT). Adds Dr. Volker Tresp, who is responsible for day-to-day management of Theseus and is a specialist in data mining, machine learning and decision support at CT, "This is a vast area because it opens up the entire field of picture, video, multimedia and content archives for deep exploration as they relate to security, robotics, entertainment, environmental sciences, and much more."

Specifically, a research area within Theseus known as "Medico" is building an intelligent, scalable picture archiving and search system (that could be supported by ALPHA) capable of retrieving images by content. Suppose, for instance, that a cardiologist is examining MR ­images of a patient with a pulmonary valve ­deficiency. "To help determine whether the ­deficiency warrants surgery, he might ask Theseus to show him images of pulmonary valves that look similar to the one he is looking at in terms of morphology and function before and after surgery," says Dr. Dorin Comaniciu, head of the Integrated Data Systems department at Siemens Corporate Research and one of the ­initiators of Theseus Medico.

Communicative Cameras. But the areas of application for this kind of search engine extend well beyond medical uses. Says Ramesh Visvanathan, PhD, head of the Real-time Vision and Modeling Department at SCR, "In the context of the Theseus project, our Vision Center of Competence in Munich is defining metadata languages for the automatic identification of video content. In terms of security applications, for instance, this will mean that cameras will be able to track a target of interest by describing it in a standardized language and passing the information from one camera to another." The technology would thus make it possible to follow an intruder as he or she leaves one camera’s field of view and enters the area monitored by another camera.

And what about the quality of the images that intelligent systems select? Regardless of whether an image originates in a surveillance camera or a medical database, the highest quality must be guaranteed if the evaluation of its content is to be reliable. Image retrieval systems therefore need  a way of ensuring selection of the best available images. Work now in the pipeline at Beijing’s Tsinghua University that is sponsored in part by Siemens may provide an answer. "The idea is to develop an assistant that will select the best and most relevant images for doctors from large data sets," says Comaniciu. Trained by using the criteria doctors themselves use for selecting images, the assistant may even be able to enhance images that are less than perfect.

Algorithms and Automation. Just as intelligent assistants are rapidly reproducing in the health-care universe, they are also beginning to populate other areas—particularly in industry. In steel production, for instance, the trend toward total automation is leading to increasing use of decentralized intelligence.

"Depending on the grade of steel, the manufacturing components involved may have individual strategies for monitoring and managing each step while taking a collective view of the process," says Dr. Michael Metzger, a specialist in steel industry solutions at Corporate Technology in Munich. He explains that this boils down to the use of "algorithms stationed near associated actuators working together to solve a control problem within a community of machines." Such systems must, furthermore, be able to learn at lightning speed. "In order to accomplish this," says Metzger, "these systems are based on control and optimization process models that are themselves based on relationships derived from physics and expert knowledge. But they must also be able to learn from the huge amount of data produced by an ­automation system, thus enabling the control system to respond optimally in real time to ­variables such as rolling force and temperature," he explains.

As in health care, a process of customization is in full swing here. This begins with expert knowledge and data mining, which discover key parameters, such as deformation history and cooling rate for a given grade of steel. Then, to optimize results for a particular order, the entire production process is simulated—including neural networks and learning algorithms. Once optimized in the virtual world, the information is transferred to the rolling mill and put to the test. Values for each process step are taken and compared against the simulated (and thus optimized) values. "As a result," says Metzger, "the models learn how to improve themselves based on this comparison. Ultimately," he adds, "such systems will provide decision support and finally decision automation."

Digital Repairmen. Not only do learning systems keep track of what works best under a complex variety of circumstances. They also keep an eye on the long-term factors that cause machine wear and tear, and predict when service should be performed with a view to minimizing downtime. With this in mind, in 2007 Siemens established a strategic program called the Machine Monitoring Initiative. "The project will tap basic research throughout the organization in data mining, learning systems and decision support," says Claus Neubauer, a data integration specialist at SCR. The results will be used to automate the prediction and scheduling of maintenance on everything from power, rail and communication networks to MR scanners and windmill gearboxes.

Predicting when machines will need maintenance and which parts will need to be replaced may sound like a tall order, but what about predicting whether a customer will actually purchase a wind park or an MR scanner? Surprisingly, agents are already zeroing in on this kind of information as well. Research ­conducted at SCR has come up with an agent technology that is "70 to 80 % accurate," says Amit Chakraborty, who leads the Modeling and Optimization program at SCR. "In ­providing this decision support, the agent takes many factors into account, including customer reliability, competitors, and sales force information," he adds.

Intelligence Everywhere. Naturally, given the fact that they are weightless, not particularly expensive to produce, and capable of incrementally increasing the productivity of hardware, invisible agents will eventually pop up just about everywhere. The trend toward decentralized intelligence in highly automated production facilities will have its counterparts in traffic and rail management, building and home automation, safety and security technology, power generation and distribution, and of course health care. The implications of these invisible entities for entertainment, information accessibility, security, environmental protection, and the way humans communicate, organize, work and live could be profound.

"We should keep in mind that this is all about solutions that support human activities," says CT’s Raffler. "Based on this, agents will understand what we are looking for, present results more intelligently than is now possible, answer questions, deal with large bodies of unstructured data, compose services, and propose new processes for solving problems."

Information, something we produce more of with every passing second, will become increasingly valuable as we learn to mine it, combine its streams, and refine its messages. What lies ahead, in short, is a harvest without end.
                                                                                                              Arthur F. Pease