Comprehensive software reviews to make better IT decisions

Google’s AI System Spots Breast Cancer More Accurately Than Radiologists

The artificial intelligence (AI) system developed by Google Health more accurately identifies breast cancer than human experts, reports The Guardian (and others). The system has been tested in the UK and the US, and the results are published in Nature, one of the most prestigious scientific journals in the world.

The system produced 5.7% fewer false positives and 9.4% fewer false negatives in the US data. (False positives are cases where the system mistakenly labelled the image as cancerous while the patient in fact did not have cancer. False negatives are the opposite: these are the cases where the system missed the tumors.)

The UK results were more modest: there were 1.2% fewer false positives and 2.7% fewer false negatives compared to human experts.

The difference in performance seems to do with how breast cancer screening works: in the US, women are seen every one to two years and the mammograms are examined by one radiologist. In the UK, the screening period is every three years, and two radiologists review the results. (A third expert usually gets involved in cases of disagreement.)

It is also worth keeping in mind when considering the results that the AI system received less information than the human experts: it had no access to patient histories, for example, or prior mammograms, notes Google Blog on this topic.

The system was developed over two years in collaboration with DeepMind (a Google subsidiary), Cancer Research UK Imperial Centre, Northwestern University, and Royal Surrey County Hospital.

Our Take

While the results are quite promising, two things are important:

First, the performance of the AI system slips when tested on images from different mammogram systems. The study reported in Nature used images from machines provided by a single manufacturer. This is known as data capture bias, and it is a common problem in image recognition systems.

Google and the University of Toronto researchers write in an unrelated paper that “the performance of a model can vary depending on what instruments were used to capture the input to the model. For example, a face detection model may perform differently depending on the camera’s hardware and software, including lens, image stabilization, high dynamic range techniques, and background blurring for portrait mode. Performance may also vary across real or simulated traditional camera settings such as aperture, shutter speed and ISO. Similarly, video and audio input will be dependent on the choice of recording instruments and their settings.” And since mammograms are a type of image, factors impacting the performance of an image recognition system also apply to those interpreting mammograms.

The second point that many people make, including the creators of the Google AI system, is that the breast cancer recognition system is not meant to replace radiologists. It will rather complement them, as a “second pair of eyes and a safety net,” says Dr. Caroline Rubin, Vice President for Clinical Radiology at the Royal College of Radiologists.

And some countries simply don’t have enough radiologists – in the UK, for example, 8% of hospital posts for breast radiology are unfilled, with many older radiologists retiring and not enough new ones joining in, perhaps scared away by the AI advances such as the one reported here.

Want to Know More?

Looking to learn where else AI and ML are used in healthcare? See our blueprint Get Started With AI.

To learn about data capture bias and other machine/algorithmic biases, including data and design biases, consult our blueprint Mitigate Machine Bias.