Source: Penn State
Date: November 1, 2006
Post to:

Researchers Teach Computers How To Name Images By 'Thinking'

Penn State researchers have "taught" computers how to interpret images using a vocabulary of up to 330 English words, so that a computer can describe a photograph of two polo players, for instance, as "sport," "people," "horse," "polo."

The new system, which can automatically annotate entire online collections of photographs as they are uploaded, means significant time-savings for the millions of Internet users who now manually tag or identify their images. It also facilitates retrieval of images through the use of search terms, said James Wang, associate professor in the Penn State College of Information Sciences and Technology, and one of the technology's two inventors.

The system is described in a paper, "Real-Time Computerized Annotation of Pictures," given at the recent ACM Multimedia 2006 conference in Santa Barbara, Calif., and authored by Jia Li, associate professor, Department of Statistics, and Wang. Penn State has filed a provisional patent application on the invention. Major search engines currently rely upon uploaded tags of text to describe images. While many collections are annotated, many are not. The result: Images without text tags are not accessible to Web searchers. Because it provides text tags, the ALIPR system-Automatic Linguistic Indexing of Pictures-Real Time-makes those images visible to Web users.

ALIPR does this by analyzing the pixel content of images and comparing that against a stored knowledge base of the pixel content of tens of thousands of image examples. The computer then suggests a list of 15 possible annotations or words for the image.

"By inputting tens of thousands of images, we have trained computers to recognize certain objects and concepts and automatically annotate those new or unseen images," Wang said. "More than half the time, the computer's first tag out of the top 15 tags is correct."

In addition, for 98 percent of images tested, the system has provided at least one correct annotation in the top 15 selected words. The system, which completes the annotation in about 1.4 seconds, also can be applied to other domains such as art collections, satellite imaging and pathology slides, Wang said. The new system builds on the authors' previous invention, ALIP, which also analyzes image content. But unlike ALIP which characterized images by incorporating computational-intensive spatial modeling, ALIPR characterizes images by modeling distributions of color and texture.

The researchers acknowledge computers trained with their algorithms have difficulties when photos are fuzzy or have low contrast or resolution; when objects are shown only partially; and when the angle used by the photographer presents an image in a way that is different than how the computer was trained on the object. Adding more training images as well as improving the training process may reduce these limitations-future areas of research.

A demonstration of the ALIPR system can be found at

In a companion paper also presented at the ACM conference, the researchers describe another of their systems-one that can use annotations in a retrieval process. This new system leverages annotations from different sources, human and computer. The researchers, who have built a prototype of the system, are working on testing it in real-world situations. That paper, "Toward Bridging the Annotation-Retrieval Gap in Image Search by a Generative Modeling Approach," was authored by Ritendra Datta and Weina Ge, Ph.D. students in computer science and engineering; Li; and Wang.

"Our approach aims at making all pictures on the Internet visible to the users of search engines," Wang said. Research on both systems was supported by the National Science Foundation.


Can't find it? Try searching ScienceDaily or the entire web with:


Science Video News

The 2006 Robot World Contest recently took place at South Korea's Convention and Exhibition...  > watch video

Jump to: < prev | next >

Computers With Human-Like Vision Could Strengthen Security And Surveillance, UCLA Researchers Say (November 7, 2001) -- Unmanned military vehicles could distinguish cave entrances from shadows and locate other hazards if they had a sense of vision similar to humans, say researchers at UCLA's Henry Samueli School of ... > full story

Animated 3-D Boosts Deaf Education; "Andy" The Avatar Interprets By Signing (March 13, 2001) -- "Andy" the avatar talks with his hands. He is a 3D animation, displaying a distinct personality and natural facial expressions that help him interpret words and phrases for hearing-disabled viewers ... > full story

Software Advance Helps Computers Act Logically (June 18, 2005) -- Computers just respond to commands, never "thinking" about the consequences. A new software language, however, promises to enable computers to reason much more precisely and thus better reflect ... > full story

UF Psychologists: Computer Anxiety New Illness Of High-Tech Age (July 31, 1998) -- To the fear of math, flying and other stressful problems of modern high-tech life add 'computerphobia,' say University of Florida psychologists.Just as a single bad experience can discourage people ... > full story

Teaching Computers To Replace Lost Sounds (March 25, 2002) -- Mike Savic can't recapture the 18 missing minutes of the Watergate tapes, but he can teach computers to deliver sounds that have been damaged in transmission. His research will aid military ... > full story

Visonary Computers May Put Hockey On Cell Phones (May 21, 2004) -- They may never appreciate the poetry of a sprawling glove save, or the thrill of an overtime winner, but computers are a step closer to 'seeing' the sport of ice hockey, thanks to research at the ... > full story

Using Computers To Sort Out Facts From Opinions (September 26, 2006) -- In "information extraction," computers scan text for words and phrases that identify subjects, objects and specific types of information in order to understand the highly variable ways in which human ... > full story

Technology Helps Disabled Kids Find Their Voice (February 20, 2006) -- Laptop computers that combine features from popular toys with innovative technology have rapidly accelerated the learning and communication ability of disabled children, Penn State researchers say. ... > full story

The Mathematics Of A Lampshade (June 7, 2001) -- Try to solve the following maths problem: does x^3+y^2+1 produce the same form as x^3+3y^2+xy^2? For cubic equations, it's possible to solve this problem, but mathematicians found things more ... > full story

Quantum Computer Could Solve Problems In A Few Months That Would Take Conventional Computers Millions Of Years (September 13, 2001) -- How to build a super fast computer that uses the bizarre properties of quantum physics is the aim of a project by computer scientists Fred Chong of the University of California, Davis, Isaac Chuang ... > full story

Webcam -- A web camera (or webcam) is a real time camera whose images can be accessed using the World Wide Web, instant messaging, or a PC video calling ... > full article

Speech recognition -- Speech recognition technologies allow computers equipped with a source of sound input, such as a microphone, to interpret human speech. The challenge for developers of Automatic Speech ... > full article

Streaming media -- Streaming media is media that is consumed (read, heard, viewed) while it is being delivered. Streaming is more a property of the delivery system than the media ... > full article

Application software -- Application software is a loosely defined subclass of computer software that employs the capabilities of a computer directly to a task that the user wishes to ... > full article

Computing -- Originally, the word computing was synonymous with counting and calculating, and a science that deals with the original sense of computing mathematical ... > full article

Malware -- Malware ("malicious software") is software designed to infiltrate or damage a computer system, without the owner's consent. The term describes the intent of the creator, rather than any particular ... > full article

HTTP cookie -- An HTTP cookie is a packet of information sent by a server to a World Wide Web browser and then sent back by the browser each time it accesses that server. Cookies have been of concern for Internet ... > full article

Computer software -- Computer software (or simply software) is that part of a computer system that consists of encoded information (or computer instructions), as opposed to the physical computer equipment (hardware) ... > full article

Cyber-bullying -- Cyber-bullying (cyberbullying, online bullying) is the use of electronic information and communication devices such as e-mail, instant messaging, text messages, mobile phones, pagers and defamatory ... > full article

Computer-generated imagery -- Computer-generated imagery is the application of the field of computer graphics to special effects. CGI is used in movies, television programs and commercials, and in printed ... > full article

Statistics for Business and Economics and Student CD-ROM, Fifth Edition
The strength of this book has been its unerring accuracy and statistical precision. The role of computers and statistical software has been thoroughly integrated throughout. This edition focuses on ... > read more

The Photoshop Channels Book
One big advantage Photoshop professionals have always had was the understanding of channels. It was their secret weapon, and it enabled them to do things, and work in an entirely different way ... > read more

Last Child in the Woods : Saving Our Children from Nature-Deficit Disorder
“I like to play indoors better ’cause that’s where all the electrical outlets are,” reports a fourth grader. But it’s not only computers, television, and video games ... > read more

Get the Facts on Anyone (Get the Facts on Anyone)3rd Edition
Looking for a long-lost friend? Perhaps siblings separated at early ages? Or what if you'd like to know more about a prospective tenant or employee? Get the Facts on Anyone can help. Investigative ... > read more

Feel the Fear and Do It Anyway
Dynamic and inspirational, FEEL THE FEAR AND DO IT ANYWAY is filled with concrete techniques to turn passivity into asssertiveness. Dr. Susan Jeffers, teaches you how to stop negative thinking ... > read more

Cracking the AP Calculus AB and BC Exams, 2006-2007 Edition (College Test Prep)
The Princeton Review realizes that scoring high on the AP Calculus AB & BC Exams is very different from earning straight A’s in school. We don’t try to teach you everything there is to ... > read more

Cracking the AP Chemistry Exam, 2006-2007 Edition (College Test Prep)
The Princeton Review realizes that scoring high on the AP Chemistry Exam is very different from earning straight A’s in school. We don’t try to teach you everything there is to know about ... > read more

LabVIEW for Data Acquisition
Preface I've been interested in gadgets and computers since grade school, particularly gadgets connected to computers. My first real gadget-to-computer project, at age 16, was an alarm for my car ... > read more

The Immunoassay Handbook, Third Edition
Immunoassays are biochemical tests that rely on antibodies (a special type of protein) to bind to specific molecular structures (antigens). They have become the most commercially successful ... > read more

Space Mission Analysis and Design, 3rd edition (Space Technology Library) (Space Technology Library)
This practical handbook for Space Mission Engineering draws on leading aerospace experts to carry readers through mission design, from orbit selection to ground ops. SMAD III updates the technology, ... > read more

Text: small | med | large
Also search ScienceDaily or the web with Google:
Health & Medicine Mind & Brain Plants & Animals Space & Time Earth & Climate Matter & Energy Computers & Math Fossils & Ruins