Projects > Media Annotation

( riemann.ist.psu.edu/IMAGE, www-db.stanford.edu/IMAGE )

Inspired by the fact that the Riemann Hypothesis remains as one of the most important unsolved problems in mathematics, the RIEMANN group attempts to address the problem of intelligent media annotation, one of the most important unsolved problems in computer and information sciences.

Various projects that lay within media annotation (demos on more than 200,000 images).

a-LIP

Automatic Linguistic Indexing of Pictures

a-LIP annotates images based on 600 trained concepts.

Click here for more on a-LIP

SIMPLIcity

Semantics-sensitive Integrated Matching for Picture LIbraries (SIMPLIcity) is a content-based image search engine. Instead of searching for images based on keywords, the user first selects or inputs an image and then the system produces images from the database that have similar characteristics.

Click here for more on SIMPLIcity

Art and Cultural Heritages

This project focuses on the preservation, retrieval, and dissemination of digital imagery for Asian art and cultural heritages.

Click here for more on Art and Cultural Heritages

Story Picturing Engine

The story picturing engine automatically illustrates a story by attaching images to keywords in the story.

Click here for more on the Story Picturing Engine

Satellite Imagery

The Satellite Imagery project aims at distinguishing certain characteristics of a satellite image and has impacts on many contexts including the military/security agencies and environmentalists.

Click here for more on Satellite Imagery

Virtual Microscope

This system provides progressive zooming of very large images using wavelets.

Click here for more on the Virtual Microscope

CLUE

Cluster-based image retreval

The Cluster-based Image Retrieval system bases its results on many different semantics that the computer finds similar between the original set of images.

Click here for more on CLUE

WIPE

Wavelet Image Pornography Elimination / Detection

The Wavelet Image Pornography Elimination (WIPE) filter tells the user the level of indication that an image is objectionable. The results are based on the image's pixels.

Click here for more on WIPE

UFM

Unified Feature Matching (UFM)

A content-based image retrieval system similar to SIMPLIcity, UFM uses a fuzzy matching technique as an alternative to IRM.

Click here for more on UFM

Some Real-world Applications of Our Research

1. terragalleria.com: a fabulous site with more than 10,000 high quality photos taken by Q.-T. Luong. Try the following SIMPLIcity queries:

2. Airliners.net: A site with almost 1,000,000 aviation images. (try a SIMPLIcity search: query 1, query 2, query 3) (Info: . external query, user comments)
3. Mindat.org: A site with some 50,000 images of minerals. (try a SIMPLIcity search: query 1, query 2, query 3 )

4. Memorynet.org: A site with images of global culture and heritages.

NOTE: If you run an online community site and would like to use SIMPLIcity for free to manage your images by visual similarity, please contact us at jwang @ ist.psu.edu to obtain a copy.

Intelligent Media Annotation

The broad heading of Intelligent Media Annotation covers projects that include studies on content-based image search, automatic learning-based linguistic indexing of pictures, and various ways to browse a large collection of images. The projects in the tables above explore these different areas. Media Annotation is a way for systems to recognize the content of images so that it may categorize, compare, or annotate images so that a user may manipulate the information. These developing systems are advanced because they get progressively better at their task. This makes them learning-based systems. For example, instead of learning that picture A is a person, it learns the characteristics make up a picture with a person contained therein. This allows the system to apply concept of "person" to any picture with those characteristics.

The possibilities for Intelligent Media Annotation technology are endless and have numerous impacts in a variety of sectors including education, business, military, healthcare, and the society as a whole. Students and researchers may find more digital imagery information available with these advanced searching techniques. Teachers or professors may also use the Story Picturing Engine to automatically add relative images to their lectures instead of having to search for the various topics.

There are many impacts this technology could have on the business industry as well. For example, online search engines can utilize this technology to provide better services for their customers as seen in the searches done within the SIMPLIcity project. Art companies may also use this, as shown in the Advancing Digital Imagery for Asian Art and Cultural Heritages project. By searching for similar images, the consumer may find more artwork to their liking.

As seen in the Satellite Imagery project, the military is able to take a great amount of satellite imagery information and use Media Annotation to flag images that are potentially dangerous. The same idea can be used with video surveillance images.

The Wavelet Image Pornography Elimination (WIPE) project also has future potential in the healthcare industry. This system is designed to locate and then block out certain information. In healthcare, this can help to eliminate the parts of an image that release patient identity. By automatically removing the patient's identity (ie. blurring their eyes), the image may be used in any number of medical journals or other sources of information.

Finally, Media Annotation has an impact on society as a whole. People will be able to go online and do detailed, and therefore more accurate, searches for digital images using SIMPLIcity or Cluster Based Image Retrieval technologies. The technology could also be applied to their personal store of digital images. Instead of the hard drive becoming a "digital shoebox" full of random pictures, media annotation could categorize, search, and annotate the images. A user is then able to find all of the pictures of the beach in one easy search instead of sorting through each folder of images. Possibilities for this technology are endless and have impacts on a wide range of contexts. Below is more information on the people behind the research, projects within this area, and links to more information on this subject.

Project Heads

The project is headed by Prof. James Z. Wang (jwang @ ist.psu. edu) and Prof. Jia Li (jiali @ stat.psu. edu) of The Pennsylvania State University. Some research projects were started when they were with the Database group, Information Systems Lab, and the Biomedical Informatics program of Stanford University. This content-based image search and automatic learning-based linguistic indexing project was started in 1995 when James developed an art image retrieval system for the Stanford University Libraries. He has later worked for the IBM QBIC project, the NEC AMORA project, and the AI Center of SRI International. From 1999 to 2000, their research was funded by the National Science Foundation (NSF) under the TID project. The technology research has been continued at Penn State University since 2000, with partial funding from NSF, PNC Foundation, and SUN Microsystems. If you have any questions or comments about the image retrieval project, or if you are interested in using SIMPLIcity, please send an email message.

Project Background

This content-based image search and automatic learning-based linguistic indexing project was started in 1995 when James developed an art image retrieval system for the Stanford University Libraries. He has later worked for the IBM QBIC project, the NEC AMORA project, and the AI Center of SRI International. From 1999 to 2000, their research was funded by the National Science Foundation (NSF) under the TID project. The technology research has been continued at Penn State University since 2000, with partial funding from NSF, PNC Foundation, and SUN Microsystems. If you have any questions or comments about the image retrieval project, or if you are interested in using SIMPLIcity, please send an email message.

Related Publications

A full list of the main related publications can be found here.

James Z. Wang, Jia Li and Gio Wiederhold, ``SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture Libraries,'' IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 9, pp. 947-963, 2001. (download)
Jia Li and James Z. Wang, ``Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach,'' IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, no. 9, pp. 1075-1088, 2003. (download)