The SMASH project initially proposed to create a software framework that makes it easy for students and practitioners to analyze and manipulate video, image and audio data in large scale. The vision was to create a library that makes written program code automatically scalable to cloud computing so that a successfull experiment in the small could be immediately repeated in the large.

The outcomes of this award are as follows. We organized the creation of a research corpus of 100 Million images and 1 Million videos from Flickr (YFCC100m). The corpus contains various annotations, such as tags, title, descriptions, geo-tags, time stamps, etc. As far as we know, it is the largest openly available research corpus available for multimedia data as of today. Amazon agreed to host the corpus as part of their Open Data Initiative. Lawrence Livermore National Lab hosts the corpus for goverment research purposes. We then created the Multimedia Commons initiative to further the creation of annotation and code sharing around this dataset. We also contributed the orginally proposed scalable multimedia analysis framework (Smash) based on Amazon's cloud tools and Jupyter Notebook. Smash allows students to use Python to perform various multimedia analysis experiments directly on 100M images and 1M videos. The corpus and it's infrastructure have been widely adapted into the research community as indicated by a) mainstream media coverage (including CNN, BBC News, and Forbes) and b) references to the main article in the Communications of the ACM, at a rate of more than two citations per week. 

The result of this NSF grant allows for empirical studies at never-before-seen scale. The images and videos show many facts that would have to otherwise be analyzed as part of interviewing, traveling and/or performing lab experiments. With the help of undergraduate supplement funding we also created the Multimedia Commons search engine which allows users to create subcorpora for their specific research questions. An example result of using this search engine by an undergraduate student for his research idea was the proposed redefinition of a difficulty metric for Origami tutorial videos. The paper was not only accepted at the 2018 International conference on Origami in Science, Mathematics and Education, the student also won a scholarship. 

The results of the Smash project will continue to have impact in teaching and research at UC Berkeley and in the multimedia community as a whole beyond the scope of the NSF funding, especially since large industry players such as IBM and Google reportedly rely on YFCC100m and Multimedia Commons and are actively funding research with it.

Last Modified: 07/31/2019
Modified by: Gerald Friedland