My PhD dissertation is now available. It lays out interaction designs, design framework, data structures, and algorithms for sketching embodied representations of algebra.

PhD Dissertation pdf

The dissertation was nominated by the MIT Media Lab for ACM SIGCHI Outstanding Dissertation Award.

Nazmus Saquib, Rubaiat Habib, Li-Yi Wei, Gloria Mark, Deb Roy 

CHI 2021 Paper

Mathematical models and expressions traditionally evolved as symbolic representations, with cognitively arbitrary rules of symbol manipulation. The embodied mathematics philosophy posits that abstract math concepts are layers of metaphors grounded in our intuitive arithmetic capabilities, such as categorizing objects and part-whole analysis. 

We introduce a design framework that facilitates the construction and exploration of embodied representations for algebraic expressions, using interactions inspired by innate arithmetic capabilities. We instantiated our design in a sketch interface that enables construction of visually interpretable compositions that are directly mappable to algebraic expressions and explorable through a ladder of abstraction. 

The emphasis is on bottom-up construction, with the user sketching pictures while the system generates corresponding algebra. We present diverse examples created by our prototype system. A coverage of the US Common Core curriculum and playtesting studies with children point to the future direction and potential for a sketch-based design paradigm for mathematics.

Nazmus Saquib, Rubaiat Habib, Li-Yi Wei, Wilmot Li (CHI 2019 paper)

Augmented and mixed-reality technologies enable us to enhance and extend our perception of reality by incorporating virtual graphics into real-world scenes. One simple but powerful way to augment a scene is to blend dynamic graphics with live action footage of real people performing. This technique has been used as a special effect for music videos, scientific documentaries, and instructional materials incorporated in the post-processing stage.

As live-streaming becomes an increasingly powerful cultural phenomenon, this work explored how to enhance these real-time presentations with interactive graphics to create a powerful new storytelling environment. Traditionally, crafting such an interactive and expressive performance required technical programming or highly-specialized tools tailored for experts. 

Our approach is different, and could open up this kind of presentation to a much wider range of people. Our system leverages the rich gestural (from direct manipulation to abstract communications) and postural language of humans to interact with graphical elements. By simplifying the mapping between gestures, postures, and their corresponding output effects, our UI enables users to create customized, rich interactions with the graphical elements.

News Coverage:

• Synced Review 2019. MIT & Adobe Introduce Real Time AR Tool for Storytelling

• 2-minute Papers 2019. Augmented Reality Presentations Are Coming!

Impact: To date, an eclectic group of users used the prototype to ideate and create demos. One patent has been filed, and there is a work-in-progress livestream Adobe product being developed based on this work.

Supplementary Examples: A wide range of users could use the system to make an impressive array of examples and possibilities. Some examples can be seen here: cooking instruction video, astronomy research presentation, interior design, meditation tutorial.

Nazmus Saquib, Ayesha Bose, Dwyane George, Sepandar Kamvar. Sensei: Sensing Educational Interaction. Ubicomp 2018 (IMWUT). (pdf)

Sensei (Sensing Educational Interaction) is a dynamic range based distributed sensor network that aid the observation process in early childhood Montessori classrooms. The system is currently deployed in Wildflower Schools. In a busy classroom, teachers hardly have time to observe each and every child to learn about their needs. Sensei helps teachers make sense of their classrooms. 

Unobtrusive sensors provide a way to tap into classroom interactions data that can be interpreted with machine learning algorithms to provide insights to teachers that would otherwise have been lost from a busy classroom. The observation tools create a novel human-machine interface to empower teachers and students in personalizing the curriculum.

Proximity sensing radio sensors are embedded in children’s shoes, learning materials, and selected landmarks in the classrooms. By logging proximity data, we can reconstruct the daily social network, teacher-student time distribution, and learning time. Based on this data, we provide unique insights to each teacher about their teaching style and the time they spend with each child. We can also understand the kind of lessons a child is interested in. A visualization dashboard lets a teacher explore such data and compare with his/her own intuition about the classroom.

This novel way to understand classroom dynamics has already helped teachers make better sense of what each child needs, as evident from our deployment and interviews at different schools.

News Coverage

• Quartz 2020. The Montessori schools embracing kid-tracking devices

• Edsurge 2018. Kid-Tracking Sensors May Not Be the Wildest Thing About This Montessori Model

• Edscoop 2018. What happens when student-tracking technology meets Montessori’s century-old teaching style?

• NHK World TV (Japan) aired a documentary on emerging educational technology around the world, where Sensei was featured.

Impact

The work led to the establishment of Wildflower School's Innovation Lab, which acquired $3M funding from Chan-Zuckerberg Initiative and Omidyar Network, among others, to scale and deploy Sensei in Montessori schools across USA.

Nazmus Saquib, Deb Roy. Children-Centered Sensing in Early Childhood Classrooms. CHI 2018 EA. (pdf)

A prototype and a case study in Montessori philosophy inspired sensing is presented: designing unobtrusive sensor networks to understand and reflect on a child's learning progress, by instrumenting existing Montessori learning materials using distributed sensing techniques. Swarm robots communicate and collaborate in teams by using IR and other communication methods. The same techniques can be used to track the compositionality of blocks and other Montessori learning materials. In this work, I develop the necessary hardware and sensing techniques.

Nazmus Saquib, Mairaj Syed, Danny Halawi

We retrace early middle-eastern history by mining 1400 ancients text books and creating a citation network of 50k scholars (3M edges) who passed down historical statements over 350 years, from 632 AD to 1000 AD. By reconstructing the citations and using biographical information of these people, we are able to reveal unique insights about the nature of scholarship, biases and information diffusion in the early stages of middle-eastern history.

I founded and co-led the project along with Mairaj Syed, an associate professor of religious studies at UC Davis. This is an ongoing research project that started in 2011, which now has a formal name: Hikmah Lab. 

Impact: The project has garnered significant attention in the historian and digital humanities communities. For example, the digital humanities community has organized a full conference dedicated to the technical methods and analysis developed by my team, to be held in January 2021. The flyer for the conference can be found here.

Grant: Middle East in the Wider World Grant from UC.

Nazmus Saquib, Elizabeth Christoforetti, Sep Kamvar (MIT Media Lab Technical Report)

Placelet is a foot traffic analytics system utilizing privacy-aware computer vision. The system was designed and developed to collect data on pedestrian behavior in an urban setting, with an aim to understand economic activity in an area. Video stream is analyzed in real time to produce (intermediate) contour data using optical flow, and the contour results are uploaded to a server for further analysis. No video is saved on device. I led the technical team for the project and contributed to ideation and design, firmware for data collection schedule and network protocol, and the video analysis algorithms. 

Impact: The system was deployed in three stores in Downtown Boston, and also in several public places in collaboration with the Boston City Council. Additional sensor units capable of capturing noise and air quality were also constructed as part of the project.

Grant: The project won a Knight Foundation prototype grant in 2015.

News Coverage: Boston Globe, Fast Company

Naimul Hoque, Nazmus Saquib, Syed Masum Billah, Klaus Mueller (CSCW 2020 Paper, video)

Several prominent studies have shown that the imbalanced on-screen exposure of observable phenotypic traits like gender and skin-tone in movies, TV shows, live telecasts, and other visual media can reinforce gender and racial stereotypes in society. Researchers and human rights organizations alike have long been calling to make media producers more aware of such stereotypes. While awareness among media producers is growing, balancing the presence of different phenotypes in a video requires substantial manual effort and can typically only be done in the post-production phase. The task becomes even more challenging in the case of a live telecast where video producers must make instantaneous decisions with no post-production phase to refine or revert a decision. In this paper, we propose Screen-Balancer, an interactive tool that assists media producers in balancing the presence of different phenotypes in a live telecast. The design of Screen-Balancer is informed by a field study conducted in a professional live studio. Screen- Balancer analyzes the facial features of the actors to determine phenotypic traits using facial detection packages; it then facilitates real-time visual feedback for interactive moderation of gender and skin-tone distributions. Our user study revealed that the participants were able to reduce the difference of screen times between male and female actors by 43%, and that of light-skinned and dark-skinned actors by 44%, thus showing the promise and potential of using such a tool in commercial production systems.