In this paper we introduce a novel interface combining spatial and continuous tangible interaction for creating and manipulating audio-visual effects. Our goal is to provide a ready-to-use, “hands-on” interface that does not need protracted explanation to the user, yet provides possibilities for expression. Therefore, our interface exploits the three-dimensional topology of physical sand, which is distributed over a tabletop surface. We discovered, that users of the system were engaged by the natural interaction and playful manner of the installation, as it resembles the play in a sandbox. We demonstrate an artistic setup that produces ambient soundscapes using a Lattice Boltzmann based particle simulation running through a deformable landscape. Visual feedback is front-projected onto the sand as well as the user’s hand. The user can explore and change the landscape by using his or her hands and use spatial gestures via on-body projection to control AR content and further settings. The focus of this work lies on the simultaneous interaction with sand and the user’s own body, and it’s contribution to audio-visual installations.

Our prototype system was tested with potential users in a small informal study and was overall well received. Users had fun exploring the different forms of interaction techniques to control the particle simulation and soundscape, and were amazed by the possibilities of on-body interaction. In future we plan to further evaluate our system in a formal study and compare interaction and user experience to similar interfaces. The system was successfully deployed as an indoor room installation, reducing it’s components to a minimum. Further deployments are planned.