Computer Vision & Semantic Reasoning – This part of the project comprises an Ontology module for semantic reasoning and various machine learning algorithms for Object, scene, and person recognition. This combination of
Object/Person Localization and semantic reasoning are linked together to facilitate a platform where different detected entities from computer vision algorithms are communicated to the Ontology. The semantic interpretation in association with different scenes and time stamps provides the capability of informing the user of where and when items of interest are detected in response to relative, predefined, queries that the user can send to the HIPI system.
Haptograms – A haptogram is a meaning-bearing synthetically generated haptic pattern to transfer semantic content, typically formed by activation of a sequence of actuators, communicated as stimuli to the body. The shape and meaning of the current set of haptograms are inspired by Social Haptic Communication to facilitate recognition and learnability. The novelty lies in providing a new mode of communication for people with deafblindness based on a simple language without the need for human intervention or close physical proximity.
Haptogram Design Toolkit: Software – An application software tool (written in C#) was created for designing vibrotactile patterns and activation of vibration motors. It enables fast and accurate drawing, and activation of vibrotactile patterns, or haptograms, on different grid sizes. Multiple motors can be activated simultaneously or sequentially on one or more haptic screens in a standalone vs. combined way. It opens up the way to personalized haptogram libraries leading to language variants according to preference types. Novelty lies in the functionalities offered.
Haptogram Design Toolkit: Hardware – To enable use of numerous actuators positioned on different body parts as haptic screens, two different printed circuit boards (PCBs) were developed each with 32 and 40 channels respectively as part of the haptogram design toolkit. Each channel of these boards can be connected to an actuator. These boards can be used separately or daisy-chained together to enable the use of a larger set of actuators. For example, by connecting three 32 channel boards we will be able to connect with 96 actuators which can be placed on different appropriate parts of the body for conveyance of tactile messages by haptograms
Tactile board software – This was developed as a user-feedback and interaction system for communication with the SUITCEYES HIPI. This device facilitates communication for individuals with deafblindness, enabling them to initiate and engage in social interaction with others without the need for an intervener. Users can create words and simple sentences by drawing a multi-stroke pattern on a 4×4 grid of dots displayed on a tablet. This pattern is translated into speech and text displayed on-screen and is fully customizable. This devise provides new communication possibilities to people with deafblindness.
Tactile board cover – An especially designed soft cover for tablet computers for guiding the fingers of users with deafblindness so that the user knows where the finger is on the gall surface of the device. in this way the above mentioned haptograms or other sign systems can be fed from the user.
One-to-many communication – A haptic system that enables transmission of information from distance (whether in the same room or in different countries) and delivery in haptic format through haptograms to multiple users with deafblindness simultaneously. Novelty lies in enabling remote one-to-many haptic communication, potentially opening the possibilities for mass haptic communication.
Chairable – A portable, easy-to-use textile-based haptic communication station as a tool for experiments and one-to-many haptic communication for conferences and educational settings enabling the use of haptograms or other sign system. The textile is to put on the back of chairs and is constructed in such a way that it fits to a wide spectrum of chairs, armchairs and office chairs.
Gamified scenarios – Specifically designed for (and with) users with deafblindness. There are a number of games (e.g., Easter Egg Hunt and Keep Your Distance) developed in the project that utilize sensor technologies, vibro-tactile vibration motors and QR markers, enabling the user to engage with playful learning and or recreational activities.
Smartphone App – A mobile game prototype for learning the pattern of Social Haptic Communication concepts. PatRec is a multiple-choice quiz game that is connected to the chairable that contains a 3×3 array of vibration motors emulating different SHC signs. Players collect scores and badges whenever they guess the right SHC vibration pattern, leading to continuous engagement and a better position on the leaderboard. We report the technical implementation of the game and the findings from a user evaluation.
The HIPI – The core technological innovation in SUITCEYES has been the development of a Haptic Intelligent Personalised Interface, or the HIPI that brings together computer-vision and haptic feedback system through textile-based soft interfaces for conveying environmental Information in a haptic form. The HIPI consists of various technological components that can be used individually or coupled with one another to achieve varying objectives. At an example, a HIPI can include a smart textile-based garment that contains a Raspberry Pi 4 minicomputer; Intel RealSense D435i depth camera; and a set of vibration motors formed into a belt around the waist and in a 4×4 grid on the backend connects them together. Information about the immediate environment can be captured using images and distance information from the depth camera, and the proximity of Bluetooth Low Energy Beacons in the environment can be ducted to assist with orientation. This information can then be processed, and signals relayed to the wearer as vibration patterns to alert them to objects of interest in the environment. The innovation lies in the integrated and wearable nature of the device. Different versions of the HIPI are further described below.
Different versions of the HIPI
Red Dress – The Red Dress was the very first version of the HIPI, used for the purposes of experimenting with how to attach various actuators to garments in a flexible way while still keeping the design attractive. In that prototype, the electronic components were place in the belt, which could easily be removed (in connection with the wires and actuators) to render the dress washable. While attractive, the related experiments identified the need to embed the wiring in the garment in other ways.
Black Dress – This prototype was created to experiment with addressing the shortcomings of the red-dress by providing piping as a means of concealing wires. It also offered in-built spaces inside the garment to house actuators to provide close proximity to the body and to remove the risk of accidental detachments from the outside of the garment. The insights gained through the internal experiments with this prototype informed further developments in the project.
Chessboard Vest – While each of the Red and the Black dresses were used for internal experiments within the project, a constraint with these prototypes was their design to fit specific sizes and body shapes. By this point in the project, extended experiments involving different study participants were needed. To enable such experiments, more flexibility was needed. The Chessboard Vest was therefore designed to offer this flexibility. This single garment enabled personalisation for a tight fit for different body sizes and shapes. This prototype was used in multiple studies both by the partners for internal experiments with actuators, haptogram design, etc., but also in psychophysics experiments and studies involving external study participants.
Gaming Green – This prototype was designed mainly for gamification experiments enabling flexibility for addition of components on top of the garment. For this, this prototype offered strong Velcro outer and magnetic panels, which together would enable attachment of components (e.g., cameras, sensors) onto the garment in a flexible manner to experiment with positioning of sensing components to allow experiment with and design of different games. This prototype was used for internal testing as well as in gamification experiments.
Gaming Grey – Based on the findings from experiments with the Gaming Green prototype, the Gaming Grey prototype was created. In this version, the components such as actuators, sensors, and camera were positioned within the garment. A zipper system also enabled some flexibility in the size and shape of the wearer. This prototype was used in gamification experiments as well as in connection to the tactile-board development and testing.
Multipanel Suit – This prototype was designed to enable placement of actuators on different body parts to enable experiments with multi-screen haptogram design that would emulate Social Haptic Communication that utilizes different locations on the body. This prototype consists of 10 different sub-parts (for lower arms upper arms, lower legs and upper legs, back and shoulders) that can be used individually or in combination with one another. This prototype was used in conjunction with developing the PCBs with extended channels, and production of actuators with longer wiring to reach different body parts.
Chairable 1– In order to enable an effective testing involving many participants the “chairable” was constructed. This was a tactile input matrix to be put on the back of an office chair such as available in the office environments where psychophysical experiments were to be performed. It offered a flexible solution with removable vibrators to be placed freely, and adaptable to the experiment to be performed. What is also valuable is that the problem of a tight fit to the body for good tactile contact (while simultaneously having test persons with different sizes) could be avoided by using this chair-solution. This prototype was used in psychophysics experiments.
Chairable 2 – As chairs are different, e.g., some have arm support, some have divided back, etc., a next generation of chairables was constructed that could be put on the back of a chair and also be wrapped around for adopting to a wider spectrum of chairs. Chairable 2 also contained an arrangement of pockets for control boards and batteries, etc., rendering it as an stand-alone solution without a need for additional tables etc to hold the related electronic components. This prototype was used both in psychophysics experiments as well as the development of one-to-many and distance haptic communication experiments.
Black Vest – Based on several toile-like pre-prototypes (and previous Red Dress for positioning of actuators and Gaming green for positioning of sensors) the Black Vest HIPI garment was constructed and produced for including both sensor capability (concentrated to the front) and actuator capability (including the vibrator matrix for the rear part of the garment). Testing from the point of manufacturability was conducted. The Black Vest (HIPI) was used in computer vision and active object search experiments. It was, for example, used for walking (blindfolded) in a room and identifying faces and objects (as demonstrated at the final review meeting).