EUD4GSH

Overview

The increasing availability of sensors, connected objects and intelligent services has made it possible to obtain smart homes able to support automations involving dynamic composition of objects, services, and devices. Since each household has specific needs and preferences which can also vary over time, it is crucial to empower its users to directly control such automations, even when people have no programming experience. Existing software development life cycles typically involve only technical people and are slow to respond to the evolving needs of different types of user, also because professional developers often lack the domain knowledge to address user requirements. Thus, a viable way to make applications better comply with users' expectations is to make users capable of 'programming' the intended behaviour through suitable automations.

Automations can be helpful to support achievement of various types of goals. In this project, we will focus on how to support goals consistent with the principles of a circular economy (such as energy saving and waste recycling). To this goal, a set of tools and technologies will be designed, prototyped, and deployed in real home environments to analyse their impact in daily life, and actualbility to empower end users in realising their automation needs. They include meta-design tools for creating new automations that better fit their needs and controlling those active in the home, innovative modalities to control and create automations based on conversational agents and augmented reality, and a home Digital Twin useful for supporting simulations and what if analysis to understand the potential impact of specific automations on aspects relevant for circular economy.

In the project, AI-based techniques will also be used to suggest recommendations to users, by providing rule-based automations judged appropriate in specific situations, also explaining to them the what, why and how of such recommendations. Explanations will also be provided by the system to help users evaluate specific non-functional requirements of automations, i.e. their "sustainability", such as their impact on energy consumption or saving. Specific techniques will also be developed to highlight potential errors inadvertently included while users specify rules, or issues existing in their current rule sets (e.g. conflicts between rules): in particular, clear and understandable explanations will be provided to users, describing the problem and why a specific modification would be needed to resolve it.

The project will follow a human-centred approach in which end users will be continuously involved starting from the requirement phase through interviews, questionnaires, and focus groups; usability laboratory tests on the initial prototypes; and longitudinal studies during which the developed prototypes will be deployed in some homes for analysing their use and impact over longer periods of time in the wild.

Objectives

The project will pursue three main objectives:

• Intelligent EUD Multimodal App

  As a first objective, this project would like to investigate and experiment novel EUD techniques based on simple and natural conversational dialogues, combined with augmented reality and intelligent recommendations, for the creation and control of automations in a home environment. They will be integrated in a multimodal app for creating automations specified in terms of trigger-action rules. The goal will be to overcome previous work on EUD for trigger-action programming, which in general has proposed visual editors with conceptual representations of possible automation elements that tend to be large, comprehensive, static, abstract EUD tools detached from the user's real context. With those tools it may not be straightforward how to navigate the large number of elements, and understanding what real element they refer to may sometimes require technical knowledge. This requires the design of novel solutions for creating relevant automations, supported also by intelligent recommendations, and monitoring the smart home behaviour in a more direct and immediate way.

• Green Smart Home Digital Twin

  The state-of-the-art also highlighted a lack of approaches to IoT ecosystems and smart homes allowing the simulation of automations created by the end users to investigate their effectiveness before their actual deployment. In this way, automations suggested by recommendation systems can be assessed through a previous evaluation of their suitability to the case at hand, taking into account that automations for a green smart home should limit energy and water consumption, foster material recycling, and improve health and wellbeing of the inhabitants. The choice of proper automations, their combination according to the inhabitants' preferences, habits, and needs, and the assessment of their suitability to circular economy require the implementation of relevant algorithms and dashboards. This project aims to address this problem through the design and development of a DT of the domestic environment including the house, the connected smart devices, and the home inhabitants. This DT will make it possible to simulate different 'what-if' scenarios based on the IoT automations defined by the end users through EUD techniques and AI-based recommendations. Simulations will allow users to evaluate in advance the effects of the automations in terms of different parameters (e.g., energy consumption), and, after their analysis, to approve their deployment in the domestic environment. More specifically, the simulations: will provide information about energy consumption trends determined by the end-user definition of automations related to domestic appliances, lighting, heating, etc.; will allow the exploration of alternative automations in the form of 'what-if' scenarios; will provide recommendations of specific automations aimed at reducing waste and expense; will provide explanations about the validity of the AI-based recommendations so that the user may understand the motivations underlying the recommended alternatives

• User Experience of the Green Smart Home

  Most studies of toolkits' uses for smart homes have been based on ideation workshops, e.g. end-user programming uses that afford convenient living (e.g., automating devices for morning and returning-home routines), energy savings, and security. In particular, a detailed study by Bellucci et al. [2019] pinpointed 11 possible use areas: comfort, safety, parenting, resource conservation, health and fitness, home maintenance, reminders about objects, entertainment, security, cooking, and education. However, what people articulate in workshop settings may not culminate in actual use. In scientific literature, not much has been published on in-the-wild studies of smart home technology. We thus plan to contribute to this aspect by deploying the GSH platform developed and integrated in the project in a number of homes for its actual use, in particular for the aspects relevant for supporting ecological style of life, such as promoting energy saving, waste recycling, …

Partners

The CNR unit is the Human Interfaces in Information Systems Laboratory at ISTI, in Pisa. Its research activity is in methods and tools to support user interface designers, software developers, and end-users in obtaining systems that can be accessed from different contexts of use in such a way as to improve usability, accessibility, and user experience. It has expertise in EUD authoring tools and platforms for IoT personalisation, which have been deployed in different domains (ambient-assisted living, industry, smart retail, smart home, and humanoid robots). The EUD tools developed follow various interaction paradigms (visual vizards, block-based, conversational, mobile augmented reality)

The UNIBS unit belongs to the research group Artificial Intelligence, Human-Computer Interaction and Robotics of the Department of Information Engineering, University of Brescia. The unit has studied and applied EUD methods and techniques in different application domains: mechanical engineering, medical diagnosis, e-government, health and wellness, smart home, and robotics. Recently, the unit members have explored the use of the conversational and multi-modal interfaces of commercial virtual assistants to support smart home tailorability by end users. The unit is also interested in the design and evaluation of the interaction with Digital Twins.