Robotic delivery service in combined outdoor-indoor environments: technical analysis and user evaluation

Robotic solutions for delivery tasks in urban and unstructured areas have represented a solid and considerable ﬁeld of research in recent years. The aim of the proposed paper is to present the technical feasibility and usability of a robotic solution able to carry items from outdoor areas up to the user’s apartment and vice-versa. The proposed solution is based on three heterogeneous mobile platforms, working in three diﬀerent environments (domestic, condo-minium, outdoor), able to cooperate among themselves and with other machines in the framework (i.e. the elevator of the condominium). The evaluation was performed in realistic environments involving 30 end-users.


Introduction
In recent years, research progress in robotics has heavily driven the spread of robotic solutions in different fields of applications, including defense, rescue, security, healthcare, and agriculture. In particular, logistic applications have been investigated thoroughly and have resulted relevant success cases such as the Kiva robots used in Amazon's warehouses 1 2 . Furthermore, service robotics for logistic applications have been successfully installed in hospitals. These devices are based on mobile platforms that can navigate and move safely in a human populated environment. A survey of the current state of the art of these systems is proposed in [? ]. Moving the focus to the urban context, in 2016, DHL, the global market leader in the logistics industry, conducted an extensive analysis of robotics in logistics, proposing futuristic scenarios where the supply chains will be strongly automated and the door-to-door delivery will be performed by a system of coordinated robots 3 . In this direction, new innovative start-ups have been founded aiming to perform robotic delivery, such as Dispatch Robotics 4 15 and Starship Technologies 5 . In addition, service robotics in urban environments have increased importance in assisted living applications, such as assistance to elderly or disabled individuals, by enabling independent living and autonomy. This paper present the results, in terms of technical feasibility and usability, of two developed services in the context of real urban environments: shopping 20 delivery and garbage collection service. These two services were considered useful by 75% and 62% of individuals, respectively, out of more than 100 people who participated in the end-users' needs analysis [? ]. These experimental scenarios, described in Sec. 4, were implemented using a system composed of three heterogeneous mobile platforms and common 25 agents (i.e. elevator) able to cooperate among themselves and in three different environments [? ]: domestic, condominium, and outdoor (Sec. 3). As introduced, the implemented solution has been tested extensively to understand its reliability and the usability experienced by the involved end-users (Sec. 5). To conclude, the feasibility of the described solution is presented and ex-30 plained through analysis of both technical and usability aspects.
The paper is structured as follows: • Sec. 2 features an overview of the current state of the art is provided.
• Sec. 3 contains a description of the system, while Sec. 4 presents details of the scenario and describes the strategy to provide the proposed services.

35
• Results of the performed technical tests and the usability evaluation by end users are shown in Sec. 5.
• Sec. 6 contains a discussion of the work performed.

Related Works
In the current state of the art, the use of mobile robots for transportation 40 of goods has been explored extensively. The main characteristic of the proposed system is a "robot relay" strategy where, to link the carriages in outdoor and indoor environments, items are physically exchanged between mobile platforms, as typically performed in production chains in industrial and structured environments. To the best of the authors' knowledge, the relay strategy has been already used in robotics for 70 telecommunication purposes, as explained in [? ], but this is the first time that this strategy has been employed in transportation tasks.
Regarding the indoor carriage, the proposed solution involved the use of an elevator already present in the environment to perform multi-floor navigation.
This aspect was investigated also in [? ], where the CoBots platform has the 75 functionality of riding elevators with human help, and in [? ], where the recognition method of the buttons and path planning algorithms for navigating in the elevator were studied.
Going beyond the current state of the art, the work described in the paper aimed to develop a feasible solution to provide delivery services from outdoor to 80 indoor environments (and vice versa) based on heterogeneous mobile platforms working in heterogeneous environments, focusing on analysis of the reliability and usability of the system.

Description of the System
The proposed system implements two services that involve three different  communicates with the control PC, continuously sending its internal status and asking for the task that must be done. All the communications are implemented via the Client/Server TCP mechanism.

90
To develop the multi-agent system described above, one of the first prob-

Agents
As mentioned, three mobile robots were used in the system. They here briefly described in this section. • It provides a mobile interface that is able to autonomously navigate toward users to facilitate the provision services and information, specifically with cognitively frail people [? ] • In the role of providing services and information, physical embodi- Control PC: The Control PC is reachable from all agents of the system. It runs a status server, which collects all the status data of the agents, and 165 a task server, which waits for agent requests, sending back the task that a specific agent has to accomplish. This module also has a Plan Manager, which stores the sequence of tasks (plan) to execute in order to accomplish the shopping or the garbage service. The Control PC receives a service request from the speech or the tablet interface of the DoRo robot. When a 170 service starts, the Plan Manager sends the tasks to the task server module.
When a task is requested of a robot, the robot starts to execute the task.
The Plan Manager waits until the robot finishes, looks at the status server, and then requests the next task. Some tasks could also run also in parallel (e.g. lift and robot control).

Methodology
The aim of the work conducted and described in the paper was to demonstrate the maturity of a developed system both from a technical point of view and the usability of end users point of view.
For this reason, the proposed services were tested in realistic environments 180 and tried by end-users to evaluate them thoroughly.

Experimental Scenarios
The system implements robotic services for shop delivery and garbage collection. Therefore, items have to be delivered from outdoors (a shop) to indoors (user apartment) in the first case and from indoors to outdoors (discharge area) 185 in the latter case.
The overall strategy for the shopping service is shown in Fig. 2, where steps of the process are depicted. The steps are actions generated by the Scenario Plan Manager: The same protocol can be performed from the user's apartment to the outdoor environment to provide garbage collection.

205
From a technical point of view, the crucial points of the services were identified at particular sub-tasks of the procedure: • In outdoor navigation, in areas close to buildings due to degradation of quality of GPS signal • In the link between outdoor and indoor navigation, where the carried item 210 is moved from outdoor robot to condominium platform (or vice versa) • In multi-floor navigation, where the condominium platform uses the elevator to move from one floor to another.
Outdoor navigation in areas close to buildings. In outdoor navigation, the localization is commonly based on GPS data. However, the degradation of the in the described outdoor navigation task, which has the goal of the entrance of the building.
For these reasons, a solution that combines GPS and AMCL to provide position and orientation of the robot by switching automatically in real time 220 between these two systems, depending on their estimates, was investigated. environments, a mechanism of haulage, crossing the edge of heterogeneous areas, was implemented (see Fig. 3). This is mechanically based on sets of rollers mounted on both outdoor and condominium platform.
Platforms, and therefore rollers, can be aligned using a docking strategy by which the condominium robot is able to move closely and precisely toward

User Evaluation
To evaluate the usability of the proposed Robot-Era services, the System to obtain a general overview on the scores of each item. Afterward, the score contribution of each item is determined from 0 to 4. For positively worded items (1, 3, 5, 7, and 9), the score contribution is the scale position minus 1.
For negatively worded items (2, 4, 6, 8, and 10), the score contribution is 5 minus 290 the scale position. To obtain the overall SUS score, the sum of the item scores is multiplied by 2.5. Thus, SUS scores range from 0 to 100. The interpretation of the score is: not usable (0-59 points), usable (60-79 points), and excellent (80-100 points), as used in [? ]. Finally, to investigate if socio-demographic factors, such as gender, age and marital status, and technology knowledge impact on the 295 usability, non-parametric tests, such as Wilcoxon rank test and Kruskal-Wallis rank test, were applied.

Results
Experiments were run in a realistic scenario at the DomoCasaLab of the BioRobotic Institute, part of the Echord++ Rif. 8 . 300

Technical Results
Outdoor navigation in areas close to buildings.  The path is divided into three sections, A, B, and C. Sections are defined corresponding to the change of localization system. In Fig. 5, the path of the robot is blue when GPS is selected and used, and red when AMCL is selected and used. In Fig. 6, the combination of GPS and AMCL σ standard deviations along the whole path is represented. Two peaks can be identified, corresponding with

Robot Cooperation.
The success rate of the sub-task was measured equal to 96% on 60 tests (58 success, 2 failure).
However, to achieve a complete reliability on this task, a deeper analysis was 335 performed to implement a recognition of the error mechanism.
The variability due to the manual teaching process was evaluated in 15 different trials, in which the procedure was repeated by the same operator.  Standard deviation in the x, y position and orientation was computed (Tab. 3).
The intrinsic variability of the procedure was evaluated in 30 additional 340 trials, related to three different processes of the docking procedure. In Tab.
4, data are reported divided, for each sub-session and related to the global evaluation, considering all three trials as a whole. The resulting data confirm the contributions of both cited factors in the final variability of the docking procedure.

345
Considering the data retrieved from the analysis described earlier, two strategies were added to manage the appearance and resolution of misalignments: 1. To reduce the variability introduced by the operator, reference of the outdoor robot was provided online to aid the operator during the teaching phase.   and ±2 deg on orientation, the procedure was simply restarted before the activation of rollers. In this way, even if there was a hypercorrection of the procedure, namely the correction of right performances, the probability of a failure occurring was reduced. In other words, each navigation task is characterized by a balance among accepted tolerance and requested pre-360 cision that the navigation planner must respect. Using a simple failure recovery enhances the success rate of the operation, without changing the navigation planner which would cause a loss of generality.
Use of elevator and multi-floor navigation. The success rate of the sub-tasks was measured as 100% on 30 tests.

365
As mentioned, to prove the repeatability and robustness of this sub-task, a focused evaluation was performed. On 30 tests performed, current position of condominium robot was logged. Positions are plotted in Fig. 7, while in Fig. 8 the same paths are referred to the relative maps.

Discussion and conclusion
Even if not quantitatively investigated in the current state of the art, the user experience can be affected by the reliability of the system. In other words, particularly in new and innovative solutions as the one proposed, failure that occurred in the system during the services can be self-attributed by the endusers, decreasing their perception of usability of the system. Furthermore, differently from other technological solutions (as logistics or industrial), the analysis of technology readiness level of innovative systems addressed to be used by non-expert end users has to include not only technical aspects but also more user-related factors.

415
For these reasons, the analysis performed and presented in this paper has been focused on both aspects to provide a complete evaluation of the system developed.
To better understand the process, a deeper analysis was performed of the most difficult sub-tasks: outdoor navigation in areas close to buildings, docking 420 process for transport of items between outdoor and indoor environment (and vice versa), and navigation inside the elevator for multi-floor navigation. The results obtained showed the repeatability and robustness of the system.
From the user's point of view, considering the overall SUS scores, the proposed system was evaluated positively from the usability perspective by twenty 425 elderly participants. The level of autonomy as well as the technology skills variables did not impact the perceived usability significantly. Moreover, the overall usability of the system is not related to age, gender, and educational level. These results suggest that the proposed system was developed to be usable for a wide segment of the elderly population.

430
Concerning the 10 items of the SUS questionnaire, the results are positive because the positively phrased items have a mode value equal to 4, while the negatively phrased items have a mode value equal to 2 (excepted for Item4).
In particular, Item1 indicates that participants would like to use the proposed system frequently and assessed a positive rating (Mode:4). In effect, 15 elderly 435 people agreed and strongly agreed with Item1. Moreover, most older volunteers (19 of 30) felt very confident using the system during the experimentation (Item9, Mode:4) and only 9 of them found the robotics system very cumbersome to use (Item8, Mode:2). These results suggest that the elderly users were satisfied by the evaluated system. volunteers had enough trust in the capabilities of the described system.
As a final result, the developed solution could be considered usable by endusers for of the developed services.
The choice of SUS was made because this evaluation tool is a quick assessment of the usability of a product. The tool is easy for participants to 460 administer and to complete, giving a global view of subjective assessments of usability. However, the SUS is not an objective measure of a systems usability, but it is a measure of perceived usability. For these reasons, the assessment of perceived usability has some limitations because participants could be inconsistent and prone to bias while rating their own usability experience after 465 interacting with a system. Moreover, an evaluation tool based on a Likert scale could be subject to distortion such as the central tendency bias, in which the user avoids the use of extreme response categories, or the acquiescence bias, in which the user always agrees with statements. However, developing survey instruments with positively and negatively worded items, such as the SUS, could 470 avoid response bias [? ]. Keeping in mind these limitations, in this study the Cronbachs Alpha test was applied to verify the internal consistency of the SUS survey. The Cronbachs Alpha of the SUS was .862, above the threshold of .7 for being acceptable [? ], so the outcome of the survey can be considered reliable.
However, for future work, the usability of the robotics system will be evaluated 475 with a mixed method based both on quantitative approaches, such as SUS, and qualitative ones, such as the interview, to avoid distortion or bias. Furthermore, an objective measure of a systems usability will be done by calculating metrics such as the success rate, the time required to complete a task, or the error rate.
In conclusion, in both the shopping delivery and the garbage collection ser-

Acknowledgement
The work described was supported by the Robot-Era and Echord++ projects, founded by the European Community's Seventh Framework Programme (FP7/2007-