Digital twin of Human robot collaboration enabled through sensor data fusion

Human-Robot Collaboration (HRC) allows human operators to work side by side with robots in close proximity where the combined skills of the robots and operators are adopted to achieve higher overall productivity and better product quality. Compared with the traditional robotic production, HRC empowers a manufacturing paradigm shift towards a high level of flexibility, adaptability, and controllability. However, robots used today are often controlled by native rigid codes that cannot support an efficient HRC. In addition, HRC state prediction is paramount importance in any HRC system. To support intuitive collaboration with robots from human sides in a safe environment, it is important to use and fuse data from multiple sensors, instead of single sensor, to percept HRC environments. Also, a virtual world of HRC system is created in the front end, and execution and monitoring are performed at the back end.

Targeting this purpose, EU ODIN project will deploy a digital twin enabled through sensor data fusion in a HRC system. Within the context, digital twin and sensor data fusion are two core modules. Digital twin has been proposed for industrial applications such as assembly, waste electrical and electronic equipment recycling and predictive maintenance. Sensor data fusion can be used to improve the quality of information output and system performance. By using advanced technologies such deep learning, edge computing and fog computing with digital twin, data can be analysed and converted into commands/actions for system operation.

A digital twin of the HRC system will be developed to fuse data from cameras, gesture sensors, microphones and robot controllers. In parallel, advanced tools of 3D simulations and virtual commissioning offers the critical base of creating the digital twin. Through the developed digital twin system, the HRC system can be expected to measure the relative positions between humans and robots and recognise and predict system status, where a deep learning system will be developed to process the fused data. Within such context, the commands for the HRC system can be generated for safe and smooth operation. The new HRC system with the digital twin will be able to offer a safer operation environment to the pilot line compared to conventional systems with single sensor data input.

In the EU ODIN project, the Digital Twin modules are expected to simulate and validate alternative system configurations signifies a huge reduction in the cost, time and complexity of reconfiguring/operating large-scale robotics systems. The ability to continuously update the digital representation with sensor data coming from the shopfloor further extends its use as a monitoring and decision-making support tool.