Distributed Hierarchical MCU Sensing Network with On-Device Neural Inference for AR-Assisted Robot Teleoperation

Author : Oleg Kapshii, Igor Kravets, Oleksandr Karpin, Nazar Kotlyar

Abstract : Remote teleoperation of robotic manipulators deprives the operator of direct tactile and spatial awareness. Cameras alone cannot provide gripper contact state, pressure distribution, or workspace intrusions. Raw sensor transmission is not a viable alternative - two three-fingered hands equipped with 5×7 capacitive tactile arrays sampled at 1 kHz generate over 4 Mbps of continuous data, far exceeding practical wireless budgets. This paper proposes a distributed hierarchical sensing architecture that eliminates raw data transmission by performing progressive abstraction at each processing layer on the robot. Dedicated PSoC 4 microcontroller nodes manage single sensors - a tactile finger array, an inertial measurement unit, or an ultra-wideband radar. They run hardware-accelerated scanning and forward only compact contact descriptors on detected events. An optional aggregation layer, implemented as one or more PSoC 6 devices where topology requires it, consolidates node outputs and manages local communication with BLE. A PSoC Edge device forms the main inference subsystem, running parallel neural network classifiers on NN accelerators to identify contact geometry, detect motion anomalies, and assess workspace occupancy. It then composes sparse semantic events for wireless delivery to the operator. The operator receives these events as augmented reality overlays on the robot camera view: tactile contact maps on a virtual hand model, anomaly indicators, and proximity zone alerts. This provides richer situational awareness than cameras alone without consuming additional wireless bandwidth. An estimation based on published component specifications shows that the architecture reduces onboard sensor throughput by four to five orders of magnitude before transmission. It makes wireless delivery of rich tactile and spatial feedback practical on standard low-power radio links.

Keywords : Teleoperation, distributed sensing, edge computing, augmented reality, neural network inference.

Conference Name : International Conference on Augmented Reality with Artificial Intelligence (ICARAI-26)

Conference Place : Warsaw, Poland

Conference Date : 22nd May 2026