Consumer Technology

Xilinx All Programmable SoCs and MPSoCs Enabling Vision Guided Robots & Drones

Guest blog from Giles Peckham

Embedded vision can be split into two high level categories: systems that perceive the environment or systems that perceive the environment and take action. Vision guided robots and drones take action, needing to respond to situations by going from sensing to processing, analysing, deciding, communicating and controlling.

At the moment, one of the hottest topics is the civil use of drones with applications across commercial, medical, agricultural, broadcast and law enforcement. For many of these applications, drones offer significant advantages, such as lower cost of ownership and operation. Developing drones for these applications presents several challenges though, including the deployment of machine learning, sensor fusion and embedded vision, the creation of a secure solution and maintaining a power-efficient overall solution.

Hexacopter with crop sprayer flying in the sky

Architecturally, drones have several critical subsystems, precise motor control in real-time, software defined radio to provide bi-directional communication and finally a precise embedded vision system. A precise embedded vision system uses a high frame rate and has the ability to process and act upon the image faster than a human, while many applications will use a multiple-camera approach to observe the environment, some also combine cameras designed to observe different elements of the electromagnetic spectrum as in infrared applications.

At the algorithmic level, along with pattern and object recognition, algorithms such as Simultaneous Localisation and Mapping (SLAM) and dense optical flow, are used to provide an enhanced perception and obstacle-avoidance capability.

For drone developers, the All Programmable Zynq®-7000 SoC or Zynq® UltraScale+™ MPSoC devices offer a unique solution to the challenge at hand. These devices provide high-performance logic coupled with high-performance ARM® A53or A9 processors for a tightly integrated heterogeneous processing unit. For real-time control, as may be required for motor control, the Zynq UltraScale+ MPSoC also provides a real-time processing unit that contains dual ARM® R5 processors capable of implementing a safety processor. This heterogeneous processing unit enables efficient segmentation of the functionality within either the processor or programmable logic.

Using an ARM-based processing system, it is possible to perform very sophisticated, real-time vision analytics and decision making at the edge. Coupled with the programmable logic fabric, the SoCs provide the ability to detect objects in real-time and then classify them using the processor system. Once classified, the vision-guided robotic system or drone can then take the appropriate action for the class of object detected. To enable the real-time decision making at the edge, developers are deploying machine learning inference techniques.

reVISION™ provides the ability to accelerate several OpenCV functions into the Programmable Logic (PL) within the selected Zynq-7000 or Zynq UltraScale+ MPSoC device. This enables a more responsive image processing-system as pipelines are inherently parallel in nature. Within an autonomous drone or robot application, machine learning inference is often used to enable decision making. To support this, reVISION also makes it possible to implement machine learning inference engines within programmable logic. reVISION can take network and training information directly from a Caffe prototxt file definition of the network. By using programmable logic, designers can accelerate an inference engine and thus provide a more responsive, power-efficient solution.

Due to the remote nature of vision guided robots and drones, the optimal design solution must also consider security to prevent unauthorised modification or accessing of the system and its data or malicious hacking. Although developing a secure design must be considered from the system level downwards, an All Programmable Zynq-7000 SoC or Zynq UltraScale+ MPSoC provides several device and system-level security aspects that can be put into effect with ease. With these devices, developers can encrypt and authenticate the boot and configuration process, along with supporting Trustzone technology. With Trustzone, the development team can create orthogonal worlds, a practice that limits software access to the underlying hardware, with the use of a hypervisor. Meanwhile the inbuilt XADC or System Monitor can be used to monitor device voltages and temperatures along with external parameters to provide for an anti-tamper approach.

Vision guided robotics and drones are experiencing significant growth as the applications they are used for rapidly expand. Developers can best meet challenges of generating a precise vision system with optimal performance per Watt by using an All Programmable Zynq-7000 SoC or Zynq UltraScale+ MPSoC designed with the help of the reVISION stack. This gives them the opportunity to use industry-standard high-level frameworks and libraries to design a power efficient, responsive, flexible and secure system.

Author Bio

Giles has more than 30 years’ experience in the semiconductor industry, starting with the design of ASSPs for consumer applications at Philips Semiconductors (now NXP) before moving on to FAE and marketing roles for gate array and standard cell products and finally a sales role in the same organisation. After five years in IP product marketing and international sales roles at European Silicon Structures, the e-beam direct-write ASIC company, Giles recognised the increasing potential for FPGAs and joined Xilinx. Whilst at Xilinx, he has held a number of technical and commercial marketing roles in EMEA before being promoted to run the group. Giles holds a BSC in Electronic Engineering and Physics from Loughborough University, UK and a Professional Postgraduate Diploma in Marketing from the Chartered Institute of Marketing in the UK. Giles is based in the Xilinx EMEA office in London.