Artificial Intelligence and Edge Computing are two key modern-day technologies that are revolutionising cameras and imaging solutions, be it home, industrial or enterprise applications. Cameras powered by AI, also known as Smart Cameras or Edge Cameras, are finding their way to a wide range of applications, such as Robotics, Industrial, Healthcare, ADAS and Autonomous Vehicles among others. Powered by advanced AI-enabled system on chips (SoCs), Smart Cameras are designed to run compute-intensive workloads at the edge of network. In addition, superior IoT capabilities providing high-speed data exchange enable fast and real-time communication between the camera and Cloud.

Smart Cameras / Edge Cameras with Vision Analytics come with superior image processing, machine learning, and security features, in addition to integrated AI Algorithms that help make intelligent decisions at the edge. This case study outlines Mistral’s expertise in designing an Edge Camera for Vision Analytics and AI/DL Applications with 4K@30 video capturing and integrated AI-Video analytics algorithms.

The Customer

The customer is a technology company offering smart windows for automatic in-house climate control and energy saving.

Customer Requirement

The customer had been exploring AI technologies to provide intelligence to the products they offer. The idea was to automatically control the environment inside a building, by regulating the natural light entering the room through the window glasses. The customer approached Mistral to develop an Edge Camera that comprises of essential sensors and AI Vision Analytics to accurately detect, cognize and interpret the environmental conditions of the room (including temperature and light) and automatically trigger commands that enable their product to ensure optimum climatic conditions in the room.

Solution Provided

Requirement Analysis

Mistral conducted a comprehensive requirement study to capture functional aspects of the camera. Based on the requirement analysis, Mistral proposed a design and development activity with the following scope,

• Hardware Design using NVIDIA Jetson Nano SoM as the processing core
• Drivers and BSP development with Linux as High-level OS
• OpenCV for Vision Analytics
• Integration of various sensors
• Camera tuning and sensor integration
• Proto build and supply

Hardware Design

The camera was designed around NVIDIA Jetson Nano, one of the smallest Artificial Intelligence (AI) platforms available in market, powered by Quad-core ARM A57 @ 1.43 GHz CPU and the 128-core Maxwell GPU. The Jetson Nano SoM comes with 4GB LPDDR4 memory, 16GB eMMC Flash and plenty on I/O options, including a MIPI CSI connector, 4 USB ports (1x USB 3.0 and 3 x USB 2.0), gigabit ethernet port, and 40 GPIO pins, which makes it a perfect platform for the edge camera.
Mistral custom designed a carrier board for Jetson Nano, which exercises all the features of the SoM. The carrier board provides I/O accessibility and power to the Jetson Nano. The carrier card is integrated with a 4K Camera Module, IR Thermal Sensor, UWB + Bluetooth + Accelerometer combo chip, Digital MIC and SD Card Slot, in addition to PoE and several interfaces.

The 4K Camera Module is interfaced with the Jetson Nano over 4-Lane CSI. The camera captures 4K@30 video for Vision Analytics. The IR Thermal Sensor is connected over I2C for real-time thermal measurements. The UWB + Bluetooth + Accelerometer chip is connected to the SoM over SPI (can be connected over UART as well) and configured as a real-time location system (RTLS) to aid real-time people tracking.

The platform is provided with an 802.3af Power over Ethernet (PoE) for data streaming and constant power supply and USB Ports for system programming & configuration. Though, the Jetson Nano offers 16GB eMMC Flash, however, Mistral provided an SD Card slot for additional storage and to run edge AI – ML applications. The design also includes a USB to UART converter for Host communication.

Software Design

Mistral ported Linux Kernel 4.9 (high-level OS) on the platform. The Linux Kernel was enabled with support for UWB + Bluetooth + Accelerometer chip, digital MIC, 4K camera sensor and an option for camera temperature measurement. Mistral developed Python codes for IR-sensor and 4K camera streaming using OpenCV and GStreamer. Mistral also developed and ported Linux kernel driver for IR sensor on the carrier card.

Booting of the platform was enabled over an SD-CARD based filesystem [Bootloaders & Kernel loaded from internal EMMC]. Mistral also developed a GPIO Kernel driver to control the reset of all the peripherals such as UWB transceiver IC with, IR-sensor & MIC.

Implementation of OpenCV

Mistral implemented OpenCV for advanced image processing and vision analytics within the Camera. OpenCV was tested for face detection using jetson-inference engine on the platform. OpenCV based Edge-ML Application for face mask detection & social distancing and Occupancy Detection Application for object detection and people counting were provided by the customer. Mistral ported these applications to custom designed carrier card and additionally modified the Occupancy Detection Application by adding filters for noise reduction and better performance.

Test Applications

Mistral created test utilities to,

(i) Monitor long run tests to observe current consumption & temperature throttling points
(ii) To set different power modes, observe total power consumption & performance
(iii) To perform all peripheral interface tests.

Read the full case study to know more about design challenges and how these challenges were addressed.