Streaming Cameras on Embedded Systems
This technical application note provides a summary and instructions for streaming Teledyne FLIR machine vision cameras using FlyCapture2 on ARM-based embedded boards. It includes examples of some of the more commonly used embedded boards: ODROID-XU; Samsung Exynos 5250 Arndale; and NVIDIA Jetson TK1, TX1, TX2 and DRIVE PX. The benchmark results show that embedded boards are able to support high-speed machine vision applications.
System Configuration
 |
These boards do not support Power over Ethernet (PoE), therefore all GigE models were powered externally via the GPIO. All GS3-U3 models were powered externally via the GPIO. The CM3 model was powered via the USB3 interface. |
| ODROID-XU Board Specification | |
| CPU | Exynos5 Octa Cortex™-A15 - 1.6 GHz quad core |
| GPU | PowerVR SGX544MP3 |
| RAM | 2 Gbyte LPDDR3 |
| Host Adaptor Driver | exynos-xhci |
| Operating System | Ubuntu 13.10 with 3.4.67 linux kernel |
| Software | FlyCap 2.6.3.3 Linux ARM |
| Camera and Firmware | GS3-U3-23S6M (firmware 2.3.3.0) |
| Samsung Exynos 5250 Arndale Specification | |
| CPU | Exynos5 Octa Cortex™-A15 - 1.6 GHz quad core |
| GPU | ARM Mali T-604 |
| RAM | 2 Gbyte DDR3 |
| Hard Drive | SanDisk Extreme II SSD 480GB (SDSSDXP-480G) |
| Hard Drive Enclosure | Oyen Digital MiniPro 2.5-inch eSATA 6 Gb/s, USB 3.1 hard drive enclosure |
| Host Adaptor Driver | xhci-hcd |
| Operating System | Arndale raring server release 13.09 with 3.12 Linux kernel |
| Software | FlyCap 2.6.3.3 Linux ARM |
| Camera and Firmware | GS3-U3-23S6M (firmware 2.3.3.0) |
| NVIDIA Jetson TK1 Specification | |
| CPU | Tegra K1 SOC |
| GPU | Kepler GPU with 192 CUDA cores |
| RAM | 2 Gbyte |
| Host Adaptor System | xhci-hcd |
| Operating System | Ubuntu 14.04 |
| Software | FlyCap 2.6.3.3 Linux ARM |
| Camera and Firmware | GS3-U3-23S6M (firmware 2.3.3.0) GS3-PGE-23S6C (firmware 1.6.3.0) FL3-GE-03S1M (firmware 1.26.3.0) |
| NVIDIA Jetson TX1 Specification | |
| CPU | Cortex A57 |
| GPU | Maxwell GPU with 256 CUDA cores |
| RAM | 4 Gbyte |
| Host Adaptor System | xhci-hcd |
| Operating System | Ubuntu 14.04 |
| Software | FlyCap 2.9.3.43 Linux ARM |
| Camera and Firmware | GS3-U3-23S6M (firmware 2.22.3.0) GS3-PGE-23S6M (firmware 1.10.3.0) |
| NVIDIA Jetson TX2 Specification | |
| CPU | Cortex A57 |
| GPU | Pascal GPU with 256 CUDA cores |
| RAM | 8 Gbyte |
| Host Adaptor System | xhci-hcd |
| Operating System | Ubuntu 16.04 |
| Software | FlyCap 2.11.3.121 Linux ARM |
| Camera and Firmware | GS3-U3-23S6M (firmware 2.25.3.0) GS3-PGE-23S6M (firmware 1.13.3.0) BFLY-PGE-23S6M (firmware 1.61.3.0) |
| NVIDIA DRIVE PX Specification | |
| CPU | Tegra X1 64-bit processor |
| GPU | Maxwell GPU with 256 CUDA cores |
| RAM | 4 Gbyte |
| Host Adaptor System | xhci-hcd |
| Operating System | Ubuntu 15.04 with 3.18.21-tegra-g06aec38 Linux NVIDIA kernel |
| Software | FlyCap 2.9.3.43 Linux ARM64 |
| Camera and Firmware | BFLY-PGE-50S5C-C (firmware 1.53.3.0) BFLY-PGE-23S6M-C (firmware 1.40.3.0) CM3-U3-50S5M-CS (firmware 1.10.3.0) |
| Note: To achieve maximum bandwidth: GigE cameras require a Gigabit adapter connected directly to an on-board USB port. USB3 cameras require a custom kernel as described in the NVIDIA DRIVE PX Getting Started. |
System Setup
ODROID-XU Getting Started
1. To boot from the microSD card, ensure boot media SW1-1 is set to OFF and SW2-2 is set to ON.
2. Download xUbuntu 13.10 image on a Linux PC running Ubuntu.
3. Unpack the files. For an .xz extension, use xz -d image.img.xz on cmd line.
4. Insert the microSD card in the Ubuntu PC.
Note: Windows users can use Windows tool Win32DiskImager.exe to write an image to the microSD card.
5. Clear the destination media by entering dd if=/dev/zero of=/dev/sdX bs=4M
Note: /dev/sdX where X denotes the microSD card’s file storage.
6. Flash your image: sudo dd if=image.img of=/dev/sdX bs=4M
7. Enter sync before removing the media to make sure that all data is written.
8. Download FlyCapture2 ARM hard float package (flycapture.2.6.3.2_armhf.tar.gz or newer).
9. Retrieve dependency using command sudo apt-get install libglademm-2.4-dev
10. Extract FlycCapture2 ARM hard float package and run flycap2-conf as root.
11. Reboot ODROID-XU.
12. In the terminal, change to FlyCapture2 ARM hard float directory’s bin folder.
13. Run the command export LD_LIBRARY_PATH=./../lib
14. Start FlyCapture2 using command ./FlyCap2
Samsung Exynos 5250 Arndale Getting Started
1. Download the Linaro image for Arndale. This configuration used Version 13.11.
2. Insert the microSD card in the Ubuntu PC.
3. Write the image to a microSD card.
Note: Windows users can use Windows tool Win32DiskImager.exe to write an image to the microSD card.
4. Download the FlyCapture2 ARM hard float package (flycapture.2.6.3.2_armhf.tar.gz or newer).
5. Extract the Flycapture2 ARM hard float package and run flycap2-conf as root.
6. Reboot.
7. In the terminal, change to Flycapture2 ARM hard float directory’s bin folder.
8. Run the command export LD_LIBRARY_PATH=./../lib
9. Start FlyCapture2 using command ./FlyCap2
NVIDIA Jetson TK1 Getting Started
|
Make sure you have at least 15 GB of free space on your Linux machine. This file copying process takes approximately one hour. The flashing process ends with 'Reset the board to boot from internal eMMC.' |
1. Follow NVIDIA’s readme file and boot up the pre-installed Ubuntu 14.04 on TK1.
2. By default, Jetson TK1 does not provide support for USB3 cameras. To enable USB3 support:
a. Go to https://developer.nvidia.com/linux-tegra-rel-19 and download both of these tar-balls:
• Driver Package: Jetson TK1
• Sample file system
b. On the Linux PC, unzip the tar files by running the command:
export RELEASE_NAME=Tegra124_Linux_R19.3.0_armhf.tbz2sudo tar xpf ${RELEASE_NAME}cd Linux_for_Tegra/rootfs/sudo tar xpf ../../Tegra_Linux_Sample-Root-Filesystem_R19.3.0_armhf.tbz2cd ../
Note: It’s important to use sudo to extract these files.
c. In the extracted Linux_for_Tegra folder, edit the jetson-tk1.conf file by uncomment the line (ODMDATA=0x6209C000) for USB2 and comment in the line for USB3 (ODMDATA=0x6009C000).
d. Enter sudo ./apply_binaries.sh
e. Connect the TK1 to the Linux machine using the micro-USB cable that comes with the TK1.
f. On the TK1, hold down the Recovery button and press the Reset button once. Wait two seconds and release the Recovery button.
g. Verify that the TK1 is connected to the Linux PC by issuing the lsusb command on the Linux PC and look for a connected usb device called NVIDIA Corp.
h. Copy the Linux_for_Tegra folder unzipped in step 2 to your Linux machine.
i. In the Linux machine, enter sudo ./flash.sh -S 14GiB jetson-tk1 mmcblk0p1
3. Reboot TK1 by pressing the Reset button.
The steps above will allow you to stream your USB3 camera at a maximum image size of 2 MB. To acquire images greater than 2 MB in resolution, add the following to the APPEND line:
usbcore.usbfs_memory_mb=1000
to this file:
/boot/extlinux/extlinux.conf
NVIDIA Jetson TX1 Getting Started
- Follow NVIDIA’s Quick Start Guide file to install the 64-bit Linux 4 Tegra R24.1 package.
Note: This step overwrites your existing file system. - Install 64-bit FlyCapture2 ARM by downloading the latest FlyCapture2 64-bit ARM and follow the readme file for installation instructions.
- Install the g++ compiler in order to compile the FlyCapture2 SDK’s C++ samples
sudo apt-get updatesudo apt-get install build-essential
The steps above allow you to stream your USB3 camera at a maximum image size of 2 MB. To acquire images greater than 2 MB in resolution, add the following to the APPEND line:
usbcore.usbfs_memory_mb=1000
to this file:
/boot/extlinux/extlinux.conf
NVIDIA Jetson TX2 Getting Started
- Follow NVIDIA’s Quick Start Guide file to install the 64-bit Linux 4 Tegra R24.1 package.
Note: This step overwrites your existing file system. - Install 64-bit FlyCapture2 ARM by downloading the latest FlyCapture2 64-bit ARM and follow the readme file for installation instructions.
The steps above allow you to stream your USB3 camera at a maximum image size of 2 MB. To acquire images greater than 2 MB in resolution, add the following to the APPEND line:
usbcore.usbfs_memory_mb=1000
to this file:
/boot/extlinux/extlinux.conf
NVIDIA DRIVE PX Getting Started
To obtain maximum bandwidth with DRIVE PX:
- USB3 cameras require the latest version of a custom kernel from GitHub, as described below.
- GigE cameras require a USB 3.1-to-Ethernet adapter connected directly to an on-board USB port.
For USB3 cameras:
- Get the latest kernel from GitHub:
# Clone source repo for mainline kernelgit clone https://github.com/torvalds/linux.git
# Get list of tags to determine appropriate release (I settled on v4.7.0-rc7 for specific technical reasons).git tag -l# Checkout specific tagged revision to local branch.# git checkout tags/<tag_name> -b <branch_name>git checkout tags/v4.7-rc7 -b nv_custom_branch - To acquire images greater than 2 MB in resolution, modify the os_args parameter of the KERNEL_PRIMARY partition in:
vibrante-<platform|ver>-foundation/utils/scripts/bootburn/<cfg_file>
Add the following to os_args:usbcore.usbfs_memory_mb=1000 -
After the kernel is built, flash the custom kernel onto the board. Because the process depends on the board and kernel version, consult the NVIDIA Vibrante Linux Development guide at /home/nvidia/PDK_Docs/ for specific instructions.
Note: You only want to re-flash the kernel, not the complete file system. To avoid flashing the complete file system, add the "-s" option (skip filesystem) to the bootburn.sh command:"./utils/scripts/bootburn/bootburn.sh -b e2379b00a -s"The sample in the documentation does not use the -s.
For GigE and USB3 cameras:
(Note: Connect the GigE camera to the PC with a USB-to-Ethernet adapter.)
- Install the g++ compiler in order to install the FlyCapture2 SDK's C++ samples:
sudo apt-get updatesudo apt-get install build-essential - Install 64-bit FlyCapture2 ARM by downloading the latest FlyCapture2 64-bit ARM and follow the README file for installation instructions.
Benchmark Results
The following tables benchmark the performance of running selected USB3 or GigE cameras on the following boards: ODROID-XU; Samsung Exynos 5250 Arndale, and NVIDIA Jetson TK1 sand TX1, and DRIVE PX.
ODROID-XU
These results are from a console application that continuously captures images using GS3-U3-23S6M. The benchmark using the console application is shown below.
Console application for ODROID-XU
| Requested Frame Rate | Processed Frame Rate | CPU Usage | Bandwidth | |
| 1280 x 960 Raw8 | 200 FPS | 200 FPS | 22% | 234 MB/s |
| 1792 x 960 Raw8 | 200 FPS | 200 FPS | 28% | 328 MB/s |
GS3-U3-23S6M benchmark results
| |
Currently, ODROID-XU officially supports up to Linux kernel 3.4.67. Expect the CPU usage on the ODROID-XU board to reduce by 50% when running Linux kernel 3.6 or higher. |
Samsung Exynos 5250 Arndale
Using FlyCapture2 to enumerate camera
These are the results from a console application that continuously captures images using GS3-U3-23S6M.
Console application for Arndale
This is a typical solution for an in-field system solution where the embedded board is portable and capable of performing basic processing as well as image/data saving. The benchmark results using the console application are shown below.
| Requested Frame Rate | Processed Frame Rate | CPU Usage | Bandwidth | Image Save Speed | |
| 1280 x 960 Raw8 | 200 FPS | 200 FPS | 24% | 234 MB/s | 190 MB/s |
| 1792 x 960 Raw8 | 200 FPS | 200 FPS | 28% | 328 MB/s | 170 MB/s |
| 1920 x 1200 Raw8 | 162 FPS | 162 FPS | 29% | 356 MB/s | 176 MB/s |
GS3-U3-23S6M benchmark results
GS3-U3-23S6M running at 1280 x 960 resolution, 24% CPU usage
GS3-U3-23S6M running at 1280 x 960 resolution, Raw8 pixel format, approximately 200 FPS
| |
The speed of image saving is dependent on the hard drive. Typical hard drive write speed is advertised with sequential write speed. When benchmarking hard drives, one should use a file size similar to the image size. |
GS3-U3-23S6M running at 1280 x 720 resolution, Raw8 pixel format, 24 FPS
NVIDIA Jetson TK1
These are the results from a console application that continuously captures images using GS3-U3-23S6M, GS-PGE-23S6C, and FL3-GE-03S1M. The resulting benchmark using the console application is shown below.
Console application for NVIDEA Jetson TK1
| Requested Frame Rate | Processed Frame Rate | CPU Usage | Bandwidth | |
| 1280 x 960 Raw8 |
200 FPS | 200 FPS | 24% | 234 MB/s |
| 1920 x 1200 Raw8 |
162 FPS | 162 FPS | 30% | 356 MB/s |
GS3-U3-23S6M benchmark results
| Requested Frame Rate | Processed Frame Rate | CPU Usage | Bandwidth | |
| 1920 x 1200 Raw8 |
45 FPS | 45 FPS | 25% | 103 MB/s |
GS3-PGE-23S6C benchmark results
| Requested Frame Rate | Processed Frame Rate | CPU Usage | Bandwidth | |
| 640 x 480 Raw8 |
120 FPS | 120 FPS | 24% | 36 MB/s |
Flea3-GE-03S1M benchmark results
NVIDIA Jetson TX1
These are the results from a console application that continuously captures images using Grasshopper3 GS3-U3-23S6M and Grasshopper3 GS-PGE-23S6M. The resulting benchmark using the console application is shown below. In our console application, we enabled maximum packet size supported on the Gigabit host controller, which is 9000 bytes.
Console application for NVIDEA Jetson TX1
| Requested Frame Rate | Processed Frame Rate | CPU Usage | Bandwidth | |
| 1280 x 960 Raw8 |
201 FPS | 201 FPS | 27% | 234 MB/s |
| 1920 x 1200 Raw8 |
163 FPS | 163 FPS | 27% | 356 MB/s |
GS3-U3-23S6M benchmark result using NVIDIA Jetson TX1
| Requested Frame Rate | Processed Frame Rate | CPU Usage | Bandwidth | |
| 1920 x 1200 Raw8 |
45 FPS | 45 FPS | 30% | 104 MB/s |
GS3-PGE-23S6M benchmark result using NVIDIA Jetson TX1
NVIDIA Jetson TX2
These are the results from a console application that continuously captures images using Grasshopper3 GS3-U3-23S6M, Grasshopper3 GS-PGE-23S6M, and Blackfly BFLY-PGE-23S6M. The resulting benchmark using the console application is shown below. In our console application, for GigE cameras we enabled maximum packet size supported on the Gigabit host controller, which is 9000 bytes.
Console application for NVIDEA Jetson TX1
| Requested Frame Rate | Processed Frame Rate | CPU Usage | Bandwidth | |
| 1920 x 1200 Raw8 |
163 FPS | 163 FPS | 25% | 375 MB/s |
GS3-U3-23S6M benchmark result using NVIDIA Jetson TX1
| Requested Frame Rate | Processed Frame Rate | CPU Usage | Bandwidth | |
| 1920 x 1200 Raw8 |
45 FPS | 46 FPS | 25% | 105 MB/s |
GS3-PGE-23S6M benchmark result using NVIDIA Jetson TX2
| Requested Frame Rate | Processed Frame Rate | CPU Usage | Bandwidth | |
| 1920 x 1200 Raw8 |
41 FPS | 41 FPS | 25% | 94 MB/s |
BFLY-PGE-23S6M benchmark result using NVIDIA Jetson TX2
NVIDIA DRIVE PX
These are the results from a console application that continuously captures images using BLFY-PGE-23S6M, BFLY-PGE-50SM-C and CM3-U3-50SM. The resulting benchmark using the console application is shown below.
Console application for NVIDEA DRIVE PX
| Requested Frame Rate | Processed Frame Rate | CPU Usage | Bandwidth | |
| 2448 x 2048 Mono16 |
18 FPS | 18 FPS | 80% | 85 MB/s |
BFLY-PGE-23S6M benchmark results
| Requested Frame Rate | Processed Frame Rate | CPU Usage | Bandwidth | |
| 2448 x 2048 Mono16 |
9 FPS | 9 FPS | 80% | 92 MB/s |
BFLY-PGE-5056M benchmark results
| Requested Frame Rate | Processed Frame Rate | CPU Usage | Bandwidth | |
| 1920 x 1200 Mono16 |
19 FPS | 19 FPS | 40% | 23 MB/s |
CM3-U3-5056M benchmark results
Troubleshooting
Image Transfer Fails to Start when Image Size is Bigger than 2 MB
Cause: The USBFS buffer size is too small (16 MB by default). Use the following command to check usbfs_memory_mb size.
cat /sys/module/usbcore/parameters/usbfs_memory_mb
Solution: Increase the memory by adding the following to the APPEND line:
usbcore.usbfs_memory_mb=1000
to this file:/boot/extlinux/extlinux.conf
If this method fails to set the memory limit, run the following command:sudo sh -c 'echo 1000 > /sys/module/usbcore/parameters/usbfs_memory_mb'
Camera is Detected but Cannot Stream
Cause: The USB3 bus power provided by the ARM board is not sufficient to power the camera.
Solution: Power the USB3 camera using a GPIO cable.
GigE Camera Frame Rates are extremely low on Jetson TK1, TX1, or TX2 board
Cause: The socket buffer size by default is too low.
Solution: Increase socket buffer size by typing in the following in Terminal:
sudo sysctl –w net.core.rmem_max=33554432sudo sysctl –w net.core.wmem_max=33554432sudo sysctl –w net.core.rmem_default=33554432sudo sysctl –w net.core.wmem_default=33554432