Creating OCF based RealSense applications using NodeJS

By wanghj
Access the Open Connectivity Framework and Intel® RealSense™ Technology all from JavaScript!
Level: Beginner
1h
Since JavaScript becomes one of the most popular programming language, Intel Open Source Technology Center (OTC) Web team devotes to adding JavaScript APIs from native C/C++ etc., API for more web developers. They can use JavaScript to provide a fast, smooth, and pleasant developers from the out-of-box experience, to a working prototype, all the way to production and deployment. This article illustrates how to create applications using Open Connectivity Foundation™ (OCF) JavaScript API and Intel® RealSense™ JavaScript API.
RealSense
OCF
JavaScript
minnowboard

Introduction

This article illustrates how to create applications using Open Connectivity Foundation™ (OCF) JavaScript API and Intel® RealSense™ JavaScript API with following features:

  • Controlling LED light by distance: Use Person Tracking based on RealSense SDK with Intel® RealSense™ 3D Camera ZR300 to detect people with distance information to control LED light color according to changes of person position.
  • Controlling LED light/buzzer by person recognition: Identify persons with different LED colors and buzzer status with ZR300 camera.

Since JavaScript becomes one of the most popular programming language, Intel Open Source Technology Center (OTC) Web team devotes to adding JavaScript APIs from native C/C++ etc., API for more web developers. They can use JavaScript to provide a fast, smooth, and pleasant developers from the out-of-box experience, to a working prototype, all the way to production and deployment.

Node.js, OCF and RealSense JavaScript Technologies

Node.js is a JavaScript runtime built on the Google V8 JavaScript engine. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient.

The OCF is dedicated to ensuring secure interoperability in Internet of Things (IoT) for consumers, businesses and industries by delivering a standard communications platform, a bridging specification, an open source implementation - IoTivity and a certification program allowing devices to communicate regardless of form factor, operating system, service provider, transport technology or ecosystem. OCF lets you customize and control devices, help keep your home safe, make home life easier, more distinct, and more enjoyable.

Intel RealSense technology is fundamentally re-shaping the future of technology by equipping devices with the ability to see, understand, interact with, and learn from their environment. The RealSense with depth perception technology that enable you to create new and innovative ways to connect with the world.

Architecture

Intel OTC Web team is developing several open source projects to extend the use of the IoTivity and RealSense technology:

  • iotivity-node provides the node binding for IoTivity, so developers can fast prototype the OCF devices using Javascript
  • The IoT REST API server enables cloud services to access resources on the OCF network, etc.
  • node-realsense offers following Node.js modules of Intel RealSense technology which have been registered as NPM packages:

Precondition

It requires Node.js environment with Ubuntu 16.04.1 LTS on Minnowboard Turbot and PC, set Linux and NPM proxy if you're behind a proxy.

  1. Please refer to this guidance to install Ubuntu 16.04 LTS on MinnowBoard
  2. Install Node.js, v6.11.2 is preferred.

The MinnowBoard Turbot will be configured as an OCF server, managing the connection to two sensors, and exposing them via the OCF rest api.

Once you have Ubuntu OS running on MinnowBoard Turbot, you need to install some additional packages:

  • mraa
  • Iotivity
  • iotivity-node
  • iot-rest-api-server

Install dependencies:

copy
$ $ $
sudo apt install uuid-dev libcurl4-openssl-dev libboost-all-dev sqlite3 libsqlite3-dev glib2.0-dev scons git npm install mraa npm install iot-rest-api-server@0.6.0

Confirm the setup is ready by using Node.js in interactive mode:

copy
# The module mraa requires root access sudo node > var mraa = require('mraa') undefined > var pin = new mraa.Gpio(12) undefined > var ocf = require('iotivity-node') undefined > .exit

Congratulations! You just ran your environment in the Node.js interactive mode, the OCF Server test environment is ready now. Let's launch your first OCF Server.

Establish connection between sensors and MinnowBoard Turbot

Please refer to MinnowBoard Turbot pin number. The mapping sensors are derived from the corresponding .js files( means buzzer.js, rgb_led.js), please refer to this table to connect sensors.

Sensor Sensor Pin MinnowBoard Pin Notes
Buzzer GND GND Pin 2 Grove Buzzer
VCC 3V Pin 4
SIG GPIO Pin 12 GPIO used for signal
RGB LED GND GND Pin 1 Grove Chainable RGB LED
VCC 5V Pin 3
DI GPIO Pin 14 GPIO used for serial data
CI GPIO Pin 16 GPIO used for serial clock

Mount RGB LED and buzzer with MinnowBoard Turbot as shown in the following picture:

Known issue: Once power on board, buzzer sensor will turn on before launch the sensor server file

Launch OCF Server scripts and discover server resources via iot-rest-api-server

Launch sensor servers

Run below commands to launch all sensor servers.

copy
$ $ $ $ $ $
cd ~ git clone https://github.com/01org/node-realsense.git cd node-realsense/samples/ocf-realsense/ocf-servers npm install export NODE_DEBUG=rgb_led sudo -E node rgb_led.js &

The above should display the following:

copy
$ $
export NODE_DEBUG=buzzer sudo -E node buzzer.js &

Discover LGB LED and buzzer resources

Open a new terminal, launch iot-rest-api-server and discover the running RGB LED and buzzer resources.

copy
$ $
cd ~/node-realsense/../node_modules/iot-rest-api-server node index.js &

Verify there is no error message, and show message like:

copy
API Server started on 2017-09-12T09:10:38.674Z [port:8000, https: false] )

Now we can discover the resources:

copy
$
./test/oic-get /res

The output should contain below information:

The hrefs "/a/buzzer" and "/a/rgbled" should be included in the output.

The following will get and set the value of rgbled. First it determines the device id for the rgbled, and then use that device id to get and set the rgbled.

copy
$ $ $
export DEVICE_ID=`./test/oic-get /res |awk -F '"' '{for(;i++<NF;){tmp=match($i, "/a/rgbled");if(tmp){print $(i-6);break;}}}'` ./test/oic-get /a/rgbled?di=${DEVICE_ID} ./test/oic-post /a/rgbled?di=${DEVICE_ID} ../../node-realsense/samples/ocf-realsense/demo1/test/red.json

Determine hostname and IP address of MinnowBoard Turbot

In order for the client system (the PC) to connect to the MinnowBoard Turbot, the PC will need to know the hostname or IP address of the MinnowBoard Turbot. You can determine this information from the command line. On the MinnowBoard Turbot run the following:

copy
$
nslookup $(hostname) |grep Address |cut -d ":" -f2 |tail -1

You should see something like:

$ nslookup $(hostname) |grep Address |cut -d ":" -f2 |tail -1 192.168.1.23

This will be used on the PC for the value for API_SERVER_HOST later.

The PC will communicate with the Intel RealSense camera and perform facial recognition. The PC will use OCF to communicate with the sensors exposed by that MinnowBoard Turbot system.

Make sure your PC has installed Ubuntu 16.04, connect ZR300 with USB 3.0 and install Intel RealSense SDK for Linux, start to run these applications when SDK is installed.

Install Intel RealSense SDK for Linux

Add RealSense repository, open terminal and run commands

copy
$ $ $ $ $
echo 'deb "http://realsense-alm-public.s3.amazonaws.com/apt-repo" xenial main' | sudo tee /etc/apt/sources.list.d/realsense-latest.list sudo apt-key adv --keyserver keys.gnupg.net --recv-key D6FB2970 sudo apt update sudo apt install git

Install SDK runtime, Person Tracking, Object Library and SLAM samples:

copy
$
sudo apt install librealsense-samples beignet-opencl-icd

Test ZR300 Camera

Plug in ZR300 Camera and install camera access to determine if your camera is functioning correctly.

copy
$ $
sudo apt install librealsense-utils cpp-capture

Install RealSense SDK to develop your own RealSense applications

copy
$
sudo apt install librealsense-object-recognition-dev librealsense-persontracking-dev librealsense-slam-dev

Start Run commands to start applications:

In the following, you will need to set the API_SERVER_HOST to the IP address of the MinnowBoard Turbot:

copy
$
export API_SERVER_HOST=192.168.1.13

NOTE: Replace 192.168.1.13 with the MinnowBoard Turbot's IP address determined previously.

Controlling LED light by distance

This app displays live color preview from camera within a browser and draw red rectangles around the person(s) detected in the camera frame with a color dot indicating the center of mass for the detected persons in the frame.

It also displays the Person ID (pid) for every person and the distance from person to camera. Upon the distance, app could control the light turn on/off automatically.

When people in the 0.5 ~ 1 meter from camera, light will be on blue, and when people move to 1 ~ 1.5 meter from camera light will be on red. The expected of light color should be as below:

Distance(Meter) Expected Behavior
0.5 ~ 1.0 Blue
1.0 ~ 1.5 Red
1.5 ~ 2.0 White
2.0 ~ 2.5 Green

Start to run:

copy
$ $ $
cd demo1 npm install node main.js

The above Node.JS application provides a web service on port 8000 that allows you to see the results. You can connect to the server locally:

copy
$
xdg-open http://127.0.0.1:8000/view.html

or connect to it from another system on the same network via your PC's IP address.

Known issue:

This application doesn't support recognized multi-user at the same time as it can't control multi-led which using same resourceType at present.

Controlling LED light/buzzer by person recognition:

This app illustrates how to register new users to the database and identify them when they appear in the scene. It displays the live color preview from the camera within a browser and draw red rectangles around the person(s) detected in the camera frame with a color dot indicating the center of mass for the detected persons in the frame.

It also displays the Person ID (pid) for every person detected which is temporary id for the person. Mouse click in any of the detected person's rectangle, activates registration for that person and saves the information in a database (database name currently hardcode) with a Recognition ID (rid) assigned to that person, when this operation executes correctly, the color of rectangle around the person will turn to green.

Clicking on the load database button, loads the database that has information about the recognized persons, displays a green rectangle and the rid of already registered person when they are present in the FOV.

Start to run:

copy
$ $ $
cd ../demo2 npm install node main.js

Person can be recognized and the recognition database can be stored when clicking the person to track him/her. The stored database can also be loaded.

The above Node.JS application provides a web service on port 8000 that allows you to see the results. You can connect to the server locally:

copy
$
xdg-open http://127.0.0.1:8000/view.html

or connect to it from another system on the same network via your PC's IP address.

Adding faces to the database as 'recognized'

When person is not recognized, light is on red and buzzer turns on:

Click person's box on the screen, the person will be recognized and store in database. The light will be on green. (If you want to clear the database, remove DB file ./demo2/person_recognition_id.db.)

Based on OCF framework and RealSense for implementing human-computer interaction techniques, setting up a smart home or connecting an entertainment center to IoT devices can be simple, secure and hassle-free.

There are thousands of Node.js packages that you can use to build IoT applications. The IoT Node.js packages described in this article is a typical example that incorporates these technologies. IoT and Web developers can quickly turn their ideas into working prototypes. After all in the world of IoT, the speed of innovation is the key to success. Intel OTC Web Team also contributes to leading initiatives such as WebGL, WebVR, Sensors, Robotics, Zephyr.js, NW.js and others that help develop and deploy intelligent services.

Ingredients

Others
MinnowBoard Turbot
A powerful PC with an extendable IO interface. https://minnowboard.org
Intel RealSense Camera ZR300
https://newsroom.intel.com/chip-shots/intel-announces-tools-realsense-technology-development/
Grove Chainable RGB LED
http://www.seeedstudio.com/depot/twig-chainable-rgb-led-p-850.html?cPath=156_157
Grove Buzzer
http://wiki.seeed.cc/Grove-Buzzer/
Desktop/Laptop with USB3.0, running Ubuntu 16.04LTS

Steps

1 total steps
1
DJ Injection