Vision configuration¶
The vision component is composed of a set of independent modules, each responsible for one particular vision task (person detection, face recognition, QR code recognition, etc.). The implementation is located in the sparc/external_platform/vision folder, with the modules distributed in the following folder hierarchy :
- facenet - contains FfaceNnet network for face recognition and Haar cascades for face detection
- image_provider - contains the script that gets the stream of images from the robot
- qrcodes_handler - contains the script that generates, detects and recognizes QR codes
- segmentation - contains Resnet-18-8s network for person segmentation
- tracking - contains implementation of SORT algorithm for object tracking
- yolo2 - contains YOLOv2 network for object detection
- data_processor.py - auxiliary file for processing data that comes from the vision modules
- vision_manager.py - file for combining all the modules’ outputs into a single vision result
- run_vision.py - main file for running the vision component as a stand-alone application
Software Prerequisites¶
To test the vision component both as a stand-alone application or within the big project, the machine must install all the following required packages:
- Cython (version >=0.27.3)
- H5py (version >= 2.7.1)
- Matplotlib
- Naoqi (version >=2.5)
- NumPy
- OpenCV-Python (version >=3.3.0.10)
- Python (version 2.7)
- Pillow
- Psutil
- PyCrayon (version >=0.5)
- PyTorch
- PyZBar
- Requests
- TensorFlow (version >=1.2)
- Scikit-image
- Scikit-learn
- SciPy
Testing the vision component¶
python run_vision.py [-r] [-v]
The script gets input images from a video camera or the robot’s camera and displays 3 images:
- RGB image - input image with overlapped detections
- Blue bbox - detected person. The text above represents the person ID, accuracy of detection and distance to the person
- Yellow bbox - detected object. The text above represents the object name, accuracy of detection and distance to the object
- Pink bbox - detected QR code. The text above represents the QR code ID, accuracy of detection and distance to the QR code
- Green/Red bbox - recognized/unrecognized face. The text above represents the person name and accuracy of recognition
- Depth image - depth image with overlapped bboxes representing detected people
- Segmented image - black and white image, where the white pixels represent the segmented detected person tensorflow==1.2 Scipy scikit-learn matplotlib Pillowrequests Psutil
Numpy
Scikit-image
pyzbar
Naoqi 2.5.5 (if robot input required)
Testing the vision component¶
python run_vision.py [-r] [-v]
The script gets input images from a video camera or the robot’s camera and displays 3 images:
- List item
Optional arguments: -r, –robot_stream = use the input from the robot’s camera. By default the script uses the video camera of the machine. -v, –verbose = display information about execution time.
Fine-tuning the face recognition module¶
The input directory that contains the training images to fine-tune the pre-trained model are in sparc/external_platform/vision/facenet/input_dir.
The pre-trained model is located in sparc/external_platform/vision/facenet/pre_model, while the fine-tuned model is located in sparc/external_platform/vision/facenet/classifier.
- Adding a new face to the database
- Add a new folder containing a set of images of the new face in sparc/external_platform/vision/facenet/input_dir. The folder name should match the person name.
- Align the face dataset using python aligndata_first.py`
- Retrain the last layer of the pre-trained model using
python create_classifier_se.py
- Removing existing faces
- Delete the folder associated with the face to be removed from sparc/external_platform/vision/facenet/input_dir
- Retrain the last layer of the pre-trained model using
“python create_classifier_se.py
Generate new QR codes¶
The images representing the existing QR codes are located in sparc/external_platform/vision/qrcodes_handler/qr_codes folder. To generate new QR codes:
Delete existing images from qrcodes_handler/qr_codes
Modify the generate_QRcodes function in qrcodes_handler/qrcodes_handler.py and add each new QR code as follows:
new_qr_code = pyqrcode.create(‘new_qr_code_name’) new_qr_code.png(‘./qr_codes/new_qr_code_image.png’, scale=20)
Run
python qrcodes_handler.py