OpenMVG (open Multiple View Geometry): an open source multi-view stereo geometry library. This is a well-known open source library for handling multi-view stereo geometry in CV. It believes in simplicity and maintainability, and provides a powerful set of interface, each module has been tested and strives to provide a consistent and reliable experience.
Address: github
Documents: documents
openMVG can:
Solve the problem of accurate matching of multi-view stereo geometry;
Provide a series of feature extraction and matching methods needed for SfM;
Complete SfM tool chain (correction, estimation, reconstruction, surface treatment, etc.);
openMVG strives to provide highly readable code to facilitate secondary development by developers. The core functions are as streamlined as possible, so you may need other libraries to complete your system. openMVG is divided into several large modules:
Core library: core algorithm implementation of each function;
Sample: teach you how to use it;
Tool chain : That is to say, they can be used together (feature matching of disordered image sets, SfM, processing color and texture);
#0. Installation (win10 VS2013)
The first step is of course Clone the code from github, and then follow the BUILD instructions. What needs to be noted is:
template lt; class Tgt; inline T operator|(T x, T y){
return static_castlt; Tgt; (static_castlt; intgt; (x) | static_castlt; intgt; (y));
};
It is recommended to compile together with opencv, the method is in CMakeLists.txt Modify the corresponding option in the file to ON, and then add an entry called OpenCV_DIR in the cmake GUI. The value is the path to the opencv you have installed.
openMVG is very well written and provides good support for Windows, so after cmake, use VS to open the generated openMVG.sln solution and compile it. The compilation time is slightly longer. The VS2013 I use does not support the new C feature: constexpr, so it is recommended that you use VS2015 or a newer version. If you must use VS2013, you can do this: directly remove the place where constexpr is in the src/openMVG/cameras/Camera_Common.hpp file. , or you can change it to a template function:
When running the sample, you encounter a pit here: DenseStoraage.h line 86 error: R6010 Assertion failed, which is an assertion error and will not occur in release mode. appears, but it almost always appears in debug mode. The reason is clear by opening the URL given by Eigen: the data structure is unaligned arrays. This problem is still very annoying for people who are new to openMVG. The openMVG code is very elegant. Many data types are extended from template classes or functions, and various specific methods are extended through virtual functions. It also relies heavily on the Eigen library. Therefore, it brings obstacles to the positioning problem.
After a day of hard work, the reason was finally confirmed:
The Regions class defined in the file regions.h contains the stl container vector of fixed-size vectorizable Eigen types. According to the solution provided by Eigen, you need to do is:
//Original
typedef std:: vectorlt; FeatureTgt; FeatsT;
//Changed to
typedef std :: vectorlt; FeatureT, Eigen: : aligned_allocatorlt; FeatureTgt; gt; FeatsT;
//Other similar places must be changed, including the function that returns vector, it is best to add EIGEN_MAKE_ALIGNED_OPERATOR_NEW
#1. Core library
#1.1 Image
#Image Container
openMVG provides a basic class used as an image container: Imagelt; Tgt; , T represents the pixel type. This class can handle grayscale, RGB, RGBA or custom type images. Usage is very simple:
// A 8-bit gray image:
Imagelt; unsigned chargt; grayscale_image_8bit;
// Multichannel image: (use pre- defined pixel type)
// A 8-bit RGB image:
Imagelt;RGBColorgt; rgb_image_8bit;
// 8-bit RGBA image
Imagelt; RGBAColorgt; rgba_image_8bit;
Imagelt; Rgbalt; unsigned chargt; gt; rgba_image2_8bit;
The RGBColor, RGBAColor, etc. here are types defined by openMVG based on Eigen. It is specifically defined in pixel_types.hpp.
#Image I/O
openMVG supports image files in ppm/pgm, jpeg, png, and tiff formats, examples:
Imagelt; RGBColorgt; rgb_image_gray;
bool bRet = ReadImage("Foo.imgExtension", amp; rgb_image);
#Drawing operations
Used to draw circles, ellipses on images, Straight lines etc.
#1.2 Numerical values
openMVG repackages Eigen's basic types and algorithms for easier calling. For example, Vec2 represents a two-dimensional point (char type).
#1.3 Features
This module mainly provides the encapsulation of feature containers, including features, feature descriptions, key point sets, etc. They are basically template classes, such as a sift feature description You can do this:
// SIFT like descriptor
typedef Descriptorlt; float, 128gt; siftDescriptorData;
#1.4 Camera
This module provides abstract classes for different camera models, including:
#smallhole camera model
The simplest camera model, as shown in the figure: The camera model includes internal parameters and external parameters. The key The words are the familiar ones: projection matrix, rotation and translation matrix, focal length, principal point, etc. Please refer to the description for details. Look at an example: PinholeCamera class provided by openMVG:
/// Pinhole camera P = K[R|t], t = -RC
struct PinholeCamera
{
//Constructor
PinholeCamera(
const Mat3 amp; K = Mat3::Identity(),
const Mat3 & R = Mat3::Identity(),
const Vec3 & t = Vec3::Zero())
: _K(K), _R(R), _t(t)
{
_C = -R.transpose() * t;
P_From_KRt(_K, _R, _t, amp; _P);
}
PinholeCamera(const Mat34 amp; P)
{
_P = P;
KRt_From_P (_P, amp; _K, amp; _R, amp; _t);
_C = -_R.transpose() * _t;
}
/ // Projection matrix P = K[R|t]
Mat34 _P;
/// Intrinsic parameter (Focal, principal point)
Mat3 _K;
/// Extrinsic Rotation
Mat3 _R;
/// Extrinsic translation
Vec3 _t;
/// Camera center
Vec3 _C；
};
#1.5 Multi-view geometry
This part is a comparison of basic and One of the important modules includes:
Solution algorithm for n (gt; = 2) perspectives in multi-view set geometry;
Synthesize these solution algorithms for robust General framework for estimation - Kernel;
The document explains the concepts of homography matrix, eigenmatrix, essential matrix, position matrix, etc. It is very well explained. It is recommended to read the document carefully.
A brief explanation:
Homography matrix: describes the relationship between two projection planes;
Eigen matrix: the relationship between the same scene under imaging by two cameras, also It is the relationship between the imaging positions of point A on the object under two viewing angles;
Essential matrix: established based on the intrinsic matrix and internal parameter matrix, describing the relative relationship between the camera and the intrinsic matrix position;
p>
Position matrix: estimates the absolute position of the camera (converted into a minimization problem to solve);
Kernel: a class that combines solvers, data, measurement schemes, etc., this class Will be used to robustly estimate the above parameters and matrices;
#1.6 Linear Programming
A toolset for solving linear optimization (parameter estimation) in multi-view geometries, documentation .
#1.7 Robust estimation
Provides a series of robust estimation methods, such as: Max-Consensus, Max-Consensus, AC-Ransac A Contrario Ransac, etc.
#1.7 Matching
The interfaces provided include: NNS, K-NN, FLANN, KVLD, Cascade hashing Nearest Neighbor, etc. These interfaces can be used on two- or three-dimensional point sets, as well as higher-dimensional feature description sets.
#1.8 Tracking
Tracking in multi-view geometry refers to finding the corresponding feature points (the positions of the same point in different viewing angles) in a series of pictures.
#1.9 sfm
The sfm module provided by openMVG contains a series of methods and data storage interfaces for dealing with SfM problems, such as camera position estimation, structure measurement, BA, etc. The SfM_Data class contains all inputs to SfM:
struct SfM_Data
{
/// Considered views
Views views; // Contains image file name, id_view, id_pose, id_intrinsic, image size.
/// Considered poses (indexed by view.id_pose)
Poses poses; // Three-dimensional position of the camera
/// Considered camera intrinsics (indexed by view.id_cam)
Intrinsics intrinsics; // Camera internal parameters
/// Structure (3D points with their 2D observations)
Landmarks structure; // 3D points associated with 2D view features
}
The following is an example:
#1. features_siftPutativeMatches
This example does Here are a few things (direct official translation):
Extract the SIFT features of the two images (using the non-free vlsift) and form a feature description;
Match the two images according to the feature descriptor Feature points on the image (BRUTE_FORCE_L2 method);
Display the matching results;
You will encounter an Assertion failed assertion error at the beginning. See the previous article for how to deal with it. After successful operation, three files will be generated in the project directory: 00_images.jpg, 01_features.jpg, 02_siftMatches.svg.
#2. features_affine_demo
This example is a sample of image MSER (reference) and TBMR feature extraction. MSER (Maximally Stable Extremal Regions) maximum extreme value stable region is a pair Image grayscale is invariant to affine transformations, which may be why this example is named. I don’t know much about the TBMR (tree-based Morse regions) algorithm. You can search for it in Google Scholar.
#3. features_image_matching
This example gives an example of using the Image_describer interface to extract feature descriptors, and matching and displaying the results. In the example, you can choose SIFT, AKAZE_MLDB or AKAZE algorithm. An introduction to AKAZE can be found here. It is a more stable feature detection algorithm than SIFT. The part about parsing input parameters in the program can be commented out (if you have used VS2013 debug), and directly modify the value of sImage_describer_type for testing.
#4. features_kvld_filter and features_repeatability
These two are also about feature extraction and matching. In the kvld example, due to the previous changes to the content in regions.h, some function interfaces Corresponding changes must also be made, specifically adding alignment function options to the vector.
#5. multiview_robust_homography_guided and multiview_robust_fundamental_guided
These two examples estimate the homography matrix and the eigenmatrix, and can in turn determine the matching points based on this information. Both samples took a long time to run (when the resolution was high), and the first one encountered an error when using another photo. It was probably an error during DoG, and I didn’t go into details...
#6. exif_Parsing
Extract EXIF ??information, compile it and execute it through the command line. The given parameter format is: --imafile Your photo path, use / slash in the path.
Another article: learn openMVG-Installation and Introduction
#7. multiview_robust_essential
Estimate the essential matrix and calculate the 3D structure. This example can be run directly, and the generated point cloud is very sparse and has no color information.