Siamese Neural Network

Siamese neural networks, also known as twin neural networks, use shared weights to compare two input vectors and compute comparable outputs, with applications in face recognition, handwriting recognition, and text matching. The networks are trained using triplet loss or contrastive loss, with Euclidean distance as a common metric. DeepFace is a notable example of a face recognition system based on this architecture.

Siamese neural network A Siamese neural network sometimes called a twin neural network is an artificial neural network https://en.wikipedia.org/wiki/Artificial neural network that uses the same weights while working in tandem on two different input vectors to compute comparable output vectors. 1 cite note-1 2 cite note-2 3 Often one of the output vectors is precomputed, thus forming a baseline against which the other output vector is compared. This is similar to comparing fingerprints https://en.wikipedia.org/wiki/Fingerprint but can be described more technically as a distance function for locality-sensitive hashing https://en.wikipedia.org/wiki/Locality-sensitive hashing . citation needed https://en.wikipedia.org/wiki/Wikipedia:Citation needed It is possible to build an architecture that is functionally similar to a twin network but implements a slightly different function. This is typically used for comparing similar instances in different type sets. citation needed https://en.wikipedia.org/wiki/Wikipedia:Citation needed Uses of similarity measures where a twin network might be used are such things as recognizing handwritten https://en.wikipedia.org/wiki/Handwriting recognition checks, automatic detection of faces https://en.wikipedia.org/wiki/Face detection in camera images, and matching text queries with indexed documents 4 . The perhaps most well-known application of twin networks are face recognition https://en.wikipedia.org/wiki/Face recognition , where known images of people are precomputed and compared to an image from a turnstile or similar. It is not obvious at first, but there are two slightly different problems. One is recognizing a person among a large number of other persons, that is the facial recognition problem. DeepFace https://en.wikipedia.org/wiki/DeepFace is an example of such a system. In its most extreme form this is recognizing a single person at a train station or airport. The other is 3 cite note-:0-3 face verification https://en.wikipedia.org/wiki/Face verification , that is for example, to verify whether a photo in a passport matches the face of the passport's owner. The twin network might be the same, but the implementation can be quite different. Learning edit /w/index.php?title=Siamese neural network&action=edit§ion=1 Learning in twin networks can be done with triplet loss https://en.wikipedia.org/wiki/Triplet loss or contrastive loss https://en.wikipedia.org/wiki/Contrastive loss . For learning by triplet loss a baseline vector anchor image is compared against a positive vector truthy image and a negative vector falsy image . The negative vector will force learning in the network, while the positive vector will act like a regularizer. For learning by contrastive loss there must be a weight decay to regularize the weights, or some similar operation like a normalization. A distance metric for a loss function may have the following properties 5 cite note-5 - Non-negativity: - Identity of Non-discernibles: - Commutativity: Triangle inequality https://en.wikipedia.org/wiki/Triangle inequality : In particular, the triplet loss algorithm is often defined with squared Euclidean which unlike Euclidean, does not have triangle inequality distance at its core. Predefined metrics, Euclidean distance metric edit /w/index.php?title=Siamese neural network&action=edit§ion=2 The common learning goal is to minimize a distance metric for similar objects and maximize for distinct ones. This gives a loss function like - are indexes into a set of vectors - function implemented by the twin network The most common distance metric used is Euclidean distance https://en.wikipedia.org/wiki/Euclidean distance , in case of which the loss function can be rewritten in matrix form as Learned metrics, nonlinear distance metric edit /w/index.php?title=Siamese neural network&action=edit§ion=3 A more general case is where the output vector from the twin network is passed through additional network layers implementing non-linear distance metrics. - are indexes into a set of vectors - function implemented by the twin network - function implemented by the network joining outputs from the twin network On a matrix form the previous is often approximated as a Mahalanobis distance https://en.wikipedia.org/wiki/Mahalanobis distance for a linear space as 6 cite note-6 This can be further subdivided in at least Unsupervised learning https://en.wikipedia.org/wiki/Unsupervised learning and Supervised learning https://en.wikipedia.org/wiki/Supervised learning . Learned metrics, half-twin networks edit /w/index.php?title=Siamese neural network&action=edit§ion=4 This form also allows the twin network to be more of a half-twin, implementing a slightly different functions - are indexes into a set of vectors - function implemented by the half-twin network - function implemented by the network joining outputs from the twin network Twin networks for object tracking edit /w/index.php?title=Siamese neural network&action=edit§ion=5 Twin networks have been used in object tracking because of its unique two tandem inputs and similarity measurement. In object tracking, one input of the twin network is user pre-selected exemplar image, the other input is a larger search image. The twin network's job is to locate the exemplar inside of the search image. By measuring the similarity between exemplar and each part of the search image, a map of similarity score can be given by the twin network. Furthermore, using a Fully Convolutional Network, the process of computing each sector's similarity score can be replaced with only one cross correlation layer. 7 cite note-7 After being first introduced in 2016, Twin fully convolutional network has been used in many High-performance Real-time Object Tracking Neural Networks. Like CFnet, 8 StructSiam, SiamFC-tri, 9 cite note-9 DSiam, 10 cite note-10 SA-Siam, 11 cite note-11 SiamRPN, 12 cite note-12 DaSiamRPN, 13 cite note-13 Cascaded SiamRPN, 14 cite note-14 SiamMask, 15 cite note-15 SiamRPN++, 16 cite note-16 Deeper and Wider SiamRPN. 17 cite note-17 18 cite note-18 See also edit /w/index.php?title=Siamese neural network&action=edit§ion=6 References edit /w/index.php?title=Siamese neural network&action=edit§ion=7 ↑ cite ref-1 Bromley, Jane; Guyon, Isabelle; LeCun, Yann; Säckinger, Eduard; Shah, Roopak 1994 . "Signature verification using a "Siamese" time delay neural network" https://papers.neurips.cc/paper files/paper/1993/file/288cc0ff022877bd3df94bc9360b9c5d-Paper.pdf PDF . Advances in Neural Information Processing Systems . 6 : 737–744. ↑ cite ref-2 Chopra, S.; Hadsell, R.; LeCun, Y. June 2005 . "Learning a Similarity Metric Discriminatively, with Application to Face Verification". 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR'05 . Vol. 1. pp. 539–546 vol. 1. doi https://en.wikipedia.org/wiki/Doi identifier : 10.1109/CVPR.2005.202 https://doi.org/10.1109%2FCVPR.2005.202 . ISBN https://en.wikipedia.org/wiki/ISBN identifier 0-7695-2372-2 https://en.wikipedia.org/wiki/Special:BookSources/0-7695-2372-2 . S2CID https://en.wikipedia.org/wiki/S2CID identifier 5555257 https://api.semanticscholar.org/CorpusID:5555257 . 1 cite ref-:0 3-0 2 cite ref-:0 3-1 Taigman, Y.; Yang, M.; Ranzato, M.; Wolf, L. June 2014 . "DeepFace: Closing the Gap to Human-Level Performance in Face Verification". 2014 IEEE Conference on Computer Vision and Pattern Recognition . pp. 1701–1708. doi https://en.wikipedia.org/wiki/Doi identifier : 10.1109/CVPR.2014.220 https://doi.org/10.1109%2FCVPR.2014.220 . ISBN https://en.wikipedia.org/wiki/ISBN identifier 978-1-4799-5118-5 https://en.wikipedia.org/wiki/Special:BookSources/978-1-4799-5118-5 . S2CID https://en.wikipedia.org/wiki/S2CID identifier 2814088 https://api.semanticscholar.org/CorpusID:2814088 . ↑ cite ref-4 "Siamese Recurrent Architectures for Learning Sentence Similarity" https://ojs.aaai.org/index.php/AAAI/article/view/10350 . ↑ cite ref-5 Chatterjee, Moitreya; Luo, Yunan. "Similarity Learning with or without Convolutional Neural Network" http://slazebni.cs.illinois.edu/spring17/lec09 similarity.pdf PDF . Retrieved 2018-12-07. ↑ cite ref-6 Chandra, M.P. 1936 . "On the generalized distance in statistics" http://library.isical.ac.in:8080/jspui/bitstream/123456789/6765/1/Vol02 1936 1 Art05-pcm.pdf PDF . Proceedings of the National Institute of Sciences of India . 1. 2 : 49–55. ↑ cite ref-7 Fully-Convolutional Siamese Networks for Object Tracking arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 1606.09549 https://arxiv.org/abs/1606.09549 ↑ cite ref-8 "End-to-end representation learning for Correlation Filter based tracking" https://www.robots.ox.ac.uk/~luca/cfnet.html . ↑ cite ref-9 "Structured Siamese Network for Real-Time Visual Tracking" http://openaccess.thecvf.com/content ECCV 2018/papers/Yunhua Zhang Structured Siamese Network ECCV 2018 paper.pdf PDF . ↑ cite ref-10 "Triplet Loss in Siamese Network for Object Tracking" http://openaccess.thecvf.com/content ECCV 2018/papers/Xingping Dong Triplet Loss with ECCV 2018 paper.pdf PDF . ↑ cite ref-11 "Learning Dynamic Siamese Network for Visual Object Tracking" http://openaccess.thecvf.com/content ICCV 2017/papers/Guo Learning Dynamic Siamese ICCV 2017 paper.pdf PDF . ↑ cite ref-12 "A Twofold Siamese Network for Real-Time Object Tracking" http://openaccess.thecvf.com/content cvpr 2018/papers/He A Twofold Siamese CVPR 2018 paper.pdf PDF . ↑ cite ref-13 "High Performance Visual Tracking with Siamese Region Proposal Network" http://openaccess.thecvf.com/content cvpr 2018/papers/Li High Performance Visual CVPR 2018 paper.pdf PDF . ↑ cite ref-14 Zhu, Zheng; Wang, Qiang; Li, Bo; Wu, Wei; Yan, Junjie; Hu, Weiming 2018 . "Distractor-aware Siamese Networks for Visual Object Tracking". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 1808.06048 https://arxiv.org/abs/1808.06048 cs.CV https://arxiv.org/archive/cs.CV . ↑ cite ref-15 Fan, Heng; Ling, Haibin 2018 . "Siamese Cascaded Region Proposal Networks for Real-Time Visual Tracking". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 1812.06148 https://arxiv.org/abs/1812.06148 cs.CV https://arxiv.org/archive/cs.CV . ↑ cite ref-16 Wang, Qiang; Zhang, Li; Bertinetto, Luca; Hu, Weiming; Torr, Philip H. S. 2018 . "Fast Online Object Tracking and Segmentation: A Unifying Approach". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 1812.05050 https://arxiv.org/abs/1812.05050 cs.CV https://arxiv.org/archive/cs.CV . ↑ cite ref-17 Li, Bo; Wu, Wei; Wang, Qiang; Zhang, Fangyi; Xing, Junliang; Yan, Junjie 2018 . "SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 1812.11703 https://arxiv.org/abs/1812.11703 cs.CV https://arxiv.org/archive/cs.CV . ↑ cite ref-18 Zhang, Zhipeng; Peng, Houwen 2019 . "Deeper and Wider Siamese Networks for Real-Time Visual Tracking". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 1901.01660 https://arxiv.org/abs/1901.01660 cs.CV https://arxiv.org/archive/cs.CV .