Please use this identifier to cite or link to this item:
http://hdl.handle.net/1942/25919Full metadata record
| DC Field | Value | Language |
|---|---|---|
| dc.contributor.author | LI, Yanzhe | - |
| dc.contributor.author | CLAESEN, Luc | - |
| dc.contributor.author | Huang, Kai | - |
| dc.contributor.author | Zhao, Menglian | - |
| dc.date.accessioned | 2018-04-16T15:04:41Z | - |
| dc.date.available | 2018-04-16T15:04:41Z | - |
| dc.date.issued | 2018 | - |
| dc.identifier.citation | IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 29 (4), p. 1179-1193 | - |
| dc.identifier.issn | 1051-8215 | - |
| dc.identifier.uri | http://hdl.handle.net/1942/25919 | - |
| dc.description.abstract | Depth image-based rendering (DIBR) techniques have drawn more attention in various three dimensional (3D) applications nowadays. In this paper, a real-time high-quality DIBR system which consists of disparity estimation and view synthesis is proposed. For disparity estimation, a local approach that focuses on depth discontinuities and disparity smoothness is presented to improve the disparity accuracy. For view synthesis, a method that contains view interpolation and extrapolation is proposed to render high-quality virtual views. Moreover, the system is designed with an optimized parallelism scheme to achieve a high throughput, and can be scaled up easily. It is implemented on an Altera Stratix IV FPGA at a processing speed of 45 frames per second (fps) for 1080p resolution. Evaluated on selected image sets of the Middlebury benchmark, the average error rate of the disparity maps is 6.02%; the average peak signal to noise ratio (PSNR) and structural similarity (SSIM) values of the virtual views are 30.07 dB and 0.9303, respectively. The experimental results indicate that the proposed DIBR system has the topperforming processing speed and its accuracy performance is among the best of state-of-the-art hardware implementations. | - |
| dc.description.sponsorship | This work was supported in part by the Belgian Flemish Research Council and the Chinese Ministry of Science and Technology bilateral cooperation under Project G.0524.13. This paper was recommended by Associate Editor S. Shirani. | - |
| dc.language.iso | en | - |
| dc.rights | IEEE copyright | - |
| dc.subject.other | multi-camera systems; FPGA; VLSI; System-on-Chip; SoC; 3D-TV; Stereo Vision; Image Processing; DIBR; disparity calculation | - |
| dc.title | A Real-Time High-Quality Complete System for Depth Image-Based Rendering on FPGA | - |
| dc.type | Journal Contribution | - |
| dc.identifier.epage | 1193 | - |
| dc.identifier.issue | 4 | - |
| dc.identifier.spage | 1179 | - |
| dc.identifier.volume | 29 | - |
| local.format.pages | 15 | - |
| local.bibliographicCitation.jcat | A1 | - |
| dc.description.notes | Huang, K (reprint author), Zhejiang Univ, Inst VLSI Design, Hangzhou 310058, Zhejiang, Peoples R China. liyz@vlsi.zju.edu.cn; luc.claesen@uhasselt.be; huangk@vlsi.zju.edu.cn; zhaoml@vlsi.zju.edu.cn | - |
| dc.relation.references | [1] N. A. Dodgson, “Autostereoscopic 3D displays,” Computer, vol. 38, no. 8, pp. 31–36, Aug 2005. [2] C. Fehn, “Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV,” in Proc. SPIE Conf. on Stereoscopic Displays and Virtual Reality Systems, vol. 5291, 2004, pp. 93–104. [3] S. F. Tsai, C. C. Cheng, C. T. Li, and L. G. Chen, “A real-time 1080p 2D-to-3D video conversion system,” IEEE Transactions on Consumer Electronics, vol. 57, pp. 915–922, May 2011. [4] http://vision.middlebury.edu/stereo/. [5] R. Szeliski, Computer Vision: Algorithms and Applications. Springer, 2010. [6] X. Xu, X. Xie, and Q. Dai, “Real-time 3D video synthesis from binocular stereo camera,” in 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video, May 2008, pp. 133–136. [7] C. Riechert, F. Zilly, P. Kauff, J. Guther, and R. Schafer, “Fully automatic stereo-to-multiview conversion in autostereoscopic displays,” in The best of IET and IBC, vol. 4, Sep 2012, pp. 8–14. [8] F. Seitner, M. Nezveda, M. Gelautz, G. Braun, C. Kapeller, W. Zellinger, and B. Moser, “Trifocal system for high-quality inter-camera mapping and virtual view synthesis,” in 2015 International Conference on 3D Imaging (IC3D), Dec 2015, pp. 1–8. [9] M. Dumont, Real-Time View Interpolation for Eye Gaze Corrected Video Conferencing. PhD thesis, Hasselt University, Belgium, 2015. [10] C.-K. Liao, H.-C. Yeh, K. Zhang, V. Geert, T.-S. Chang, and G. Lafruit, “Stereo matching and viewpoint synthesis FPGA implementation,” in 3D-TV System with Depth-Image-Based Rendering, New York, USA: Springer, 2013, pp. 69–106. [11] A. Akin, R. Capoccia, J. Narinx, J. Masur, A. Schmid, and Y. Leblebici, “Real-time free viewpoint synthesis using three-camera disparity estimation hardware,” in 2015 IEEE International Symposium on Circuits and Systems (ISCAS), May 2015, pp. 2525–2528. [12] L. Zhang, K. Zhang, T. S. Chang, G. Lafruit, G. K. Kuzmanov, and D. Verkest, “Real-time high-definition stereo matching on FPGA,” in Proceedings of the 19th ACM/SIGDA International Symposium on Field Programmable Gate Arrays. ACM, 2011, pp. 55–64. [13] Y. Shan, Y. Hao, W. Wang, Y. Wang, X. Chen, H. Yang, and W. Luk, “Hardware acceleration for an accurate stereo vision system using minicensus adaptive support region,” ACM Trans. Embed. Comput. Syst., vol. 13, no. 4s, pp. 132:1–132:24, Apr. 2014. [14] M. Jin and T. Maruyama, “Fast and accurate stereo vision system on FPGA,” ACM Trans. Reconfigurable Technol. Syst., vol. 7, no. 1, pp. 3:1–3:24, 2014. [15] W. Wang, J. Yan, N. Xu, Y. Wang, and F. H. Hsu, “Real-time highquality stereo vision system in FPGA,” IEEE Trans. Cir. and Sys. for Video Technol., vol. 25, no. 10, pp. 1696–1708, 2015. [16] Y. Li, C. Yang, W. Zhong, Z. Li, and S. Chen, “High throughput hardware architecture for accurate semi-global matching,” in 22nd Asia and South Pacific Design Automation Conference (ASP-DAC), Jan 2017, pp. 641–646. [17] H. J. Chen, F. H. Lo, F. C. Jan, and S. D. Wu, “Real-time multi-view rendering architecture for autostereoscopic displays,” in Proceedings of 2010 IEEE International Symposium on Circuits and Systems, May 2010, pp. 1165–1168. [18] Y. K. Lai, Y. C. Chung, and Y. F. Lai, “Hardware implementation for real-time 3D rendering in 2D-to-3D conversion,” in 2013 IEEE International Symposium on Circuits and Systems (ISCAS2013), May 2013, pp. 893–896. [19] P. F. Jin, S. J. Yao, D. X. Li, L. H. Wang, and M. Zhang, “Real-time multi-view rendering based on FPGA,” in 2012 International Conference on Systems and Informatics (ICSAI2012), May 2012, pp. 1981–1984. [20] J. Wang and L. A. Rønningen, “Real time believable stereo and virtual view synthesis engine for autostereoscopic display,” in 2012 International Conference on 3D Imaging (IC3D), Dec 2012, pp. 1–6. [21] D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” International Journal of Computer Vision, vol. 47, no. 1, pp. 7–42, 2002. [22] H. Hirschm¨uller and D. Scharstein, “Evaluation of stereo matching costs on images with radiometric differences,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 9, pp. 1582–1599, Sept 2009. [23] G. F¨uhr, G. P. Fickel, L. P. Dal’Aqua, C. R. Jung, T. Malzbender, and R. Samadani, “An evaluation of stereo matching methods for view interpolation,” in 2013 IEEE International Conference on Image Processing, Sept 2013, pp. 403–407. [24] N. Y.-C. Chang, T.-H. Tsai, B.-H. Hsu, Y.-C. Chen, and T.-S. Chang, “Algorithm and architecture of disparity estimation with mini-census adaptive support weight,” IEEE Trans. Cir. and Sys. for Video Technol., vol. 20, no. 6, pp. 792–805, Jun. 2010. [25] W. S. Fife and J. K. Archibald, “Improved census transforms for resource-optimized stereo vision,” IEEE Trans. Cir. and Sys. for Video Technol., vol. 23, no. 1, pp. 60–73, Jan 2013. [26] F. Tombari, S. Mattoccia, and L. Di Stefano, “Segmentation-based adaptive support for accurate stereo correspondence,” in Proceedings of the 2Nd Pacific Rim Conference on Advances in Image and Video Technology, 2007, pp. 427–438. [27] S. Rogmans, J. Lu, P. Bekaert, and G. Lafruit, “Real-time stereobased view synthesis algorithms: A unified framework and evaluation on commodity GPUs,” Signal Processing: Image Communication, vol. 24, no. 1, pp. 49 – 64, 2009. [28] C. V´azquez, W. J. Tam, and F. Speranza, “Stereoscopic imaging: filling disoccluded areas in depth image-based rendering,” in Proc. SPIE, vol. 6392, 2006, p. 63920D. [29] J. Overes, Occlusion filling in depth-image-based rendering. Master thesis, Delft University of Technology, The Netherlands, 2009. [30] K. Zhang, J. Lu, Q. Yang, G. Lafruit, R. Lauwereins, and L. V. Gool, “Real-time and accurate stereo: A scalable approach with bitwise fast voting on CUDA,” IEEE Trans. Cir. and Sys. for Video Technol., vol. 21, no. 7, pp. 867–878, July 2011. [31] M. A. Vega-Rodr´ıguez, J. M. S´anchez-P´erez, and J. A. G´omez-Pulido, “An FPGA-based implementation for median filter meeting the real-time requirements of automated visual inspection systems,” in Proceedings of the 10th Mediterranean Conference on Control and Automation, Lisbon, 2002. [32] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, April 2004. [33] C. Ttofis, C. Kyrkou, and T. Theocharides, “A low-cost real-time embedded stereo vision system for accurate disparity estimation based on guided image filtering,” IEEE Transactions on Computers, vol. 65, no. 9, pp. 2678–2693, Sept 2016. [34] C. Ttofis and T. Theocharides, “Towards accurate hardware stereo correspondence: A real-time FPGA implementation of a segmentationbased adaptive support weight algorithm,” in Proc. Design Autom. Test Eur. Conf. Exhibit. (DATE), 2012, pp. 703–708. [35] S. Jin, J. Cho, X. D. Pham, K. M. Lee, S. K. Park, M. Kim, and J. W. Jeon, “FPGA design and implementation of a real-time stereo vision system,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 20, no. 1, pp. 15–26, Jan 2010. [36] L. Zhang and W. J. Tam, “Stereoscopic image generation based on depth images for 3D TV,” IEEE Transactions on Broadcasting, vol. 51, no. 2, pp. 191–199, June 2005. [37] T. C. Yang, P. C. Kuo, B. D. Liu, and J. F. Yang, “Depth imagebased rendering with edge-oriented hole filling for multiview synthesis,” in 2013 International Conference on Communications, Circuits and Systems (ICCCAS), vol. 1, Nov 2013, pp. 50–53. [38] Z. Sun and C. Jung, “Real-time depth-image-based rendering on GPU,” in 2015 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery, Sept 2015, pp. 324–328. 15 [39] P. J. Lee and Effendi, “Nongeometric distortion smoothing approach for depth map preprocessing,” IEEE Transactions on Multimedia, vol. 13, no. 2, pp. 246–254, April 2011. [40] K.-H. Chen, C.-H. Chen, C.-H. Chang, J.-Y. Liu, and C.-L. Su, “A shape-adaptive low-complexity technique for 3D free-viewpoint visual applications,” Circuits, Systems, and Signal Processing, vol. 34, no. 2, pp. 579–604, Feb 2015. [41] P.-C. Kuo, J.-M. Lin, B.-D. Liu, and J.-F. Yang, “High efficiency depth image-based rendering with simplified inpainting-based hole filling,” Multidimensional Systems and Signal Processing, vol. 27, no. 3, pp. 623–645, Jul 2016. | - |
| local.type.refereed | Refereed | - |
| local.type.specified | Article | - |
| dc.identifier.doi | 10.1109/TCSVT.2018.2825022 | - |
| dc.identifier.isi | 000464149700020 | - |
| item.fulltext | With Fulltext | - |
| item.contributor | LI, Yanzhe | - |
| item.contributor | CLAESEN, Luc | - |
| item.contributor | Huang, Kai | - |
| item.contributor | Zhao, Menglian | - |
| item.accessRights | Restricted Access | - |
| item.validation | ecoom 2020 | - |
| item.fullcitation | LI, Yanzhe; CLAESEN, Luc; Huang, Kai & Zhao, Menglian (2018) A Real-Time High-Quality Complete System for Depth Image-Based Rendering on FPGA. In: IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 29 (4), p. 1179-1193. | - |
| crisitem.journal.issn | 1051-8215 | - |
| crisitem.journal.eissn | 1558-2205 | - |
| Appears in Collections: | Phd Theses Research publications | |
Files in This Item:
| File | Description | Size | Format | |
|---|---|---|---|---|
| A_Real-Time_High-Quality_Complete_System_for_Depth_Image-Based_Rendering_on_FPGA.pdf Restricted Access | Published version | 6.22 MB | Adobe PDF | View/Open Request a copy |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.