超分辨率圖像重建方法綜述

蘇衡; 周杰; 張志浩

doi:10.3724/SP.J.1004.2013.01202

超分辨率圖像重建方法綜述

doi: 10.3724/SP.J.1004.2013.01202

蘇衡^1,2,
周杰¹,
張志浩¹

1.
清華大學(xué)自動(dòng)化系北京 100084;
2.
北京葫蘆軟件技術(shù)開(kāi)發(fā)有限公司北京 100084

基金項目:

國家自然科學(xué)基金重大國際(地區)合作研究項目(61020106004);國家自然科學(xué)基金(61005023, 61021063);國家杰出青年科學(xué)基金項目(61225008);教育部博士點(diǎn)基金(20120002110033) 資助

詳細信息

作者簡(jiǎn)介:
蘇衡 2012 年獲清華大學(xué)自動(dòng)化系博士學(xué)位. 現就職于北京葫蘆軟件技術(shù)開(kāi)發(fā)有限公司, 擔任研究專(zhuān)員. 主要研究方向為超分辨率圖像重建技術(shù).E-mail: heng.su@hulu.com

計量
- 文章訪(fǎng)問(wèn)數: 6016
- HTML全文瀏覽量: 272
- PDF下載量: 7939
- 被引次數: 0
出版歷程
- 收稿日期: 2011-08-31
- 修回日期: 2013-01-29
- 刊出日期: 2013-08-20

Survey of Super-resolution Image Reconstruction Methods

1.
Department of Automation, Tsinghua University, Beijing 100084;
2.
Beijing Hulu Inc., Beijing 100084

Funds:

Supported by Key International (Regional) Joint Research Pro- gram of National Natural Science Foundation of China (6102010 6004), National Natural Science Foundation of China (61005023, 61021063), National Science Fund for Distinguished Young Scholars (61225008), and Ph. D. Programs Foundation of Min- istry of Education of China (20120002110033)

摘要

摘要: 由于廣泛的實(shí)用價(jià)值與理論價(jià)值,超分辨率圖像重建(Super-resolution image reconstruction, SRIR 或 SR)技術(shù)成為計算機視覺(jué)與圖像處理領(lǐng)域的一個(gè)研究熱點(diǎn), 引起了研究者的廣泛關(guān)注. 本文將超分辨率圖像重建問(wèn)題按照不同的輸入輸出情況進(jìn)行系統分類(lèi), 將超分辨率問(wèn)題分為基于重建的超分辨率、視頻超分辨率、單幀圖像超分辨率三大類(lèi). 對于其中每一大類(lèi)問(wèn)題, 分別全面綜述了該問(wèn)題的發(fā)展歷史、常用算法的分類(lèi)及當前的最新研究成果等各種相關(guān)問(wèn)題, 并對不同算法的特點(diǎn)進(jìn)行了比較分析. 本文隨后討論了各不同類(lèi)別超分辨率算法的互相融合和圖像視頻質(zhì)量評價(jià)的方法, 最后給出了對這一領(lǐng)域未來(lái)發(fā)展的思考與展望.
- 超分辨率圖像重建 /
- 計算機視覺(jué) /
- 圖像處理 /
- 方法綜述
Abstract: Because of its extensive practical and theoretical values, the super-resolution image reconstruction (SRIR or SR) technique has become a hot topic in the areas of computer vision and image processing, attracting many researchers' attentions. This paper categorizes the SR problems according to their input and output conditions into three main categories: reconstruction-based SR, video SR and single image SR. For each category, the development history, common algorithm classes and state-of-the-art research achievements are reviewed comprehensively. We also analyze the characteristics of different algorithms. Afterwards, we discuss the combination of different super-resolution categories and the evaluation of image and video qualities. Thoughts and foresights of this field are given at the end of this paper.
- Super-resolution image reconstruction /
- computer vision /
- image processing /
- survey

HTML全文

參考文獻(83)

[1]	Capel D, Zisserman A. Computer vision applied to super resolution. IEEE Signal Processing Magazine, 2003, 20(3): 75-86
[2]	Tsai R Y, Huang T S. Multiframe image restoration and registration. Advances in Computer Vision and Image Processing, 1984, 1: 317-339
[3]	Borman S, Stevenson R. Spatial Resolution Enhancement of Low-resolution Image Sequences: A Comprehensive Review with Directions for Future Research, Technical Report, Laboratory Image and Signal Analysis, University of Notre Dame, 1998
[4]	Chaudhuri S. Super-Resolution Imaging. Boston: Kluwer Academic Publishers, 2001
[5]	Park S C, Park M K, Kan M G. Super-resolution image reconstruction: a technical overview. IEEE Signal Processing Magazine, 2003, 20(3): 21-36
[6]	Van Ouwerkerk J D. Image super-resolution survey. Image and Vision Computing, 2006, 24(10): 1039-1052
[7]	Katartzis A, Petrou M. Current trends in super-resolution image reconstruction. Image Fusion: Algorithms and Applications. New York: Academic Press, 2008
[8]	Sun J, Zhu J J, Tappen M F. Context-constrained hallucination for image super-resolution. In: Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). San Francisco, CA: IEEE, 2010. 231-238
[9]	Tai Y W, Liu S C, Brown M S, Lin S. Super resolution using edge prior and single image detail synthesis. In: Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). San Francisco, CA: IEEE, 2010. 2400-2407
[10]	Farsiu S, Robinson M D, Elad M, Milanfar P. Fast and robust multiframe super resolution. IEEE Transactions on Image Processing, 2004, 13(10): 1327-1344
[11]	Rhee S, Kang M. Discrete cosine transform based regularized high-resolution image reconstruction algorithm. Optical Engineering, 1999, 38(8): 1348-1356
[12]	Katsaggelos A K, Lay K T, Galatsanos N P. A general framework for frequency domain multi-channel signal processing. IEEE Transactions on Image Processing, 1993, 2(3): 417-420
[13]	Nguyen N, Milanfar P. An efficient wavelet-based algorithm for image superresolution. In: Proceedings of the 2000 International Conference on Image Processing. Vancouver, BC, Canada: IEEE, 2000, 2: 351-354
[14]	Ji H, Fermuller C. Robust wavelet-based super-resolution reconstruction: theory and algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(4): 649-660
[15]	Lertrattanapanich S, Bose N K. High resolution image formation from low resolution frames using delaunay triangulation. IEEE Transactions on Image Processing, 2002, 11(12): 1427-1441
[16]	Sanchez-Beato A, Pajares G. Noniterative interpolation-based super-resolution minimizing aliasing in the reconstructed image. IEEE Transactions on Image Processing, 2008, 17(10): 1817-1826
[17]	Nasonov A V, Krylov A S. Fast super-resolution using weighted median filtering. In: Proceedings of the 20th International Conference on Pattern Recognition (ICPR). Istanbul: IEEE, 2010. 2230-2233
[18]	Lin S C, Chen C T. Reconstructing vehicle license plate image from low resolution images using nonuniform interpolation method. International Journal of Image Processing, 2007, 1(2): 21
[19]	Stark H, Oskoui P. High-resolution image recovery from image-plane arrays, using convex projections. Optical Society of America, Journal, A: Optics and Image Science, 1989, 6(11): 1715-1726
[20]	Banham M R, Katsaggelos A K. Digital image restoration. IEEE Signal Processing Magazine, 1997, 14(2): 24-41
[21]	Patti A J, Sezan M I, Murat T A. Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time. IEEE Transactions on Image Processing, 1997, 6(8): 1064-1076
[22]	Kim J Y, Park R H, Yang S. Super-resolution using pocs-based reconstruction with artifact reduction constraints. In: Proceedings of the 2005 Visual Communications and Image Processing, 5960, 2005. 59605B
[23]	Patti A J, Altunbasak Y. Artifact reduction for set theoretic super resolution image reconstruction with edge adaptive constraints and higher-order interpolants. IEEE Transactions on Image Processing, 2001, 10(1): 179-186
[24]	Tom B, Katsaggelos A. Iterative algorithm for improving the resolution of video sequences. In: Proceedings of the 1996 SPIE, 2727, SPIE, 1996. 1430
[25]	Yu J, Xiao C B, Su K N. A method of gibbs artifact reduction for pocs super-resolution image reconstruction. In: Proceedings of the 8th International Conference on Signal Processing. Beijing, China: IEEE, 2006, 2: 1-4
[26]	Hennings-Yeomans P H, Baker S, Kumar B V K V. Simultaneous super-resolution and feature extraction for recognition of low-resolution faces. In: Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Anchorage, AK: IEEE, 2008. 1-8
[27]	Babacan S D, Molina R, Katsaggelos A K. Total variation super resolution using a variational approach. In: Proceedings of the 15th IEEE International Conference on Image Processing (ICIP). San Diego, CA: IEEE, 2008. 641-644
[28]	Chantas G, Galatsanos N, Likas A, Saunders M. Variational bayesian image restoration based on a product of t-distributions image prior. IEEE Transactions on Image Processing, 2008, 17(10): 1795-1805
[29]	Chantas G K, Galatsanos N P, Woods N A. Super-resolution based on fast registration and maximum a posteriori reconstruction. IEEE Transactions on Image Processing, 2007, 16(7): 1821-1830
[30]	Su H, Wu Y, Zhou J. Super-resolution without dense flow. IEEE Transactions on Image Processing, 2012, 21(4): 1782-1895
[31]	Bose N K, Lertrattanapanich S, Koo J. Advances in superresolution using L-curve. In: Proceedings of the 2001 IEEE International Symposium on Circuits and Systems. Sydney, NSW: IEEE, 2001, 2: 433-436
[32]	Yuan Q Q, Zhang L P, Shen H F, Li P X. Adaptive multiple-frame image super-resolution based on U-curve. IEEE Transactions on Image Processing, 2010, 19(12): 3157-3170
[33]	Huang Li-Li, Xiao Liang, Wei Zhi-Hui, Zhang Jun. A fast decoupling algorithm for image super-resolution reconstruction of space-invariant system. Acta Automatica Sinica, 2010, 36(2): 229-236(黃麗麗, 肖亮, 韋志輝, 張軍. 空間移不變系統圖像超分辨復原的快速解耦算法. 自動(dòng)化學(xué)報, 2010, 36(2): 229-236)
[34]	Wang Q, Tang X O, Shum H. Patch based blind image super resolution. In: Proceedings of the 10th IEEE International Conference on Computer Vision (ICCV). Beijing, China: IEEE, 2005, 1: 709-716
[35]	Baboulaz L, Dragotti P L. Exact feature extraction using finite rate of innovation principles with an application to image super-resolution. IEEE Transactions on Image Processing, 2009, 18(2): 281-298
[36]	Sun Yan-Yue, He Xiao-Hai, Song Hai-Ying, Chen Wei-Long. A block-matching image registration algorithm for video super-resolution reconstruction. Acta Automatica Sinica, 2011, 37(1): 37-43(孫琰玥, 何小海, 宋海英, 陳為龍. 一種用于視頻超分辨率重建的塊匹配圖像配準方法. 自動(dòng)化學(xué)報, 2011, 37(1): 37-43)
[37]	Shen H F, Zhang L P, Huang B, Li P X. A map approach for joint motion estimation, segmentation, and super resolution. IEEE Transactions on Image Processing, 2007, 16(2): 479-490
[38]	He Y, Yap K H, Chen L, Chau L P. A nonlinear least square technique for simultaneous image registration and super-resolution. IEEE Transactions on Image Processing, 2007, 16(11): 2830-2841
[39]	Su H, Tang L, Wu Y, Tretter D, Zhou J. Spatially adaptive block-based super-resolution. IEEE Transactions on Image Processing, 2012, 21(3): 1031-1045
[40]	Takeda H, Farsiu S, Milanfar P. Kernel regression for image processing and reconstruction. IEEE Transactions on Image Processing, 2007, 16(2): 349-366
[41]	Takeda H, Milanfar P, Protter M, Elad M. Super-resolution without explicit subpixel motion estimation. IEEE Transactions on Image Processing, 2009, 18(9): 1958-1975
[42]	Protter M, Elad M. Super resolution with probabilistic motion estimation. IEEE Transactions on Image Processing, 2009, 18(8): 1899-1904
[43]	Protter M, Elad M, Takeda H, Milanfar P. Generalizing the nonlocal-means to super-resolution reconstruction. IEEE Transactions on Image Processing, 2009, 18(1): 36-51
[44]	Gunturk B K, Altunbasak Y, Mersereau R M. Super-resolution reconstruction of compressed video using transform-domain statistics. IEEE Transactions on Image Processing, 2004, 13(1): 33-43
[45]	Krämer P, Hadar O, Benois-Pineau J, Domenger J P. Super-resolution mosaicing from mpeg compressed video. Signal Processing: Image Communication, 2007, 22(10): 845-865
[46]	Xu Z Q, Zhu X C. Super-resolution reconstruction of compressed video based on adaptive quantization constraint set. In: Proceedings of the 1st International Conference on Innovative Computing, Information and Control. Beijing, China: IEEE, 2006, 1: 281-284
[47]	Xu Z Q, Gan Z L, Zhu X C. Compressed video super-resolution reconstruction based on regularized algorithm. In: Proceedings of the 8th International Conference on Signal Processing. Beijing, China: IEEE, 2006, 2
[48]	Begin I, Ferrie F P. Comparison of super-resolution algorithms using image quality measures. In: Proceedings of the 3rd Canadian Conference on Computer and Robot Vision. Washington, DC, USA: IEEE Computer Society, 2006. 72
[49]	Mudenagudi U, Banerjee B, Kalra P K. Space-time super-resolution using graph-cut optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(5): 995-1008
[50]	Belekos S P, Galatsanos N P, Babacan S D, Katsaggelos A K. Maximum a posteriori super-resolution of compressed video using a new multichannel image prior. In: Proceedings of the 16th IEEE International Conference on Image Processing (ICIP). Cairo: IEEE, 2009. 2797-2800
[51]	Segall C A, Katsaggelos A K, Molina R, Mateos J. Bayesian resolution enhancement of compressed video. IEEE Transactions on Image Processing, 2004, 13(7): 898-911
[52]	Su H, Wu Y, Zhou J. Adaptive incremental video super-resolution with temporal consistency. In: Proceedings of the 18th IEEE International Conference on Image Processing (ICIP). Brussels, Belgium: IEEE, 2011. 1149-1152
[53]	Kong D, Han M, Xu W, Tao H, Gong Y H. A conditional random field model for video super-resolution. In: Proceedings of the 18th International Conference on Pattern Recognition (ICPR). Hong Kong, China: IEEE, 2006, 3: 619-622
[54]	Zibetti M V W, Mayer J. Simultaneous super-resolution for video sequences. In: Proceedings of the 2005 IEEE International Conference on Image Processing (ICIP). Genova: IEEE, 2005, 1: I-877
[55]	Dai S Y, Han M, Xu W, Wu Y, Gong Y H. Soft edge smoothness prior for alpha channel super resolution. In: Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Minneapolis, MN: IEEE, 2007. 1-8
[56]	Mallat S, Yu G S. Super-resolution with sparse mixing estimators. IEEE Transactions on Image Processing, 2010, 19(11): 2889-2900
[57]	Pentland A, Horowitz B. A practical approach to fractal-based image compression. In: Proceedings of the 1991 Data Compression Conference. Snowbird, UT: IEEE, 1991. 176- 185
[58]	Freeman W T, Pasztor E C. Learning low-level vision. In: Proceedings of the 7th IEEE International Conference on Computer Vision (ICCV). Kerkyra: IEEE, 1999, 2: 1182- 1189
[59]	Chang H, Yeung D Y, Xiong Y M. Super-resolution through neighbor embedding. In: Proceedings of the 2004 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington, DC, USA: IEEE, 2004, 1. I-275-I-282
[60]	Sun J, Zheng N N, Tao H, Shum H Y. Image hallucination with primal sketch priors. In: Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). Madison, WI, USA: IEEE, 2003, 2: II-729-36
[61]	Fan W, Yeung D Y. Image hallucination using neighbor embedding over visual primitive manifolds. In: Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Minneapolis, MN: IEEE, 2007. 1-7
[62]	Yang J C, Wright J, Huang T, Ma Y. Image super-resolution as sparse representation of raw image patches. In: Proceedings of the 2008 IEEE Conference of Computer Vision and Pattern Recognition (CVPR). Anchorage, AK: IEEE, 2008. 1-8
[63]	Adler A, Hel-Or Y, Elad M. A shrinkage learning approach for single image super-resolution with overcomplete representations. In: Proceedings of the 11th European Conference on Computer Vision (ECCV). 2010. Berlin, Heidelberg: Springer-Verlag, 622-635
[64]	Kim K I, Kwon Y. Single-image super-resolution using sparse regression and natural image prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(6): 1127-1133
[65]	Yang J C, Wright J, Huang T S, Ma Y. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 2010, 19(11): 2861-2873
[66]	Wang J J, Zhu S H, Gong Y H. Resolution enhancement based on learning the sparse association of image patches. Pattern Recognition Letters, 2010, 31(1): 1-10
[67]	Wang J J, Zhu S H, Gong Y H. Resolution-invariant image representation and its applications. In: Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition CVPR 2009. Miami, FL: IEEE, 2009. 2512- 2519
[68]	Liu C, Shum H Y, Freeman W T. Face hallucination: theory and practice. International Journal of Computer Vision, 2007, 75(1): 115-134
[69]	Zhang W, Cham W K. Learning-based face hallucination in dct domain. In: Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Anchorage, AK: IEEE, 2008. 1-8
[70]	Hu Y, Lam K M, Qiu G P, Shen T Z. From local pixel structure to global image super-resolution: a new face hallucination framework. IEEE Transactions on Image Processing, 2011, 20(2): 433-445
[71]	Glasner D, Bagon S, Irani M. Super-resolution from a single image. In: Proceedings of the 2009 IEEE 12th International Conference on Computer Vision. Kyoto: IEEE, 2009. 349- 356
[72]	Zhao W, Sawhney H S. Is super-resolution with optical flow feasible? In: Proceedings of the 7th European Conference on Computer Vision. London, UK: Springer-Verlag 2002. 599-613
[73]	Costa G H, Bermudez J C M. Are registration errors always bad for super-resolution? In: Proceedings of the 2007 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Honolulu, HI: IEEE, 2007, 1: I-569- I-572
[74]	Baker S, Kanade T. Limits on super-resolution and how to break them. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(9): 1167-1183
[75]	Lin Z C, Shum H Y. Fundamental limits of reconstruction-based superresolution algorithms under local translation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(1): 83-97
[76]	Tanaka M, Okutomi M. Theoretical analysis on reconstruc- tion-based super-resolution for an arbitrary PSF. In: Proceedings of the 2005 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). San Diego, CA, USA: IEEE, 2005, 2: 947-954
[77]	Robinson D, Milanfar P. Statistical performance analysis of super-resolution. IEEE Transactions on Image Processing, 2006, 15(6): 1413-1428
[78]	Wang Z, Bovik A C, Lu L G. Why is image quality assessment so difficult? In: Proceedings of the 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). Orlando, FL, USA: IEEE, 2002, 4: IV-3313-IV-3316
[79]	Damera-Venkata N, Kite T D, Geisler W S, Evans B L, Bovik A C. Image quality assessment based on a degradation model. IEEE Transactions on Image Processing, 2000, 9(4): 636-650
[80]	Wang Z, Bovik A C, Sheikh H R, Simoncelli E P. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600-612
[81]	Sheikh H R, Bovik A C. Image information and visual quality. IEEE Transactions on Image Processing, 2006, 15(2): 430-444
[82]	Sheikh H R, Sabir M F, Bovik A C. A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Transactions on Image Processing, 2006, 15(11): 3440-3451
[83]	Seshadrinathan K, Bovik A C. Motion tuned spatio-temporal quality assessment of natural videos. IEEE Transactions on Image Processing, 2010, 19(2): 335-350