MOBILE3DTV project finalized

Mission accomplished! The MOBILE3DTV project ended and has provided core technology for new mobile 3DTV systems. Summarizing our work in pictures, here are the posters that we presented at the presentation of the final end-to-end MOBILE3DTV system at ICT2010 in Brussels.




Read more!

Quality of Experience (QoE) - Current standardization activities

The assessment of the Quality of Experience (QoE) has become more important in audiovisual quality assessment and MOBILE3DTV has started to evaluate user experience for mobile 3D television and video. Within standardization bodies the topic of Quality of Experience has been included into standardization activities. The goal of ETSI HF STF 354 (Specialist Task Force 354 within the Human Factors Group of ETSI) is to provide requirement guidelines for real-time multimedia services aiming at providing a good QoE. Goal is to provide objective and subjective measures of user experience for given communication situations, service prescriptions and levels of QoS [1]. The guidelines as the expression of measure are available in ETSI EG 202 534 [2] and ETSI TR 102 535 [3].

Within the International Telecommunication Union ITU standardization activities towards QoE are targeted in ITU-T SG12, the lead study group on quality of service and quality of experience within the ITU. Goal of ITU-T SG12 to be able to a) measure quality parameters in next generation networks and b) measure their impact on QoE. The challenge of providing good QoE for new multimedia systems is study item of question Q13/12 “QoE, QoS and performance requirements and assessment methods for multimedia including IPTV”. Main study items are the relation of end-user requirements to system parameters and to identify simple and efficient analysis techniques to measure and monitor QoE. All results will be included into the development of new recommendations.

References:

1. Brooks, P., Hestnes, B., Heiestad, S., Aaby, C. (2006). “Communicating Quality of Experience data for the development of multimedia services”. Proceedings of the 20th International Symposium on Human Factors in Telecommunication, Sophia Antipolis, France, March 21-23 2006, (available from http://www.hft.org/HFT06/HFT_06_programme.htm).
2. ETSI EG 202 534: "Human Factors (HF): Guidelines for real-time person-to-person communication services". ETSI Guide, 2007
   
3. ETSI TR 102 535: "Human Factors (HF): Guidelines for real-time person-to-person communication services; future requirements". ETSI Technical Report, 2007
   

Read more!

Standards related with coding of Mobile 3DTV content

Standardization of digital audio and video is investigated by the Moving Picture Experts Group (MPEG), a working group of ISO/IES and the corresponding standards are issued with ISO/IES designations [1].

MPEG-C, Part 3
The purpose of ISO/IEC 23002-3 Auxiliary Video Data Representations (MPEG-C part 3) is to support all those applications where additional data needs to be efficiently attached to the individual pixels of a regular video. In ISO/IEC 23002-3 it is described how this can be achieved in a generic way by making use of existing (and even future) video codecs available within MPEG. ISO/IEC 23002-3 consists of an array of N-bit values which are associated with the individual pixels of a regular video stream. These data can be compressed like conventional luminance signals using already existing (and even future) MPEG video codecs. The format allows for optional subsampling of the auxiliary data in both the spatial and temporal domain. This can be beneficial depending on the particular application and its requirements and allowing for very low bitrates for the auxiliary data.

The specification is very flexible in the sense that it defines a new 8-bit code word aux_video_type that specifies the type of the associated data, e.g., currently a value of 0x10 signals a depth map, a value of 0x11 signals a parallax map. New values for additional data representations can be easily added to fulfill future demands. The specification is directly applicable to 3D video as it allows specifying such video in the format of single view + associated depth, where the single channel video is augmented by the per-pixel depth attached as auxiliary data. As such, it is susceptible to efficient compression. Rendering of virtual view (at least one in case of stereo) is required at the receiver side. The specification has been standardized since 2007 [2], [3], [4].

MVC
3D video (3DV) and free viewpoint video (FVV) are new types of visual media that expand the user’s experience beyond what is offered by 2D video. 3DV offers a 3D depth impression of the observed scenery, while FVV allows for an interactive selection of viewpoint and direction within a certain operating range. A common element of 3DV and FVV systems is the use of multiple views of the same scene that are transmitted to the user. Multiview Video Coding (MVC, ISO/IEC 14496-10:2008 Amendment 1) is an extension of the Advanced Video Coding (AVC) standard that provides efficient coding of such multiview video. The overall structure of MVC defines the following interfaces: The encoder receives N temporally synchronized video streams and generates one bitstream. The decoder receives the bitstream, decodes and outputs the N video signals. The video representation format is based on N views. For the case of stereo-video, that is two separate views coded together. A promising extension is to study view subsampling, i.e. one full resolution view + one subsampled view. The idea behind this approach is that the human visual system is capable to retrieve the stereo with the quality of the better channel. MVC is standard since 2008 (version 1) [5], [6].

3D Video Coding
3D Video Coding (3DVC) is a standard that targets serving a variety of 3D displays. Such displays here in focus present N views (e.g. N = 9) simultaneously to the user, so-called multi-vied displays. For efficiency reasons only a lower number K of views (K = 1,..,3) shall be transmitted. For those K views additional depth data shall be provided. At the receiver side the N views to be displayed are generated from the K transmitted views with depth by depth image based rendering (DIBR). This application scenario imposes specific constraints such as narrow angle acquisition (<>K out of N views, augmented with K depth sequences. This representation related to stereo-video generalizes the possibilities of MPEG-C, Part 3 and MVC, i.e. the two separate views can be coded together or can be reduced to single view + depth with the second view to be synthesized at the receiver. 3DVC is an ongoing MPEG activity, and a standard is expected in 2011 [7], [8], [9].

References:

1. Moving Pictures Experts Group, http://www.chiariglione.org/mpeg/
   
2. ISO/IEC JTC1/SC29/WG11, “Text of ISO/IEC FDIS 23002-3 Representation of Auxiliary Video and Supplemental Information”, Doc. N8768, Marrakech, Morocco, January 2007.
   
3. ISO/IEC JTC1/SC29/WG11, “Text of ISO/IEC 13818-1:2003/FDAM2 Carriage of Auxiliary Data”, Doc. N8799, Marrakech, Morocco, January 2007.
   
4. ITU T and ISO/IEC JTC 1, “Advanced video coding for generic audiovisual services”, ITU-T Rec. H.264 and ISO/IEC 14496-10 AVC, 2003, most recent Version: 2005.
   
5. ISO/IEC JTC1/SC29/WG11, “Text of ISO/IEC 14496-10:200X/FDAM 1 Multiview Video Coding”, Doc. N9978, Hannover, Germany, July 2008.
   
6. ISO/IEC JTC1/SC29/WG11, “Joint Multiview Video Model (JMVM) 8”, Doc. N9762, Archamps, France, May 2008.
   
7. ISO/IEC JTC1/SC29/WG11 “Overview of 3D Video Coding”, Doc. N9784, Archamps, France, May 2008.
   
8. ISO/IEC JTC1/SC29/WG11 “Description of Exploration Experiments in 3D Video Coding”, Doc. N9783, Archamps, France, May 2008.
   
9. ISO/IEC JTC1/SC29/WG11 “Applications and Requirements of FTV”, Doc. N9466, Shenzhen, China, Oct. 2007.

Read more!

Current user research in the context of Mobile3DTV

Related user studies in the field of 3DTV and especially in the field of mobile TV, one can realize a big change in the field of user-centered design. Current research has introduced user-centered approaches to study user’s needs and expectations about novel systems and services. This allows better understanding of the impact of critical system components. The cyclic character of user-centered design is standardized in ISO 13407 [1].

Subjective quality has usually been studied quantitatively based on requirements of the International Telecommunication Union (ITU) [2] offering guidelines to study quality perception studies in controlled, laboratory environments. In the context of mobile television, these studies, which base on quantitative measurements of ITU recommendations, have focused on evaluating the impact of coding errors, low bitrates, and impact of changes in image size or framerates as well as bitrate share between audio and visual modalities (e.g. [3], [4], [5]). Further, there are studies into transmission errors over DVB-H channel and their impact on the perceived quality [6], but still these studies are rare.

With respect to stereo-coding, there are studies on stereo-coding errors in images and videos [7]. However, user studies in the field of 3DTV have focused more on the impact of different factors on presence, the users feeling of being there. In stereo-video content presentation, the most prominent additional features that influence the perceived quality are depth information, perceived sharpness, or naturalness (i.e. the most truthful or realistic reproduction of content) [8][9]. All these factors contribute to the concept of presence, the involvement of the user in the content; the feeling of being there [10]. Nevertheless, some work has reported simulator sickness symptoms caused by the use of stereo-video presentation. This particular impact of stereo vision adversely affects enjoyment and factors creating simulator sickness are not yet known in depth [11].

In contrast to the recommendations of the ITU, current studies in the field of mobile TV include user characteristics, usage contexts and the user’s goal of viewing. Currently there are available results of field trials or studies conducted in several countries [12][13] as well as results of prospective focus groups and online surveys [14][15]. In contrast, user studies into the field of 3DTV are still rare and Freeman’s focus group [16] is the only study which examines user requirements of 3DTV more broadly.

From the methodological point of view, there is a lot of work in progress to close the shortcomings of ITU recommendations. Researchers have introduces contextual quality evaluation procedure that is conducted in parallel to the controlled laboratory assessments [17]. Additionally, evaluation of the experienced quality of critical system components targets the future acceptability of the quality of components which were developed in isolation from the end-product [18]. A third step to understand experienced quality is to understand the interpretation of constructed quality and experienced quality factors [19].

Finally, there is one available study into the user experience of mobile 3D television. Following a methodological triangulation of online survey, focus groups and probe study to elicit user requirements, Jumisko-Pyykkö et al. [20] present a broad view on user experience of mobile TV. In all the three studies, they targeted the user experience factors of user, system and services, and contexts. Concluding their study, they present first guidelines to design user experience of mobile 3D television.

References:

1. ISO 13407. 1999. Human-centered design processes for interactive systems. International Standard, the International Organization for Standardization.
2. ITU-R BT.500-11, Methodology for the Subjective Assessment of the Quality of Television Pictures, Recommendation ITU-R BT.500-11, ITU Telecom. Standardization Sector of ITU, 2002.
3. H. Knoche, J. D. McCarthy, and M. A. Sasse, “Can small be beautiful?: assessing image resolution requirements for mobile tv,” in MULTIMEDIA ‟05: Proceedings of the 13th annual ACM international conference on Multimedia. New York, NY, USA: ACM Press, 2005, pp. 829–838.
   
4. S. Winkler, S. C. Faller, “Maximizing audiovisual quality at low bitrates,” Workshop on Video Processing and Quality Metrics for Consumer Electronics. Scottsdale, United States of America: January 2005.
   
5. S. Jumisko-Pyykkö, “I would like to see the subtitles and the face or at least hear the voice”: Effects of Picture ratio and Audio-video Bitrate Ratio on Perception of Quality in Mobile Television. To appear in Personalized and Mobile Digital TV Applications in Springer Multimedia Tools and Applications Series, 2007
6. S. Jumisko-Pyykkö, Vadakital, V., Liinasuo, M. , Hannuksela M. M., Acceptance of Audiovisual Quality in Erroneous Television Sequences over a DVB-H Channel. Proceedings of Second International Workshop in Video Processing and Quality Metrics for Consumer Electronics, Scottsdale, USA, January 2006.
7. L. B. Stelmach, Tam, W. J. (1998). Stereoscopic image coding: effect of disparate image-quality in left- and right-eye views. Signal Processing: Image Communications, 14:111–117.
8. H. de Ridder, (1996). Naturalness and image quality: Saturation and lightness variations in color images of natural scenes. Journal of Imaging Science and Technology, 40:487–493.
9. A. Berthold (1997). The influence of blur on the perceived quality and sensation of depth of 2D and stereo images. Technical report, ATR Human Information Processing Research Laboratories.
   
10. W. IJsselsteijn, de Ridder, H., Freeman, J., and Avons, S. (2000a). Presence: Concept, determinants and measurement. Proceedings of the SPIE, 3959:520–529.
11. Häkkinen, J., Liinasuo, M., Takatalo, J., and Nyman, G. 2006. Visual comfort with mobile stereoscopic gaming. Proceedings of SPIE. Vol. 6055
12. Chipchase, J., Yanqing, C., Jung, Y., and Design, N. 2006. Personal Television: A Qualitative Study of Mobile TV Users. Lecture Notes in Computer Science. Vol. 4471, pp.195-204
13. Södergård C. (ed.). 2003. Mobile television – technology and user experiences, Report on the Mobile –TV Project. Espoo: VTT Publications 506.
14. Knoche, H. O., McCarthy, J. D. 2004. Mobile Users Needs and Expectations of Future Multimedia Services. In Proceedings of the WWRF12. Nov. 2004, Toronto, Canada
    
15. Carlsson, C., Walden, P. 2007. Mobile TV - To Live or Die by Content, Proc 40th HICSS 51b.
16. Freeman, J., Avons, S. E. 2000. Focus Group Exploration of Presence through Advanced Broadcast Services. In Proc. SPIE 3959 , 3959-3976.
17. Jumisko-Pyykkö, S., Hannuksela, M. M. Does Context Matter in Quality Evaluation of Mobile Television? To appear in the Proceedings of 10th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI 2008)
18. Jumisko-Pyykkö, S. Kumar Malamal Vadakital, V., Hannuksela, M.M. Acceptance Threshold: Bidimensional Research Method for User-Oriented Quality Evaluation Studies. To appear in International Journal of Digital Multimedia Broadcasting, 2008.
19. Strohmeier, D. Wahrnehmungsuntersuchung von 2D vs. 3D Displays in A/V-Applikationen mittels einer kombinierten Analysemethodik, Diploma Thesis, Ilmenau University of Technology, Germany, 2007
20. Jumisko-Pyykkö, S., Weitzel, M., Strohmeier, D. "Designing for User Experience: What to Expect from Mobile 3D TV and Video? " . Proceedings of the First International Conference on Designing Interactive User Experiences for TV and Video. October 22 - 24, 2008, Silicon Valley, California, USA.
    

Read more!

Mobile auto-stereoscopic displays

Mobile 3D displays are designed to fulfil specific requirements. They should be sufficiently small and thin for a handheld device and should provide autostereoscopy – the ability to create 3D effect without requiring glasses. Furthermore, backwards compatibility is a desirable feature – the possibility to switch the display back to “2D” mode, when 3D content is not available. Presently, there are several announced products or prototypes of reconfigurable 3D displays.

A display produced by Sharp uses electronically switchable reconfigurable parallax barrier display based on a patterned retardation film [96]. When the “3D mode” is on, the light passing through the subpixels is selectively blocked, in order different subsets of the screen image to be seen with each eye. These subsets form the “left” and “right” views, respectively. This display has been used to produce the Sharp AL3DU laptops (discontinued). The same parallax barrier has been used for a prototype of portable MVC decoder and player, developed by Nokia Research Center [1].

A reconfigurable 2D/3D technology from Ocuity Ltd. uses a Polarisation Activated Microlens array [2]. The microlens array is made from a birefringent material such that at the surface of the lens there is a refractive index step for only one of the polarisations. A switchable polariser selects either the unrefracted light for 2D displays (i.e. the cylindrical microlens array has no impact) or the refracted light.

A display family announced by NEC provides 2D compatibility by using Horizontally Double-Density Pixels (HDDP) configuration and fixed lenticular lenses. The HDDP configuration is composed of the RGB color arrangement in horizontal stripes with the pixels doubling the horizontal resolution. When the display is to be used in 2D mode, the same image data is written in the two adjacent pixels. With this approach, the 2D/3D switching can be done entirely by software means, and different parts of the displays can be in different modes [3].

Newsight has been marketing 3D displays based on wavelength-selective optical filters and with sizes ranging from 8.4’’ to 54’’. They claim their technology allows also manufacturing portable displays of size of 2’’ or so [4].

A single user 3D display technology developed by SeeFront utilizes lenticular optical layer mounted in front of a LCD panel. A camera tracks the positions of the eyes so to guide a software module which adaptively fractionizes the stereoscopic views [5]. The optical system can meet design requirements for various size displays including portable displays with small viewing distance. Only 3D viewing mode is provided.

Beside the above reviewed display solutions, auto-stereoscopic displays based either on lenticular optics or parallax barrier technology have been marketed by a number of companies [6] - [9].

References:

1. K. Willner, K. Ugur, M. Salmimaa, A. Hallapuro, J. Lainema, ‘Mobile 3D Video Using MVC and N800 Internet Tablet’, 3DTV-CON 2008, Istanbul, Turkey. Pp. 69-72.
   
2. G. J. Woodgate, J. Harrold, “Autostereoscopic display technology for mobile 3DTV applications”, in Proc. SPIE Vol.6490A-19 (Stereoscopic Displays and Applications XVIII), 2007
   
3. S.Uehara, T.Hiroya, H. Kusanagi; K. Shigemura, H.Asada, “1-inch diagonal transflective 2D and 3D LCD with HDDP arrangement”, in Proc. SPIE-IS&T Electronic Imaging 2008, Stereoscopic Displays and Applications XIX, Vol. 6803, San Jose, USA, January 2008
   
4. Newsight Displays, http://www.newsight.com/3d_products/displays/
   
5. SeeFront 3D Technology, http://www.seefront.com/seefront_3d_technology.php
   
6. Alioscopy, http://www.alioscopy.eu
   
7. MasterImage, http://www.masterimage.co.kr/new_eng/solution/technology.htm
   
8. 3DIS, http://www.3dis.co.kr/en/eproducts.htm
   
9. SpatialView, http://www.spatialview.com/index.cfm
   

Read more!