Comparison of Remote Subjective Assessment Strategies in the Context of the JPEG Pleno Point Cloud Activity

In this work we compare two different options to perform on-line subjective quality assessment experiments in the context of the Call for Evidence on JPEG Pleno Point Cloud Coding. A deep-learning based point cloud codec submitted to the Call was tested against current MPEG point cloud compression methods. The first option is based on participants downloading the entire set of stimuli and running a set of scripts in MATLAB to perform the experiment. The second option involves the participants accessing a server on the web and viewing and judging the stimuli using a web browser. Quality scores compiled using both methods were compared showing strong correlation. A second analysis compared the quality scores with those obtained in a prior laboratory-based study using higher resolution screens. The entire study also brought to light each option's unique advantages and disadvantages that make each one better suited to specific types of subjective evaluation contexts and situations.