Table of Contents
Installing the Program
The Launcher Window
The Collect Ratings Window
4. The Review Ratings Window
Customizing the Program
The Review Ratings Window
The Review Ratings window allows users to view and compare previously saved annotation files, as well as watch media files without recording ratings. The Review Ratings window opens to display a mostly blank window with some rectangular panes, some buttons,
and a menu bar.
To open a media file, users should select Open Media File from the Media menu and select the desired audio or video file using the prompt. This will display the first frame of the media file in the black pane. Users can then press the Play button to watch the
media file. If annotation file(s) are loaded alongside the media file, left-clicking on the rating axes at the bottom of the screen will jump/seek the media file to the corresponding time in the media file.
To load one or more annotation files, users should select Add Annotation File(s) from the Annotations menu or left-click the Add Annotations button. More than one annotation file can be loaded at once by selecting multiple files in the prompt (e.g., by holding
the Ctrl or Shift keys). This operation can be performed before, after, or without opening a media file. In order to be viewed simultaneously, all loaded annotation files must have the same length (i.e., sampling rate and duration).
Once loaded, the name(s) of the annotation file(s) will appear on the upper right pane with a colored index number next to it. This color corresponds to the color of annotation series displayed in the centroid and point cloud plots, as well as the time-series
plots in the rating axes. When annotation files are first loaded, the centroid plot is displayed. This plot shows each file's mean as a solid circle and two times its standard deviation for each axis as a semi-transparent ellipse. Point cloud or "smoke"
plots for individual files can be displayed by left-clicking on the corresponding annotation filename in the upper right pane. Left-clicking the first row ('Annotation Files') will return to the centroid plot. Previously added annotation files can
be removed by selecting them in the upper right pane and selecting Remove Selected File/All Files from the Annotations menu or left-clicking the Remove Selected/All Files buttons.
Users can left-click the Show/Hide Mean Plot button to toggle the mean plot display. When toggled on, the mean of all loaded annotation series will be displayed in red in both rating axes and the point cloud axis; all previously loaded annotation series will
be changed to light grey while the mean plot display is toggled on. Users can export this mean series by selecting Export Mean Series from the Annotations menu.
The Analyze Ratings button opens a new window that displays information about each loaded annotation file's Mean and Standard Deviation (SD) for each dimension. The overall reliability for each dimension is also displayed. Various
Intraclass Correlation Coefficients
are used as estimates of inter-rater agreement and reliability (McGraw and Wong, 1996
), while various
RWG scores are used as estimates of inter-rater agreement in the presence of restricted variance (James, Demaree, and Wolf, 1984
LeBreton et al., 2003
The Review Ratings window before any annotation files are loaded
The Review Ratings window with two annotation files loaded and a centroid plot
The Review Ratings window with the mean of two annotation files and a smoke plot
The Analyze Ratings window for the two example annotation files
Customizing the Program