Skip to content

Video Review and Scoring

This document is meant to provide TDT users with steps for reviewing their data-synchronized videos using OpenScope, third-party media players, or replaying it using a demo MATLAB Script. There is also information for scoring the videos using OpenScope.

The most common synchronized video file that TDT users will have is using a USB Camera but this applies to RV2 video tracker as well.

Notes for Third-Party Video Recording

If you are recording video with a third party software (a common example is EthoVision/ Noldus camera or a camera with GPIO to talk to TDT hardware), then you will want to make sure that you have a way to synchronize your video with your TDT data.

You will not be able to review the video side-by-side your data using OpenScope in this case, but you will still be able to align your video with TDT data if you have a TTL event that timestamps when frames were captured or requested. This most often will be either your Third Party camera sending a TTL pulse to your TDT processor when a frame has been captures (you will save the TTL input in Synapse), or you will send a TTL pulse from Synapse to request a frame to be captured from your external camera.

Best practice would be to start your TDT recording first, then your video, if you are sending TTL pulses externally to TDT, or start your video software first then run TDT if you are sending frame capture requests from TDT to your external camera. These timestamps will then allow you to align the camera frames to your TDT data in post-processing.

View Video Synchronized with Other Data

You can review data alongside video files using OpenScope. Here are several Lightning Videos that demonstrate opening OpenScope:

From within Synapse

Opening OpenScope directly

Observing videos associated with your recordings

Please note that if you are using OpenScope on a machine other than your Synapse computer, or you have relocated your data, then you may need to add in .ini files for Scope to recognize your tanks and block (only needed if Scope is not seeing your data).

How To Score Videos or Timestamp Data Based on Behavior In Video

To add epoc scoring to your videos: See Video Review and Epoch Annotation in the OpenScope manual.

In brief, first you must open the data in OpenScope and then open the associated video. Then open the 'Notes Window'. Once you begin animating the data, if you put the Notes Window into 'Record Mode' then it will begin to track epoc events in the Cam1 epocs (this is the red arrow). With this set up, you can add your notes by manually setting them or using the rapid annotation with the keypad.

Once the notes have been attached to an epoc event (this is the little number next to the epoc mark), they will be put into the data block itself in association with the Cam1 epoc event. Thus, when you import the data into MATLAB using TDTbin2mat and look at the data.epocs.Cam1 event, you will see that there is a Cam1.notes field as well. These are your scored events.

Below are notes and images that show you how to use the Epoc Annotation/ Notes feature in Open Scope. This allows you to mark your data ("score" your video) to add timestamps to your block where events of interest occur. This can be very helpful for post processing and analysis.

OpenScope Video Frames - video frame timestamp matches epoc 73

OpenScope Single Frames - zoom on the timelines

If you zoom in far enough you will see individual epoc events, which you can click and move frame by frame. Arrow keys still jump by one span of the x-axis (minimum 0.5 seconds).

Select View > Notes Window to open the annotation window. From there, add the list of possible notes. Click 'Record Mode' to enable keyboard scoring. Whenever a number key is pressed, the note associated with that number will be attached to the nearest Cam1 epoc event.

OpenScope Notes Window

A red arrow marks the currently tracked Cam1 epoc. If you have multiple epoc events, you can change which event gets the notes by right-clicking on the name of it.

OpenScope Notes Attached to Epoc Event

Use arrow keys (turn off "Animate") to move left and right through stream. Note that this will jump by the time span of the x-axis (minimum 0.5 seconds). For finer resolution, you will need to expand the time viewer window and click on individual camera timestamps.

Use Hot Keyboard numbers '1', '2', etc per the Note Index to add a Note to currently tracked epoc (red flag, epoc 73, Note Index 1 in this case).

OpenScope Setting Notes

Or right click near Cam1 epoc you want to mark and pick 'Note', then set note accordingly, Epoc 70, Note 2 in this case.

Manually Setting Note

Individual or all notes can be removed. The remaining notes get written to the data block, and can be used for analysis, such as generating peri-event plots.

Note Data Imported Into MATLAB

View Data in Another Video Viewer

VLC Media Player is a wonderful open source media player. There are others, too, that users might want to use to review the video files. The main issue with all of these is that the video players might assume a wrong and static frame rate for the video and the recorded AVI files will not have the same length as the recordings. The most common outcome of this is that the video ends up being ¾ the length of the recorded data upon view.

OpenScope uses the actual Cam1 timestamps to mark when the frame was pulled from the camera. Often, the real camera rate will be 15 fps but a video player will assume 20 fps when reading the data, which is how you get a '45 min' video from a real 1-hour video. If you wanted to read the video in MATLAB or Python to plot alongside your data, then you can use the Cam1 epocs to know the exact time and frame number that corresponds to your data time.

VLC media player can, however, read .sub files that correlate to the real recording time and overlay it on the video. If you want VLC media player to display the timestamp of the video while it is playing, then you can open the AVI file in OpenScope using the video viewer first. This will create a .sub file (in the data block folder) which you can then use in VLC media player to display the timestamps of each frame as it is playing.

Below is MATLAB code that will also generate a sub file. Replace 'block' with your block name, and it will make a file called 'output.sub' in that folder:

    block = 'F:\TESTAVI';
    outfile = [block filesep 'output.sub'];
    data = TDTbin2mat(block, 'STORE', 'Cam1');
    s = cell(1, length(data.epocs.Cam1.onset));

    for i = 1:length(data.epocs.Cam1.onset)
        t = data.epocs.Cam1.onset(i);
        d = data.epocs.Cam1.data(i);
        s{i} = sprintf('{%d}{%d}%.6f\n', d-1, d, t);
    end

    fid = fopen(outfile, 'w')
    fwrite(fid, cell2mat(s));
    fclose(fid)

Plot My Data Alongside Video

Your TDT data will contain timestamps for video frames as well as the continuous data. You can import data into MATLAB or Python and playback, review, and score your video in that environment.

Note

When importing TDT data and video files into MATLAB or Python, you may notice that the number of camera epoc events does not match the number of video frames.

What you are seeing is normal and is explained in this Tech Note. You can just use the number of frames that matches the length of your Cam1.onset array, starting from index 0 and just chop off the extra video frames at the end.

TDT has an example for overlaying time-locked video with a photometry trace using MATLAB. The script and example data can be found in the MATLAB SDK.