Vectorscopes can seem mysterious and even perhaps a little intimidating. However, you don't need a phD in engineering to understand how to read one. ***
-seems like some crazy engineering stuff; Don't be scared!
***You don't have to be a geek to read one. Don't need thick glasses to read one.
Composite video signals consist of two elements: luma and chroma. In simple terms, luma contains the "black and white" information in a television signal, while chroma contains the "color" information in a signal. Vectorscopes provide a visual representation of the chroma signal.
Totally achromatic (colorless) values will appear at the very center of the vectorscope. The stronger the chroma value, the farther away from the center of the vectorscope. The angle around the center of the vectorscope indicates the hue of colors.
An example reading, using color bars:
Sony Vegas' vectorscope conveniently labels the colors Red, Blue, Green, Magenta (blue+red), Cyan (blue+green), and Yellow (red+green). Roll-over the image to see how the trace (i.e. the white dots) corresponds to particular colors.
Chroma values are proportional to their associated luma values- higher luma have higher chroma values, assuming that saturation stays the same. By extension, higher/brighter exposure means higher chroma values.
Note that the trace on the left extends much further than the one on the right. Remember: vectorscopes do not display saturation. When matching scene colors, a vectorscope cannot be used to match saturation reliably, unless the luma values are the same.
If you need to match scene colors, the fastest approach would be to match colors numerically. Unfortunately, not all NLEs/editing applications provide the tools to do so. The most accurate approach (perceptually) is to use your eye. Numerical approaches do not account for the quirks of human vision such as surround effects. In any case, a vectorscope is unnecessary in matching colors.
***Now, I've just told you that software***
Hardware vectorscopes have a different usage and can be useful in different ways than the vectorscope in your NLE/editing application. In general, NLE-based vectorscopes are somewhat useful as an aid in color correction (see discussion above). They are not very useful when dealing with analog signals (i.e. making a betaSP master) or for field/studio-recording use. They do not replace hardware vectorscopes for those uses.
The manufacturer of analog vectorscopes (i.e. Tektronix, Leader, etc.) will have documentation on how to install and use/read their products. I won't bother duplicating their information. Rather, I will point out the differences between analog vectorscopes and those found in NLEs.
Note: Hardware vectorscopes can measure both analog signals (i.e. composite, component) and digital signals (SDI, HD-SDI). For brevity's sake, I will mainly talk about hardware vectorscopes that measure analog signals.
If setting white balance on a camera, a vectorscope can be used to check that the white balance is accurate. If the white balance is off, the majority of the trace be slightly off the center of the vectorscope. Analog vectorscopes offer the ability to magnify the display to zoom in. This makes the center of the display easier to see.
In a NLE, there are better way to do things. Simply use the NLE's color correction filter and eyedropper/sample a neutral color. The color corrector should provide some sort of visual indication about the change in white balance. If you are still unsure about white balance, increase the chroma in the signal (this setting is sometimes marked as "saturation"). **
A vectorscope can be useful for lining up the color bars on a tape. Most analog vectorscopes will have targets that show where the trace from color bars should end up. This aids in setting video levels when printing to analog formats. The targets in NLE-based vectorscopes tend to be way too large and imprecise. Also, NLE-based vectorscopes can only measure digital levels, not analog ones.
Analog vectorscopes will also have markings for 75% and 100% color bars. This allows the vectorscope to work in conjunction with both types of color bars. 100% color bars are generally unnecessary for video editing. They are more useful for particular engineering uses.
Camera alignment charts from companies like DSC Labs aid in matching cameras through their menu settings (i.e. color matrix). Most NLE-based vectorscopes are not designed for this purpose (i.e. no gain control, no marking for 75%/100% color bars, no live monitoring).
NLE-based vectorscopes tend to have some unexpected behaviours compared to hardware vectorscopes. They may even have settings that affect the accuracy of the readings.
The waveform monitor in Final Cut Pro only scans the title safe area of an image. Any values in the overscan area (including illegal values) will not be displayed.
The WFM does not scan every video line.
The WFM does not show values lower than black level. In most but not all cases, these values are clipped at black level (0) before processing. This behaviour means that the WFM cannot show all illegal colors.
I believe these behaviours apply to the vectorscope too (unconfirmed).
The settings for the Video Scopes in Vegas will affect the readings on the vectorscope. The settings are fairly confusing. Vegas' Video Scopes can only measure digital levels. However, they are designed to simulate analog scopes. Personally speaking, I would not expect this sort of behaviour from a vectorscope. Because the Video Scopes do not know what type of digital-analog conversion takes place, it has settings that allow the user to specify the details of the digital-analog conversion.StudioRGB: This setting is to specify the color space of the codec currently in use. The default codec for Vegas 6 is the Vegas DV codec, which uses "studio RGB" color space. Most other DV codecs use "computer RGB" color space.
The "7.5 IRE setup" settings is used to specify black level of the analog signal. This setting can be especially confusing since it can be confused with other issues surrounding 7.5 IRE setup. Most North American DV equipment does not follow engineering standards, converting digital to analog with incorrect black level [see 7.5 IRE Setup Demystified]. If accuracy is the goal, the digital-analog convertor connected to the system should be able to maintain correct levels. Secondly, Vegas' "7.5 IRE setup" setting should reflect the television standard that the analog signal is being formatted for (7.5 IRE black level for non-Japanese NTSC, 0 IRE for Japanese NTSC and PAL).
For non-Japanese NTSC, an unfortunate drawback is that there is no marking for 7.5IRE. It becomes difficult to spot where exactly picture black level lies.
One drawback to the vectorscope in Vegas is that it does not show what the underlying settings are. It is not possible to tell if the settings are correct without looking in the settings menu.
*Please note that these behaviours are only correct for the time of writing. Software upgrades may change the behaviour of vectorscopes in the NLEs mentioned.
**I have not mentioned other NLEs since I am not very familiar with them.
Rec. 709 defines a different set of luma co-efficients than Rec. 601, resulting in a different color space. This changes the locations of the color bar targets. Some NLE-based vectorscopes do not account for this difference.
Typically, SD formats use Rec. 601 color space and HD formats use Rec. 709 color space.