Tag Archives: inspection

Display video difference with ffmpeg’s overlay filter

I was seeking a method to show the difference between two videos and found this method using some of the recent features of ffmpeg. This process could be useful to illustrate how lossy particular encoding settings are to a video source. An original digital video and a lossless encoding of it should show no difference; whereas, a high-quality lossy encoding (like an h264 encoding at 1000 kilobits per second) should show visual differences compared to the original. The less efficient the codec, the lower the bitrate, or the more mangled the transcoding process, the greater the difference will be between the pixel values of the original video and the derived encoding.

Here’s what I used:
ffmpeg -y -i fileA.mov -i fileB.mov -filter_complex '[1:v]format=yuva444p,lut=c3=128,negate[video2withAlpha],[0:v][video2withAlpha]overlay[out]' -map [out] fileA-B.mov

To break this command down into a narrative, there are two file inputs fileA.mov and fileB.mov to compare. The second input (fileB.mov) is converted to the yuva444p pixel format (YUV 4:4:4 with an alpha channel), the ‘lut’ filter (aka lookup-table filter) sets the alpha channel to 50% (the ‘128’ is half of 2^8 which is the bit depth of the pixel format), and then the video is negated (all values are inverted). In other words one video is made half-transparent, changed to its negative image, and overlaid on the other video so that all similarities would cancel out and leave only the differences. I know there are a few flaws in this process since depending on the source this may invoke a colorspace or chroma subsampling change that may cause additional loss than what exists between the two inputs (but close enough for a quick demonstration). This process also is intended to compare two files that have a matching presentation, same amount of frames, and same presentation times for all frames.

Here is an example of the output. This first one depicts the differences between an mpeg2 file (found here) and an mpeg1 derivative of it. Closer to middle gray indicates no visual loss in the encoding, but deviations from middle gray show how much was lost (unfortunately YouTube’s further encoding of the demonstration embedded here flattens the results a bit).

Here’s another version of the output, this time comparing the same mpeg2 file with a prores derivative of it. Here it is very difficult to discern any data loss since nearly the whole frame is middle gray. There is still some deviation (prores is not a lossless codec) but the loss is substantially less than with mpeg1.

Here’s another example with different material. In this case an archivist was digitizing the same tape on two different digitization stations and noticed that, although the waveforms and vectroscopes on each station shows the same levels, the results appearto be slightly different. I took digitized color bar videos from each of the two stations and processed them through yuvdiag to make videos of the waveform and vectroscope output and then used the comparison process outlined above to illustrate the differences that should have been illustrated by the original waveform monitor and vectroscope.

The results showed that although the vectroscope and waveform on each of the two digitization stations showed the same data during the original digitization that at least one of them was inaccurate. By digitizing the same color bar through both stations and analyzing the resulting video with yuvdiag we could see the discrepancy between the chroma and luma settings and calibrate appropriately.

learn on film | work on data

This blog will (mostly|probably) be about moving image archiving, audiovisual processing, and use of open source software for archival objectives. I may also throw in posts about silent film, independent and grassroots media, and technology manipulation. This is only the first post so all this is a prediction, but at the moment most of my writing veers towards these topics eventually.

At work I recently started acquiring film rewinds, splicers, and various tools for archival work with film. I’ve enjoyed getting reacquainted with these tools since I haven’t had much opportunity to use them since graduating from the L. Jeffrey Selznick School of Film Preservation in 2004. These tools remind me of how different my archival education is from my later archival experience and how, although film archivists and digital media archivists share goals, their tools and skill requirements are practically in alternative universes. Back in school I would crank through film from one reel to another with my fingers resting on the edge of the print so that I could examine the frames, splices, and sprockets by sight and by feel to assess the structural and visual condition of the print. The film benches were stocked with tools for fixes such as edge-repair tape, brushes, trichlorethylene, and other machines and bottles. Beyond the tools, film archiving work was packed with tricks of the trade and experience from experiments. In film preservation there was a variety of challenges along with decades of rich technical expertise and creative solutions.

After graduating I became the archivist of a digital media collection (no film) at Democracy Now! Although I understood the aims of audiovisual archiving and rewards of the work, I started my first archival job with little experience in the means or methods necessary to preserve a digital collection. There seemed a particular gap of knowledge within the media archiving community about preservation strategy for non-file-based digital media, such as the DAT and DV tapes that I worked with. With film materials, the archivist could interact with the content through two methods. Firstly, the archivist could project the film and see what it looks like and how it renders itself to an audience. Secondly, an archivist could handle a film print on a bench and spool through a print manually to feel for edge damage, detect physical abnormalities, and make repairs. It is hard to imagine a film archivist working with only one of these methods for media handling, both are essential. However, once out of school and into a digital archive I could initially only use that first type of handling by viewing how a media file ‘projected’ through QuickTime or VLC. The bits inside, codec structure, and architectural patterns of digital media were lost to me. For me, going from a film-based education to responsibilities over a digital collection illuminated a gap in the technical handling skills within the moving image archiving profession. Although I understood the chemistry and structure of sprockets and film frames, I had no practical knowledge of DIF blocks and GOP headers or how to handle them, which seemed particularly important when digital media could be non-standard, broken, or obscure.

A further obstacle was that, at least at the time of my education, many film archivists were slow to respond to the incoming predominance of digital representations of media and how this affected preservation. In one lesson we would learn to use specialist methods and tools to bring a deteriorated nitrate print to a healthy enough state for duplication; however, in another lesson we were taught that digital media will deteriorate in a manner where suddenly the media is completely unworkable and beyond the possibility of repair. From this I tried to imagine the opposite of this situation, such as an unskilled projectionist blindly forcing a fragile film print through a projector, the edge damage of the initial frames causing the projector’s teeth to further damage the film, and fail to present the media. Unplayable and broken digital media may be fixed just as an unplayable film print may be fixed. In both cases success may be limited or unfeasible, but the creative restoration efforts that are possible within a film archive do have a parallel in the digital realm, even though there has been little overlap between these two types of expertise.

One of the distinctions of the Selznick School of Film Preservation is that, although the examples were primarily with film, they teach well the means and reasons for an archivist to have a high degree of technical control over the collections that they manage. In my later work I would struggle to gain the same understanding and control over digital media collections that I had been taught with film collections. With the current educational options for moving image archivists it can still be a challenge to break beyond n00b status to grasp the inner workings of digital media formats, codecs, decoders, and muxers. For me, the best enablers for this have been the use and collaboration with open source projects such as ffmpeg, cdparanoia, dvgrab, mediainfo and others. I hope to highlight some of these applications and their relevance to media archivists in upcoming posts.