Netflix researchers have created a new type of AI-powered green screen technology that can produce realistic visual effects for film and television in real time.
Green screen technology is commonly used to capture images of actors that can then be inserted into the foreground of virtual or pre-recorded scenes. To do this, the actors are filmed against a bright green background, which is easily isolated and digitally removed.
This process can be done automatically with reasonable accuracy, such as in television weather forecasts, but can be thrown off by green clothing or by thin or transparent objects, such as strands of hair. When more precision is needed in movies or TV series, specialist operators manually tweak the settings, sometimes taking hours to perfect a shot.
In an attempt to create a technique that is fast and accurate, Netflix has come up with a method it calls Magenta Green Screen (MGS). The actors are filmed against a backdrop of bright green LED lights while being illuminated from the front by red and blue lights, which together create a magenta glow (see video, above).
Because digital cameras work by taking an individual red, green, and blue value for each pixel, this technique has the effect of creating a green channel that records only the background, with the foreground appearing black, and red and blue channels. that record only the foreground. leaving the background black. Together they create the magenta and green appearance.
Film editors can replace the green channel in real time, realistically and instantly placing actors in the foreground of another scene, even with potentially tricky areas, like clear bottles or the area around strands of hair, working without issues.
“Computers have already provided such powerful tools to make many things easier,” he says. Paul Debevec, one of the Netflix researchers involved. “[This is] another thing we can make easier, so that the talented artists we have can focus on the art, making things look better.”
But there is a problem with the method. Because the foreground is only shot in blue and red, the actors look tinged with magenta. To solve this, Netflix uses artificial intelligence to bring the full range of colors back to the foreground, using a photograph of the actors in normal lighting as a reference to create a realistic-looking green channel. This AI works quickly, but not yet in real time, though fast techniques like averaging the red and blue channels to create an approximation of a green channel work effectively enough for the director to monitor while shooting.
drew lahat at the Geiger Post video production company in California says the industry always welcomes new methods, but the technique involves many complex steps, which still may not be conducive to the fast-paced nature of shooting movies and television.
“There are a variety of aspects of this technology that, from a practical perspective, make me wonder if it’s worth it,” he says. “From what I understand, unless you have a great green screen in a fully controlled space, this technique falls apart quickly.”
