Netflix researchers have developed a new type of AI-powered greenscreen technology that can produce realistic visual effects for film and television in real time.
Green-screen technology is routinely used to capture images of actors that can then be inserted into the foreground of virtual or pre-recorded scenes. For this, actors are filmed against a bright green background, which can easily be digitally isolated and removed.
This process can be done automatically with reasonable accuracy, such as with television weather forecasts, but it can be triggered by green garments or by transparent or fine objects, such as strands of hair. When greater precision is needed in films or television series, specialist operators adjust the settings manually, sometimes taking hours to perfect a shot.
In an effort to create a technique that is both fast and accurate, Netflix has come up with a method called Magenta Green Screen (MGS). Actors are filmed against a backdrop of bright green LEDs as they are lit from the front with red and blue LEDs, which combine to create a magenta glow (see video above).
Because digital cameras work by taking an individual red, green, and blue value for each pixel, this technique has the effect of creating a green channel that captures only the background, making the foreground appear black, and red and blue channels that capture only the background. capture foreground , which makes the background appear black. Together these create the magenta and green appearance.
Film editors can replace the green channel in real time by placing the actors realistically and directly in the foreground of another scene, with even potentially tricky areas, such as clear bottles or the area around hair strands, working smoothly.
“Computers have already provided such powerful tools to make a lot of things easier,” says Paul Debevec, one of the Netflix researchers involved. “[This is] another thing we can make easier so that the talented artists we have can focus on being an artist, actually making things look better.
But there is a problem with the method. Because the foreground is only captured in blue and red, the actors look magenta-tinted. To solve this, Netflix uses artificial intelligence to bring the full color range back to the foreground, using a photo of the normally exposed actors as a reference to create a realistic-looking green channel. This AI works quickly, but not yet in real time, although fast techniques, such as averaging the red and blue channels to create an approximation of a green channel, work effectively enough for the director to monitor while filming.
Drew Lahat at video production company Geiger Post in California says the industry always welcomes new methods, but the technique involves many complex steps, which may not yet be conducive to the fast-paced nature of film and television shooting.
“There are several aspects of this technology that, from a practical perspective, make me wonder if it’s worth it,” he says. “From what I understand, this technique falls apart quickly unless you have an excellent green screen in a fully controlled room.”
Subjects: