According to the report reaching us from our sources, YouTube is currently working on its own take on stories which is at this time being experimented on in its beta mode with a small group of top creators. At the moment, the company is including the next logical feature from Snapchat and Instagram to those stories: Augmented Reality (AR) green screen filters called mobile real-time “video segmentation”.
The technology which is explained in full on Google’s research blog seems to be in actual fact quite laudable. It is an easy task to figure out where the foreground ends and the background begins if you have a depth-sensing camera (like the iPhone X’s front-facing array) or plenty of processing time and no battery to think about (like a desktop computer). On mobile, however, and with an ordinary RGB image, it is not so much as easy. And, if doing a still image is hard, a video is even more difficult, since the computer has to do the calculation at a minimum of 30 times per second. This is why the company employed the use of neural network architecture to identify and separate a subject from the background. It is of a like net effect as a green screen, giving creators opportunity to insert whatever they want as their new desktop backdrop in real time, but it has been made all the more impressive because it doesn’t require the use of an actual green screen to make separating the subject easier.
The system is however not yet 100 percent perfect, and it has a lot in common with portrait mode features on newer smartphones which are essentially performing the same sort of task in the sense that there is still some blurring or ghosting around the edges where the masking is not quite there. Nevertheless, it is still an impressive technology to see running on a smartphone.
In this its early stage, Google’s plan is to use the beta of YouTube stories which is already limited to further test and improve on the technology and as they progress towards that, the green screen technology could show up in the other Augmented Reality (AR) services and apps in the future.