Meta’s New AI SAM 2 Can Segment Anything, Including Video

It’s been an exciting few weeks for Meta as far as its progress with artificial intelligence (AI) and generative artificial intelligence (GenAI) goes.

Recently, Meta introduced its new AI Studio, which would allow users to create, share, discover and chat with two types of AIs, one being AI characters built by content creators and the other AI characters based on user’s interests.

This followed Meta’s release of its Llama 3.1 405B, a new model that represented its first frontier-level open-source generative AI model.

The move came as Mark Zuckerberg voiced his support for open-sourcing AI software and models, which even saw a blog post titled “Open-source AI Is the Path Forward” being published.

The company has been pushing on various fronts to advance the application and research of AI and other technologies through collaboration, which included bringing AI tools to WhatsApp Business, sharing its mixed reality operating system with third-party hardware companies.

In its latest move, Meta has made an announcement pertaining to its April 2023-released Meta Segment Anything Model (SAM), one that will benefit image and video editing, AI application development, computer vision systems training and more.

So, what updates did Meta reveal about SAM and what benefits will they bring? Let’s explore!
 

What’s New With Meta’s Segment Anything Model?

 

• Through a news release published on its website, Meta announced the release of Meta Segment Anything Model 2 AKA SAM 2, the latest and newest version of its Meta Segment Anything Model.

• Segmentation refers to the capability of identifying which pixel belongs to which object in an image or a video.

• Meta’s previously released SAM helped the company develop AI-powered image editing tools in its apps, which included Backdrop and Cutouts on Instagram and WhatsApp.

• Furthermore, this capability spurred diverse applications in science, medicine and other industries, including being “used in marine science to segment sonar images and analyze coral reefs, satellite imagery analysis for disaster relief, and in the medical field, segmenting cellular images and aiding in detecting skin cancer.”

• Meta’s SAM 2 brings these capabilities to videos.

• The model can segment any object in an image or video and consistently follow it across all frames of a video in real-time.

• This has been a challenge faced by existing models, especially when objects can move fast, change in appearance and be concealed by other objects.

• “We solved many of these challenges in building SAM 2.”

   

What Benefits Does SAM 2 Offer?

 

• SAM 2 offers enhancements to numerous real-world applications, while also enabling many potential use cases.

• This includes using generative video models to create new video effects and unlock new creative applications.

• It could also aid in faster annotation tools for visual data to build better computer vision systems.

• SAM 2 outshines previous capabilities and achieves better video segmentation performance with three times less human-in-the-loop interactions.

• Meta shared its research on SAM 2 through a published paper accessible by everyone, as the company keeps to its “open science” approach.

• Additionally, Meta is sharing its SA-V dataset, which was used in building SAM 2. It includes approximately 51,000 real-world videos and more than 600,000 masklets.

• The company is also sharing SAM 2’s code and model weights with a permissive Apache 2.0 license. This allows anyone to build their own experiences.

• Meta is also releasing a web-based demo experience, which enables real-time interactive segmentation of short videos and applies video effects on the model predictions.

 

What Did Meta Say About SAM 2?

 

• Through the news release, Meta said, “We believe this research can unlock new possibilities such as easier video editing and generation, and allow new experiences to be created in mixed reality.”

• “SAM 2 could also be used to track a target object in a video to aid in faster annotation of visual data for training computer vision systems, including the ones used in autonomous vehicles.”

• “It could also enable creative ways of selecting and interacting with objects in real-time or in live videos.”

• “Keeping with our open science approach, we’re sharing our research on SAM 2 so others can explore new capabilities and use cases.”

• “We’re excited to see what the AI community does with this research.”

Do you think Meta’s new SAM 2 coupled with its open-source collaboration stance will help it boost its customer base and capture a bigger market share in the AI industry?

Do you think its rivals need to make similar moves or introduce AI tools with improved capabilities to remain competitive?

Let us know in the comments below!