How to detect Android motion photos in Flutter

May 09, 2023
neeraj@ente.io

Android Motion Photos and iOS Live Photos are similar features available on Apple and Android devices, respectively. A motion photo is an image that captures a brief moment of video and sound before and after the shutter button is pressed.

Ente Photos was one of the first end-to-end encrypted photo storage apps to add complete support for iOS Live Photos. Our supporters have been asking us to add support for the Android version of Live Photos. To our surprise, we could not find any open-source library that could reliably identify motion photos in Kotlin/Java or Dart. So, we started exploring, and this post is a short summary of our findings.

Format

In iOS, a Live Photo consists of two separate files: one image and one video. Both of them are linked together by a common identifier. On the other hand, in Android, the Motion Photo is represented by a single file. The image bytes are followed by the video bytes at the end, as shown in the figure below.

Github action permissions

Unlike in iOS, in the Android ecosystem, there is no fixed specification for motion photos. Various vendors have their own terminology and slightly different implementations. For example, in Samsung, it is called Motion Photo, in Motorola it's Active Shots, and in OnePlus it's Live Photos, among others.

We were surprised to find that there is no Kotlin/Java library that works out of the box for identifying Android motion photos. The primary reason for this could be the lack of any official documentation, as different vendors might implement it slightly differently.

Detect Motion Photo

There is no official documentation available on the format of Motion Photos. Luckily, there are various open source scripts written in different languages that work for a subset of various formats of Motion Photos. Based on our limited research, there are broadly two methods to detect Motion Photos:

  • Parse the XMP data for identifying motion photos and the video offset.

    For example, Motion Photos taken from Pixel devices have a key name of GCamera:MotionPhoto (GCamera:MicroVideo in older versions). To extract the Video offset (from end), you can locate the value of the Item:Length key. For older versions, the key name for the Video offset might be GCamera:MicroVideoOffset instead of Item:Length.

  • Traverse the file types and look up for known values of File Type Box (ftyp) values for the video.

    File Type Box is a box or atom in the ISO base media file format (ISO/IEC 14496-12) used to identify the type of a multimedia file. If a Motion Photo contains MP4 video, then it will contain the value “mp4”.

The second method might not be the most elegant or performant way to detect Motion Photos, but in our experience, it's the most reliable way to support formats from various manufacturers.

Motion photo package

We have published a package called motion_photos on pub.dev based on our findings. This package can be used to detect and extract video from motion photos."

To use this package, add a dependency in pubspec.yaml file.

dependencies:
 ...
 motion_photos:

Sample code to detect and get video from a motion photo.

final motionPhoto = MotionPhotos(motionImagePath);
final isMotionPhoto = await motionPhoto.isMotionPhoto();
final videoFile = await motionPhoto.getMotionVideoFile();

If you notice that the package doesn’t work for any particular variant of motion photo, please create a Github issue or a pull request with the fix.

Shoutout to our rockstar contributor Muhesh who did most of the research and created this plugin.

If you'd like to hang out with a bunch of engineers building privacy-friendly apps, join us on Matrix or Discord.