Memoji: AR Innovation and Ecosystem Integration

Intro

I miss the live Apple keynote, just hear that crows reacting. Welcome to a new episode of the A-Positive Podcast where I observe the Apple culture and my daily use of Apple hardware, software, and services. This episode is about Memoji, the personalized 3D Memojis that animate in real time to mirror our facial expressions. I will explore the technology around Memoji, how it leverages Apple’s hardware, software and the implementation across Apple products.

Memoji was first introduced in 2018 as an evolution of Animoji, which launched with the iPhone X in 2017,

The feature used advanced hardware and software to let you create cartoon-like characters based on a few specific emoji, like a dinosaur, a chicken or even an alien face. With Animoji, Apple you can send animated messages, appear as their Memoji in video calls, and even dress up their Apple Watch with their favorite character.

The Technology

Let’s start in 2017, Apple introduced the iPhone 10, also known as the iPhone X, but certainly not the correct pronunciation, with a revolutionary new way to unlock the device. Unlike previous models that relied on Touch ID using fingerprints, the iPhone ten introduced Face ID, utilizing facial recognition to unlock the phone. This innovative feature necessitated the development of new sensors that enabled facial recognition to function even in low light conditions and prevent spoofing using photos or images. Apple referred to this technology as the TrueDepth camera system, which projects infrared dots onto the user’s face and captures depth data to create a 3D map of their face. This foundation forms the basis for Animoji and Apple’s Memoji.

Apple’s Memoji showcase the company’s exceptional augmented reality (AR) and machine learning capabilities. At the heart of this feature lies the TrueDepth camera system, which projects over 30,000 infrared dots onto the user’s face. An infrared camera then reads this pattern to generate an accurate depth map. Using this depth data and imagery, Apple’s ARKit framework performs robust face tracking. In fact, ARKit can detect and analyze approximately 50 distinct facial muscle movements, such as eyebrow raising, jaw movements, and blinking, in real time. These movements are represented as “blendshapes,” which capture expressions, enabling a 3D Memoji to mirror the user’s facial expression instantly. For instance, if you smile, frown, or stick out your tongue, your Memoji replicates these movements with remarkable accuracy and synchronization.

Apple’s machine learning models interpret sensor data and drive animations smoothly. The dedicated Neural Engine in Apple’s A-series and M-series chips are crucial here. They are optimized for real-time face detection and expression mapping. Apple’s first Neural Engine in 2017, made for iPhone 10, could process 600 billion operations per second, but current chips perform many trillions, showcasing the immense ML power available for features like Memoji. This on-device ML, processes incoming camera data to track facial landmarks and emotion cues, rendering the Memoji’s movements instantly. The result is a fluid, lifelike animation.

The algorithm even generates automated lip-sync for speech, ensuring the Memoji’s mouth moves in time with the user’s voice. All this computation occurs locally on the device, utilizing Apple’s chips and ARKit APIs, eliminating the need for an internet connection. This design ensures low latency (for natural-feeling animations at 30-60fps) while aligning with Apple’s privacy stance, as the facial data remains within the device.

Memoji everywhere

So, we can be clear that Memoji feels seamlessly integrated into the Apple experience. It’s also integrated in almost all Apple devices and services. Apple has integrated Memoji functionality into various apps, services, and devices, creating a consistent and user-friendly experience.

First up, iMessage. Memoji first debuted in the native Messages app on iPhone, iPad, and Mac. you can create and customize your personal Memoji character directly within the Messages interface. Once created, you can record short Memoji videos with audio, where your Memoji speaks and moves exactly as you do. Additionally, you can send Memoji stickers — which are static images in poses like smiling or giving a thumbs-up.

Starting in iOS 13, Apple made these stickers available system-wide, even to devices without the TrueDepth camera, enabling virtually all modern iPhones and iPads to participate.

Secondly. Contacts. Memoji has also been integrated into the contact system. Since iOS 13, you can set your Memoji as your iMessage profile photo, representing yourself to others. This means that when someone messages you, you might see their smiling Memoji face as their contact image, adding a personal touch to conversations. And iOS 17 introduced Contact Posters, allowing you to choose how you appear on someone else’s iPhone when you call them. You can set a Memoji as that image for a personalized caller ID.

Memoji, initially designed for texting, has been extended to FaceTime as well. During FaceTime calls, you can replace your live camera feed with a Memoji mask, transforming into your animated character. Kids really like this feature and this always puts on a smile on their face when a face transforms into a dinosaur or unicorn.

This feature tracks your head movement and expressions in real time, creating a seamless AR filter that overlays your Memoji over your face. Group FaceTime also supports this feature, allowing each person to appear as a different Memoji during a call. For this feature to work, you do need a device with the TrueDepth camera to capture the necessary facial data.

This implementation demonstrates Apple’s strategy of using Memoji to unify the user experience of using an iPhone. But it doesn’t stop there. Since iPad Pro models also got Face ID using the same sensors, even more devices support full Memoji recordings and FaceTime Memojis, just like the iPhone.

And with Apple Watch you can also create and edit Memoji right from your wrist using the dedicated Memoji app on the Watch. The app offers a scaled-down version of the editor, allowing you to choose many combinations of features, including skin tone, hairstyle, eyes, outfits and more. Once created, these Memoji sync via iCloud and can be used on all your devices.

One unique feature of Apple Watch is the ability to set your Memoji as a watch face. Your character’s face will appear on the watch dial and even perform with animations when you tap on it. This personalization makes the Watch even more engaging and relatable, literally bearing your cartoon likeness. The integration of Memoji into the Watch demonstrates how it extends beyond communication and into device personalization. Even without a camera on the Watch, Apple ensures that your custom Memoji is accessible and viewable, showcasing the cross-device continuity of Memoji.

I’ve seen this being used for people’s loved ones, but also to recreate a Santa during the Holiday season.

Throughout the system, your Memoji can be your Apple ID profile image, which means it shows up in apps like Settings, shared family albums, or in Find My. Additionally, since Apple treats all emoji and stickers similarly, Memoji are available in third party apps as well expanding outside the regular ecosystem. All these touch points mean your personalized Memoji is always accessible, whether you’re sending a message through Whatsapp or posting on social media, your Memoji can represent you.

Even the Messages app on Mac gained the ability to create and send Memoji stickers, similar to iOS. Apple essentially ported the iOS Memoji editor to the Mac, allowing you to design your Memoji using a keyboard and mouse.

These Memoji sync across devices, ensuring that a Memoji created on a Mac appears on iPhone. And the following year, Apple further enhanced the feature by enabling an animated Memoji as user profile pictures on Mac. You can set your account photo to be an animated Memoji, with the Memoji’s face subtly animating and reacting, such as sleeping when the Mac goes to sleep. This subtle touch brings Mac accounts to life and mirrors a user’s identity across devices.

While Macs lack a TrueDepth camera, the system still displays pre-made animations and stickers. It’s these integrations on macOS that ensure Memoji is not a mobile-only feature but an integral part of the Apple experience on desktop. From Messages conversations on Mac to the login screen, your Memoji persona remains consistently present.

Engagement and Personalization

Memoji have proven to be more than just a gimmick; they actively enhance user engagement by making digital communication more personal, expressive, and enjoyable. By allowing you to create a digital caricature of yourself, Apple tapped into a powerful form of personalization that resonates with all ages. Each Memoji is essentially a custom emoji representing you, which is a significant departure from the generic smiley-face emoji.

The creation process itself, with choices like hair style and face shape, gives you a sense of ownership and identity with your device. From freckles or accessories like hats and even AirPods, enabling people from all cultures and styles to create a Memoji that truly reflects their appearance and personality. This inclusive range of options, which has expanded over time to include religious headwear, hearing aids or oxygen tubes, ensures that people can see themselves represented in their technology, which leads to a connection and increased frequent use of this feature.

Especially in Messages this is one of those features than truly enhances a digital conversation instead of drifting away from a real conversation. It adds on to emotion, facial expressions, tonality and use of other non-verbal signs. The tapback feature for example is also one of these additions to Messages that really enhances the experience of a digital conversation. It’s not just a sticker to react, but rather expressions that might occur during conversations, including distinctive sounds and a written “haha” instead of a laughing my ass off emoji, which probably is hardly ever the case.

Anyway, back to Memoji.

From an engagement standpoint, Memoji transforms messaging into a playful and creative activity. Instead of static text or the same old yellow emoji, you can send something that animates by winking, exactly as someone would in real life. This unique feature often evoke stronger reactions from friends, making conversations more engaging and enjoyable.

For instance, a joking message accompanied by your Memoji’s dramatic eye-roll or big grin can land much better than the words alone. This not only makes conversations more lively but also keeps you within Apple’s messaging ecosystem because it’s enjoyable to use.

And with most things Apple, it’s not just a one time effort to show off a feature for that year’s Worldwide Developer Conference. Apple continually introduces new styles, stickers, and customization elements, prompting you to revisit and update your Memoji. Many enjoy tweaking their Memoji to match a new hairstyle or outfit, effectively treating the Memoji as a digital doll or extension of themselves.

This creative investment leads to greater attachment to Memoji and an increased usage. For example, when Apple added outfits and clothing options, you could dress up your Memoji for different seasons or moods, further personalizing the experience. The ability to personalize also ensures that no two Memoji are identical. Your friends’ Memoji will have a unique detail, making using them in group chats entertaining. It’s fun to see a friend’s Memoji pop up with a signature accessory or expression that you recognize from real life.

This uniqueness and novelty factor keeps you engaged and can even drive device adoption. Would a features like Memoji be a reason for you why you enjoy the iPhone over other platforms?

Updates to Memoji

Talking about the ongoing investment in to Memoji, let’s see how Apple kept adding enhancements that keep Memoji fresh and relevant.

In iOS 13 Apple introduced additional personalization options, such as braces on teeth, piercings, new hairstyles, makeup, and even the ability to add AirPods to your ears.

By iOS 15, Apple added over 40 new outfit choices, allowing your Memoji to be dressed in anything from a casual hoodie to a suit. They also introduced more styles for people who wear glasses and multicolored headwear, which is useful for representing sports team colors or a country’s flag colors. The addition to accessibility-focused options, enables you to equip your Memoji with a hearing device, an oxygen tube, or a soft helmet, ensuring that people with diverse medical devices or needs see themselves represented.

Since iOS 16, you have the option to customize body shape, and the expansion of the Memoji stickers library, which are probably even more populair, with poses like the shaka sign, a hand wave, and a lightbulb idea, to provide more ways to react in chats.

With iOS 17 we got an overhaul of the iMessage app drawer, where Memoji stickers got grouped alongside emoji, making them easier to use across apps.

Overall, Apple’s consistent stream of Memoji updates, ranging from fun cosmetic additions to broader integration into the UI, suggests that Memoji are not a mere novelty but a growing aspect of the ecosystem. Apple often highlights these additions during WWDC or product launches, emphasizing the enduring popularity of Memoji. By responding to user desires, such as increased personalization, more ways to use Memoji, and staying attuned to cultural trends, like adding masks to Memoji during the 2020 pandemic, or seasonal outfits, keeps you engaged with Memoji year after year.

Ecosystem Integration

I had no idea I had so much to say about Memoji but before I wrap up I want to highlight the ecosystem integration once more. As one of the key strengths of Memoji lies in Apple’s ecosystem strategy.

Apple ensures a seamless cross-device experience for your Memoji. If you create or edit a Memoji on your iPhone, it will automatically appear on your iPad, Mac, and Apple Watch. You might start a message on your phone with a Memoji sticker, reply later from your Mac with the same Memoji, and see it as your watch face at a glance. This kind of consistency is typical of the Apple ecosystem, reducing friction and making using multiple Apple products feel like a unified experience rather than separate platforms.

In daily use, your personalized emoji “self” is always at your fingertips when communicating or customizing, subtly encouraging you to use iMessage/FaceTime on all your devices rather than switching to non-Apple services. And if you are not aware of this, you will quickly know what your were use to and missing out on.

It’s a testament to Apple’s seamless integration of hardware, software and services. With the use of custom Apple silicon, in this case the Neural Engine, the TrueDepth camera, and ARKit software. It works out of the box and effortlessly on you’ devices.

Memoji and Animoji are exclusive to Apple’s platforms, making it difficult to send live Memoji on Android phones or use them in non-Apple video calls. This exclusivity creates an incentive for friends and family to communicate through Apple’s services. I often get remarks if I use a live reaction during video calls with non-Apple devices. Seeing balloons drift around me with depth like they are really there always have people amazed. Asking how do you do that.

On top of all this, Apple’s ecosystem also prioritizes privacy, and Memoji benefits from this. Face data used to animate Memoji remains on the device, not uploaded to servers and not collecting visual data. This is an advantage over some third-party apps that process facial images in the cloud.

Integrating Memoji at the OS level ensures on-device processing for performance and privacy. Over time, this commitment builds user trust in biometric features, making them integral to daily life, like a trusted friend that brings joy while respecting privacy.

Conclusion

Apple’s Memoji technology exemplifies how the company leverages advanced technologies like Augmented Reality and Machine Learning to create a user-centric feature. It provides joy, personal connection, and seamless integration across devices. Technologically, it’s impressive, but more importantly, it’s tailored to human use, enhancing communication and self-expression.

By spreading Memoji across iPhones, iPads, Macs, and Apple Watches, and regularly updating it with new content, Apple has made it an enduring part of the ecosystem’s appeal. As Apple’s ecosystem expands, with new platforms like visionOS, it’s likely that our digital personas is pioneered by features like Memoji, and will play an even more significant role in our daily interactions with technology. The success of Memoji so far demonstrates how effectively Apple integrates technology and user experience to strengthen the bond between you and the Apple ecosystem.

Links

Apple previews Animoji technology https://www.apple.com/jo/newsroom/2018/01/apple-previews-ios-11-3/ Introduction of Memoji https://podcasts.apple.com/nl/podcast/apple-events-video/id275834665?i=100043069265 Blendshapes that ARKit uses to describe faces https://arkit-face-blendshapes.com