System design of Meta's FoA: Part 1 Instagram Feeds
Welcome to our new series on uncovering the system design of Meta's family of apps (FoA)!
Meta operates some of the world's most influential platforms. Those include Facebook, Instagram, WhatsApp, Threads, and the metaverse. Ever wondered what makes these apps tick? In this series, I'll explain how Meta is able to achieve reliability, scalability, and security across its family of apps. From the seamless experience of Instagram feeds to the WhatsApp messages, which have strong end-to-end encryption. We will discover the engineering marvels and cutting-edge technologies that keep billions of people around the world connected every day. Get ready to delve into the digital backbone of Meta!
In this first post, we will review how Instagram's Feeds work. With over a billion users, Instagram relies on complex backend systems to provide a smooth user experience. What happens when a user posts on Instagram, and how are the feeds generated? Let's take a look step by step.
Post Creation
A user uploads a photo or video with a caption using the Instagram mobile app. The media, along with its metadata, is sent to Instagram's backend servers over HTTP using GraphQL APIs.
Technologies Used
React Native is used to build Instagram's mobile app for both iOS and Android platforms.
The media file and post data are sent to Instagram's backend servers via secure HTTPS connections using RESTful APIs.
Storage
Once the data is received by the backend servers, the media is stored in a distributed storage system. Metadata (caption, tags, location) is saved in databases.
Technologies Used
TAO (The Associations and Objects) is a high-performance service developed by Meta. Instagram uses it for storing, caching, and querying the graph.
Cassandra is used for high-availability storage. It stores user interaction data and large-scale metadata for Instagram.
Everstore handles big data storage. Instagram uses it, especially for storing media and blobs.
ZippyDB is Facebook's key-value store. It is strongly consistent and globally distributed.
Indexing
Indexing involves extracting metadata from posts and making an inverted index using Unicorn. This allows for fast retrieval of relevant posts, enabling Instagram to provide users with a personalized feed that reflects their interests and interactions.
Technologies Used
Unicorn is a search backend used by Instagram. It indexes and retrieves posts based on their metadata. Unicorn is built on top of a combination of technologies, including:
Lucene: Lucene is an open-source search library that provides the core indexing and searching functionality for Unicorn. It enables efficient indexing and retrieval of documents based on their metadata.
Inverted Index: An inverted index is a data structure used by Unicorn that maps terms, such as hashtags and locations, to the posts that contain them. This enables fast retrieval of posts that match specific criteria.
Term-based indexing: Unicorn uses term-based indexing to create an index of all the unique terms in the metadata, along with the IDs of the posts that contain those terms.
Feed Generation
The post is added to the feeds of the users' followers. The feed service queries the database and generates the feed, adding the new post. Machine learning algorithms rank the post based on many factors, including engagement predictions, recency, and relationship strength.
Technologies Used
Python is a programming language used by Instagram to implement the feed generation logic.
Django is a web framework used by Instagram to handle HTTP requests and responses.
Machine learning algorithms are used in feed generation, including CNNs and RNNs to analyze user behavior and predict engagement, Naive Bayes to filter out irrelevant posts, and matrix factorization to generate recommendations.
Conclusion:
Instagram's feed generation process is a complex system that uses various technologies and algorithms to provide users with a personalized and up-to-date feed. By understanding the underlying design and technologies, we can gain insights into building scalable and reliable systems for billions of users.
Stay tuned for more exciting discoveries as we continue to explore the system design of Meta's family of apps! You'll learn about the latest technologies and innovations that power these platforms, and how you can apply these concepts to your own projects and endeavors. Whether you're a developer, entrepreneur, or simply curious about the tech world, our future posts will have something for everyone. So, keep an eye out for our next post and join us on this journey of discovery!