product DS @ meta

I had a chat with a staff DS at Meta in the Gen AI product and growth team.

It inspired me to pursue product DS, because I believe it's a good skill set to have if I'm building a startup, having product sense is useful.

DS at Meta

  • Mission: driving impact through data-informed decision making.
  • Misconception: It's not just about using fancy machine learning techniques, but about leveraging data to improve products and user experiences.
  • Specialization vs. Full-Stack: While smaller companies might require full-stack data scientists, Meta's scale allows for more specialized roles. This enables deeper expertise in specific areas of data science.

key areas of focus

  1. product
    • Use both descriptive and predictive analytics to uncover insights
    • Examples:
      • Analyzing user engagement with Meta AI
      • Determining which countries or languages to prioritize for expansion
      • Deciding on UI elements like whether to show a search bar in WhatsApp
  2. ai eval
    • Goal: Create the best AI assistant possible
    • Key metrics: User acquisition, engagement (daily/monthly active users), retention
    • Evaluation methods:
      • Instruction following accuracy
      • Academic benchmarks (continuously evolving)
      • Custom evaluation metrics tailored to Meta's specific needs
  3. product-market fit analysis
    • View product adoption as a funnel
    • Key metrics:
      • Adoption rate (e.g., percentage of WhatsApp users using Meta AI)
      • Retention (monthly return rate)
    • Use techniques like A/B testing and holdout tests to measure impact
  4. gen ai advancements
    • Develop retrieval mechanisms for real-time knowledge supplementation

essential skills

  1. hard skills
    • Strong foundation in statistics and probability
    • Engineering mindset for building and implementing solutions
  2. soft skills
    • Effective communication to convey insights and build trust
    • Product sense (especially for product scientists)
    • Analytical thinking and attention to detail
  3. passion
    • Think deeply about products and user experiences, obsess over the details
    • Question design decisions and try to reverse-engineer product changes

some resources for Meta DS interviews