As the Software Engineer, Product Recommendations at Klaviyo, you’ll help build the machine learning–powered systems that decide which products to show to whom and when across our platform. You’ll work on large-scale backend and data systems that turn billions of behavioral events into real-time, personalized product recommendations that drive revenue for merchants of all sizes. You’ll join the Product Recommendation team, partnering closely with Machine Learning Engineers, AI Engineers, other engineers, Product Managers and Designers to design, build, and operate services and data pipelines that power our recommendation features end to end—from data ingestion and feature generation to ranking models and APIs exposed in product. This is a hands-on backend role with a strong focus on building scalable systems and data processing frameworks, with prior ML system experience as a plus (not a hard requirement). How you’ll make a difference Design, build, and operate backend services that power product recommendations across Klaviyo experiences (email, SMS, KAgent, onsite, etc.), with a focus on reliability, performance, and clear APIs. Build and maintain large-scale data processing pipelines (e.g., using Apache Spark or similar frameworks) that transform raw events and catalog data into high-quality features and inputs for recommendation models. Collaborate with ML engineers to productionize recommendation models—defining interfaces, feature contracts, and deployment patterns for batch and/or real-time inference. Build ML/AI systems such as vector search that power recommendation, semantic search, and agentic use cases. Implement and evolve data and service observability (metrics, logging, tracing, dashboards) to ensure recommendations are correct, fast, and available when customers need them. Contribute to and improve shared data frameworks, libraries, and patterns that make it easier to build new recommendation use cases and iterate quickly. Work with product managers to break down complex recommendation initiatives into clear milestones, helping balance experimentation speed with reliability and technical soundness. Partner on data-driven decision making and A/B testing—ensuring recommendation systems are instrumented with the right metrics, and helping interpret results to guide future iterations. Participate in on-call and incident response for the systems you own, driving follow-ups that improve the resilience and operability of our recommendation stack. Transform workflows by putting AI at the center, building smarter systems and ways of working from the ground up—for example, using AI to accelerate development, automate tests, or better monitor and debug recommendation behavior. Share knowledge and mentor other engineers on working with large-scale data frameworks, distributed systems, and best practices for integrating ML into production systems. Who you are 3+ years of software engineering experience, including building and operating backend services in production. Strong focus on backend and distributed systems at scale; you’ve worked on high-throughput or highly available services and care about latency, reliability, and operability. Proficient in Python, and comfortable working in at least one modern language used for backend/data work (e.g., Java or Scala). Proficient with big data frameworks such as Apache Spark (or similar technologies like Flink, Beam, etc.) for building batch or streaming pipelines. Comfortable with cloud-native architectures (AWS preferred) and container orchestration (e.g., Kubernetes); able to work with infrastructure and CI/CD pipelines as part of your day-to-day development. Comfortable with data-driven decision making and A/B testing—you understand how to instrument experiments, read results, and fold learnings back into the system. Comfortable designing and querying data models in relational or analytical datastores (e.g., Postgres, MySQL, data warehouses). Familiarity with modern DevOps practices (CI/CD, monitoring, alerting) and how they apply to large-scale data and recommendation systems. Proven track record of owning projects end-to-end—from design and implementation through rollout, monitoring, and iteration—ideally across multiple components or services. Excellent collaborator and communicator: you can explain tradeoffs to technical and non-technical partners and work effectively with ML Engineers, Software Engineers, PMs, and other teams. You’ve already experimented with AI in work or personal projects, and you’re excited to dive in and learn fast. You’re hungry to responsibly explore new AI tools and workflows, finding ways to make your work smarter and more efficient. Nice to haves Previous experience working on product recommendation systems or adjacent ML-powered features (ranking, personalization, search, or similar). Experience in AI/ML systems and products, such as integrating models into production systems or building features powered by ML. Experience training machine learning models (e.g., for ranking, prediction, or personalization), even if you don’t consider yourself a full-time ML engineer. Experience with ML and distributed compute frameworks such as Ray or similar tools. Experience partnering with data science or ML teams to productionize models (feature stores, offline/online parity, model deployment and monitoring). Experience with additional data technologies (e.g., Kafka, Kinesis, Redis, feature stores, or vector databases). Background in e-commerce, marketing tech, or consumer personalization products We use Covey as part of our hiring and / or promotional process. For jobs or candidates in NYC, certain features may qualify it as an AEDT. As part of the evaluation process we provide Covey with job requirements and candidate submitted applications. We began using Covey Scout for Inbound on April 3, 2025. Please see the independent bias audit report covering our use of Covey here
Read Less
