Luigi the 10x Plumber: Containerizing & Scaling Luigi in Kubernetes

Luigi the 10x Plumber: Containerizing & Scaling Luigi in Kubernetes

Presented By Walter Menendez
Walter Menendez
Walter Menendez
Senior Data Engineer at GIPHY

Walter Menendez is a Senior Data Engineer at GIPHY, based in New York. At GIPHY, he is responsible for the development and maintenance of all of GIPHY's data pipelines, including on-site impression collection, data warehousing, and search indexing. Formerly at BuzzFeed, Walter spent three years there optimizing the internal data warehousing ecosystem, empowering the analytical approach that BuzzFeed uses for the content creation cycle. Walter studied at MIT where he earned a BS in Computer Science and Engineering.

Presentation Description

We love data because it lets us make data-driven decisions that improve the quality of results our users get from our search engine. Recently, Giphy moved their Luigi pipelines from legacy infrastructure on managed AWS EC2 instances to a new containerized ecosystem inside of Kubernetes to increase the overall latency of our most critical ETLs.

In this talk, Walter will share the exact limitations of Giphy's old infrastructure, how they identified these bottlenecks, and how decided to overhaul their system as they containerized it. Additionally, the speaker will cover how they redesigned Luigi pipelines to work around its core limitations while maximizing its strengths.

Presentation Curriculum

Luigi the 10x Plumber: Containerizing & Scaling Luigi in Kubernetes
32:47
Hide Content