When Hyungtae Kim joined Snowflake, he stepped into an AI landscape still taking shape. The challenge wasn’t just about integrating AI into Snowflake’s vast data ecosystem, it was about redefining what’s possible.

With GPUs introducing new complexities, workloads ranging from massive batch processing to real-time inference, and a growing need for AI infrastructure that could scale seamlessly, Hyungtae and his team began designing the foundation for AI workloads at an entirely new scale. The result? Snowflake Cortex. A suite of AI features built to power the next generation of data intelligence.

In this Q&A, Hyungtae shares the journey of building Cortex from the ground up, the technical hurdles his team overcame, and how they’re driving continuous improvements to help customers solve bigger, more complex problems with AI.

Introducing Cortex Infrastructure: Built[By] Hyungtae Kim

Let's go back to the beginning. What is Snowflake Cortex, and how did the AI infrastructure team begin building it?

I joined Snowflake in 2023 through the acquisition of Neeva. At the time, I remember the transition from a startup to a company at Snowflake’s scale felt both exciting and challenging. But when I first arrived, Snowflake’s AI efforts were still in their early stages. The question wasn’t just where AI fit within Snowflake’s ecosystem—it was how we could make the biggest impact.

Initially, we looked at Search as a natural entry point, but it quickly became clear that Snowflake was making a significant investment in AI. We saw that as an opportunity to build something bigger: a foundational AI platform that could support a wide range of workloads beyond just Search. That’s how we started shaping the Cortex platform.

Laying the Groundwork: Turning an Idea Into Infrastructure

How did Snowflake build the infrastructure foundation for Cortex?

One of the first projects my team tackled was designing a co-pilot. But as we assessed the broader AI initiatives, like catalog search and search integration, we quickly realized that a common service was needed to power LLM inference across multiple products. This led to the formation of a new team to build the infrastructure component of Cortex.

We were doing a lot of firsts at Snowflake and started asking questions like:

How do we build a service on top of GPUs?
How do we schedule and deploy AI jobs?
How do we implement multi-machine communication properly?

Snowflake’s existing compute infrastructure was optimized for traditional workloads, but GPUs introduced an entirely new set of challenges. Through rapid iteration, we developed the AI Reference Cluster—an infrastructure layer that incorporated stability mechanisms, capability management, and prioritization scheduling. This became the foundation for key Cortex services, including inference and training.

Scaling the Impossible: The Unique Challenges of AI Infrastructure

What are the biggest challenges in building scalable AI infrastructure at Snowflake?

The challenges we face in AI infrastructure aren’t just about engineering—they’re economic. GPUs are scarce, expensive, and inherently unstable. Their higher failure rate compared to traditional compute resources forces us to rethink systems design, prioritizing efficiency, resource sharing, and resilience.

The other unique challenge that makes this work so interesting is that we service a lot of really big and really small workloads. What that means is that we have customers running an LLM inference service on, let’s say, a 6 million row table. This is a giant batch processing workload that has specific expectations and characteristics, and we need to support that in our system. Then, on the other hand, we have interactive experiences, like Copilot chat and Cortex Analyst. Both of these have very different expectations. We need to be able to service both types of workloads in one system, which introduces a lot of unique challenges: servicing both of these extremely different workloads while meeting customer expectations at the same time.

Less Complexity, More Possibility: What Cortex Delivers for Customers

What makes this feature so significant to Snowflake and our customers?

Cortex is a game-changer. It’s a fully managed Snowflake service that abstracts away the complexity of AI infrastructure for our customers. They don’t need to think about provisioning GPU machines or tuning inference pipelines; we handle it for them.

By integrating AI directly into the Snowflake ecosystem, we empower customers to unlock new capabilities without the typical infrastructure headaches. Snowflake’s vision has always been about simplifying data-driven innovation, and Cortex extends that philosophy into the AI space.

For example, Headset managed to reduce a workload that used to take 20 minutes to 20 seconds by migrating to Snowflake Cortex, removing the need for auxiliary services and enhancing data governance.

Engineering Without a Blueprint: Turning Possibility Into Platform

What makes Snowflake a great place for engineers to make an impact in AI infrastructure?

Cortex represents a rare opportunity to work at the cutting edge of AI infrastructure. The industry is advancing rapidly, hardware changes annually, and model sizes continue to grow exponentially. So, working at Snowflake is less about applying existing solutions and more about solving novel technological challenges.

The problems that we're dealing with are generally quite unique, and they don’t have a clear counterpart in the industry to pull inspiration from. Every day, we get exposure to this rapidly evolving, fast-paced industry, so we’re working with leading AI experts and some of the best researchers doing AI inference, optimization design, and systems design at Snowflake.

It’s a fast-moving, green field, so these are new challenges that have never been solved before. It’s a very exciting place to grow, learn, and explore new things every day.

The Why Behind the Work

Building something meaningful takes focus, collaboration, and inspiration. What keeps you motivated?

It’s definitely the team. We’ve kept that sort of startup DNA at Snowflake. We’re still moving fast and breaking things (much less frequently), just at a bigger scale to keep up with the pace of innovation in this field. I’m constantly learning from my colleagues in areas like system stability and customer-driven consistency. The pace of execution is fast, and we’re in a unique position to figure out how AI transforms workloads in real time.

Our focus isn’t just on building AI features; it’s on enhancing what customers can achieve with rapidly evolving AI capabilities. Snowflake provides the perfect environment for experimentation, allowing us to tackle novel technical challenges while integrating our work into real-world use cases. One of our key partners in this journey has been the Cloud Engineering team. Since the Neeva acquisition, we’ve worked as a unified group to accelerate the development of GPU capabilities, bridging gaps in compute infrastructure and redefining Snowflake’s approach to AI.

The Culture That Makes Cutting-Edge Work Possible

How would you describe the culture of the team?

We’re balancing short-term innovation with long-term business goals. It’s challenging, but it’s not ambiguous. There’s clear direction and focus from the company. It’s just a matter of how fast we can get there.

Snowflake’s engineering culture is all about continuous evolution. You’re pushed to grow, but you’re also supported along the way. No one is set up to fail, and the expectations are high because the potential for impact is enormous. There’s always more to build, more problems to solve, and more opportunities to drive meaningful change.

For engineers looking to work on cutting-edge AI infrastructure, there’s no better place to be.

The Future of AI Infrastructure at Snowflake

Last but certainly not least, what’s next for Cortex?

My vision for Cortex is simple: make it the fastest, most reliable, and most cost-efficient platform for AI workloads. We’re constantly refining our approach to ensure customers can harness AI effectively without friction. The goal is to make Cortex the default choice for AI within Snowflake - a seamless, intuitive platform where innovation happens effortlessly.

From idea to execution, hear more stories like Hyungtae’s at careers.snowflake.com/builtby.

Hyungtae Kim is a Principal Software Engineer shaping the next generation of AI infrastructure for Snowflake Cortex. With a deep background in distributed systems and machine learning, he plays a key role in building scalable, high-performance AI services that power Snowflake’s enterprise AI capabilities. Before joining Snowflake, he worked on large-scale search and AI-driven systems at Neeva and Google. He holds both a Bachelor’s and Master’s Degree in Computer Science and Machine Learning from Columbia University.