Second Spectrum Reduces Kubernetes Hosting Costs by 90% Using AWS Load Balancer Controller and Amazon EKS

Growing out of a research group at the University of Southern California, Second Spectrum was founded in Los Angeles in 2013. TV viewers of National Basketball Association, Major League Soccer, or Premier League games or matches have seen Second Spectrum technology in action—for example, the use of computer vision to track players’ movements. As a startup, Second Spectrum seeks to quickly experiment with new tool offerings and push the most viable ones into production. To do that, the company needs efficient routing so that it can provision certain parts of its architecture in a replicable way.

For application deployment, Second Spectrum has used containerization since 2014. “Kubernetes is a beast to wrangle,” says Sheldon Kwok, senior director of infrastructure at Second Spectrum. “Because there are so many moving parts, it is extremely time consuming to understand and manage Kubernetes internals in house.

”Within this containerized environment, Second Spectrum has relied on Application Load Balancer since its release in 2016. Application Load Balancer operates at the request level (layer 7), routing traffic to targets like containers or internet protocol addresses. However, the team still faced an inefficiency challenge. “There was a whole extra network step: you had to route through the Kubernetes proxy on a NodePort that wouldn’t necessarily be in the same Availability Zone as your final service,” says Kwok. The Second Spectrum team was keen on eliminating this extra step to improve performance. 

The engineers at Second Spectrum were intrigued when AWS released AWS Load Balancer Controller in 2020, which supported pods, or small groups of application containers that can be deployed and managed in concert. “The direct-pod internet protocol target registration from AWS Load Balancer Controller enabled us to clean up and consolidate our configuration into a simple-to-use Kubernetes input,” says Kwok.

Second Spectrum received further support from Amazon EKS, which handled the Kubernetes API server for the company and served as a more efficient alternative to the self-managed Kubernetes cluster. “Using Amazon EKS takes away all the headache that comes with managing Kubernetes,” says Kwok. “It lets us work on more product-related features.” While improving productivity, the streamlined managed environment of Amazon EKS also enables the company to save on costs: “We previously spent about $1,000 per month just running one Kubernetes cluster for the API servers,” says Kwok. “Amazon EKS costs us around $100 per month.”

AWS Load Balancer Controller and Amazon EKS enabled Second Spectrum to monitor the Kubernetes pods directly and skip the extra step of first going through the instances. With this new configuration, the team noticed a performance improvement of around 5 ms per request—a small but important win when it comes to delivering near-real-time tracking and visualizations for an enhanced viewing experience. This AWS-automated process also greatly reduces the potential for human error in the configuration chain. “The rate at which we experience problems with load balancing and availability should go down by about 75 percent,” says Eric Lui, vice president of engineering operations at Second Spectrum.

The Second Spectrum team estimates that the simpler configuration saves at least 2 weeks of staff time per year because engineers are better equipped to self-serve rather than wait for the infrastructure team. Factor in the use of Amazon EKS—which removes time spent training new hires on operating self-managed Kubernetes clusters—and the expected time savings adds up to approximately 3 full-time equivalent months per year. The company can use just one Application Load Balancer for its entire namespace, and AWS Load Balancer Controller can dynamically scale to handle an influx of incoming requests and automatically stop routing to proxy servers that go down.

The Second Spectrum team sees this new configuration as an important step in its move toward a DevOps approach to software engineering. “We’re looking to put these tools in the hands of developers so that they can take responsibility for their own load balancing configuration: getting it deployed, testing it, and setting up health checks,” says Lui. “That unlocks the capability for people to be more autonomous and more productive. In terms of impact to the organization, that is a key piece.”

Now that the Second Spectrum engineers don’t spend as much time managing Kubernetes clusters, they have more time to explore other opportunities. Going forward, the team plans to take advantage of price savings for elastic workloads by migrating those workloads to AWS Fargate, a serverless compute engine for containers that works alongside Amazon EKS. Also, the new routing configuration is scalable and replicable, so the company can stand up a similar configuration in Europe.

As Second Spectrum continues advancing sports analysis, it expects AWS offerings will play a key role. “AWS is very open to customer requests. We often ask for new features, and our Technical Account Manager has mentioned that other customers have asked for something similar,” says Kwok. “We try to stay on top of all the new functionality AWS provides to make use of AWS as much as possible.” 

Second Spectrum was founded in Los Angeles in 2013. As the official tracking provider for the National Basketball Association, Major League Soccer, and the Premier League, Second Spectrum provides an advanced machine understanding of games that unlocks fast technology-driven insights.

• Reduced Kubernetes hosting costs by 90%
• Expects to reduce error rate by 75%
• Expects to save 3 full-time equivalent months per year in engineering and new-hire training time
• Improved application response time by 5 ms per request