file name

Leap

what Goran did

Node.js / MongoDB / AWS / Terraform
Custom API
Unit tests
Integration with 3rd party services
AWS infrastructure controlled by Terraform

Activities

Node.js development
MongoDB database management
API development and maintenance
Migration from EC2 to AWS Lambda
Infrastructure management using Terraform

Description

Leap is NBC's API written in Node.js, utilizing MongoDB as its database. The API handles a significant load, such as during the Olympic Games, where it processes around 80 million requests per hour. This project marked a new and exciting experience after many years of programming in Drupal, PHP, and relational databases.

The primary job of Leap is to gather JSON files from iStreamPlanet and MPX, store them, and provide API access to query and transform that data into specific formats.

The primary consumers of the API are end-user applications such as the NBC Sports App, Telemundo Deports, NBC Sports Gold, and other video-driven applications.

Challenges

Strict Deadlines: There was no room for delays. Initially, I was part of a team, but in the second phase, I co-led the project with an architect.

Undocumented Code: The custom API initially lacked documentation, making it challenging to understand how it worked and how to troubleshoot issues.

High Traffic Management: Managing an API with constant high traffic was critical. The API consists of two main parts:

Ingestors: These continuously gather JSON files from specific servers, process them, and import them into the database. NBC constantly releases new content such as TV programs and events, and the ingestors never stop monitoring, collecting, and processing these files.
API: This part of the application is used by most NBC services.

Maintenance of EC2 Machines: After the initial project completion, another issue arose: NBC lacked sufficient personnel to maintain the EC2 machines.

Solutions

Migration to AWS Lambda: To address the maintenance issue, the architect proposed migrating everything to AWS Lambda. The concern was whether Lambda could handle the high volume of traffic efficiently. My task was to migrate the entire API from EC2 servers to Lambda.

Learning and Implementation: Despite having no prior experience with DevOps on AWS, the client was satisfied with my Node.js application development and paid for a month of learning to ensure I could handle this task.

Terraform: I decided to control the entire infrastructure using Terraform, which proved to be an excellent decision.

Results

Successful Migration: Successfully migrated the API to AWS Lambda, ensuring it could handle the high request load.

Improved Maintenance: The migration reduced the need for extensive manual maintenance of EC2 machines, streamlining operations and improving efficiency.

Enhanced Infrastructure Management: Using Terraform to manage the infrastructure provided better control and flexibility.