The Virtues of a Programming your ETL Pipeline using a Fluent Interface

Introduction: The Fluent Interface Pattern

This week marked a new chapter in my career. I started a new job as a Data Engineer for Synaptiq building ETL pipelines that feed into ML and AI models. Looking back at my (nearly) year long career at Sequoia (A First American Startup), I cant help but feel grateful for the opportunity to have worked with such a talented team. I learned a lot about Data Engineering. One of the most underrated things I learned in that role came in the form of a design preference from my Team lead. Often he would ask me to write additional transforms and apply them in pipelines using the Fluent API

Continue reading »

Development Containers in VSCode and Github Codespaces!

Development Containers

Intro

If you're involved in a collaborative project, you're likely familiar with dev containers. They serve as a crucial tool for ensuring consistency in development environments across team members. This means you're guaranteed to be working with the same Python version, have the necessary drivers, and more.

In my current role, we leverage dev containers in conjunction with Poetry not only to enforce a standardized development environment but also to establish a uniform Python virtual environment.

Continue reading »