All speakers

Mark Weiss

Senior Software Engineer, Beeswax

Mark Weiss is a Senior Software Engineer at Beeswax, the online advertising industry’s first extensible programmatic buying platform, where he focuses on designing and building data processing infrastructure and applications supporting reporting and machine learning. He has previously held various engineering individual contributor and leadership roles, and has worked on ETL systems and data-driven distributed platforms for much of his career. Mark has spoken previously at DataEngConf NYC, and regularly speaks and mentors at the NYC Python Meetup. He is also blogs and hosts the podcast "Using Reflection" at http://www.usingreflection.com, and can be found on Github, Twitter and LinkedIn under @marksweiss. He lives in Brooklyn, NY

Mark Weiss

Sessions / 2018 / 1 talk

  • Data Pipeline Frameworks: The Dream and the Reality

    There are several commercial, managed service and open source choices of data pipeline frameworks on the market. In this talk, we will discuss two of them, the AWS Data Pipeline managed service and the open source software Airflow. These frameworks have very different feature sets and operational models, however, they have both benefited us and fallen short of our needs in similar ways. To understand the reasons, we analyze our experience of first building a data processing platform on Data Pipeline, and then developing the next generation platform on Airflow. We find that managed service and open source framework are leaky abstractions and thus both frameworks required us to understand and build primitives to support deployment and operations. Likewise, we discuss the necessity of implementing cross-cutting aspects such as logging, monitoring, security and configuration, which arises from the shortcomings of existing pre-implemented components. Generalizing from specific pain points and solutions, we posit that almost any organization building a data platform using a pipeline framework or service will run into many of the same issues, because opinionated framework/service implementations will conflict with an organization's existing code, preferences and procedures. So where is the line? What value can you expect to get from a data pipeline framework or service? What will you need to wrap, integrate with or fully implement yourself? To develop a robust data pipeline platform for your organization, you will need to bridge the gap between the framework dream and production reality. This talk will help you do that.

Sessions / 2017 / 1 talk

  • Applying Design To Solve Scaling Problems and Evolve an Architecture

    Beeswax is a high-scale internet-ad buying platform that processes over a million events per second, and differentiates from competitors by gathering and processing that event data and making it all available to customers. The business has grown quickly, forcing it to address scalability and flexibility issues with its existing data processing architecture. Faced with resource and time constraints, Beeswax was able to both solve pressing operational concerns and create the architectural flexibility to add new data sources in the present and leverage new data processing technologies in the future by applying design patterns to the existing data processing technology stack. In this talk, we will describe the challenges, commonly occurring in data engineering, and approaches that we undertook to address them.

Ready to take this stage?

The next edition is programmed by practitioners. Tell us what you built and what you learned.

Apply to be a speaker