A look back on 2018A look back on 2018

Many data scientists are familiar with word embedding models such as word2vec, which capture semantic similarity among words and phrases in a corpus. However, word embeddings are limited in their ability to interrogate a corpus alongside other context or over time. Moreover, word embedding models either need significant amounts of data, or tuning through transfer learning of a domain-specific vocabulary that is unique to most commercial applications. In this talk, I will introduce exponential family embeddings. Developed by Rudolph and Blei, these methods extend the idea of word embeddings to other types of high-dimensional data. I will demonstrate how they can be used to conduct advanced topic modeling on datasets that are medium-sized, which are specialized enough to require significant modifications of a word2vec model and contain more general data types (including categorical, count, continuous). I will discuss how we implemented a dynamic embedding model using scikit-learn and Tensor Flow and our proprietary corpus of job descriptions. Using both categorical and natural language data associated with jobs, we charted the development of different skill sets over the last 3 years. I specifically focus description of my results on how data science and quantitative skill sets have developed, grown and pollinated other types of jobs over time. I will specifically discuss the following: Introduction to word embeddings models (word2vec, GLoVE) focussing on barriers to real-world/industrial implementation Background on exponential family embeddings (with reference to Rudolph and Blei), focussing on applications of multivariate and Bernoulli models. Description of data used to train the model (size, types of data as well as processing steps that we optimized with) Description of results from model What 'fringe' data science skills have become 'core' data science skills What data science skills have pollinated other types of roles? Where are the primary functions/roles where there is pollination What are 'new' data science skills that are emerging?

Maryam JahanshahiResearch Scientist, TapRecruit

When we talk about data, it's easy to be optimistic - we're inundated with stories about how data has revolutionized business, industries, and governments. But if your data is bad, you're likely to end up worse off than if you had no data. Bad data has the power to cause us to have high confidence in bad decisions at massive scale. This talk will look at the effect of bad data in a machine learning context. We'll cover what can happen when you have bad data and how you can use tools to avoid bad data the future.

Rigel SwavelySenior Machine Learning Engineer, Clarifai

Foursquare is a technology company focused on building a trusted and independent platform for understanding how billions of people move through the real world. This talk will cover the history of Foursquare, from its roots in consumer apps to its recent pivot to building a location technology platform. Lessons learned and to be discussed with examples include: designing the right metrics to track, the value of feature engineering, and the importance of researching your data.

Stephanie YangData Scientist & Technical Lead, Foursquare

At WayUp, the leading platform for connecting college students and young professionals to internships, part-time jobs, and entry-level roles, the technology recommends job listings to users immediately after they join the site and create a profile. In this talk, I discuss how constraints from the business model and site design pushed us to build a custom system that matches the user's profile to job descriptions, even without the interaction data that drives most common recommendation systems. With limited resources, but high expectations, it was important to make good architecture choices that would let the system scale up and evolve with business priorities. Some choices that have paid off include a two-step narrow-down approach using both business rules and machine learning, microservices and proper separation of concerns, careful choices of off-the-shelf technologies and tools, and designing for testability and iteration.

Harlan HarrisDirector of Data Science, WayUp

The talk will give an overview of how the scrappy data engineering team at TripleLift evolved its data pipeline to keep up with its rapid growth which is currently processing tens of billions of events a day. Emphasis will be placed on the major turning points and decisions that required us to tackle the same problems in a different way - both due to new scales of data as well as growing business requirements. The talk will cover the following data technologies and how they were used and modified over the years: Kafka, Redshift, Secor, Spark, Spark Streaming, VoltDB, and Druid.

Dan GoldinVP Engineering, TripleLift

Wes Chow walks through a narrow slice of computing history and illustrates the problems that occur when practicing engineers don't learn the lessons of the past. Along the way, he examines incentives and business models, and argues how we as consumers can better align our behavior to make for a more robust open source ecosystem.

Wes ChowDirector of Engineering, Advanced Analytics, Cortico

Businesses derive value from their customer base. This makes it critical to be able to forecast the value of a customer to the business. At the same time, modeling customer behavior is not a straightforward measurement -- the data generating process is subtle, data is characteristically sparse & stochastic, and heterogeneity abounds. We'll look at business analysis from a customer-centric mindset, where analysis of growth and retention is at the core. We'll dive into the model of Fader & Hardie, which provides a principled, probabilistic model to forecast customer lifetime value (CLV) from a stream of purchases. We'll motivate multilevel models and show how they can account for customer heterogeneity in purchase behavior. We'll analyze predictive fits using lifetimes, an open-source implementation of several useful Bayesian CLV models. We'll look across businesses and briefly cover insights that come from looking at large-scale datasets of customer purchase behavior. CLV estimates have a multitude of business uses, across: * forecasting of cashflow, profitability and demand * investment & valuation * customer base segmentation * allocation of marketing spend, or * monitoring the health of the business Data scientists and business analysts should leave this talk with another tool in their predictive modeling toolkit. More importantly, they should have clarity on how customer lifetime value is defined, how it can be reasoned about, and how forward-looking estimates can be fit from purchase data.

Brian BloniarzDirector of Data Science, Second Measure

While the vast majority of talks at DataEngConf are unabashedly technical, we recognize there are many deeply technical attendees who are thinking about founding, or joining, a startup. In this session we'll feature a unique panel of technical founders who we've hand picked due to their technical ingenuity combined with their business success. Moderated by Pete Soderling, founder of DataEngConf, the session will dig into insights from our panelist's experiences as engineering leaders and product visionaries and uncover startup hacks they used in building data-oriented tools and bringing them to market. Listen and learn to discover how their views can help you develop your early stage product and go-to-market strategy and prepare you, as an engineer, for the types of challenges common to starting a company.

William FalconAI Researcher, Facebook / NYU

Streaming applications can be designed to balance or favor one or more of latency, throughput, memory consumption, or CPU load. In order to scale growing real-time applications well, properties like replayability, at-least-once and exactly-once processing, and out-of-order processing drive decisions that need to be made inside the streaming application and by data producers and consumers. This presentation discusses some useful design patterns for streaming applications that help deliver great value and an exceptional digital personalization experience to our customers, with personalized responses for Capital One's Eno chatbot as an example.

Andreas MarkmannData Engineering Manager, Capital One

Machine learning has revolutionized the capability of businesses to create personalized experiences via real-time, individual predictions and recommendations. But what happens when one must make thousands of decisions for thousands of individuals at the same time? At Dia&Co, a plus-size women’s styling service, we recently faced such an obstacle when building out a brand new product line for the business. This talk will explore how we combined modern machine learning with classical operations research techniques to scale personalization in the face of constraints inherent to a retail business. The basics of operations research will be introduced before demonstrating how to solve a simple version of our real-world problem using all open source libraries. I will then reveal the gory details of productionizing this work, from testing to gracefully handling failures of convergence. Finally, I will cover the journey from the coldest of starts, with zero data, to synthesizing machine learning with the operations research problem.

Ethan RosenthalMember of Technical Staff, Runway