Where Insights Come From

Abid Ali, instructor for the University of Chicago’s eight-week “Data Engineering” course, compares data engineering to a skyscraper in downtown Chicago: “You see a beautiful building and that’s like the beauty of data science. But so much happens in the planning of the beautiful building that you don’t see and rarely know. That’s data engineering.”

Data engineers create the pipelines that gather data, compiling them from multiple sources and then loading them into data warehouses. The clean and prepared data then become the input that drives the insights of data science and the incredible recent success of LLMs like ChatGPT. Seventy percent of any data project, Ali notes, will typically be data engineering.

“That’s why most of the jobs available today are in data engineering. It’s a skillset that is very high in demand and that’s not going to change. Companies will keep building out their AI systems and they will keep searching for more and more insights using data science.”

Lifelong Learner

Growing up in Pakistan, Ali was fortunate to have an older brother studying computer science who recognized the importance of data very early on. Guided by his brother, Ali started to program in middle school; several years later he was tagging along to the database analytics firm where his brother worked. That’s where Ali later interned and started his career.

From the role of developer to tech lead and onto technical and data architect, Ali’s career advanced as he leveraged his expertise in data for higher roles in consulting and portfolio management. Encouraged to gain experience abroad, he took consulting roles in strategic infrastructure analysis for major banks in the UK and the USA, eventually settling in Chicago. More recently, Ali was tapped by a Silicon Valley-based venture capital firm to work in the still new and centrally important area of customer success.

“When you work in data, you are required to learn many technologies and many new skills. You’re just thrown into projects. So you have to be very adaptable. In my case, having the communication skills to layer on top of my technical base became—and still is—a very valuable asset.”

Throughout this time, Ali also continued his education. He focused on business strategy, first earning an MS in information science, then a PhD in organizational leadership, and most recently an MBA. He is now plotting a course for his second PhD, this time in education.

A lifechanging event took place in 2016 when Ali became a citizen of the United States. For more than a decade he’d wrestled internally over whether to stay in the country’s high-pressured work environment, but now he found himself with breathing room.

“That’s when I decided to start teaching,” he says. “With the knowledge I’d gained working in data for 20 years, I knew I had the technical expertise to help students. But my life experience is important too. It allows me to connect with people going through the same challenges I had and help them and give them encouragement.”

Since then, he’s taught in all the scenarios—online, in person, and hybrid—at the University of Chicago and other world-class universities across the United States. “I’ve taught in so many modes, but I’ve also learned in so many modes, and worked in those modes too, and so now connecting with students in class comes as naturally to me as connecting with colleagues at work.”

When it comes to “Data Engineering,” Ali believes the critical challenge for instructors is keeping students engaged with lively material. He studies in his own time audience engagement techniques he hopes will sustain his students’ enjoyment of the class. He also takes the feedback from students very seriously.

“I tell students to let me know what is working and what is not. I am always modifying aspects of the course—not the curriculum, but the way I teach the curriculum and the assignments that go with it.

“Ultimately, the course gives students considerable insight into how these processes work,” Ali says. “They learn the language of data engineering so that they can engage the technical side of data and understand the spectrum of things that can be done with it. What they leave with isn’t a hammer that turns everything they see into nails, but an elaborate toolbox whose contents can be used across the variety of business cases professionals encounter today.”