My colleague called it the thrash. It is one of the worst parts of life as a Senior Data Scientist (DS). You don’t know what to work on next. The ideas from the Product Manager (PM) are missing, vague, or technically naive. The ideas on the Quarterly Business Plan (QBP), in the backlog, or being discussed by your coworkers don’t match your areas of expertise or don’t seem as if they could ever be all that impactful. They don’t spark joy. Maybe you have some ideas of your own but they aren’t aligning with what you are hearing from your Engineering Manager (EM), Tech Lead, Director, or PM.
When you are stuck in the thrash, you probably aren’t “producing” in the usual way. You aren’t committing code. You find it hard to explain what you have been working on during sprint planning meetings or at stand up. All of this may make you or your EM or PM feel as if you aren’t being productive enough. This is especially true if you have that backlog or those ideas from your PM.
I have a lot of contacts in academia and I imagine some of you are thinking: “Welcome to my world!” A good chunk of your work life involves thinking up what you or your students should work on next. Whatever you work on has to be something really novel. This can definitely be stressful but, at least in my brief time in academia, it never felt quite as bad as the DS thrash. This might be because idea exploration was the job. It might be because no one was tracking my work on a weekly basis, had some larger ambitions that required (frequent) contributions from me, or was in any real position of authority over me.
I did once work with an experimental scientist who really struggled with something like the thrash when doing public policy work. She was used to being constrained by the time and money required to perform experiments. She couldn’t grapple with the murkiness of the thrash. She felt disappointed that she couldn’t do research and publish as fast as her ambitions. She didn’t know what to put down for Budget Justification in grant applications. “Am I supposed to ask them to give me money and time just to think? And to fund my travel around the world meeting other people for dinner? How much is enough? How much is too much?”
I steered my experimental scientist friend towards organizations funding relatively well defined research projects to be completed during relatively well defined time windows. When she submitted proposals, she found the grants were won by the people who had submitted the project ideas and, in some cases, already done the projects. But, more than anything else, the project ideas were “lame.” This is the price you pay for skipping the thrash.
You may not have to deal with the thrash if your employer is very organized or if you are working in a more “junior” capacity, regardless of actual job title. The price you pay may include having to work on lame projects. You will also encounter the younger brother of the thrash, the feeling of not knowing how to accomplish the specific tasks which have been assigned to you. This too can feel awful, particularly if people are asking you when you’ll be done with your tasks or pressuring you to finish them quickly. From my outsider perspective, it seems like being (and learning to be) a Software Engineer (SWE) involves constant battles with the micro-thrash. Maybe this is why burnout is common.
Let’s talk about some solutions.
Speaking with the Operations (Ops) or business people at your company can be helpful. They battle with very tangible issues that can often be mitigated with a little DS magic. If nothing else, maybe you could automate, partially automate, or improve some process that takes hours and hours of busy work now. Hopefully you can improve the logic at the same time. Just speaking with the Ops and business people, you will learn about parts of the business and the industry that you would otherwise not be exposed to.
You may come to appreciate the ideas of your PM or EM more, given this new context. Or you may find out that those ideas are as bad or worse than you thought. Hopefully you find some better ideas that do spark joy, for you, for those Ops and business people, and for everyone.
I find it funny when some companies want to route all communication through the PMs. Those poor PMs. Trying to herd all the SWE and DS cats. Translating the technical gibberish the SWE and DS cats produce when they can be bothered to document or show up for stand up into business english. Now also responsible for explaining the nuances of the business to those cats. All without any real authority. When there is this resource of the Ops people readily available. One of the real revelations for me at my first tech job at Uber was just how great the Ops people were (are?). If you want to really know about transportation, about your city, about microeconomics, … talk to Uber Ops. We are just starting to interact with “business” for my projects at Walmart, but I am getting a similar vibe from these people.
Speaking with people, in your organization or others, several levels higher in the org chart than you can also be helpful. The worst part of the thrash is feeling limited or constrained in what you can accomplish. You will probably feel as if you are smart and hard working enough to do great things for your employer, but you just… can’t. You might be tempted to blame your managers, but basically you don’t know how to apply your skills. That Senior Director or Vice President probably knows about a plethora of projects and project ideas that you don’t.
My colleague told me about the thrash when I was at Scoop. My brilliant colleagues had revamped “the matcher” in a matter of weeks. The matcher was the one piece of Math Programming at Scoop. I didn’t know what to work on next. My colleague wanted us to work on revamping geospatial data at Scoop, leveraging the power of h3 and kepler. The PMs were hostile to that idea. How would being smarter with geo data help the bottom line? Very much a P2. They wanted us to develop Machine Learning (ML) models of whether or not we could serve different customer requests. Making models of, basically, what our own backend systems were doing initially seemed like a questionable idea.
I spoke with a bunch of people at Scoop while still trying to move tickets across the Jira board. I spoke with an Ops guy (who coincidentally had come from Uber) and with one of the co-founders. I spoke with all of my DS colleagues. (We had a great team! Miss you all.) I realized that the geo data revamp could be paired with a pricing update that could produce real value for the company and our users. I realized that the ML models were just a way to make some of our services (e.g., pricing) aware of the dynamics of other of our services (e.g., the matcher). I ended up working on both of these project ideas. I learned a lot and felt both were meaningful. It’s a shame about COVID.
Leave a Reply