Performance engineers need to be an interesting mix of data-lovers and people-whisperers.
A month ago I had an interesting conversation with an old friend. He worked on security for a long time, and is now part of the small group of people in our ethics team working to spread good ethics practices. He knows me from my long history of working on performance. Up to now most of his ethics work has involved working with teams who are enthusiastic about finding and fixing ethical mistakes they didn’t know about. But he’s starting to go beyond that, pushing diligence work (things like “find the biases in your data,” or “think carefully about how bad actors can abuse your platform”) onto teams that are less interested in it.
Engineers: “We’re all trying to be ethical people, so we’re doing ‘ethics’ correctly already, right?”
Narrator: “They weren’t doing it correctly.”
My friend thought maybe my history on the performance team – influencing other people to work on performance problems – would give me some perspective on how he could influence other people to complete the ethics work he wanted them to do.
It turns out that people who work on performance, security and ethics at software companies have some things in common. You’re an expert in something the rest of the engineers are not. In fact, the rest of the engineers will give lip service to the value of what you work on, but they don’t really want to work on it themselves. They vaguely hope they’re doing it right, vaguely fear the consequences of doing it wrong, and generally compartmentalize it away. We engineers, we’re sometimes a little too good at abstraction and encapsulation. But sometimes people need to pay attention. They need a wake-up call. So, you end up needing to influence others, and that’s not always easy.
Principle 1: Those people are humans, not robots. Appealing to pure logic is usually the wrong approach. When you are convinced that other people need to work on performance [or security or ethics], and they continue to resist all your logical arguments, it usually boils down to a couple of root causes.
- They are not convinced the work is as valuable as whatever else they’re planning to do.
- They are scared of the work.
When you’re the expert and you need to get others to wake up to your message, you can’t come down on them like a police force, expecting compliance. When you know you have the moral high ground, lording over people is the exact wrong way to gain their willing cooperation. To be effective at protecting performance [or security or ethics], you need to be a good partner. That means you need to be good at dismantling the two root causes of their resistance…
Setting priority – To convince someone to do work, you need to show them the value of the result in terms they understand. They may not speak the same language as you – and here I’m not talking about English vs. other languages, but terminology and approach. They may not think in the same units that you do – not everyone understands how milliseconds or kilobytes or frames relate to the user’s experience. You need to figure out how THEY think in order to share data that’s meaningful.
Sometimes you also need to demonstrate that you understand the value of the OTHER things they want to do, and STILL think your task is more valuable. This means you can’t just talk at them, you have to ask questions and listen. You need to learn enough about the world of that person across the table, to understand their priorities, their goals. You need to let them know that you understand and value what they’re trying to accomplish, too. Then show them how your task fits in.
Removing fear – Remember, you are the expert here. To you, this work is understandable and finite. To them, it is a hairy scary monster. They’re don’t understand what they’ll have to do, and they’re afraid it will become an awful drain on their time and energy. Once again, you have to ask questions. Ask about their capabilities, their assumptions about the work, and their scheduling concerns. As much as possible, have the process documented before you even approach them. Whenever people get confused, fix the documentation/process. Have an easy way for them to ask questions, and be responsive when they do. Be a good partner! They’ll be more willing to do their share of the work if they feel like you’re there sharing responsibility. That doesn’t mean do the work for them – they need to learn and you don’t have the time to or expertise to get into everyone’s code. But be there for them when they’re stuck.
Principle 2: You are human, too. I learned something about myself when I was on the Windows performance team. This will surprise nobody who knows me, but I tend to be soft-hearted. As a result, I let people get away with too much. I fall for their sob-stories and let them delay or shift responsibility when it would be better to hold firm or escalate to get external support.
But for years I served on a performance impact review-board, and found that my teammates on the board were different. We had some hard-liners that would not let even the smallest details go – things that really were not impactful. Together, we balanced each other. I’d convince them to let go of the small things, and they’d keep me from letting important things go. We knew we had a good-cop/bad-cop dynamic, and we used it to check ourselves. We worked on keeping interpersonal friction at a minimum, because our relationships were built on mutual respect. We tried to be united and clear in front of other teams. We had a policy to always do review meetings with at least 2 board-members, so none of us were left to operate on our own. This wasn’t needless process; we all preferred it that way.
You probably know whether you’re the softie or the hard-liner. Find someone you trust to balance you out, and always work together. That way your requests to other teams will be firm but fair.