13 architecturalist papers: getting to mainstream
(cross posted from: http://wrongtool.kostadis.com/13-architecturalist-papers-getting-to-mainstream/)
Recently one of my co-workers asked about how do I value innovation. And so I started with a formulation I described before (here: http://wrongtool.kostadis.com/a-relativistic-theory-of-innovation/).
And in our discussion, and because many years have passed since I first wrote about it, I realized I was using a more sophisticated model.
The most significant bit from that earlier post is that there are regions of innovations. And within an area, you can innovate, and the next thing to do is reasonably apparent to those who are in that region and may appear as magical if you are further behind.
When I think about innovation and products, I think about these four regions:
- The edge of human knowledge
- Innovative commercial products
- Mainstream products
- Laggard products
I also think of these regions as waves, where the innovations in the bleeding edge wage move downstream.
Most academic research is in the first region and tends to be 5–10 years away, at least, from being part of some innovative product. And most research is intellectually exciting but of marginal commercial value.
Being able to predict which of the research technologies will win is impossible, and the surface area of opportunity is huge, and the investments that are required to turn those ideas into products is vast.
The biggest commercial wins of all time are when someone can take a research idea and turn it into a commercial product. Examples of such things include — server virtualization and network virtualization, the search of the web, cloud computing.
However, as a strategic software architect, the job isn’t to find that disruptive new technology, the position is to keep a sizeable successful software business successful and relevant.
Searching for that next disruptive idea is only part of the equation.
The equally important part is managing your investments in delivering commercially viable innovations, ensuring that the mainstream market is satisfied with the product and that you never fall behind the mainstream requirements and become a laggard — primarily avoid four at all costs, always be on 3, and do some of 2 while looking for 1.
A challenge is that although a strategic software architect may be thinking about all four regions at the same time, most teams are in one place and are focused on their backlog for their region.
If your goal as a software architect is to move a team downstream from being pure research or upstream from being a laggard, you will effectively deprioritize the obvious set of next work with something different. Something more risky because it’s not the obvious next thing.
And if you are a laggard, the product will be suffering from a lot of code rot problems. And just cleaning it up may feel like the top priority.
And that would be the wrong thing.
You are a laggard because you are missing features.
If your code sucks, and your build times are horrible, etc., it’s tempting to think that you are a laggard because of that. In fact, that is irrelevant. You are a laggard because in the market you have a feature gap between the laggard and mainstream market products.
The real objective is to own most of the mainstream market and to offer innovative features that capture premium value while hoping that you discover some groundbreaking innovation.
And because you are a laggard, the market doesn’t look for you to innovate, they look for you to deliver the things that they want in the way that they want. The mainstream has defined its requirements, and you must fit into those requirements. A lot of times the temptation is to argue with the mainstream market, but that’s the wrong strategy. The mainstream isn’t interested in innovations, it has a particular point of view, and you need to satisfy it.
There may be innovative ways to deliver those solutions, but they have to meet the requirements of that market even if those requirements are — for lack of a better word — wrong.
What this means is that you have to add — sometimes — even more technical debt to get to the mainstream market so that you can have the revenue dollars and credibility to innovate.
In any new situation, the first job of a strategic software architect is to validate if the product meets the mainstream requirements and if not, all energy has to be focused on that.
Warning signs that you are not meeting mainstream requirements include declining market share, the field losing trust, emergence of direct competitors offering the same kind of product that is bought and deployed in the same way, decaying investment in the core product.
You have to direct your first set of investments and energy at getting out of the laggard space.
At NetApp when I joined the storage management team and was made their architect, this was precisely the situation we were in. The product, DataFabric Manager (aka DFM) had had marginal investment, and suddenly storage management had become a thing that customers cared about. Our field referred to DFM as Doesn’t F*king Matter and our customers were livid.
The first thing we had to do was identify the set of feature gaps that we needed to close to make the product not suck. And to be fair, the product didn’t suck given the investment. What had happened was that the market requirements had shifted on the team and the company had not made the investments to meet those requirements.
Once we identified those feature gaps, we had to code like mad. I remember sitting in a room with the head of engineering, and he said: Kostadis, I need you to disappear for the next few months. We need to code like mad. And I can’t have you distract anyone with your great next set of ideas.” The engineering head was dyslexic, and he never wrote anything. When he wanted to emphasize things he would take a pen and write it down, and he wrote “Code like mad!!” And he underlined it twice. I wish I still had that paper.
And so we coded like mad and got out of the laggard space.
I like to joke this is the easy part. The requirements are understood, the team top-to-bottom understands what it needs to build, the only problem is you have to go very very very fast and have to be very very very disciplined. The hard part is that a laggard product has a lot of internal challenges — a lack of resources, talent and brain drain, and a lot of internal corporate vultures. The internal corporate vultures are the worst. Instead of fixing the laggard product they decide that the right answer is to buy another product, or to do a rewrite or to fire everyone or to do anything but fix the feature gaps.
It’s mind-boggling. In fact, if you walk into a situation where everyone wants to re-write the core product, it probably is indicative of a laggard product.
And the right answer isn’t to do a rewrite but to fix the critical feature gaps.
So when we delivered that next DFM release, we didn’t advance the architecture, we didn’t solve the underlying technical problems, and we may have added some tech debt, but we got to the mainstream, and that set us up for trying to get to the innovative product space.
And we did.