A paradigm shift is needed to end Feature Starvation

Thijs Petter

Our CEO Steven ten Napel wrote in his blog about the need for a paradigm shift in software product innovation. In this blog, I want to argue that the paradigm shift needs both modesty and ambition.

Many of us have been in IT for quite some time, maybe more than 10, 20 or 30 years. And that may make us feel like seasoned professionals. But does it also make us good craftsmen? Many crafts have a traditional lifecycle for learning the tricks of the trade based on the medieval Guild system. You start as an apprentice, after a couple of years of hard work and decent performance you become a journeyman, and then, after maybe 20 years you might have a chance to become a master.

In essence, it is nothing but a recognition from others in the same branch of your experience and skills in the trade. Experience and skills – both must develop over the years for others to recognize that you’re good at what you do.

Modesty

But there’s one more thing: tradition. Guilds build on a tradition that shaped up over the course of hundreds of years, maybe even thousands. What is a tradition? Perhaps we can see it as the collective knowledge and experience of all people in the trade results in best practices to avoid re-inventing the wheel all the time.

And this is where it becomes interesting for us IT professionals: how much of tradition exists in our trade? Maybe 50 years? How does that compare to other fields such as Construction, Law, or Medicine?

Perhaps you feel proud because your software greatly improves construction work, or legal work or helps doctors a lot. And you should be. But at the same time, I would argue that modesty is a good habit when reflecting on our daily work. IT as a trade hasn’t been in business that long. We’re all rookies. Our tradition is still taking shape. Today’s hype is too often tomorrow’s old news.

Yesterday our application had to be written in Java with Hibernate and web-enabled to reduce client installations or the Microsoft variant. And today it must be SaaS-enabled with multi-tenancy support, exposed as an open REST-based API with native Apps on top of that and oh yeah, we also need to play the platform game.

Gartner developed the Hype Cycle Model to describe this shaping up of tradition. And do not think it is only technology and architecture related: at the same pace we see change and innovation in organizational processes, from waterfall to scrum and agile through all sorts of collaboration tools.

And it is not always for the better. How often does a hype actually turn into something productive? And even if it does for the market, you can still ask yourself: will it also add value to my software? Choices made in the past may prevent options now. So how to understand the impact of choices today on the options of tomorrow? The trap of adopting a new technology or method – just for the sake of technology is always near to R&D departments and software engineers. But how can we showcase the business value of new technology if it takes more than two years to rewrite our product?

Ambition starts with knowing where you stand

But modesty should not withhold us from being ambitious. In this series of blog posts, I want to explain trends from a root cause analysis perspective. Yes, we can be modest on the success of new hypes, but in almost every case, the starting point of the hype was a good solution to an existing problem that got spread around. Perhaps many people make their own interpretation of the solution causing the hype to derail, but at least the problem was a real problem. Take for example waterfall and scrum. A while after the scrum hype took off, everyone was doing scrum – that is: everyone was doing their own version of scrum. So it derails a bit. But the fundamental problem is in the lack of agility in the waterfall process. A real problem.

Related  Partnerships in Software Product Development

Looking at the current situation we are in:

  • How did we end up here?
  • What are the fundamental mistakes we made in our architecture?
  • Why did we run out of options?

Many mistakes and choices were made unknowingly and with best knowledge and advice back then. It is easy to judge history, it is better to be modest and learn from it.

Let’s look at the facts and why they came into existence.

  • Our application is big and complex
  • We suffer from feature starvation – it takes too long to deliver new functionality
  • We suffer from technology lag – still use Delphi, Progress or VB
  • It is difficult and time-consuming to test all usage paths of our product
  • The product has too many configuration options, and each customer uses different settings
  • It is hard to remain compatible across releases
  • Our database schema cannot be reduced or changed, because our customers directly read from it for their reporting purposes, and changes break those reports
  • We tried to make new, web-based user interfaces for specific roles and it looks good, but now we also replicated logic across all of them
  • We have a backlog of many tickets and ideas, and it is discouraging to see it only grow
  • We never have time to cleanup technical debt
  • There is so much undocumented code, and the guy who wrote it is long gone
  • Our release frequency is couple of times per year, but we are not able to increase it

In a way, the first bullet states it all: the application is big and complex. It didn’t start like that, but now it is like that. And if we start from scratch and rebuild it with the best of our knowledge today, we’d probably need another 3 years to rewrite all the functionality in such a way that all of our customers’ current needs are satisfied, and by that time … Well, it simply is not an option.

So how do we modernize it in a gradual manner? In every business and industry, it is a good tradition to reduce complexity by breaking things up into pieces. Divide & conquer!

There is a good chance that the application already consists of a set of functional modules that may even show up in our catalog as different pieces with different prices.

There is a good chance that our code base is sort of componentized accordingly.

And there is a bigger chance that in the end, we’re storing it all in a single database, which effectively de-componentizes the whole shebang into a big monolith.

Ambition

So our strategy must be to break it up again in pieces or components. And our goal must be to do component level deployment in order to achieve more agility in our software development process and kill the feature-starvation.

And that means we have to reinvent the processes on how to develop, test and deploy components that act as a seamless whole for our customer.

This is where modern day technology and architectural patterns come to play. Maybe you are familiar with terms like Containerization, DevOps, Functional Programming, Domain-Driven Design, Event Sourcing, etc. Maybe you think they are yet another passing hype. But we will zoom in on a number of them in subsequent blogs and focus on the core problem they solve and how they are truly required to make a paradigm shift forward. They add value on their own, but in combination will give a new grip on the software development process.

Related  Optimize Your Software Development with Distributed Teams

DevOps should really be OpsDev

Deploying components that need to work in conjunction is a challenge where new technologies like Docker and Kubernetes and platforms like Amazon AWS and Microsoft Azure can help. Continuous integration and continuous deployment have emerged into the DevOps hype. But to shift the mind, developers must be more modest. It is not the development that drives the business, but the customers. And they work with what Ops delivers. So better join in at Ops, rather than inviting them to your dev floor.

APIs must be designed by the Product Manager

Componentization also means that our components can no longer connect with each other through the database. Instead, they have to use formal contracts at API level. Is this where we apply micro-services? Yes and no. More than something technical, it means something to our product management. We have to redesign our contracts from an information management perspective and think through what data ownership really means. It is too simple to pass the internal foreign key in the external API.

Listen! You need to inform the world

Opening up systems can be done through a User Interface or through an Application Programmers Interface. The API perspective has had attention from the late 90s onwards. But an API is something that the system only listens to. To achieve collaborative componentization it is essential that a component informs about its data changes. This is the essence of Event Sourcing technology.

Configuration is a programming language

Many of the customer products that we encounter have a sort of a stack that helps developer productivity. This can be a 4GL framework or a set of libraries and frameworks that help decrease 3GL productivity. In both places, there is little support for configuration and customization other than at the framework level itself. Settings are done through property files, XML files, here and there in database tables. And no one has an overview of the whole, except for some really experienced and smart people in the customer implementation team.

In a way, the whole configuration and settings that can be used on the application form a programming language for the implementation consultants. But is it a sound language? Domain-Driven Design along with Domain Specific Languages can really help make the next step in this.

And, what if such a language would be visual? A picture says more than 1000 words.

Ambition with modesty should end feature starvation

In this series of tech blogs, I want to dive deeper into each of these topics. Covering a bunch of technologies, patterns, and approaches, collectively they will help an organization to become more agile and future-ready in software development. And when applied with discipline, they will turn the feature starvation back into a continuous feature stream up to the point of agile budgeting.

Where do these patterns and technologies come from, what problems they are trying to solve, and how can we apply them in our own product stack are important questions here. Please join us in our ambitious journey of shaping up a tradition of modest craftsmanship.

 

Thijs Petter

CTO

1 Comment

  1. Iwan 1 week ago

    Hi Thijs,

    I like your post. It is really insightful and very recognizable. Although I do feel that the ‘guild’ aspect is a bit overrated. It is correct that the concept is as you describe it, but its implementation was predominantly political and flawed due to human nature. It was a power-structure, which was the result of the principle that power-corrupts. That as a sidenote. Btw, I think that the traditional guild structure is a cause for stovepipe thinking and consequently stovepipe organisations. The guild variant in Agile concepts is a means of knowledge sharing. The traditional guilds where a means of perfecting trades. The result is an organisational foundation of waterfall-prone businesses: Competence Centers. (https://www.arc-e-tect.com/2018/02/the-incompetence-of-cc-in-agile-world.html and https://www.arc-e-tect.com/2018/03/the-bright-future-of-cc-in-agile-world.html are two articles I wrote on the topic)

    IMHO the problem we’re facing in IT is something we’ve faced in other areas as well, which is that we want to model the new according to the old. In order to grasp concepts, we often use terminology that we know from similar concepts. Which initially means getting a better understanding about the new, but eventually prohibits the evolution of the new.
    Key aspects I think of the predicament we as an IT centric world are in are stemming from found in the area communication. Not in that particular order, they are extremely related. We tend to forget that dialog is synchronous communication, otherwise two parties have their own monolog. Dialog through asynchronous communication channels are therefore ineffective and inefficient. From an ego-centric perspective this is counter-intuitive, since once you’ve said your piece you can continue with your work, but from a let-us-work-together-as-a-team perspective it is all but effective or efficient)
    This is stemming from a focus on efficiency in our processes instead of effectivity. IT as a cost center should be as efficient as possible many CFO’s think, but as a revenue generator it should be as effective as possible. Many CMO’s feel. But as long as we see IT as an enabler, as a tool, as a means instead of as a product we will keep on thinking that we should consider software development as a manufacturing process more than an engineering process.

    DevOps shouldn’t be OpsDev, instead it should be BizValDel (Business Value Delivery). Dev and Ops are not roles, but activities and responsibilities. We’re inherently limiting our abilities to grow and establish a trade, by sticking with the old paradigms, the old designations, the old roles, the old trades.

    Just my €0.02. And thanks for sharing your thoughts, it really got me thinking to look at it (IT) differently.

    Iwan

Leave a reply

Your email address will not be published. Required fields are marked *

*