A paradigm shift is needed to end Feature Starvation

Our CEO Steven ten Napel wrote in his blog about the need for a paradigm shift in software product innovation. In this blog, I want to argue that the paradigm shift needs both modesty and ambition.

Many of us have been in IT for quite some time, maybe more than 10, 20 or 30 years. And that may make us feel like seasoned professionals. But does it also make us good craftsmen? Many crafts have a traditional lifecycle for learning the tricks of the trade based on the medieval Guild system. You start as an apprentice, after a couple of years of hard work and decent performance you become a journeyman, and then, after maybe 20 years you might have a chance to become a master.

In essence, it is nothing but a recognition from others in the same branch of your experience and skills in the trade. Experience and skills – both must develop over the years for others to recognize that you’re good at what you do.

Modesty

But there’s one more thing: tradition. Guilds build on a tradition that shaped up over the course of hundreds of years, maybe even thousands. What is a tradition? Perhaps we can see it as the collective knowledge and experience of all people in the trade results in best practices to avoid re-inventing the wheel all the time.

And this is where it becomes interesting for us IT professionals: how much of tradition exists in our trade? Maybe 50 years? How does that compare to other fields such as Construction, Law, or Medicine?

Perhaps you feel proud because your software greatly improves construction work, or legal work or helps doctors a lot. And you should be. But at the same time, I would argue that modesty is a good habit when reflecting on our daily work. IT as a trade hasn’t been in business that long. We’re all rookies. Our tradition is still taking shape. Today’s hype is too often tomorrow’s old news.

Yesterday our application had to be written in Java with Hibernate and web-enabled to reduce client installations or the Microsoft variant. And today it must be SaaS-enabled with multi-tenancy support, exposed as an open REST-based API with native Apps on top of that and oh yeah, we also need to play the platform game.

Gartner developed the Hype Cycle Model to describe this shaping up of tradition. And do not think it is only technology and architecture related: at the same pace we see change and innovation in organizational processes, from waterfall to scrum and agile through all sorts of collaboration tools.

And it is not always for the better. How often does a hype actually turn into something productive? And even if it does for the market, you can still ask yourself: will it also add value to my software? Choices made in the past may prevent options now. So how to understand the impact of choices today on the options of tomorrow? The trap of adopting a new technology or method – just for the sake of technology is always near to R&D departments and software engineers. But how can we showcase the business value of new technology if it takes more than two years to rewrite our product?

Ambition starts with knowing where you stand

But modesty should not withhold us from being ambitious. In this series of blog posts, I want to explain trends from a root cause analysis perspective. Yes, we can be modest on the success of new hypes, but in almost every case, the starting point of the hype was a good solution to an existing problem that got spread around. Perhaps many people make their own interpretation of the solution causing the hype to derail, but at least the problem was a real problem. Take for example waterfall and scrum. A while after the scrum hype took off, everyone was doing scrum – that is: everyone was doing their own version of scrum. So it derails a bit. But the fundamental problem is in the lack of agility in the waterfall process. A real problem.

Looking at the current situation we are in:

• How did we end up here?
• What are the fundamental mistakes we made in our architecture?
• Why did we run out of options?

Many mistakes and choices were made unknowingly and with best knowledge and advice back then. It is easy to judge history, it is better to be modest and learn from it.

Let’s look at the facts and why they came into existence.

• Our application is big and complex

• We suffer from feature starvation – it takes too long to deliver new functionality

• We suffer from technology lag – still use Delphi, Progress or VB

• It is difficult and time-consuming to test all usage paths of our product

• The product has too many configuration options, and each customer uses different settings

• It is hard to remain compatible across releases

• Our database schema cannot be reduced or changed, because our customers directly read from it for their reporting purposes, and changes break those reports

• We tried to make new, web-based user interfaces for specific roles and it looks good, but now we also replicated logic across all of them

• We have a backlog of many tickets and ideas, and it is discouraging to see it only grow

• We never have time to cleanup technical debt

• There is so much undocumented code, and the guy who wrote it is long gone

• Our release frequency is couple of times per year, but we are not able to increase it

In a way, the first bullet states it all: the application is big and complex. It didn’t start like that, but now it is like that. And if we start from scratch and rebuild it with the best of our knowledge today, we’d probably need another 3 years to rewrite all the functionality in such a way that all of our customers’ current needs are satisfied, and by that time … Well, it simply is not an option.

So how do we modernize it in a gradual manner? In every business and industry, it is a good tradition to reduce complexity by breaking things up into pieces. Divide & conquer!

There is a good chance that the application already consists of a set of functional modules that may even show up in our catalog as different pieces with different prices.

There is a good chance that our code base is sort of componentized accordingly.

And there is a bigger chance that in the end, we’re storing it all in a single database, which effectively de-componentizes the whole shebang into a big monolith.

Ambition

So our strategy must be to break it up again in pieces or components. And our goal must be to do component level deployment in order to achieve more agility in our software development process and kill the feature-starvation.

And that means we have to reinvent the processes on how to develop, test and deploy components that act as a seamless whole for our customer.

This is where modern day technology and architectural patterns come to play. Maybe you are familiar with terms like Containerization, DevOps, Functional Programming, Domain-Driven Design, Event Sourcing, etc. Maybe you think they are yet another passing hype. But we will zoom in on a number of them in subsequent blogs and focus on the core problem they solve and how they are truly required to make a paradigm shift forward. They add value on their own, but in combination will give a new grip on the software development process.

DevOps should really be OpsDev

Deploying components that need to work in conjunction is a challenge where new technologies like Docker and Kubernetes and platforms like Amazon AWS and Microsoft Azure can help. Continuous integration and continuous deployment have emerged into the DevOps hype. But to shift the mind, developers must be more modest. It is not the development that drives the business, but the customers. And they work with what Ops delivers. So better join in at Ops, rather than inviting them to your dev floor.

APIs must be designed by the Product Manager

Componentization also means that our components can no longer connect with each other through the database. Instead, they have to use formal contracts at API level. Is this where we apply micro-services? Yes and no. More than something technical, it means something to our product management. We have to redesign our contracts from an information management perspective and think through what data ownership really means. It is too simple to pass the internal foreign key in the external API.

Listen! You need to inform the world

Opening up systems can be done through a User Interface or through an Application Programmers Interface. The API perspective has had attention from the late 90s onwards. But an API is something that the system only listens to. To achieve collaborative componentization it is essential that a component informs about its data changes. This is the essence of Event Sourcing technology.

Configuration is a programming language

Many of the customer products that we encounter have a sort of a stack that helps developer productivity. This can be a 4GL framework or a set of libraries and frameworks that help decrease 3GL productivity. In both places, there is little support for configuration and customization other than at the framework level itself. Settings are done through property files, XML files, here and there in database tables. And no one has an overview of the whole, except for some really experienced and smart people in the customer implementation team.

In a way, the whole configuration and settings that can be used on the application form a programming language for the implementation consultants. But is it a sound language? Domain-Driven Design along with Domain Specific Languages can really help make the next step in this.

And, what if such a language would be visual? A picture says more than 1000 words.

Ambition with modesty should end feature starvation

In this series of tech blogs, I want to dive deeper into each of these topics. Covering a bunch of technologies, patterns, and approaches, collectively they will help an organization to become more agile and future-ready in software development. And when applied with discipline, they will turn the feature starvation back into a continuous feature stream up to the point of agile budgeting.

Where do these patterns and technologies come from, what problems they are trying to solve, and how can we apply them in our own product stack are important questions here. Please join us in our ambitious journey of shaping up a tradition of modest craftsmanship.