Sandro Mancuso is a renowned software craftsman and speaker, author of The Software Craftsman Book and long-time friend of Codecamp. In 2020, he sat down and had an interesting talk with Vlad Iliescu (Head of AI, Strongbytes) and Dan Nicola (CEO, Strongbytes and Codecamp) about Software Modernisation. Check out what they had to say on the subject.

15 minutes read. Don’t worry! It will totally make sense even if you read it in parts. We’ll even split it for you!

5 minutes read

Vlad: I think one of my favorite software development stories, a story that to this day continues to shape how I view software development and how I think it should be created and maintained, is that of the great Netscape rewrite some 23 years ago. For those who don’t know, at that time there were two main competing web browsers (none of them were called Chrome by the way). They were Microsoft’s Internet Explorer and Netscape’s Netscape Navigator. At some point, the people at Netscape decided to rewrite their browser because the code was ugly. It was hard to maintain, it had some bugs, the app was slow. Long story short this rewrite went so well that in the end it bankrupted the company because massive rewrites take a lot of time and a lot of effort which means you don’t have as many resources so you can’t add new features to your product like your competitors do, which means you fall behind and then you lose market share. The thing that stayed with me through these years was that I would never ever rewrite something that works unless I got a really good reason for doing it. Sandro, would you say this rewrite attempt fits your definition of software modernization or is it something totally different?

Sandro: I think that there are different types of things that you do in a software modernisation. You can modernize systems at many different levels, so a rewrite is just one of these things, one that I would very very rarely recommend.  For me the rewrite is the last resort, so let’s talk about the different levels: you can do some small refactorings, but I would not consider them software modernization per se, I would talk more about the continuous improvement of the software at the lower level. This is a thing that you just do as you are working with the system. But then you can start escalating the size of your changes. For example you can start talking about modularization, you want to modularize the system, and that is a little bit more than just a small refactoring. Or you might want to re-architect the system, or you might want to re-platform. But if you were to completely rewrite, it would be just like rebuilding the entire app and I find that very rarely there is justification for that. I would rather just focus on smaller areas that might not be working so well and then just concentrate on that. If you don’t need to change certain things I don’t think you should.

Vlad: Not even if you want to use the latest and greatest libraries or frameworks?

Sandro: I think that we need to step back and understand what we mean by software modernization first and then what are the drivers for that modernization. To be fair, I’m not sure if using the latest and greatest libraries is the best driver. However, it is not a driver that you should ignore completely.

For example, if your system is using technologies that are very old, that are not being maintained, or if some of the technologies are being discontinued, that creates a big risk to the system, because now, you’re building a system but the technology is not even being maintained by anyone else so that is a business risk. There are other technologies that are just old and are not getting traction anymore.

If that is the case, then you might have a different reason for modernization: attracting talent.

Attracting and retaining talent is a thing that very rarely is discussed when people talk about software modernization but this is a key driver. It’s a big risk if you cannot attract or retain talent in your organization. You will have a problem in the future. So a modernization can have that as one of many other drivers as well.

Dan: I think it’s funny how sometimes, maybe all the time, the architecture of a product is actually mirroring the background and the skills of your team. If you have teams made up of database developers skilled in writing SQL languages, then your architecture will be heavily database driven. 

I have one of my own stories with software modernization. It’s actually about the first product I ever worked on and a lot of things that happened there are classic textbook. So, in my first job, I was part of a team that was building a fintech solution. It was quite innovative at the time, we’re talking about almost 20 years ago and in the end it became part of the solution that got patented for updating the statuses of transactions and whether invoices were paid or not. Like all classic scenarios, we first created the prototype to demonstrate that this works on a vertical basis and our business people wanted to demonstrate that to banks or to other investors. We started adding new features and before we knew it, it got into production. We went along for about one year and after that we reached that moment where we all gathered in a room to discuss the project, because it was harder and harder to add new features and things were actually breaking. You would refresh and everything would be ok, and when you  refreshed again, it all went to hell. We couldn’t figure out what was happening so we came to the conclusion that we needed to rewrite all of it. After a couple of meetings like that, we created the pitch and presented it to the investors and to our surprise, they agreed. We estimated about six months, but of course  it took one year and a half. Long story short, after three more years, we were in the same room with the same people, and we came to the conclusion that we needed to rewrite this again. But because we couldn’t present the same pitch as before to the bord, we investigated other ways for the rewrite, how to replace parts of the old system while adding new features at the same time. It was painful at first, but we created a way where we could replace or improve the system but without shutting it down. 

Vlad: And did you ever go through this again in the following years? 

Dan: We constantly did. Coming back to what software modernization is, I think it’s not about a big bang approach, although the name sounds like it. No, it’s a continuous process and you need to enter in that mindset. Your product and the technology behind it should advance hand in hand.

5 minutes read

Sandro: Yes, I  totally agree. For me,  continuous software modernization is not a thing that you do once, it is a continuous improvement, a process,  but also, where it matters. Normally, I would  say that software modernization is a continuous process of improving strategic systems and the goal of improving those strategic systems is because you want, and it sounds a bit cliche, you want that business agility. So if there is something in the business that you are not able to do anymore, like in your example, Dan, after rushing a little bit in the beginning of the project, you started slowing down and were not able to add features as fast as the business needed, this is a constraint. It’s a constraint to the business. 

Dan actually did mention quite a few points I’d like to discuss. One of them was building a product from scratch. 

There is the famous story of having a proof of concept or prototype become a product, and to be fair, for a lot of organizations that’s exactly the right thing to do. Because what you want at that point is quick feedback on the idea and on the investment. Is this even a business? You should be able to answer the question whether it is a good investment and if you should continue, as fast as possible and if you need to push stuff out, push stuff out. Once the idea is validated, that’s when you need to start thinking, okay so if we’re gonna really put this in production, we need to start building new features while modernizing. So there’s a continuous process, and, in most of the cases, it should work in parallel, as you are adding. So this is one of the many different ways of modernizing code. 

For example you can modernize through new features. Then another thing is how do you prevent the big rewrites?  One of the solutions is to modularize your system. Not always, at the beginning of a project, can we get the boundaries of the modules properly, but sometimes, just a few hours or a few days spent on this can give us at least a better understanding of what those modules would look like, so we know not to mix the payment logic with the order management or the customer management side, so they are different domains. Although those boundaries will not be very clear,  the APIs between them might have to adapt, if we are able to maintain the modularization, the modernization will always happen within a part of the system, and that goes back to what I was saying at the beginning, that I very rarely would find a reason to refactor the entire project. You only do that when everything is entangled and even then I would begin with modularizing the areas that I care about or the business cares about and the rest I would leave messy. 

Vlad: Both of you mentioned that it’s about continually updating your software and continually removing the technical debt. But some of the projects I’ve worked on just didn’t prioritize this as much as just adding new features. So I ask you, when do you draw the line and say, look, no more, we’re going to stop and we’re going to allocate, I don’t know, 30% of our development capacity to just fixing things that should have been fixed a long time ago. 

Dan: There are a lot of ways to handle technical debt and first of all, technical debt always exists for different reasons. Either because we don’t know how to do things, or we don’t know the road map and we cannot prepare the architecture of our system for what’s coming next  or because we don’t have the time, as the customer needs to go to market. They’re all valid reasons, but it has to be in our practice to focus on this and I’m going to take a step back, because one of the frequent discussions that I have is about business value in a software product and we can all agree that when most people talk about business value, they refer to the features themselves and that’s the primary source of business value. But there are also other things that can have direct and indirect sources of value. For instance, if I’m going to focus on software design, or what we put as developers, under the umbrella of refactoring, what we’re doing, is protecting our capacity to deliver features in the future at a steady pace, because otherwise the bigger your system gets, the more complex it is and you will have a harder time delivering those features. Or having  continuous and automated deployment strategies. You make your deployments easier and faster and you deploy in small bits, not a big bank from time to time. This also makes it easier for the business to test and validate things on the market. If you want to re-platform your software, then probably you want to move to the cloud and have better scalability and that is a good thing in itself, but also it involves some reduction of costs. I think we need to focus on these things in parallel with building new features and have this conversation with the stakeholders in the project. 

I’m jumping to another thing now, but I think one of the biggest impediments is that we, as a team, cannot explain or cannot convince the rest of the people involved that are either from the business side or other departments, what the value of such an initiative is.

Vlad: How do you quantify this value?

Sandro:  So this is where I think that we, the technical people, fail massively. This distinction that there is a technical problem or a technical debt separated from the business side of things is the mistake. We don’t exercise the idea that technical problems are business problems enough, so if we cannot map a technical problem to our business problem, we need to ask ourselves is this really a thing that is important, that we should be addressing? As Dan was saying, there are many reasons why you would change a system and when we see inefficiencies as developers, like I’m spending too much time waiting for the test to run or QA is very slow, if we think hard enough, it’s very easy to relate technical issues to business issues. If you don’t do that, you never know what to measure. 

For example saying that the build takes 30 minutes, what is the impact of that? I won’t know, because that’s a technical metric, but what if I said that we have 400 developers and every time someone needs to fix the code and they run it again to see if it’s gonna be green said developers must waste 30 minutes looking at their screens. How much is that worth? You can start going to many many different metrics, depending on which kind of inefficiencies you are seeing as a technical person. 

5 minutes read

Vlad: How do you manage developing new features while at the same time updating your software, making sure it’s a well-oiled machine?

Sandro: There is no magic solution. At some point you need to pay the price.  I think that the question is, how do you convince the team or the organization to pay the price? Because the longer you go along, the worse it will be. We, as developers, fail to show those trade-offs. The business will always say I want new features and it’s fine for them to do that. That’s what they are paying for. If you think about the business mindset, if we were the ones paying, we would want more features in the fastest time possible. But it can’t happen if we don’t listen to the developers. When people come and start saying to me, look we know you want those 10 features during this period of time, but we’ll deliver five at best, and if you’re not happy we’ll need to explain to them why not, so this is where we fail as developers.

Maybe we could deliver a bit more, maybe not the 10 but maybe seven, but for that we need an extra investment, and if the investors don’t do that, it’s going to get worse as we go along.  The problem is that just saying that is also not enough, because they will think that we are bullshitting them. That’s the developers just finding excuses. This is why you need to try to show them how long the process takes, show them all the inefficiencies that you have, the amount of people involved and so on. 

We also don’t need to ask them for a big investment. Let’s say you have loads of inefficiencies. How do you fix those inefficiencies, they ask, and you answer I don’t know, it’s  gonna take a long time, I need to stop building features and I reckon it’s gonna take six months or a year to fix everything. Of course you’re gonna get a no in your face and you should, because you are not helping your case. You’re going to ask them to just trust you that there are inefficiencies and keep paying you until you fix all of them and you’ll tell them when you’re done and in the meantime they’re not going to get a single feature out. And they will say, well, you are asking me for money to fix the shit you created in the first place. And that’s when the crazy conversations start. 

Dan: Nobody wants to have these conversations but try to do it. We’re in charge of development so let’s fix our own house before we get stuck in a vicious circle. You want to add some features and you hold some capacity to make improvements but at the same time, because your software is not in a good shape, you have bugs coming in from production or you have requests to retrieve some reports because your system hangs for 15 minutes and does not respond, so you need to fix that ASAP and then you get frustrated or you’re developing new features and then you end up developing software in the same way even though the pattern is wrong. At some point there needs to be an alignment and you need to be transparent about the state of your software. At the same time, the business side needs to be aware of what’s happening so you can plan that roadmap together and blend improvements with new features.

Sandro: There are a few other things that I’d like to add to this. There is the issue of our own professionalism and we at some point need to say no. But this is not an easy thing. Bob Martin spoke a lot about this and I myself wrote a lot about this. It’s not an easy thing and you are just one developer in a much bigger team. You can’t say I’m not going to do this because that’s the wrong thing and I want to refactor the entire world. That is not professional behavior. You have to work with the business side and explain to them that I know that you want this but these are the implications and you also can shorten the feedback loop as well if you are constantly talking to them.

For example, if you’re using SCRUM or some iterative approach and then every week or every two weeks you go in and show them what you did. This is something that we adopted this year in our own projects. What we do is we don’t only present what we achieved, we also present all the problems that we are facing, the risks of the project, the technical data, so when we need to make decisions, if we need to deliver something by a certain date, we, as a team, can make a decision. We’re gonna show you regularly what we do, but we are also going to share that responsibility with the business side, so this is not just the development team’s responsibility, this is a business decision. We are working on that, and by the way, on a weekly basis or every two weeks i’m going to be showing you the issues that we have so don’t come to us in six months time and say, hey, we have incurred that. There’s always a way for you to say this feature will take x month, x amount of time and you don’t need to justify every single nitty gritty thing  that you’re going to do with that company.

Codecamp: I think this is a brilliant approach. Basically you’re involving the stakeholder and making them part of the team and part of the decision so, then, when it comes to basically making the call to pay the money for making the software better, it’s going to be way easier for them, since they were part of the initial discovery session.

Sandro: We had one project that we needed to modernize. We came up with four different approaches but all of the approaches had pros and cons. We could not find a clear winner so we had to discuss them with our client and the client picked one that probably we would not have picked. The client did a risk assessment and they ended up with a very conservative approach, that was much slower and that didn’t address the immediate needs that they had, but that was a conscious decision. We had a debate with our client where we presented the risk and there was a decision to go with this approach. At every iteration we are going to be talking about that we’re going to be progressing with that approach but we’re going to keep highlighting the risk and this is a great way for you to say no because you’re not saying no. You are just working together and highlighting issues.

Dan: If there’s a pressure to get something out ASAP, the customer needs it to be released in production sooner than it’s actually feasible, either you say no, and that doesn’t always work, or, you need to cut some corners, and that will have consequences after. So instead of waiting six months and then having this conflict with the customer, remember when you asked us to do this, that’s why we have these issues and now we need to pay for them, you can have the conversation sooner and you can be honest. We can meet your deadline but we’re going to have to do it like that and we shouldn’t, so in the coming period we should find some time to fix it and do it the right way.

Sometimes the customer will say all right, if it’s like that, it’s not really an urgent thing so take your time and do it right from the beginning, or, they acknowledge it and then it’s not a surprise when you tell them that part of the code is not in a good state.

Vlad: This is also a good approach when trying to measure the progress of software modernization in general,  because when the stakeholders are familiar with what needs to be done and with what they’re postponing, then it’s easier to follow. When in a sprint you fix something, you make something better, it’s easier for them to see that progress.

Sandro: Basically, they understand what they’re paying for. This is key and certain modernization styles are very technical. We cannot always align them as there are many different ways of modernizing the system. We need to figure out a way to show progress but in order to decide how we’re going to show progress, we need to be more precise when we define the scope of the modernization. So there are far more conversations that need to happen, and we need to understand why we are modernizing this. What are we trying to solve, what are the inefficiencies, how do we know that there are inefficiencies? Is it just a gut feeling or do we have some way of measuring them? So we need to have those discussions and then decide, before we even go into the modernization, decide how we’re going to demonstrate that progress is happening and, if possible, to have some metrics that actually make sense. 

You can always find metrics, but they need to make sense, and we need to make sure we have a way to correctly measure them. It is an investment and at the end of the day you want a return on that investment. We need to be careful first of all, not to apply a standard that is higher than our normal standard. For example, I can go to our product rooms and ask how do you measure the impact of this feature that you are asking me for? How will it impact you if it is done earlier or later? Give me a number. They won’t know. A lot of people ask where are your metrics, what is your data. These are the people that don’t have any data themselves for whatever they do. First of all we need to make sure that we calibrate our conversation to the same standards, so you know we are all trying to do the best here and then find the metrics that will help us to see progress. One of our guys has a saying: tell me what you’re gonna measure and i’ll tell you how i’m gonna behave, so we need to be careful not to pick the wrong metrics because you might have an undesirable behavior.

Vlad: It doesn’t always make sense and you can’t really get away with just saying I want six months dedicated to researching this. I don’t know what will come out of it, but I’m sure it’ll make the software better, because people are going to laugh in your face. Nobody’s going to take you seriously. You do need to make a plan and to understand how you can define the progress and how you’ll know when you’re done, because if you don’t know when you’re done then you’re never going to be done.

Sandro: For me, you need to run any modernization initiative as a normal project with a backlog, with stories, with demos, so you need to be showing the cadence approach and as a normal project or at least a normal Agile project, a way to change the scope or even to stop work.

Vlad:  What approaches do we have for doing software modernization in an incremental fashion?

Dan: I think each system is different, so each application is in a different state. The way we approach is one: by getting a clear image of what’s the desired situation, where are you heading and of course, that varies, that will change in time, but what’s the next step where we say okay, now we’re happy with our software, it meets the current demands and then it will evolve. And the second would be making some sort of analysis and saying okay, what is the current state, what are the issues, bring the business people and the technical people in alignment. Then we can envision small steps that get us in that direction. In this stage we can have experiments, we might even not be sure if certain things would be beneficial or not, or there might be different trade-offs like Sandro was mentioning. But then, we could make small timebox experiments and say okay, let’s try it like that and see how it goes, do some tests and then see if that makes the system and take it from there.

Sandro: For me, there are levels as Dan was saying. There are different levels of focus. The first one is the business because before I go into the nitty gritty of how we’re gonna do that we all need to know very clearly why we are talking about modernizing our systems. There is some strategic value that you might want to talk about. For example, we might want to focus on preparing for future changes, maybe there is a huge stream of work that our system is not prepared to receive because whatever new things we want to do, there are new compliances or regulations coming in that our system either is not compliant with or needs to adapt to. So there is something strategic that we are not able to do or won’t be able to do or it will be very difficult to do with the current situation of the system. This is a strategic approach to modularization which is different from talking about inefficiencies. Maybe something is very costly and we cannot release it fast enough or there are too many bugs. This is a slightly different perspective, it’s about efficiencies, reducing costs. That’s a different take. 

Another one is experimentation. We would like to innovate, but it’s very difficult, and we can barely bring a new feature into the system. Even to innovate, we need to understand where to innovate. It needs to be easy to run a proof of concept and just isolate one part of the system and innovate in that area and collect data and so on and so forth. 

These are very different reasons for modernization, so once we all understand what the end goal is, what we’re trying to achieve, then we can start going into a little bit more detail of what we are going to do. Now that we understand what we want to achieve, let’s look at where are we today, why are we not able to do that or why it would be so difficult to do that and then start creating that path. It depends on what kind of goal it is. 

For example, you take new features, let’s say adding  a new payment method. You already accept American Express and now, you want to do Apple Pay or PayPal. If you need to touch the entire system to do this, then you have a problem, because you will have compliance and infosec issues and all these kinds of nonsense applied to the entire system instead of just being on the payment area. So basically, you can take the new feature, and design it the way that you want the future architecture to be. A separate service using a different technology and you create the foundation to migrate the existing ones to that new one. Or, you can just take a fully functional approach, you know this area is problematic, the whole payment system is problematic and we’re gonna separate the team just to focus on the many different issues that the payment area has. You’re not gonna use a feature anymore. You’re gonna take the existing issues that you have within that area and solve them in a more strategic way. 

You can have business flows as well, so, for example, one of our clients, they had internal users that have used the system, but for them to do their jobs, and sell whatever they sell and maintain their clients, they needed to go through quite a few systems or different areas in a very inefficient way and copy and paste from one system. What you can do is take a business stream and you can create a modernization that aligns to the business flow. There are usability of the screens that are technical components, hotspots when you have too many teams, that’s a very technical one. Too many teams that every time that they are working they’re having code freezes. They need to synchronize all the changes together so they can be released together and the reason is because they are working on similar parts of the system, so you want to break those things. As you can see, these are very different examples of metrics showing progress. They are significantly different from each other but they all have a business value.

Dan:  I’ve read this somewhere, way back, but then, whenever we start this sort of initiative I ask it to the customers: if you would just write a simple Hello World and you want to deploy it into production, what sort of things need to happen to get there? And then, you find out that there are various things from the development side, different teams that need to work together, that operations can be highly regulated, you need environments, you need various levels of testing and documentation.

Sandro: The value of stream mapping is a great exercise so, basically just map, for those that might not be familiar, just map all these steps from an idea, to software in production.So it has all the steps, from having the idea, generating documentation, speaking to product owners, what goes to the teams, what the teams do, how do they test, how do they deploy, what happens when things go wrong. Sometimes on the business side, they cannot precisely say what is wrong, they just aren’t happy and everyone is unhappy, maybe things are not moving. You need to be careful with the local optimizations to not impact the whole project. So the value mapping is great in this case because you can map all the steps. Not always everyone understands how many steps are involved in the process and mainly, when something goes wrong, a bug is found or a deployment doesn’t happen, you can find out how far back it goes into the process and then how long does it take to get back to where it was and if you have multiple teams how many of them are blocking each other. So once you map, that can be a great source of modernization in the right areas because now, everyone understands where the bottlenecks are and you can try to either remove the bottlenecks or make them larger, so in the end, the software will work and be productive