Saying that DevSecOps is his passion is going too far for Nico Kaag. But he would like to see security become a permanent part of (agile) software development. In practice, Nico often sees that software developers can develop excellent software, but pay insufficient attention to security. And yet the disadvantages are often in the news. “You read in the newspaper almost every day that there are vulnerabilities in software, that stuff has been stolen,” says Nico. Nico does not dare to say why there is no attention to security. ‘Maybe the knowledge is lacking. Or is it not sexy enough for the developers. Security remains a bit scary, I think.’

‘Is this different from before?” Mirna wants to know. She previously outlined the situation: ‘In the past, software development was often project-based. There were large development projects, with extensive risk assessments. Then there was the implementation process. There were disadvantages that agile working had to overcome. What have you seen change with agile working? Were we perhaps better at security in the past?’

Nico immediately shakes his head. ‘We were certainly no better in the past when it comes to security. Security was still somewhat of an unexplored area at the time. We never thought that people, consciously or unconsciously, would use software to do things that we don’t want them to do. I think that security is somehow not in the genes of the developer.’

‘Security must be part of the deliverable’

Nico compares the lack of attention to security in software development with smoking. Why do people continue to smoke even though they know it is bad for them? That’s how I see it with software development. Security still seems like a distant memory for developers. They think it’s an add-on to their work. I think it should be a permanent part of their work. Nowadays you can no longer buy a car without an anti-lock braking system (which ensures that the car remains steerable in the event of an emergency stop, ed.). I would also expect the same with the delivery of software. That there is no Cross-Site Scripting. That there are no known vulnerabilities when it is delivered. Security should be part of the deliverable and it is not yet.’

Nico points out the ‘shift left’ that is going on in the market. ‘As a security team, we are increasingly moving to the front end, the beginning of the development process. Security requirements become part of the business/non-functional requirements. This will make teams more aware that the business considers security important and perhaps even can be used as a USP (Unique Selling Point). As a result, they will have to make security an inseparable part of the product.’

‘Make the Product Owner accountable’

Nico’s aim to anchor security in agile software development did not just come out of the blue. In 2021, he completed research at The Hague Graduate School into security in agile software teams. The aim of the research was to determine the impact of AgileSecDevOps on business risks and how these risks can be influenced. In addition to a literature study on AgileSecDevOps, Nico and an information security team of six people conducted three months of research at an (anonymously described) organization with seventeen agile scrum teams.

‘You just mentioned that security requirements must become part of the business requirements. How do you ensure that security requirements are included from the start? Does that happen automatically?’ asks Mirna.

Nico: ‘That certainly doesn’t happen automatically. We first appointed a Security Champion in the research teams. We chose someone with an interest in security and we tried to motivate them to get security on the radar of the DevSecOps teams. From the information security team we of course offered support with questions and so on. And we helped the champion when we noticed that security was not receiving enough attention and priority. We also made the Product Owner accountable. If software turned out to be unsafe, the Product Owner was contacted. After all, software delivery is the responsibility of the Product Owner, not security. In other words, we allocated ownership better and more clearly. If the Product Owner didn’t cooperate, things got difficult. But understand, we did not immediately demand a ten. At first we settled for a five or six. That’s how we built it together.’

‘Allocate capacity for security maintenance in the sprint planning’

Mirna: ‘You made the Product Owner responsible. What challenge followed next?’

‘When a product is released into production, it may initially contain vulnerabilities. Or vulnerabilities are added when something new is added. I saw that there was often no room for maintenance in the subsequent sprints. The sprints were often fully planned with new functionalities. So only new functionalities were developed or functionalities were improved. But there was no room to adjust security flaws.’

“Security was assigned to the Product Owner, but was given a lower priority?” asks Mirna.

Nico: ‘Yes, and that is why we also recommended that 20-25% of the capacity be allocated and possibly spent on maintenance in the sprint planning,’ Nico explains. “Then we saw that more attention was being paid to security.”

“Was that enough?” Mirna wants to know.

Nico: ‘Sometimes yes, sometimes no. We did see that the better the teams worked ‘at the front’, the less time the team ultimately needed for security issues. Ultimately, teams turned out to be able to achieve that 20-25% time.’

The weakest link

‘What happened next in the teams?’ Mirna asks.

Nico: ‘We saw enormous differences in the maturity of the teams. The deliverables are provided by the teams, but often function in one ecosystem. This makes you only as strong as your weakest link, we saw.’

Mirna: ‘How do you, as an organization, ensure that the weakest link is seen? And how do you approach this?’

Nico: ‘As an information security team, we saw where security flaws existed in penetration tests and vulnerability scans. We then entered into discussions with the poorly performing teams. We asked about the causes. Is it lacking resources and/or knowledge? Do you not have enough time? If this was the case recently, we would have the team join a well-performing team. He could then convince the weaker team that security ‘is part of it’. That it is possible to plan and prioritize well and that it generally does not take any extra time. We also asked the question: ‘If you don’t have time to do it right the first time, where do you find the time to fix the mistake(s)?’ Did we see any differences in the CI/CD pipeline? for example, we advised to make better use of the tooling and knowledge. Ultimately, the teams started helping each other by sharing experiences and knowledge. We also placed people from high-performing teams in the weaker teams. That worked very well. Those people also enjoyed helping.’

‘Include security guarantees in assessment’

Mirna: ‘How would you describe the role and responsibility of the information security team in this process? In other words, how did you get the teams to follow you?’

Nico explains: ‘The Chief Information Security Officer in the organization was the linking pin to the management. The management team also thought it was important that secure software was built. In addition, ensuring security became an assessment criterion for people. For example, the Product Owner was assessed on the number of security incidents and findings from penetration tests. Later we also tightened the CI/CD pipeline. We did automatic vulnerability scans. If the scans showed that there were errors, you could not check in new code as a developer. As a result, you could not bring the product to production. And of course people don’t like that.’

Mirna: ‘How were the assessment criteria designed? For example, was that via KPIs or via the Dreyfus Maturity model?’

‘We used the ASAMM model (Agile Software Assurance Maturity Model, a method to analyze and improve the Software Development Life Cycle, ed.). We created a large questionnaire based on OWASP’s SAMM model. SAMM is still very ‘waterfall’. So we added Agile to that. This was also reflected in the personal assessment of the employees. Then you see the behavior change. But we also gave people a chance to achieve KPIs, right? We supported them with time, with tooling, training and things like that.’

Security as a business enabler

Mirna: ‘What did you find most difficult to implement or have carried out in the research?’

Nico: ‘To convince the business owners. For example, they thought it was expensive and took too long, while there was a rush to comply with new guidelines from supervisors or legislation. In my opinion, they ‘hid’ behind that,” says Nico. ‘The biggest challenge was to show that security is not a business disabler, but a business enabler. With safe products you have less maintenance and less misery. The time you put in at the front, you will get back at the back. I thought it was a challenge to show that. We also showed what the consequences could be if you supply an unsafe product. It costs the company money, you get dissatisfied customers and sometimes it even causes bankruptcy. That did help.’

‘How did you make that concrete?’ Mirna wants to know. ‘What metrics did you use at the different levels?’

Nico: ‘We gebruikten het aantal findings uit de penetratietesten. En het aantal meldingen door middel van responsible disclosure.’ 

Mirna: ‘Hebben jullie de uitkomst van het onderzoek ook getoetst bij andere bedrijven?’ 

Nico schudt zijn hoofd: ‘Wel hebben we eerst een nulmeting gedaan. En na een jaar hebben we nog een meting gedaan. We zagen gek genoeg niet dat het veel veiliger was geworden. Dat kwam vooral omdat er zoveel verschil was tussen de teams. Het ene team leverde wel een veilig product, maar het andere niet. Je ecosysteem blijft hierdoor dan onveilig. Je moet de teams echt zien als communicerende vaten, anders heb je een probleem.’ 

Mirna: ‘Maar hoe wist je dan dat je onderzoek succesvol was?’ 

‘We denken dat we de teams te positief hebben ingeschat bij de nulmeting. Positiever dan ze bleken bij de individuele begeleiding van de teams.’ 

Mirna: ‘En effecten van dit soort trajecten zie je vaak ook veel later.’ 

Nico knikt. ‘We zagen wel een verandering, namelijk dat het securityteam meer werd gezien als een kritische vriend in plaats van de afdeling business preventie. We hebben ook heel duidelijk het ‘comply and explain’ principe gehanteerd. Want misschien zijn er goede redenen om op een stukje een minder veilig product te leveren. Maar laten we dit dan uitleggen en beschrijven wat we eraan gaan doen om het risico te laten accepteren. Met andere woorden: maak het transparant.’

Most important factors for developing more secure software

“If you now look back at the research, what were the key success factors with a view to developing more secure software?” asks Mirna.

‘First of all, we didn’t appoint the Security Champion, but we chose someone who raised his hand and enjoyed doing it. In addition, we have also properly instructed, guided and regularly consulted with that person. We also didn’t blame the Champion if things weren’t going well yet. We rewarded when things went well. The nice thing was, you saw that other people slowly wanted to become a Security Champion. This is how people started to show the same behavior and the approach spread through the organization like an oil slick.’

It also helped that the research team was dedicated to guiding the development teams, Nico explains. ‘We didn’t have to worry about other security matters. We focused on the development teams.’

Nico also emphasizes that the teams participated in the research on a voluntary basis. ‘We asked the Product Owners to participate in our research. We showed the goals and also explained what we would do with the results. We also emphasized that we would anonymize the results. Teams could therefore not see and say that a particular team was doing poorly. Teams were also not required to participate. We were afraid that teams would otherwise give politically correct answers. Only the researchers and the Product Owners could view the results. The CISO and board only received the anonymous data. This made people feel safe and dared to make mistakes. Or give a less positive, more realistic score, I guess. And even though not everyone participated, the research allowed us to include all teams in the improvement.’

Mirna summarizes: ‘To achieve more secure software, it is – in short – important that security is recognized in the organization. That the CISO is present on the board and that the risks of unsafe software are clear. Furthermore, accountability must be placed with the Product Owner. Voluntariness is also important in appointing the Security Champion and the collaboration with and between the DevSecOps teams. In addition, you have identified a number of mechanisms, such as including security in the targets and assessment of people, in the CI/CD pipeline and in the sprint planning.’

Nico: ‘And make it part of your KPIs. Have the team report those kinds of things to the management team.’

‘Security must be seen as part of the product’

Nico emphasizes an important success factor: ‘What is especially important and what we wanted to achieve is that security is no longer seen separately from the product, but as part of the product. Safety must simply be in the software, it is an inseparable part of this.’

Mirna: ‘And the role of the Product Owner is also important in the agile way of working.’

Nico nods: ‘We have defined the product in great detail in advance. This made it clear what the Product Owner is responsible for end-to-end. We really made it DevSecOps. ‘What you build is what you run and own, so to speak. There is always a generic layer where security is centrally arranged. But as a Product Owner you have to check, for example, whether the server on which your stuff runs has been patched, you have to plan downtime, etc.’

“How did you keep the architects on track?” asks Mirna. ‘They often work in an overarching manner and often feel disconnected from the development process.’

‘It was indeed a challenge to achieve uniformity in the CI/CD pipeline from an architectural vision, otherwise the teams will build a pipeline themselves. Architecture was more focused on the business functions. It was also a challenge to get good coding guidelines for all teams and uniform error handling. How do you do input validation on a field, if a user enters something incorrectly. What information do you want to give back, so that the user and developer know what is going wrong, without providing information that a hacker can abuse? That remains a challenge.’

‘Ensure an open, learning culture’

Mirna: ‘How has the research been received within the organization where the research took place?’

Nico: ‘Positive. The CISO was behind it, which helped. But the research made the degree of safety of the product transparent and tangible. As a result, the teams saw that they had to improve. It also helped that the teams were helped to solve problems by the security team, from the People-Process-Technology Framework. We looked: is training needed? Do the processes need to be improved? Is there the right tooling? Or is there something else going on?’

“If you do agile really well, the risk is mitigated,” says Mirna. “That’s right,” says Nico. “If you apply agile properly and consistently,” Nico emphasizes. ‘The quality of software increases very quickly with agile working. If unsafe things come out in a release, you can really do the Plan, Do, Check, Act Cycle. If you see something quickly, you can solve it quickly. Fail fast, learn fast. An open, learning culture is important. We tried to create that by working with anonymous data. It’s okay to make mistakes, but you have to learn from mistakes. It was nice to see that the Product Owners started to collaborate much more with each other.’

Background 

Nico Kaag completed the research “Agile Secure Development & Operations” in 2021 for the Master Risk Management at The Hague Graduate School. The subtitle of the thesis is: “Security hand in hand with Agile Development & Operations?”. The objective of the research was to determine the impact of applying Agile SecDevOps on business risks and how these risks can be influenced. After a literature study, research was conducted at an (anonymously described) organization with seventeen agile scrum teams. To determine the impact of security in an Agile DevSecOps environment, two measurements were carried out with an identical method, ASAMM. The method is based on OWASP’s Software Assurance Maturity Model, to which Agile has been added from the Software Security Alliance. 

Recommendations from the research are aimed at regularly measuring the maturity of the teams and addressing security as early as possible in the development process, the so-called ‘shift left’. The use of Agile will stimulate continuous improvement. The chain/ecosystem is as strong as the weakest link and therefore large differences in maturity for teams, in the same ecosystem, must be avoided where possible, Nico writes in his thesis. 

Want to know more about the research or the ASAMM model? Contact Nico Kaag or Barry Derksen.