THE CLIENT: Scottish Water wanted to explore how it could create a visual resource to track data on a range of hazards such as animal faeces or the presence of lead across the whole of the water collection and treatment system. Water goes through around eight stages from falling out of the sky to coming out of the customer’s tap.
THE CHALLENGE: Time was relatively short – a visual dashboard summarising the current state of a particular hazard had to be available, and cover a large number of identified hazards, within a short time period. But the data was collected in different ways at the various stages of the water treatment process and had to undergo substantial data engineering before it could be usefully collated and displayed simply in Power BI for end users to check.
THE DELIVERY CHALLENGE: There was substantial uncertainty around whether or not the data could be extracted and manipulated in order to be displayed as expected in Power BI, as it was the first time this had been attempted. Agile provided a solution to allow developer capacity to be purchased for a 4 week sprint to allow experimentation to begin, and to allow a quick review point and potential pivot to a different approach in the event of problems, or a potential exit if the challenge proved to be too great.
THE CHALLENGE FOR ME: Scottish Water colleagues were unaware of Agile principles, and so I had to coach them as they became product owners, and were asked on a monthly basis to review progress in show and tells and retrospectives and make decisions about what the focus should be for the following sprint, or even if the work should continue.
THE OUTCOME: Over a period of five iterations, the team (comprising data scientists and engineers from Scottish Water and developers from CompanyNet) focused on two hazards out of around 30, and were able to produce a method of extracting and manipulating the data in Azure Databricks before exporting out into Power BI. This proof of concept was followed by a Minimum Viable Product for two hazards. Further sprints focused on scaling up the method for the remaining hazards in the new Risk Assessment Platform.
LESSONS LEARNED:
- Agile is an excellent methodology for experimental work, where the technical methods for producing project deliverables are unclear.
- Agile provides frequent and regular break point opportunities for work to be terminated, minimising costs for clients and controlling the risk of wasting money if the project is not going well.
- New product owners adjust with support to the greater level of involvement and decision-making that is required from them in collaborative agile practices compared to waterfall projects, where the requirements and costs are known in advance.
- Regular show and tells and demos of working software provide reassurance and focus developers’ attention on testing early in the process of creating something that works.
