IT security for critical infrastructure with the versiondog data management system

Standard solutions

People do not tend to think of sewerage services as being particularly high-tech, but that is far from the reality of the situation. The German partner water associations Emschergenossenschaft and Lippeverband (EGLV) operate 59 sewage treatment plants, 344 pumping stations and 411 drain and rainwater treatment facilities and detention basins. Most of them incorporate automated systems that are networked to allow remote monitoring and control.

Since 2009, all control devices used in EGLV facilities are managed by versiondog. Currently, their total number of automated control device projects is around 3,500. Add to that the parameter data and settings of numerous frequency converters, pressure sensors and level sensors and that brings the total number of versiondog components in use at EGLV to around 5,000.

When the Emschergenossenschaft was founded in 1899, it was the first German water association. A whole series of water associations modelled on it were to follow, including the Lippeverband in 1926. The two have worked in close cooperation from the very beginning and operate under a common organisational structure. Their areas of responsibilities have always included wastewater treatment and drainage management. New responsibilities have steadily been added, including additional districts and the upgrading and replacement of existing facilities. There is also the long-term project "Restoration of the River Emscher". Started in the 1990s and due to be completed in 2020, it includes laying new channels in order to return the river to its former level of cleanliness. Underground channels have not been constructed until recently because they would have been endangered by collapsing mineshafts due to past mining in the region. Along one notable section of the channel, between Dortmund and Dinslaken, the Fraunhofer Institute IFF of Magdeburg has collaborated in the development of an automated channel inspection system. A further notable achievement by the EGLV is the construction of detention basins that can be depended on to contain substantially more rainwater run-off than normal.

Even a brief look at these key responsibilities of the EGLV makes it clear how wide the range of challenges is that it faces—and it hints at the diversity of the automation and IT equipment that must be in use in an infrastructure that has developed over many years and decades. Although much has been rationalised and harmonised, there remains within region covered by the EGLV a great deal of structural heterogeneity, which makes effective maintenance no easy task. A relatively new factor is the need for water associations to fulfil the requirements of the IT Security Act for critical infrastructure. Companies have to demonstrate that their security measures and standards are in place and suitably robust. An important part of this is to be able to quickly resume normal operation after a disruption. One basic requirement, for example, is to be in a position to reload the software of any crashed PLC in any control unit with exactly the right version—fast, and with certainty. Other standards that need to be taken into account include BSI data security standards and the set of risk management rules produced by the German Association of Transport Sciences (DVWG) for water service providers.

It was not very long ago at EGLV that communication with electrotechnical installations was the responsibility of the maintenance department. Now, due to the merging of office and industrial networks, it is that of the IT department. There used to be a number of different remote access technologies from different manufacturers connected to the control centre. Now there is only one. Work is currently being done to enable AUVESY's software to monitor and access every single piece of automation equipment, however minor.

Ralf Hellmann, head of the maintenance department, has a clear goal: "We want to connect every installation to the versiondog system, even those that are right next door. The installations are organised in a hierarchical structure, which is clearly reflected in the versiondog software. With staff numbers limited, this helps guarantee support at all sites and keep precise track of all changes."

Wide range of functions and features as standard

According to Ralf Hellmann, even the standard version of versiondog offers all that is needed to attain the required level of quality. The software is used for the version control of all digitised data, from complete PLC programs down to the parameter lists of individual components. For the networked installations, a series of around 120 snapshots a day are made. There is no prioritisation; the important thing is that backups are made every day.

versiondog users can get even more value from the system by using it for all types of data, not just that of automated equipment. The SmartCompare comparators can be very helpful in clearly displaying the differences between two versions of an Excel file, for example. And for automated equipment, it is just as important to track the state and changes of humble level sensors and flow meters as it is of a sophisticated process control system. Where comparison results cannot be presented in text or graphical form, a binary comparison is always possible. Dr. Thorsten Sögding from Business Development at AUVESY is keen to emphasise that the range of data types with custom comparators and interfaces is constantly expanding.

Back in 2002, when Ralf Hellmann started at the Emschergenossenschaft, there was not a lot of attention paid to backing up the control programs of automated equipment. In fact, it was often done by subcontractors. Although one would expect the opposite, more time and effort was spent trying to align versions of the control software programs of different pieces of equipment than was spent making the necessary ongoing changes and adjustments.

Reducing the programming and support workload is the only way to satisfy future requirements, such as those of the Industrial IoT, without increasing staff levels. Ralf Hellmann sees the introduction of harmonised standards, especially on the control side, as the key. "We are well on the way to achieving our goal. Theoretically, we should at some point be able to program a new installation by simply copying 1:1 the software from an existing installation of the same type." Another approach being taken is to implement new projects with process control systems. In this case, as in others, the constraints for programmers are so tight that standardisation will only be possible with effective data management and by using a well-maintained and organised library of standard programming blocks.

Until as recently as 2007, the maintenance department only had three programming devices in their workshop. PCS service technician Arnes Hadzic remembers: "It was down to me to make sure that the current versions of programs were always correctly archived. But no matter how committed a person is to the job, that kind of task is impossible without the right tools. So EGLV introduced a data management system. When updates and support for the system came to an end, EGLV switched to versiondog from AUVESY."The large geographical area covered by the association made it especially important to EGLV that AUVESY be able to ensure that all users—both internal and external—would always have 100% secure and reliable access to the software. Now that the association has around 150 versiondog users from a wide variety of departments, Arnes Hadzic can say with confidence: "AUVESY was able to provide us with the perfect software for our needs."

Now that he is an experienced versiondog user and administrator, Arnes Hadzic is in a position to fully appreciate what he calls the "Apple Effect": "Maintenance personnel on-site can take one look at the software and start using it straight away". They do not necessarily need to know details, such as how devices are connected to the server via LTE. Because of its ease of use, Ralf Hellman gives working with versiondog "a solid 8" on a scale of 1 to 10. He is quick to add that if they themselves were to use all of the system's functionality "there may well be the potential for a 9 or a 10." There are, however, certain peripheral structural constraints that have so far prevented them from doing so.

Ask Ralf Hellmann if versiondog has saved the association money and he answers with a categorical 'yes'. "It's difficult to say exactly how much money we have saved in total, but the amount is so high that the precise figure isn't really important." In the past we spent a lot of time aligning versions of control programs, documenting them or rewriting them altogether.

There is no question about it, with versiondog the days of shoeboxes containing collections of disks and USB sticks (and no better way of telling which software versions belong together) are well and truly in the past. "I don't want to go back to that," says Ralf Hellmann. In fact, he would like to start taking advantage of the benefits of versiondog from the very start, i.e. when an new installation is under construction. "It would be great for us if use of the system were also compulsory for external contractors, whenever they install, set up and commission technical equipment." He goes on to emphasise that there will have to come a time when no control program data ever goes missing. "With versiondog there is never any question about who programmed what and who made which changes, and that increases quality."

The versiondog functions 'Supplier Check-Out' and 'SmartImport for supplier projects' ensure that it is completely clear in the change history what was done by external staff and when. And that is important at EGLV because although internal staff are able to make a number of programming changes themselves, contractors are very often used, especially on the larger jobs. Ralf Hellmann also mentions the benefits of the fully documented change history provided by versiondog. According to him, program changes that work well at first can lead to programming dead ends during the course of ongoing development. "With versiondog, we find errors or at least the starting point of programming problems within minutes - usually," he adds, with a smile.

Cybersecurity is not exactly a new subject, but it has come into sharper focus in recent times. Critical infrastructure, such as sewage treatment plants serving more than half a million people—of which the Emschergenossenschaft has eight—must be effectively protected. The Emschergenossenschaft and the Lippeverband are both working hard on future security solutions. They know they can count on AUVESY and versiondog for support.