To date, among the most obstinate obstacles to the widespread adoption of decision support systems (DSS) has been the enormous investment in time needed to establish such systems. A joint project between the CSIRO Divisions of Information Technology and Water Resources has delivered some lessons on how to cope with a moving target.
In an era when just about the only organisations not under pressure to change are those on their way to receivership, any decision support system (DSS) is in serious danger of becoming obsolete by the time it is finally in place.
"One of the limitations on the use of a DSS seems to be that it takes such a long time to establish," says CSIRO systems integration specialist, Dave Abel.
"You've got to assemble the resources, kick the organisation to do the work and so on. If that takes two years that might be a year longer than the organisation is interested in that project.
"If they can't get a solution in six months, they'll fall back on a thumb in the air and a guess."Now a collaborative project between the CSIRO Divisions of Information Technology and Water Resources is suggesting some useful strategies for developing decision support systems that may help organisations make them far more powerful and relevant management tools in future. For a start, the project has pioneered the adoption of an approach from the database world - that of the federated database - for integrating modelling systems into a DSS. But along the way Abel says the project has highlighted other useful strategies of interest to DSS developers.
Some three years in development, the HYDRA system is an experimental spatial decision support system (SDSS) for water quality planning developed in collaboration with the Sydney Water Corporation. The application focus is water quality management in the Hawkesbury-Nepean catchment under both urban growth and changes to the sewage treatment plants (STPs) in Western Sydney. Building on water quality modelling systems in place at Sydney Water Corporation, HYDRA is a project of astonishing complexity.
The complete environmental management information system relies on integrating numbers of spatial information systems (SIS) and computational modelling systems. Spatial information systems provide powerful facilities for assembly, fusion and visualisation of spatial data, while computational models allow analysis of the behaviour of complex physical processes.
Each of the modelling systems has a particular domain of application. As an example, the SALMON-Q system can model tidal flows in channels but not surface flows across land, while HSPF can model flows across land and in channels but not support tidal flow.
Initiated in 1994, HYDRA is designed to support "what if" analyses by allowing a user to specify a scenario (by editing an existing scenario), evaluate the outcomes of the scenario and examine those outcomes. Phase one involved developing an intuitive GUI designed for planners or managers rather than hydrologists. To achieve this, the developers provided maps and charts as the working surfaces in the GUI. Under this approach, a user can specify changes in land use or new STPs by editing the map. Clicking on an object in the map causes the object to present a menu of performable operations.
"Phase one looked at the issue of how you make this type of capability more accessible to the managers," says Abel.
"In other words, if they were going to use it, it had got to be in a form they could make sense of. We figured that if they had a map of the Hawkesbury-Nepean they could do something like playing Sim City [the game] and determine the effects of another half million people in Western Sydney by drawing in extra suburbs, dropping in new sewerage treatment plants and so on." It was equally important to ensure access to the modelling system was available and widely used outside the Corporation's specialist modelling unit. The concern was that the capability of the modellers was not being fully appreciated throughout the organisation. There was a danger this lack of awareness would lead to managers of other units deciding to analyse problems and develop solutions independently, unaware there were specialist modelling techniques and excellent data sources available elsewhere in the organisation.
"Where you've got a specialist unit that can run these models, whether it's hydrological models or a financial model or manpower planning or whatever, you can certainly establish a little cell of people that can do that properly," Abel says. "But if they're tucked away down a corridor, the effectiveness of that unit to the whole organisation is pretty limited.
"You should be able to get a manager to do a strategic planning exercise without having to call in the modellers or devote the modellers to their project for six months. Rather managers should be able to specify a problem in their terms and get the results back in their terms."The GUI initiative was designed to tackle this problem head on.
Another issue confronted during phase one was that of stocking the systems with the right data. While the project was in progress, Water Board staff were busily updating some models to make their predictions more accurate.
A stitch in time
For Abel, whose expertise lies in systems integration, the challenge was to find ways to tie the varied modelling systems together.
Abel says the Sydney Water Corporation had a very long track record in using hydrological modelling systems to understand the behaviour of river systems.
Each worked well in particular niche areas, but a major aim of the project was to "stitch" together modelling systems to help the Corporation deal with highly complex problems.
"They can do that manually," says Abel, "but it's fairly difficult."Complicating the process was the scientific complexity and extreme age of many of the models involved.
"We had the guys from Water Resources to do much of the hydrological modelling - but these systems go back to the mid to late '70s. Their inputs are modelled on FORTRAN fixed field punch cards, so they are awkward to use. And, like all systems that were conceived 20 years ago, they've had extensions bunged on them over that 20-year period," says Abel.
"They really are God-awful, and it really takes a year or two of experimentation for people to really know their way around them," he says.
"They are horrible systems."
Once phase one was completed, it was time to begin to achieve an understanding of the task of building environmental management information systems incorporating spatial information systems and multiple, pre-existing modelling systems. Primarily that involved integrating the diverse modelling packages.
That aim became the core of phase two.
Abel says special purpose modelling packages could become more accessible and more effective for decision support when integrated into a spatial information system. The trouble was, differences in scope, data models and command languages all made integration difficult. To achieve such integration, the CSIRO researchers adopted a concept from the database world - that of federated databases - and sought to apply it to DSS.
"The federated approach really deals with how you actually can couple existing systems," says Abel.
"It is not an approach used a lot yet, although it is getting more airplay now.
It's an approach we borrowed from the area of database, where the topic of federated databases goes back probably to 1988 or '89.
"The approach really comes from the corporate scene, where there are very strong motivations to hook together pre-existing databases within a corporate information system," Abel says.
Something of value
It is the recognition of the value of the information in corporate databases as a corporate asset that has prompted development of a rash of application programs capable of accessing multiple databases at a time. So-called federated database systems, or multidatabases, provide the databases support for such applications. A federated database system is defined as a collection of cooperating yet autonomous component DBMSs. Abel was confident the approach would have equal value to DSS.
"The other aspect was that if you have a number of legacy systems, and you want to move across to a new environment, that is actually a fairly difficult task," Abel says, "and the federated database idea can be used to ease that migration.
"So we borrowed ideas rather than inventing them, but of course along the way it was also necessary to extend them. I don't think the approach has been widely used elsewhere for decision support, although it is becoming a more widely used technique in other areas." And although in the HYDRA project the federated approach was used to integrate hydrological modelling systems, Abel says there was no reason why it couldn't be applied more widely to other kinds of decision support systems.
"It's something you do when you've got existing systems and you don't want to throw them away," he says. "You are trying to retrofit them for use within the new system, which can be much more cost effective than rebuilding these things.
"Often it's not a matter of replacing these other systems for use within a corporate DSS because they are mandatory inclusions."To achieve the integration also meant borrowing a concept from the networking world and porting it across to DSS.
"In networking systems they talk about wrappers," says Abel. "A wrapper is something that you put on top of an existing system to make it pretend to be a relational database or a Web server or whatever. That is a very similar approach to what we've been following."Strings and sealing waxWhile integrating the modelling systems proved reasonably effective, Abel nonetheless describes the end result of the year-long stage two as "a strings and sealing wax solution". His comments highlight just how complex and difficult such exercises can be, and point to a great need for flexibility amongst DSS developers.
"The third year basically started off by us deciding to throw away nine months of that year and then to figure out how we could do it better."According to Abel, that experience was an object lesson in the value of pilot projects and prototyping. Since many of the integration issues to be faced in such projects are difficult, Abel recommends a careful and cautious approach to such exercises, constantly evaluating and re-evaluating results as you go.
"My advice from that nine months is to build a little bit, assess it and then figure out whether you should really write that off to experience and then go back and start from scratch."As he points out, that can be a painful exercise, particularly when it means telling a team of bright young people who've poured their hearts and souls into the work that they have to go back to fundamentals and write off the work done so far. But such hard decisions will be vital if any of the DSS is to be ready in time to be useful to an organisation under constant threat of change, and so it proved with HYDRA.
Once that decision was taken and further work done, stage three, which was just completed, was to take the output of the models and make them available over the Internet.
Keeping it relevant
As organisations constantly change goals and directions, one of the challenges is to keep the DSS useful and relevant. For this as yet Abel can offer no solution.
"I don't think anybody has got a good answer to that," Abel says. "There's a team in the States working out of GTE, the telecommunications company, and they talk about future-proofing. They are arguing that you really do need to organise systems in a highly modular way so that you can replace components on the fly and change the interface.
"Unfortunately to date nobody has really provided a convincing example of how that can be done.
"The other issue is that it's a moving target too," Abel says. "Organisations can reinvent themselves in six months, and in a government business enterprise often the issues change pretty dramatically too.
"So one of the continuing issues is that problems arise much more quickly than you can actually gather the data in some sort of corporate resource to handle it."Addressing this difficulty is the thrust of a new project just under way within CSIRO, which is already receiving a good deal of attention. Although very much in its early stages, the notion is that rather than assembling all the resources locally, an organisation would only assemble those resources specific to its core area of expertise and local domain.
"So the Sydney Water Board might only gather data on water quality and the Hawkesbury-Nepean, and maintain that as a core resource," Abel says. "Where it comes to spatial data or modelling software, it would access those services over the Internet on an as needed, when needed, footing and on a fee-for-service basis.
"There are some pretty strong coordination efforts going on right now, and this Australian spatial data initiative is one of them."The researchers at CSIRO say that not only are there very strong cost motivations in doing it this way, but that the same approach makes every bit as much sense in the corporate environment. In fact Abel believes stopping the turf wars and getting coordination going between organisations will become a major part in the development of any DSS, and in ensuring that the DSS is developed in time to be of value to the organisation. And if sufficient numbers of organisations accept the notion of resource sharing as the optimum route to DSS, managers in future might be far less tempted to adopt the "thumb in the air and a guess" approach which gets so many companies in trouble today.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.