Macquarie University deploys Oracle RAC

No more database and website crashes on exam results night

Comments

If asked to determine which is of greater importance — the bar opening on time or keeping IT up and running — many university students would likely argue the former, but according to Macquarie University’s CIO, Marc Bailey, there are two occasions a year when the student response is universally IT: Exam results night.

On these nights the exam results for the university’s 35,000-strong student base are released at the stroke of midnight, with dramatic consequences for Macquarie’s student administration system.

“For those nights of the year we go from what is a relatively simple eight-hour-day, five-day-a-week business system to a mission-critical high-performance, high-demand system,” Bailey says.

Giving a sense of the degree of scale required, Bailey says the system would have some 1000 staff accessing it on an average day. On results night, that figure swells to 20,000 users within 12 hours of the results going live.

“It is a student admin system and a system of the university, but on those dates it becomes the most important system. If we say we are going to get the students [the results at midnight], we have to get it to them.”

Delivering on that promise to students was proving difficult, with the university finding that its existing system just couldn’t cope, and one occasion, crashed within five minutes of the results going live.

“We were disappointing our students and that’s no good for the reputation of the university, it’s no good for the students and no good for the IT people,” Bailey says.

So along with VMware’s Grid to help it scale up a farm of Apache Web servers on the front end, the IT team looked to Oracle’s Real Application Clusters (RAC) to help it better manage its back-end database requirements.

RAC, Oracle says, enables a single database to run across a cluster of servers, increasing fault tolerance, performance, and scalability.

Using RAC, the university is now able to deliver some 7,500 results within the first hour of their release and handle more than 2100 concurrent active user sessions. The system is also able to scale to a peak of 16,000 exam results per hour.

Bailey says that while the benefits of the RAC system are very noticeable, the complexity of its implementation was reasonably high, taking about 45 days to go from proof of concept through to validating the infrastructure, then 90 days to “cut over”.

“The Oracle database is the core of our production system at the university, and we [effectively] put our eggs into one basket, but if you build the thing correctly – build in redundancy and scale – it takes the effort out when you implement a new system; you only have to worry about application resilience rather than database resilience,” Bailey says.

“What we tried doing was building a data core in two locations, one at our hot site and one at our primary site. The crunch moment was when we migrated the data across.

Because we did leverage the data core for all our production systems it wasn’t as simple as migrating one application across; it was all our applications.”

The university has about 10 critical business systems including its finance system.

On the hardware side the university moved from a Sun ‘big iron’ set up — which, while good in it’s own day, Bailey says, was just no longer coping — to a Dell 6800 blade server environment.

“Technically this is a cluster with five logically redundant nodes (three in the primary site and two in the hot site); a couple of dozen blades,” he says.

Detailing the decision to expand its relationship with Oracle — its existing student administration system was Oracle-based — Bailey said the university recognised the fundamental importance of its databases to its business.

“For us, our relationship with oracle remains at this time about the database,” he says. “It is just one of those things we don’t want to take any risk with as it is core to what we do. We are even taking the step of working with people to make their applications work on top of Oracle.

One such instance of this, Bailey says, was to take the step of partnering with Adelaide-based Planet Software to help it get its Sonia application, which helps match students to field placement opportunities, working on Macquarie’s Oracle environment.

“It is a small company but a great fit for what we want to do, but initially it didn’t run on Oracle, and that was a problem for us as we wanted to run this thing at a scale much larger than they are used to dealing with,” Bailey says. “So, we needed to know they could leverage our redundant data core. We worked with them, got them to do a port.”

Looking at the additional benefits of the RAC deployment, Bailey says the university has also been able to reduce the time it takes to create clone databases for testing by about 90 per cent, down from around twenty to two hours.

“That is a really impressive thing about Oracle RAC in combination wit SAN technologies — it makes it much easier to replicate databases,” he says. “The reason that matters is that typically when you’re doing that you’re looking to solve a problem, so you don’t have 20 hours.

“Capacity management is easier; it is much easier to figure out quoting and allocating space, it is really easy to add another database to the system, so it makes us more agile.”