Hardware Failure Results in Zero Downtime When Running Emerald’s Primavera In-A-Box

I woke up with little stress or angst the other day despite the fact that I knew we were replacing a memory module on Emerald’s Primavera-in-a-Box solution that was running production workloads for a client in the energy industry.

Typically in the past, this kind of work would spike my anxiety levels, especially with a production server. This type of scenario requires a large amount of paperwork, many levels of approvals, along with various team members asking many complicated and sometimes ill-informed questions. This spawning frustration, firefighting modes, arguments, supplier shortfalls (delivery time shortages etc.) and egos all play a role in complicating the task at hand. The question “where were you when we implemented our current environment?” often comes to mind when dealing with the egos involved in the typical corporate event that occurs after a system fails.

Has Anyone in Your Company Ever Experienced This Kind of Stress Because of Hardware or Software Failure?

Emerald has heard from clients how 1 hour of planned maintenance turned into a 10, 20 or even a 72 hour outage. Can your business afford this kind of downtime for Primavera? What about other applications with the same level of criticality that are running on Oracle Database?

Yes we have received these kinds of calls to our support desk in the past five years, sometimes from the users or sometimes from the technical resources that are responsible for Primavera, pleading for help with a lot of panic in their voices. It could be a user that just deleted a resource hierarchy that affected everyone using Primavera (the user had access), or a technical resource that found a server disk failure and realized no proper backup currently existed and the application user was going to lose data.

You would think in 2015 these kinds of server issues could be avoided or preempted right?

Why Was Stress Reduced to Nothing?

Two years ago, the client described above decided they wanted to eliminate most of the HW/SW downtime they were experiencing with Primavera P6. The cost and risks to the business were becoming too significant to ignore and they decided to implement Emerald’s Primavera-in-a-Box solution running on the Oracle Database Appliance X4-2 with the Oracle ZFS Storage Appliance ZS3-2. With Emerald’s Primavera-In-A-Box solution the end user gets:

• Application performance (outperforms custom hw/sw solutions by 3-7 times)
• Application stability (can restore from all user errors)
• Application usability

These benefits are based on real life testing with the client in question including zero downtime during shifts where work was being scheduled in Primavera P6. Why does the business user demand such a system? It’s simple, when work being scheduled ranges from $1 to $7 million a shift it makes sense that this client believes it would be prudent to eliminate the risk of failure.

How Did We Eliminate the Guess Work?

Using Oracle Enterprise Monitoring we were notified that a memory chip on one of the nodes was starting to become suspect. We received an automatic notification to our support email and we approved the automatic service request with Oracle for a field technician. We coordinated the site visit with the datacenter and the field technician for the day after the new memory chip was to arrive. In total we were onsite with all necessary personnel and equipment in 2 business days.

What Maintenance Work Was Performed?

We shut down the node with the memory chip that was showing signs of issues and we replaced it while the users of Primavera kept working in the system. It was explained to me after that under normal circumstances this scenario would typically take 5-10 people from various technology backgrounds (DBA, Storage, networking and APPS) around 1-5 hours each to coordinate a ticket delivery and get the necessary approvals, and another 1-2 hours to execute the fix. The hardware replacement took us 15 minutes and we achieved zero downtime while users continued to use Primavera during their regular business hours.

What Cost Risk Was Avoided?

Imagine doing this maintenance work duration a 2000 man-hour shift that relied on a Primavera schedule. If we assume conservatively that the rate for the average maintenance worker is $50/hr x 2000=$100k/hr. Now let’s assume only 20% of the crews would be affected by the scheduling of work because the other crews are supportive, then we have $20k/hr. Now it is clear why even 1 hour of planned downtime can be extremely costly, not to mention the cost if that time runs longer than anticipated.

To learn more about Emerald’s Primavera-In-A-Box solution please take a look at our product page or better yet contact us.