Pro Member Gathering: NexusCon 2024 Best Case Study Recap - 'Say Cheese' Pilot Projects

For our first Nexus Pro Gathering since NexusCon, we were excited to welcome Christopher Tjiattas, Program Manager at District Control Systems, back to the (virtual) stage to once again give his award-winning case study presentation. 

“Pilot Projects: Say Cheese” won the 2024 Nexie for Best Case Study, and during Take 2, Christopher discussed what a pilot project should be, shared an example of how they can fall apart, and offered three key takeaways we all can use when designing pilot projects moving forward. Read our wrap-up below, and thanks again to Christopher for his knowledge and time!

What is a Pilot Project?

Before diving into how a pilot project can fail, and ways to avoid that, it’s important to define what exactly IS a pilot project. Christopher provided us with a checklist…

Is your pilot project…

1. Limited in scope? Pilots should be small, specific, and controllable to ensure you’re testing the exact projects and procedures you intend to, without outside influence or variability.
2. In a Test Environment? Pilots require a safe, dedicated space for testing that won’t impact other areas of business operations.
3. Providing feedback loops? Input should be received from all stakeholders for the pilot to improve over time and provide lessons moving forward. 
4. Mitigating risk? Pilots should identify risks and address them, or at least learn about what they are.
5. A decision-making tool? Ultimately, pilot projects must provide data you can use moving forward for decision-making.

Without each of these items, your pilot project may succumb to the Swiss Cheese Model, in which flaws align to cause failure.

How Pilot Projects Fail

To illustrate how a breakdown in his pilot project checklist can cause them to fail, Christopher used an example from his own work with the U.S. Department of State Overseas Bureau of Operations (OBO). With more than 190 overseas missions, all including 24/7 facilities with complex systems, the OBO presents unique challenges for building operations and maintenance. With many of these buildings located in the most remote areas of the world, far away from vendors, it can take as much as $25,000 and 3-4 days of travel for even one technician to provide onsite service.

Considering the costs, it’s understandable that the OBO was interested in remote connectivity for those systems. Unfortunately, due to security risks and other factors, that option wasn’t currently allowed in the Foreign Affairs Manual, the bible for operations throughout the State Department. However, with a champion who was able to secure funding and support, as well as hire a vendor, select sites, and develop use cases, a pilot program for remote connectivity was constructed — all in a matter of just a few weeks, without the involvement of subject matter experts (SMEs).  

You can see right away where the pilot began to fail. Without SMEs' input in the pilot program's planning stages, they could not engage until well after the project was in motion, making it more difficult to correct errors and missteps they could plainly see once they were involved. 

The pilot program’s contract with the vendor was just 26 words, creating a lack of clarity, undefined goals, and unspecific deliverables — all challenges that were difficult to overcome. This also allowed for broad use cases, developed by many stakeholders who often didn’t agree and overlapped with their goals. 

When it came to site selection, the program also opted for sites with aged and outdated HVAC control systems, requiring upgrades before the pilot program could even begin. Several of the sites already had local technicians available on-site, making the ROI for those facilities even less valuable. 

Eventually, budget cuts led to the program’s discontinuation, but that was only after several million dollars were spent. The sunk cost fallacy and confirmation bias took hold, dragging the pilot well beyond where it should have ended. 

Let’s take a quick look at the pilot program checklist to see if this project made the cut…

1. Limited in scope? No. With an unspecified contract and a broad range of goals and deliverables, this pilot was the opposite of limited.
2. In a Test Environment? Many of the selected sites weren’t optimal and introduced external problems and risks.
3. Providing feedback loops? While stakeholders met regularly, the news they brought to decision-makers often wasn’t well-received, leading to a breakdown in the feedback process. 
4. Mitigating risk? With multiple, often overlapping use cases that were broad in scope, in many ways, this project created more risk in what was already a risky endeavor.
5. A decision-making tool? Unfortunately, this pilot was doomed from the start and provided no opportunities for decision-making.

Key Takeaways for Pilot Programs

Despite this project’s failure, there were several key takeaways Christopher learned for future pilot programs.

1. Involve SMEs early. With SMEs involved from the start, pilot programs can have better-defined objectives and set goals. Sometimes, this may be something you can do yourself with internal SMEs. For other projects, it may be necessary (and more than worth it) to hire a separate consultant, apart from vendors, to make sure that the owner’s interests are represented in the early stages of project development.
2. Limit the scope. In other words, keep it simple, stupid. Don’t try to solve more issues than less. By limiting the scope, you also limit cost, risk, and the potential for failure.
3. Align stakeholders. With a limited scope, you must ensure there is no overlap in use cases, with aligned stakeholders who recognize and support the goals. If there isn’t support for that, or if more than one goal is identified, then have multiple pilot projects.

Ultimately, Christopher used these lessons and helped the OBO to design another pilot project six months later — with a much more limited scope, focused goals, with site selections that provided a high ROI. With SME input from the start, they were able to develop a project using AGILE methodology for feedback, and it has led to ongoing success. 

While final numbers are still being calculated, Christopher estimates the pilot project could save the OBO as much as $300,000 a year in energy costs, in addition to savings already achieved in travel costs for technicians.

If you’re currently considering a pilot program, or perhaps in the middle of one and wondering why it might not be going as well as planned, we hope this checklist, along with key takeaways, helps design a program that can succeed and expand beyond the initial stages into a full workflow for you and your team. 

Thanks again to Christopher Tjiattas for his knowledge, and congratulations on his Nexie Award! This one will be hard to beat in 2025!

Watch Recording

Slide Deck