.png)
A Single BACnet Routing Bottleneck at Microsoft Led to 118,000 Hours of Missed Runtime Before Anyone Recognized It as a Systemic Issue
CBRE didnât realize how much runtime Microsoftâs 80-building campus had been missing until after it fixed a nearly undetectable problem.
Months of high-volume routine comfort escalations, mainly hot/cold calls, went undiagnosed despite intensive controls troubleshooting. The culprit turned out to be a network bottleneck that had been quietly degrading operations for more than a year.
The campus runs 16,000 devices from 40 manufacturers across 800 BACnet channels, layered with four BMS platforms and an Iconics FDD application. When schedule overrides sometimes worked and sometimes didnât, the teamrepeatedly escalated to the BMS vendor. âTraditional troubleshooting continually failed,â said April Yi of CBRE.
Packet-level analysis through Optigo Network's Visual BACnet tool eventually revealed repeated âbusy routerâ messages associated with a single Allerton controller, which served as the bridge between two sides of the campus. The controller wasnât misconfigured, and no system was incorrectly integrated. However, as the system had evolved over the years, network traffic patterns between FDD and BAS became suboptimal, allowing the problem to slip through the cracks.
After reassigning Iconics servers to reduce traffic through this single-controller bottleneck, the busy router diagnostic cleared, and response times improved.
Then the team looked backward.
From June 2023 through August 2025, they identified nearly 118,000 hours during which systems didnât run as scheduled between 7:00 and 11:00 AM. âThirty-eight percent of those assets did not turn on for more than 50% of the scheduled days,â Yi said.
Energy use dipped. Complaints rose.
The lesson wasnât about a bad controller. âThe red blinking light is not always where the problem is,â said Ping Yao of Optigo Networks. Without historical network visibility, the missed runtime looked like gremlins in BAS controllers.
â
â
Register for the next Nexus Labs event
Sign up for the newsletter to get 5 stories like this per week:
CBRE didnât realize how much runtime Microsoftâs 80-building campus had been missing until after it fixed a nearly undetectable problem.
Months of high-volume routine comfort escalations, mainly hot/cold calls, went undiagnosed despite intensive controls troubleshooting. The culprit turned out to be a network bottleneck that had been quietly degrading operations for more than a year.
The campus runs 16,000 devices from 40 manufacturers across 800 BACnet channels, layered with four BMS platforms and an Iconics FDD application. When schedule overrides sometimes worked and sometimes didnât, the teamrepeatedly escalated to the BMS vendor. âTraditional troubleshooting continually failed,â said April Yi of CBRE.
Packet-level analysis through Optigo Network's Visual BACnet tool eventually revealed repeated âbusy routerâ messages associated with a single Allerton controller, which served as the bridge between two sides of the campus. The controller wasnât misconfigured, and no system was incorrectly integrated. However, as the system had evolved over the years, network traffic patterns between FDD and BAS became suboptimal, allowing the problem to slip through the cracks.
After reassigning Iconics servers to reduce traffic through this single-controller bottleneck, the busy router diagnostic cleared, and response times improved.
Then the team looked backward.
From June 2023 through August 2025, they identified nearly 118,000 hours during which systems didnât run as scheduled between 7:00 and 11:00 AM. âThirty-eight percent of those assets did not turn on for more than 50% of the scheduled days,â Yi said.
Energy use dipped. Complaints rose.
The lesson wasnât about a bad controller. âThe red blinking light is not always where the problem is,â said Ping Yao of Optigo Networks. Without historical network visibility, the missed runtime looked like gremlins in BAS controllers.
â
â
Register for the next Nexus Labs event
Sign up for the newsletter to get 5 stories like this per week:
This is a great piece!
I agree.