Join us at NexusCon 2024 - Apply to attend now
Hey friends,
As a follow-up to recent newsletters Where are the API-first companies? (last week) and The API-first Data Layer (today), this members-only deep dive unpacks this concept further by defining the independent data layer, unpacking the nuances and unanswered questions, and updating the Nexus Vendor Landscape with this new category. If you havenât read those two essays first, please do.
Enjoy and let me know what you think!
âMiddlewareââŚ
âData lakeââŚ
âData aggregation layerââŚ
âData infrastructure layerââŚ
âData transformation layerââŚ
âInfrastructure as a ServiceââŚ
We love to make up new terms and acronyms in our industry and the new wave of data layer terms are no different. Iâm not sure where the term âIndependent Data Layerâ (IDL) came from, but itâs my preference, folks, and Iâm sticking with it.
Terry Herr1 defines it as âan edge connectivity layer / middleware that is independent of the applications or application layerâ. As he recently recounted in the March Pro member gathering, the need for this layer arose with the advent of cloud-based energy analytics software. These software providers needed meter, sensor, and actuator data from the building.
Early offerings like Building Roboticsâ (Now Comfy) Trendr, Lynxspringâs BACnet data pump, Sierra Monitorâs IoT gateway, the open-source VOLTTRON did just that. But as weâll see, that original need has expanded beyond simply pumping BAS/meter data into the cloud.
A good way to explain where we are today is the gap Brian Turner talked about on the Nexus podcast:
On one side you have modern enterprise (cloud) technologies (data lakes, middleware, applications) and on the other you have on-premise systems. You fill the gap between them with a domain-specific, web-developer-grade API.
And each vendor in this category is going to approach filling that gap differently.
Therefore, we can also define it in terms of capabilities, some of which are required to fill the gap and all of which represent a wide range in todayâs vendor landscape.
While this layer started out very simple, you can see that beyond the range of capabilities, there are also a ton of nuances to explore and understand. Letâs do some exploration.
As with any layer of the smart building stack, the devil is in the details. From my vantage point, abstracting the details away with a few lines of code, as is the API-first promise, is still a few years and many unanswered questions away.
So letâs dive into the questions, rapid-fire style. When youâre done reading these, youâll understand the edges of the problem and my perspective on them. As usual, I make no guarantees on the right answers!
On one hand, you could say every building needs an IDL. But Iâd argue that a building owner really doesnât feel the pain of not having it unless they have a ton of buildings and/or theyâre pretty far along the smart building journey. And that limited and/or delayed pain could really limit the market for the IDL vendor.
Put simply, the business value of the IDL is exponentially greater when you have:
In other words, if Iâm just doing analytics on top of HVAC and metering, which honestly is the state of the art right now, how big of a deal is not having the IDL? Techy types like to say âthe future is here, itâs just not evenly distributedâ. That cliche applies here too. How long will the pain take to distribute?
Among Nexus Pro members, there seems to be a bit of a debate on whether the IDL should be just an abstraction tool for on-premise systems (capabilities 1 through 3 above) or whether it should also have the full suite of capabilities (1 through 9 above).
The former camp seems to think that the IDL should stay focused on integration with on-prem systems and leave everything else up to the application providers.
My take is similar to the one Brian Turner shared at the March gathering:
âItâs becoming less and less of a problem to get data out, especially with the new technologies and architectures implemented. And so the way we're thinking about things is less about how do I get data out of a 20 year old system that probably needs to be modernized anyway, and really focus on the newer technologies. And, and now that we have sensor clouds and all of these different streams of data, it's no longer coming from a BMS per se, it's coming from all over the place. It's now extremely important to understand how that data all relates. That's where the transformation comes in.â
I agree. I think this layer needs to go beyond in two ways:
I think this layer needs to be considered a software product in its own rightâthis isnât as simple as a data feed with context included. As Andrew said on the podcast:
What our product brings is the tools that allow us to deploy faster, collect data better and more efficiently. We monitor for changes in the network, monitor the hardware platforms we deploy, and maintain the health of those hardware platforms. We're really selling a service around reliable data acquisition. Volttron plays a central role in that, but it's really the suite of the monitoring, maintenance, and all that stuff that we're offering on top of Volttron that is really where our value comes from.
I think the companies that win this layer will build their product better than anyone else because of their singular focus.
As weâve discussed at length, itâs very confusing to go shopping for smart building software. Even if a shopper understands the value of this layer, I predict that API-second application providers will claim that they do it too. And they will probably be at least somewhat correct. After all, itâs already part of their stack.
My take is that it will take some time for the advantages of the API-first strategy to kick in. Itâs like a flywheel that takes a long time to start spinning, but once it does itâs nearly unstoppable.
But weâre not quite there yet.
Specifically, I think digital twin companies will lay claim to this layer because the value propositions have heavy overlap. And vice versa, too. I think IDL companies can and will begin to compete with API-second digital twin offerings. And let the buyer confusion continue.
Getting data from buildings is hard no matter how you do it. IDL vendors will claim to do it faster and better with modern machine learning (ML) technology, reducing the need for engineers that specialize in integration and understanding how building systems work.
Iâm a skeptic on this. Parts of this process are messy, not able to be automated, and still require a ton of integration work by people who know building systems. Â The power of ML to make it easier is unproven and unquantified. As always, Iâm happy and eager to be proven wrong here. But if Iâm right, IDL companies will be limited in growth by how many integration engineers they can hire away from others.
While the IDL gives you choice at the application layer, one could argue that, depending on the role itâs playing and its capabilities, it still creates significant vendor lock-in. In other words, youâre reducing vendor lock-in at one layer and severely increasing it at the data layer.2
This is the double-edged sword of this layer: approaching this as a complete product and filling the gap makes you the best of the best, BUT it also makes it harder to replace you. As weâll get into below, interoperability standards could help with this.
There are at least a few software application providers that are skeptical of whether IDL vendors can fulfill the value proposition at their heart of their offering. The main concern is whether the IDLâs processes and data model are robust enough to support the use cases sitting on top of it.
One example is with FDD. To do it right, you need a robust data model and you need to collect a lot of metadataâsequences of operations, design parameters, equipment sizes, etcâthat isnât necessary for other use cases. Itâs not as simple as collecting data and throwing some tags on it.
This hurdle will be an important one because IDL vendors need application providers to bring them in as their partner for that layer.
I can see it now: because an IDL can bring together silos and provide an API for it, someone will make the claim that our collective interoperability problem is solved by their solution. And while that might be halfway true, data infrastructure is a separate issue from the independent interoperability standard we know we need.
That said, an IDL vendor can play a significant role in advancing existing standards by (1) maintaining conformance to the standards as they progress and (2) updating the standards as they extend them.
The question is: what will force them to do that? I think we as a community and building owners need to force the issue here⌠because this solves the double-edged sword problem above. Interoperability standards are the only way to avoid significant vendor lock-in at this layer AND get the advantages of making it API-first.
An app store is a natural extension of the IDL but is total hype at this point.
On the recent podcast episode with Shaun Cooley, we unpacked the different types of stakeholders the IDL will impact.
It seems like the early adopters will be big companies with lots of real estate and data science teams that want to analyze the data in-house (E.g. Google). For them, itâs a no-brainer... as long as the API meets their needs.
Then there are the building owners without the internal staff to use or, frankly, simply grasp the need for this layer. They need someone to bring them a whole productâso the IDL vendor will need to partner with whoever puts the full solution together.
That brings us to other vendors. MSIs are perfect partners for IDL vendorsâif they can be convinced to transform their business model from one that sells projects and therefore benefits from not having an IDL. So are application providers that want to focus on developing software and not integrationâif they can be convinced to give up revenue and the IDL can do it better than they can internally.
My hope is that the IDL vendors create the tooling for these partners to be able to set up the IDL on their own. We canât afford to wait for the IDL vendors to do this as a centralized entity and we donât want one company to be the bottleneck.
Hopefully, youâve enjoyed my synthesis here. And hopefully, itâs obvious that almost none of these ideas are my own and Iâm grateful to our members and podcast guests for enlightening me on how this space is developing. That said, letâs try and bring the state of this layer into a few concluding bullets:
What are your thoughts and takeaways?
The Nexus Vendor Landscape has been updated with this new category. Check it out for the IDL companies I know about today. As usual, weâll keep it updated as things evolve and we get your feedback.
It may have come from him!
On the podcast, Andrew Rodgers talked about the value of VOLTTRON to avoid some of the potential handcuffs:
The value of being in an ecosystem where there are other people using that technology, um, where we have sort of this plausible, presentation to our customers that like, yes, we're a small company, but we're using this technology that is open and, you know, there are other providers for, so if you decide that, like we're not. Delivering you can fire us and you're not left holding a bag of some proprietary crap that you can't do anything with.
â
Therefore, we can also define it in terms of capabilities, some of which are required to fill the gap and all of which represent a wide range in todayâs vendor landscape.
While this layer started out very simple, you can see that beyond the range of capabilities, there are also a ton of nuances to explore and understand. Letâs do some exploration.
As with any layer of the smart building stack, the devil is in the details. From my vantage point, abstracting the details away with a few lines of code, as is the API-first promise, is still a few years and many unanswered questions away.
So letâs dive into the questions, rapid-fire style. When youâre done reading these, youâll understand the edges of the problem and my perspective on them. As usual, I make no guarantees on the right answers!
On one hand, you could say every building needs an IDL. But Iâd argue that a building owner really doesnât feel the pain of not having it unless they have a ton of buildings and/or theyâre pretty far along the smart building journey. And that limited and/or delayed pain could really limit the market for the IDL vendor.
Put simply, the business value of the IDL is exponentially greater when you have:
In other words, if Iâm just doing analytics on top of HVAC and metering, which honestly is the state of the art right now, how big of a deal is not having the IDL? Techy types like to say âthe future is here, itâs just not evenly distributedâ. That cliche applies here too. How long will the pain take to distribute?
Among Nexus Pro members, there seems to be a bit of a debate on whether the IDL should be just an abstraction tool for on-premise systems (capabilities 1 through 3 above) or whether it should also have the full suite of capabilities (1 through 9 above).
The former camp seems to think that the IDL should stay focused on integration with on-prem systems and leave everything else up to the application providers.
My take is similar to the one Brian Turner shared at the March gathering:
âItâs becoming less and less of a problem to get data out, especially with the new technologies and architectures implemented. And so the way we're thinking about things is less about how do I get data out of a 20 year old system that probably needs to be modernized anyway, and really focus on the newer technologies. And, and now that we have sensor clouds and all of these different streams of data, it's no longer coming from a BMS per se, it's coming from all over the place. It's now extremely important to understand how that data all relates. That's where the transformation comes in.â
I agree. I think this layer needs to go beyond in two ways:
I think this layer needs to be considered a software product in its own rightâthis isnât as simple as a data feed with context included. As Andrew said on the podcast:
What our product brings is the tools that allow us to deploy faster, collect data better and more efficiently. We monitor for changes in the network, monitor the hardware platforms we deploy, and maintain the health of those hardware platforms. We're really selling a service around reliable data acquisition. Volttron plays a central role in that, but it's really the suite of the monitoring, maintenance, and all that stuff that we're offering on top of Volttron that is really where our value comes from.
I think the companies that win this layer will build their product better than anyone else because of their singular focus.
As weâve discussed at length, itâs very confusing to go shopping for smart building software. Even if a shopper understands the value of this layer, I predict that API-second application providers will claim that they do it too. And they will probably be at least somewhat correct. After all, itâs already part of their stack.
My take is that it will take some time for the advantages of the API-first strategy to kick in. Itâs like a flywheel that takes a long time to start spinning, but once it does itâs nearly unstoppable.
But weâre not quite there yet.
Specifically, I think digital twin companies will lay claim to this layer because the value propositions have heavy overlap. And vice versa, too. I think IDL companies can and will begin to compete with API-second digital twin offerings. And let the buyer confusion continue.
Getting data from buildings is hard no matter how you do it. IDL vendors will claim to do it faster and better with modern machine learning (ML) technology, reducing the need for engineers that specialize in integration and understanding how building systems work.
Iâm a skeptic on this. Parts of this process are messy, not able to be automated, and still require a ton of integration work by people who know building systems. Â The power of ML to make it easier is unproven and unquantified. As always, Iâm happy and eager to be proven wrong here. But if Iâm right, IDL companies will be limited in growth by how many integration engineers they can hire away from others.
While the IDL gives you choice at the application layer, one could argue that, depending on the role itâs playing and its capabilities, it still creates significant vendor lock-in. In other words, youâre reducing vendor lock-in at one layer and severely increasing it at the data layer.2
This is the double-edged sword of this layer: approaching this as a complete product and filling the gap makes you the best of the best, BUT it also makes it harder to replace you. As weâll get into below, interoperability standards could help with this.
There are at least a few software application providers that are skeptical of whether IDL vendors can fulfill the value proposition at their heart of their offering. The main concern is whether the IDLâs processes and data model are robust enough to support the use cases sitting on top of it.
One example is with FDD. To do it right, you need a robust data model and you need to collect a lot of metadataâsequences of operations, design parameters, equipment sizes, etcâthat isnât necessary for other use cases. Itâs not as simple as collecting data and throwing some tags on it.
This hurdle will be an important one because IDL vendors need application providers to bring them in as their partner for that layer.
I can see it now: because an IDL can bring together silos and provide an API for it, someone will make the claim that our collective interoperability problem is solved by their solution. And while that might be halfway true, data infrastructure is a separate issue from the independent interoperability standard we know we need.
That said, an IDL vendor can play a significant role in advancing existing standards by (1) maintaining conformance to the standards as they progress and (2) updating the standards as they extend them.
The question is: what will force them to do that? I think we as a community and building owners need to force the issue here⌠because this solves the double-edged sword problem above. Interoperability standards are the only way to avoid significant vendor lock-in at this layer AND get the advantages of making it API-first.
An app store is a natural extension of the IDL but is total hype at this point.
On the recent podcast episode with Shaun Cooley, we unpacked the different types of stakeholders the IDL will impact.
It seems like the early adopters will be big companies with lots of real estate and data science teams that want to analyze the data in-house (E.g. Google). For them, itâs a no-brainer... as long as the API meets their needs.
Then there are the building owners without the internal staff to use or, frankly, simply grasp the need for this layer. They need someone to bring them a whole productâso the IDL vendor will need to partner with whoever puts the full solution together.
That brings us to other vendors. MSIs are perfect partners for IDL vendorsâif they can be convinced to transform their business model from one that sells projects and therefore benefits from not having an IDL. So are application providers that want to focus on developing software and not integrationâif they can be convinced to give up revenue and the IDL can do it better than they can internally.
My hope is that the IDL vendors create the tooling for these partners to be able to set up the IDL on their own. We canât afford to wait for the IDL vendors to do this as a centralized entity and we donât want one company to be the bottleneck.
Hopefully, youâve enjoyed my synthesis here. And hopefully, itâs obvious that almost none of these ideas are my own and Iâm grateful to our members and podcast guests for enlightening me on how this space is developing. That said, letâs try and bring the state of this layer into a few concluding bullets:
What are your thoughts and takeaways?
The Nexus Vendor Landscape has been updated with this new category. Check it out for the IDL companies I know about today. As usual, weâll keep it updated as things evolve and we get your feedback.
It may have come from him!
On the podcast, Andrew Rodgers talked about the value of VOLTTRON to avoid some of the potential handcuffs:
The value of being in an ecosystem where there are other people using that technology, um, where we have sort of this plausible, presentation to our customers that like, yes, we're a small company, but we're using this technology that is open and, you know, there are other providers for, so if you decide that, like we're not. Delivering you can fire us and you're not left holding a bag of some proprietary crap that you can't do anything with.
â
Therefore, we can also define it in terms of capabilities, some of which are required to fill the gap and all of which represent a wide range in todayâs vendor landscape.
While this layer started out very simple, you can see that beyond the range of capabilities, there are also a ton of nuances to explore and understand. Letâs do some exploration.
As with any layer of the smart building stack, the devil is in the details. From my vantage point, abstracting the details away with a few lines of code, as is the API-first promise, is still a few years and many unanswered questions away.
So letâs dive into the questions, rapid-fire style. When youâre done reading these, youâll understand the edges of the problem and my perspective on them. As usual, I make no guarantees on the right answers!
On one hand, you could say every building needs an IDL. But Iâd argue that a building owner really doesnât feel the pain of not having it unless they have a ton of buildings and/or theyâre pretty far along the smart building journey. And that limited and/or delayed pain could really limit the market for the IDL vendor.
Put simply, the business value of the IDL is exponentially greater when you have:
In other words, if Iâm just doing analytics on top of HVAC and metering, which honestly is the state of the art right now, how big of a deal is not having the IDL? Techy types like to say âthe future is here, itâs just not evenly distributedâ. That cliche applies here too. How long will the pain take to distribute?
Among Nexus Pro members, there seems to be a bit of a debate on whether the IDL should be just an abstraction tool for on-premise systems (capabilities 1 through 3 above) or whether it should also have the full suite of capabilities (1 through 9 above).
The former camp seems to think that the IDL should stay focused on integration with on-prem systems and leave everything else up to the application providers.
My take is similar to the one Brian Turner shared at the March gathering:
âItâs becoming less and less of a problem to get data out, especially with the new technologies and architectures implemented. And so the way we're thinking about things is less about how do I get data out of a 20 year old system that probably needs to be modernized anyway, and really focus on the newer technologies. And, and now that we have sensor clouds and all of these different streams of data, it's no longer coming from a BMS per se, it's coming from all over the place. It's now extremely important to understand how that data all relates. That's where the transformation comes in.â
I agree. I think this layer needs to go beyond in two ways:
I think this layer needs to be considered a software product in its own rightâthis isnât as simple as a data feed with context included. As Andrew said on the podcast:
What our product brings is the tools that allow us to deploy faster, collect data better and more efficiently. We monitor for changes in the network, monitor the hardware platforms we deploy, and maintain the health of those hardware platforms. We're really selling a service around reliable data acquisition. Volttron plays a central role in that, but it's really the suite of the monitoring, maintenance, and all that stuff that we're offering on top of Volttron that is really where our value comes from.
I think the companies that win this layer will build their product better than anyone else because of their singular focus.
As weâve discussed at length, itâs very confusing to go shopping for smart building software. Even if a shopper understands the value of this layer, I predict that API-second application providers will claim that they do it too. And they will probably be at least somewhat correct. After all, itâs already part of their stack.
My take is that it will take some time for the advantages of the API-first strategy to kick in. Itâs like a flywheel that takes a long time to start spinning, but once it does itâs nearly unstoppable.
But weâre not quite there yet.
Specifically, I think digital twin companies will lay claim to this layer because the value propositions have heavy overlap. And vice versa, too. I think IDL companies can and will begin to compete with API-second digital twin offerings. And let the buyer confusion continue.
Getting data from buildings is hard no matter how you do it. IDL vendors will claim to do it faster and better with modern machine learning (ML) technology, reducing the need for engineers that specialize in integration and understanding how building systems work.
Iâm a skeptic on this. Parts of this process are messy, not able to be automated, and still require a ton of integration work by people who know building systems. Â The power of ML to make it easier is unproven and unquantified. As always, Iâm happy and eager to be proven wrong here. But if Iâm right, IDL companies will be limited in growth by how many integration engineers they can hire away from others.
While the IDL gives you choice at the application layer, one could argue that, depending on the role itâs playing and its capabilities, it still creates significant vendor lock-in. In other words, youâre reducing vendor lock-in at one layer and severely increasing it at the data layer.2
This is the double-edged sword of this layer: approaching this as a complete product and filling the gap makes you the best of the best, BUT it also makes it harder to replace you. As weâll get into below, interoperability standards could help with this.
There are at least a few software application providers that are skeptical of whether IDL vendors can fulfill the value proposition at their heart of their offering. The main concern is whether the IDLâs processes and data model are robust enough to support the use cases sitting on top of it.
One example is with FDD. To do it right, you need a robust data model and you need to collect a lot of metadataâsequences of operations, design parameters, equipment sizes, etcâthat isnât necessary for other use cases. Itâs not as simple as collecting data and throwing some tags on it.
This hurdle will be an important one because IDL vendors need application providers to bring them in as their partner for that layer.
I can see it now: because an IDL can bring together silos and provide an API for it, someone will make the claim that our collective interoperability problem is solved by their solution. And while that might be halfway true, data infrastructure is a separate issue from the independent interoperability standard we know we need.
That said, an IDL vendor can play a significant role in advancing existing standards by (1) maintaining conformance to the standards as they progress and (2) updating the standards as they extend them.
The question is: what will force them to do that? I think we as a community and building owners need to force the issue here⌠because this solves the double-edged sword problem above. Interoperability standards are the only way to avoid significant vendor lock-in at this layer AND get the advantages of making it API-first.
An app store is a natural extension of the IDL but is total hype at this point.
On the recent podcast episode with Shaun Cooley, we unpacked the different types of stakeholders the IDL will impact.
It seems like the early adopters will be big companies with lots of real estate and data science teams that want to analyze the data in-house (E.g. Google). For them, itâs a no-brainer... as long as the API meets their needs.
Then there are the building owners without the internal staff to use or, frankly, simply grasp the need for this layer. They need someone to bring them a whole productâso the IDL vendor will need to partner with whoever puts the full solution together.
That brings us to other vendors. MSIs are perfect partners for IDL vendorsâif they can be convinced to transform their business model from one that sells projects and therefore benefits from not having an IDL. So are application providers that want to focus on developing software and not integrationâif they can be convinced to give up revenue and the IDL can do it better than they can internally.
My hope is that the IDL vendors create the tooling for these partners to be able to set up the IDL on their own. We canât afford to wait for the IDL vendors to do this as a centralized entity and we donât want one company to be the bottleneck.
Hopefully, youâve enjoyed my synthesis here. And hopefully, itâs obvious that almost none of these ideas are my own and Iâm grateful to our members and podcast guests for enlightening me on how this space is developing. That said, letâs try and bring the state of this layer into a few concluding bullets:
What are your thoughts and takeaways?
The Nexus Vendor Landscape has been updated with this new category. Check it out for the IDL companies I know about today. As usual, weâll keep it updated as things evolve and we get your feedback.
It may have come from him!
On the podcast, Andrew Rodgers talked about the value of VOLTTRON to avoid some of the potential handcuffs:
The value of being in an ecosystem where there are other people using that technology, um, where we have sort of this plausible, presentation to our customers that like, yes, we're a small company, but we're using this technology that is open and, you know, there are other providers for, so if you decide that, like we're not. Delivering you can fire us and you're not left holding a bag of some proprietary crap that you can't do anything with.
â
Hey friends,
As a follow-up to recent newsletters Where are the API-first companies? (last week) and The API-first Data Layer (today), this members-only deep dive unpacks this concept further by defining the independent data layer, unpacking the nuances and unanswered questions, and updating the Nexus Vendor Landscape with this new category. If you havenât read those two essays first, please do.
Enjoy and let me know what you think!
âMiddlewareââŚ
âData lakeââŚ
âData aggregation layerââŚ
âData infrastructure layerââŚ
âData transformation layerââŚ
âInfrastructure as a ServiceââŚ
We love to make up new terms and acronyms in our industry and the new wave of data layer terms are no different. Iâm not sure where the term âIndependent Data Layerâ (IDL) came from, but itâs my preference, folks, and Iâm sticking with it.
Terry Herr1 defines it as âan edge connectivity layer / middleware that is independent of the applications or application layerâ. As he recently recounted in the March Pro member gathering, the need for this layer arose with the advent of cloud-based energy analytics software. These software providers needed meter, sensor, and actuator data from the building.
Early offerings like Building Roboticsâ (Now Comfy) Trendr, Lynxspringâs BACnet data pump, Sierra Monitorâs IoT gateway, the open-source VOLTTRON did just that. But as weâll see, that original need has expanded beyond simply pumping BAS/meter data into the cloud.
A good way to explain where we are today is the gap Brian Turner talked about on the Nexus podcast:
On one side you have modern enterprise (cloud) technologies (data lakes, middleware, applications) and on the other you have on-premise systems. You fill the gap between them with a domain-specific, web-developer-grade API.
And each vendor in this category is going to approach filling that gap differently.
Head over to Nexus Connect and see whatâs new in the community. Donât forget to check out the latest member-only events.
Go to Nexus ConnectJoin Nexus Pro and get full access including invite-only member gatherings, access to the community chatroom Nexus Connect, networking opportunities, and deep dive essays.
Sign Up
Nexus Labs exists to move the smart buildings industry forward, without the fluff.
On Wednesday, we send out the renowned (and free) Nexus Newsletter.