Ph.D. researchers and assistant professor, Tarek Rakha, flying a drone in the Hinman Courtyard.

Aerial Diagnostics

Aerial Diagnostics

The Aerial Diagnostics group focuses on the use of Unmanned Aerial Vehicles (UAVs) for analytical applications in the built environment. We utilize the unparalleled vantage points drones offer to explore novel ways of surveying buildings and diagnosing anomalies with a myriad of different drone-borne sensors. Research within the group goes beyond the practical application of drones and transcends into synthesizing computational solutions based on Machine Learning (ML) and Computer Vision (CV) techniques to automate and streamline surveying and diagnostics tasks and data gathering processes.

Using the HPBL’s drone fleet, varying in sizes and payloads, the group has utilized this technology in various capacities. We employ drones to inform Building Energy Models (BEM) through the development of a pipeline that begins with a drone survey of a building and ends with an accurate representation of anomalies on a BEM ready for simulation. The group has explored the integration of drones with ASHRAE Standard 211P for Commercial Audits and integration of various Nondestructive Testing (NDT) techniques in that process. As well as utilizing aerial photogrammetry techniques coupled with thermal imaging to produce 3D models.

Example output images together with their ground truth and mIoU and AIM scores.

A Capsule Network-based Semantic Segmentation Model

For Thermal Anomaly Identification on Building Envelopes

Thermography technology is widely used to inspect thermal anomalies in building façade systems. Computer vision-based techniques provide opportunities to autonomously detect such heat anomalies to significantly improve the efficiency of decision-making for building envelope retrofitting and maintenance. In this work, we propose a novel Capsule Network-based deep learning model – CapsLab – that detects and identifies thermal anomalies by semantic segmentation. CapsLab is built based on our proposed prediction-tuning capsule (PT-Capsule) layer. Different from a traditional capsule layer, which consists of part-whole transformation and capsule-routing process, the proposed layer is composed of a prediction and tuning process, which helps decreasing the number of model parameters significantly.

Door and window detection results

Building Envelope Object Detection Using YOLO Models

Building performance significantly influences energy use and indoor thermal conditions tied to the quality of living for its occupants. Therefore, information on building envelopes is essential, especially considering that envelopes and windows can impact 50% of energy loads in the United States. However, current retrofits supporting Building Energy Modelling (BEM) tools face multiple barriers, including time consumption and labor intensity due to manual modeling and calibration processes. This paper proposes using Deep Learning (DL) -based object detection algorithms to detect building envelope components, more specificlly doors, and windows, that can be applied to building energy performance analysis, 3D modeling, and assessment of thermal irregularities. We compare four different versions of the state-of-the-art YOLO V5 model to identify which version best suits the goal of detecting these building components. Results show that YOLO V5_X provides the best performance for detection accuracy.

3D thermography CAD model

Building Envelope Anomaly Characterization and Simulation

Using Drone Time-lapse Thermography

Defects in building envelopes deteriorate over time without being visible to the human eye, while significantly impacting energy performance due to unaccounted heat transfer. Defects can be characterized in the infrared (IR) spectrum. However, IR readings are typically recorded at singular points in time, when in several cases anomalies can only be revealed at specific times of the day, possibly in different seasons of the year. This paper presents a novel workflow for 3D envelope defect characterization and modeling using aerial time-lapse IR data collection using drones. A comprehensive envelope thermal profile is developed for a case study building employing the photogrammetry software Agisoft Photoscan, which generates temporal IR inspections of building skins using multiple thermography orthomosaics. Point-cloud data is then translated into a CAD model and thermal zones for whole Building Energy Modeling (BEM) using Honeybee as a frontend to EnergyPlus to showcase the potential of inclusion of detailed 4D data.

Energy model calibration examples

Envelope Modeling Calibration using Aerial Thermography

There is a need to develop performance diagnosis frameworks and approaches for accurate Building Energy Model (BEM) simulations to develop impactful retrofitting design solutions that could make existing buildings perform closer to current efficiency measures. This paper reports on research that focuses explicitly on calibrating envelopes of existing BEMs using drones equipped with thermography sensors. The study specifically focuses on the automation of on-the-fly envelope U-value estimations and verification of calibrated envelope BEMs. The paper examines a renovated campus building in Boston, MA., representing material degradation, thermal bridging, and insulation failures using thermal imaging. 

NDT categorization

Review of NDT for Building Diagnostic Inspections

This project studies the integration of drone-borne Non-Destructive Testing (NDT) Technique sensors with ASHRAE’s Standard 211P for Commercial Building Energy Audits. The main motivation of the project lies in the fact that buildings utilizing two thirds of the electricity generated in the United States and the advent of Climate Change as the main generational and existential challenge in the 21st century. It has has become imperative for professionals and researchers in the construction industry to drive down that consumption. One avenue to do so effectively has been energy audits in which a building’s consumption is quantified and its performance measured and understood. ASHRAE’s standard 211P presents a guideline for professionals to do so, and this project aimed to streamline the process by integrating drones-based analyses with the envelope scope of the standard. The project also lays the theoretical groundwork for integration of information gathered by the NDT Techniques with Building Energy Models (BEMs).

The 3D cloud point was reconstructed with its local coordinate system by using photogrammetry techniques to process UAV-images captured in a spiral flight path. The location (3D coordinates in the local coordinate system) of the RGB and IR images for close-range façade inspection were identified by the proposed image matching and transformation algorithms.

Façade Anomalies Localization,, and Registration

Aiming at locating the detected façade anomalies in IR and RGB images to a 3D building model, this project uses computer vision techniques to identify the 3D coordinates of UAV-captured close-range façade inspection images within the reconstructed 3D reference cloud points. 


Kaiwen Chen's headshot

Kaiwen Chen, Ph.D.

Post-Doctoral Alumni

Yasser El Masri's headshot

Yasser El Masri

Ph.D. Alumni

Eleanna Panagoulia's headshot

Eleanna Panagoulia

Ph.D. Alumni

Tarek Rakha's headshot

Tarek Rakha

Associate Professor


If you can't find the information you were looking for, we'll get you to the right place.
Contact Us