2. Software Architecture and User Interfaces
App4SHM (Figure 1) is an application that interrogates the phone’s internal accelerometer to measure accelerations, estimate the natural frequencies, and compare them with a reference data set through a machine learning algorithm properly trained to detect damage in almost real time. By ‘almost real time’, we understand the time needed for data acquisition, feature extraction, and damage detection, typically taking less than two minutes altogether. App4SHM includes access to a server to run most of the computational operations and store some of the data sets (see Figure 2).
The application is composed of two software components: a mobile application and a computational platform hosted by the server. The mobile application is developed in Kotlin and is compatible with devices running Android 10 operating system (or higher). It implements a navigation mechanism with four tabs (represented by the green boxes in Figure 2), corresponding to the four steps in the damage detection process:Structure Identification (Step 1)—identification of the structure from a structural database. A new structure can be added via server;Data Acquisition (Step 2)—measurement of acceleration time series in the direction orthogonal to the screen of the smartphone (Figure 1) for a user-controlled period of time;Feature Extraction (Step 3)—estimation of the first three natural frequencies from the power spectral density of the accelerogram (using the Welch method). They are stored into a feature vector (observation);Damage Detection (Step 4)—for each new observation, a damage indicator is computed based on the Mahalanobis-squared distance (as applied in [24,25]). Subsequently, a plot of the damage indicators of the observations collected in the training database (and assumed to reflect the behavior of the undamaged structure) is presented, to which the damage indicator computed for the new observation is added in green or red if the structure is deemed undamaged or damaged, respectively.
The application is linked with a Python/Django server through REST webservices to get existing structures, perform calculations, and save the measured structural frequencies in a MySQL database (orange boxes in Figure 2). There is also a web interface (Figure 3), accessible with any browser, providing a backoffice (automatically generated by Django admin) which allows users to manage all the information stored in the database: structures, time series, power spectral densities, and natural frequencies. Using this interface, users with administrator’s privileges may access all monitoring information, including accelerograms, power spectral densities, and extracted frequencies. It is also possible to upload previously recorded observations, register new structures, and delete incorrect data sets, among other operations. This architecture has the following benefits:Multiple smartphones can be used to record acceleration time series for the same structure and centralize the information in a single database;For structures with existing training sets, it is possible to upload that information directly to the database and use it along data coming from the smartphone measurements to train the machine learning algorithms for damage detection;The web-based backoffice is accessible only to users with the right credentials, meaning that regular users cannot tamper with the information stored in the database;CPU-intensive calculations such as the Welch algorithm can be slow on certain mobile devices, so it is useful to offload those computations to the server;Facilitates the development of the application on other platforms (namely iOS) since the backoffice algorithms implemented on the server need not be changed.
2.2. User Interfaces
The user interfaces guiding the interaction between user and smartphone (Figure 4) are described below in more details.
2.2.1. Step 1—Structure Identification
This step starts by calling a webservice to get the identification of the structures for which saved frequencies already exist in the database (Figure 4a). This procedure allows the user to choose an existing structure (the amount of stored training observations is presented next to the name of each structure). It also defines the protocol for the collection of a new observation: either for adding it to the training database or for damage detection purposes. If the goal is training, the process ends with the feature extraction step and no damage indicator is computed. Conversely, under the damage detection protocol, a damage indicator is computed, but the observation is not stored in the training database. As mentioned above, the creation of a new structure is only possible via the server.
2.2.2. Step 2—Data Acquisition
In this step, App4SHM collects information from the built-in accelerometer of the smartphone with a sampling period of approximately 20 ms (50 Hz). As smartphones have imperfect sampling frequencies [16], it is not possible to guarantee a precise sampling rate. Therefore, an interpolation algorithm is implemented to correct deviations from the defined sampling frequency. As smartphones are usually laid down over the structure, only accelerometer data in the z axis (see Figure 1) is currently considered. Collected accelerations are drawn in a graph in real-time to assist the user during this step (Figure 4b). They are also locally stored in the smartphone in CSV format (a format that is recognized by most spreadsheet applications).
2.2.3. Step 3—Feature (Natural Frequencies) Extraction
The mobile application sends the acceleration data to the server, using a webservice, to extract frequencies. The frequency extraction process is comprised of two parts. First, a power spectral density analysis is done in the server using the Welch algorithm. For this purpose, the measurement interval is divided into a number of overlapping intervals. Both the number of intervals (default = 3) and the overlapping time (default = 50% of the interval) can be controlled programmatically. The power spectral density returned to the user is the average of the spectra for each of the intervals. Then, the power spectral density plot is sent back to the smartphone and shown in the mobile application (see Figure 4c). The spectral density information is stored on the smartphone. The user selects three frequencies which are then sent to the server to be saved in the database X (in case of training) or to serve as input to the damage detection algorithm. To assist the user with the selection, the average values of the training frequencies already stored on the server are presented. Unlike alternative applications like CS4SHM [20], App4SHM purposefully does not automatize the frequency extraction process from the power spectral density. This option is justified by our policy of keeping App4SHM a professional (rather than a crowdsourcing) tool. It is noted that, as opposed to CS4SHM, App4SHM has a damage detection module which may induce concern to untrained users, especially if the frequency data measured on the undamaged structure is not sufficient to be statistically relevant.
2.2.4. Step 4—Damage Detection
The damage detection step applies a Mahalanobis-based algorithm to detect abnormal observations by measuring the distance between the previously recorded frequencies and the current observation. This results in a set of damage indicators (DI), which is transmitted to and shown in the mobile application (Figure 4d). If the damage indicator is superior to a threshold defined for a level of significance of 5% of a chi-squared distribution, the application raises a red flag and outputs a warning message. Otherwise, App4SHM raises a green flag, as no damage indication is present. The 5% level of significance is a common choice in the literature (e.g., [26]), as it strikes a good balance between the risks of the algorithm yielding false positives and false negatives. However, the level of significance can be easily changed programmatically, if needed.
The choice of the Mahalanobis algorithm for damage detection is grounded in its simplicity and generalization capabilities. Its effectiveness is demonstrated in multiple papers (e.g., [25,27]) that apply it to data sets from several real-world bridges, including comparisons with alternative algorithms. The assumption of normally distributed data allows the definition of thresholds for given levels of significance. However, other algorithms can be added programmatically to App4SHM if desired, by substituting a single function on the supporting server.
As proposed by Figueiredo et al. [24,25] in the context of bridges, considering the training matrix X∈ℜn×m, where n is the number of observations and m is the number of variables, with multivariate mean vector, μ, and covariance matrix, Σ, the Mahalanobis squared distance (or DI in the context of this study), between feature vectors of the training matrix and any new feature vector (or observation) is defined as:
The assumption is that if a new observation z is obtained from data collected on the damaged system, the observation will be further away from the mean of the normal condition. On the other hand, if an observation is obtained from a system within its undamaged condition, even with operational and environmental variability, it will be closer to the mean of the normal condition.
Herein, for feature classification, it is assumed that if an observation z is extracted from an undamaged condition and it is multi-dimensional Gaussian distributed, then its DI will be Chi-square distributed with m degrees of freedom, DI~χm2. Therefore, multivariate outliers can now simply be defined as observations having DIs above a certain level of confidence.
CC BY 4.0. "Smartphone Application for Structural Health Monitoring of Bridges" by Figueiredo E, Moldovan I, Alves P et al., Sensors (Basel, Switzerland) (2022). https://doi.org/10.3390/s22218483. Licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Peer-reviewed and open access; full text from PubMed Central (PMC9655388). Headings and layout adapted for reading; figures and tables are in the original.