Verification and Validation of comparing two audio files Algorithm and Procedure

Verification and Validation of comparing two audio files Algorithm and Procedure

Colorado State University

School of Biomedical and Electrical Engineering

Prepared by Student: Minh Anh Nguyen

Email: minhanhnguyen@q.com

Purpose

The purpose of this analysis is to determine if the method of comparing two audio files can be used for detecting of human errors for lower bearing and upper bearing issues.   There are many algorithms which can be used to compare two audio files.  These algorithms are: Discrete Fourier Transform (DFT), Short Time Fourier Transform (STFT), Wavelet, and Fast Fourier Transform (FFT). In this project, the Fast Fourier Transform algorithm is used to compare two audio files.

Why use the Fast Fourier Transform?

The fast Fourier transform (FFT) is an algorithm for converting a time-domain signal into a frequency-domain representation of the relative amplitude of different frequency regions in the signal.  The FFT is an implementation of the Fourier transform. The Fourier transform is one of the most useful mathematical tools for many fields of science and engineering.  The Fourier transform displays the frequency components within a time series of data.  The result of the FFT contains the frequency data and the complex transformed result.  The FFT works perfectly when analyzing exactly one cycle of a tone.

How the Fast Fourier Transform works

The FFT takes a chunk of time called a frame (number of samples) and considers that chunk to be a single period of a repeating waveform. Most sounds are locally stationary, which means that the sound does look like a regularly repeating function over a short period of time.

The FFT is the one-dimensional Fourier transform. Assuming a signal is saved as an array in the variable X, then preforming fft(X) will return the Fourier transform of X as a vector of the same size as X.  However, the values returned by fft(X) are frequencies.

Applications of the Fast Fourier Transform

There are many applications of FFT. The FFT algorithm tends to be better suited to analyzing digital audio recordings than for filtering or synthesizing sounds. The FFT is used to determine the frequency of a signal, to try to recognize different kinds of sounds, etc. 

Problems with the Fast Fourier Transform

The FFT function automatically places some restrictions on the time series to generate a meaningful, accurate frequency response. The FFT function uses (n/2) log₂ (n), therefore it requires that the length of the time series or total number of data points be precisely equal to a 2ⁿ. As a result, FFT calculations can only be used on a fixed length waveform such as 512 points, or 1024 points, or 2048 points, etc.  Another problem with using the FFT for processing sounds is that the digital recordings must be broken up into chunks of n samples, where n always has to be an integer power of 2. When the audio is broken up into chunks like this and processed with the FFT, the filtered result will have discontinuities which cause a clicking sound in the output at each chunk boundary. For example, if the recording has a sampling rate of 44,100 Hz, and the blocks have a size n = 1024, then there will be an audible click every 1024 /(44,100 Hz) = 0.0232 seconds, which is extremely annoying to say the least. The input signal must repeat periodically and the periodic length must be equal to the length of the actual input; otherwise leakage will occur and cause both the amplitude and position of a frequency measurement to be inaccurate.

Software or tools:

The Matlab tool (Matlab R2015b) is used to compare the two audio files.

Focusrite software, Scarlett 2i2 hardware interface, and CountrymanB3 microphone are used to record the signal

The techniques for comparing two audio files

Step 1: Load or read in two audio files into the MatLab tool. These audio files are recorded by using a Countryman b3 microphone and Scarlett 2i2 software.

Step2: Truncate both signals so that their time durations are equivalent. Line up two random signals.

Step 3:  Use a Fast Fourier Transform to convert the signals from time domain to frequency domain.

Step 4:  Graph the power spectrum or magnitude of signals at different frequencies. Identify the highest magnitude in the signal.

Step 5:  Subtract signal 2 from signal 1; if the result is greater than a threshold value then both signals are not similar, and an error message will display.

Flow chart for comparing two audio files

 

                                 Figure1.  Flow chart for comparing two audio files

Audio test in Simulation mode

The following steps are the procedure to perform audio test in simulation mode

1.      Make sure audio files are loaded or read into the MatLab without any issues.

2.      Plot and listen to audio files, make sure a correct file is loaded.

3.      Verify that the FFT value is correct.

4.      Verify that two random signals are aligned.

5.      Positive test: load two audio files which have 2 different signals and verify that the test results show the difference.

6.      Negative test: load two audio files which have 2 similar signals and verify that the test result show the same thing or match.

Set up microphone

The Scarlett 2i2 hardware interface provides the tool for connecting microphones and audio signals to a computer. The signals once loaded into the machine can be routed to the audio recording and playback software.

The microphone, Countryman B3, is installed inside the Spectra Optia machine to record a signal after each loading procedure of a complex disposable tube set onto a centrifuge by users.  One then compares this signal with a correct loading signal; and if one audio file shows the setup is not correct, then an error message will display to alert users to correct the problem before they continue. There are some advantages for using this microphone to improve this medical device:  First, the area of the microphone head is small; therefore, it would not require drilling any hole inside the machine. The microphone fit perfectly in one of the existing screw holes as shown in the figure below. Second, there is no need to worry about how cleanup may affect the sound of microphone. Third, the microphone is cost effective. 

Figure2. Prototype of the microphone inside the basin of the Spectra Optia machine

The module and systems within the centrifuge basin will have a clear plastic covering to enable the operator to see and clean all sides of the module easily. The module will also come with a provided template based off of the existing basin holes to assist the field technicians with an easy installation

Summary of Results

The following figure illustrates the difference between the displays of the FFT of the repeats periodically input signals and the results of the techniques and flow chart for comparing two audio files above. 

Figure3. Matlab results for comparing two audio files, which two audio files are correctly set and error message will not display. 

Figure4. Matlab results for comparing two audio files, which one audio file is not correctly set and an error message will display.

These results showed that the method of comparing two audio files can be used for detecting of human errors for lower bearing and upper bearing incorrect loading issues.   These results also proved that this is a reliable method for identifying loading problems. The FFT of a time domain signal takes the samples and calculates a new set of numbers representing the frequencies, amplitudes, and phases of the sine waves that make up the sound. 

Verification and Validation of Prototype Algorithm and Procedure for Detecting of Loading Errors via Audio File Comparison

Some of the basic steps for verification of audio detecting a misloading issue and for debugging a computer program are: use a flow diagram which includes each logically possible action as a procedure and follows the model logic for each action for each event type as shown in figure1 above.

Validation is the testing and evaluation of the whole prototype procedure or design method for the comparison algorithm at the completion of its development to ensure compliance with the project goals stated, which, in this project, are to detect misloading of the upper bearing, lower bearing and channel.  This is a process of comparing the Matlab codes to actual system behavior, and using the differences between the two to detect loading errors, and the insights gained to improve the methods. This process is repeated until the procedure to provide analytical results of output achieve acceptable accuracy and precision under a variety of conditions.  Accuracy is the degree of agreement or closeness of the test results obtained by the procedure to the true value.  Precision is the degree of agreement among individual results.  The complete procedure should be applied repeatedly to separate and identical audio signals recorded from the machine.  It should be measured by the scatter of individual results from the mean (good grouping or correct loading) and expressed as the relative standard deviation. The results from separate samples are influenced by changes in the operational or environmental conditions.

Testing methods and scripts which are used for validation of audio detection of a misloading issue are:  shake out testing, smoke testing, functional testing, black box or system testing, integration testing, unit testing, security or penetration testing, alpha testing, and user acceptance testing.  Shake out testing checks the network facility, the license to run MatLab, the database, connectivity and the integration module.  Smoke testing is the initial set of tests to check whether the major functionality works as expected. The functional testing is to check every functionality of software to ensure it works as required.  Integration testing is to check whether all software components are combined together and work as stated in the requirement document.  Security testing is to check to see how well the computer and systems are protected against unauthorized internal and external access.  In order to show if the algorithm meets the user expectations, a user acceptance test is used.  User acceptance testing is also a black box test, a demo, and the last step before the software and project are delivered and accepted by our customer, Terumo BCT.  This also guarantees customer satisfaction.

In this project, some of the steps and methods which were used for validation of audio detecting a misloading issue were:  data validation, accuracy, precision, robustness, specificity, and/or selectivity.  Data validation is to ensure that a program operates on correct and useful data.   This means that the program reads into Matlab tool correct input files.  This validation step is very crucial for both audio and image detecting a misloading issue.  If the wrong input files load into the program, then an incorrect result will display.  One of the validation methods is a range check; this method checks that the data is within a specified range of values such as a threshold value.  To validate the algorithm application accuracy and completeness as stated in the design document as a requirement of the project, a system testing method is used.  For testing via this method, a real life test environment and all real data must be used.  The actual test result must match the expected result as shown in the figures 5-6 below. 

Both signals are similar, so no error message displays; the result proves that the comparison algorithm meets the project expectations and produces a reliable method for identifying loading problem. 

Figure 5. Prototype validation result of detecting of correctly loaded bearings

Figure 6. Prototype validation result of detecting of incorrectly loaded bearings

Evaluation of Prototype of an Algorithm for Detecting of Loading Errors via Audio File Comparison

Evaluation of a prototype is a process of judging how well the prototype’s original intended goals have been achieved.  To evaluate and enhance prototype quality throughout the entire cycle of software development, verification and validation activities are conducted.  

Sensitivity and specificity are tests that can be used to distinguish whether a medical device has or does not have misloading issue. Sensitivity is the ability of a test to correctly identify an individual misloading procedure. Specificity is the ability of a test to correctly determine the successful or unsuccessful load of each procedure. The equations to calculate and results of both sensitivity and specificity for detecting loading errors are described and showed in table 1 and figure 8 below.  In this project, the total tests are 105; out of the 105 tests: 26 tests are loaded correctly and 79 tests are loaded incorrectly (misloading problem) in seven different misloading conditions or situations.

Sensitivity =	number of true positives
	total runs in which the centrifuge has the misloading issue

Sensitivity =                  number of true positives                              

                   Number of true positives + number of false negative

                =   probability of positive test given that the centrifuge has the misloading issue

Specificity =	number of true negatives
	total without the misloading issue

Specificity =                 number of true negatives                             

                   Number of true negative + number of false positive

                =   probability of negative test given that the centrifuge is correctly loaded

The positive predictive value (PPV )=	number of true positives
	total test positives

The negative predictive value (NPV) =	number of true negatives
	total test negatives

True positives = number of individual runs in which the centrifuge has the misloading issue and a positive screening test, expect and receive an error message.
False positives = number of individual runs without the misloading issue but which have a positive screening test, expect no message but receive an error message.
False negatives = number of individual runs in which the centrifuge has the misloading issue but have a negative screening test, expect an error message but receive no error message.
True negatives = number of individual runs without the misloading issue and a negative screening test, expect and receive no error message.

Table 12: Test perform on the Algorithm for Detection of Loading Errors via Audio File Comparison

		Audio detect loading errors
		Upper/Lower bearings loaded incorrectly	Both bearings loaded correctly	Predictive values
Centrifuge misloading test	+	True Positive (TP) 73	False positive (FP) 0	Total test positive TP +FP  = 73
	-	False Negative (FN) 6	True Negative (TN) 26	Total test negative FN +  TN = 33
		Total with bearings loaded incorrectly TP+FN = 79	Total with bearings loaded correctly FP+TN = 26	Total tests TP+FP+FN+TN = 105
		Sensitivity=TP/(TP+FN) =0.924= 93%	Specificity=TN/(TN+FP)      = 1= 100%	positive predictive value = TP/(TP+FP) = 100% negative predictive value= TN/(TN+FN) = .78 = 78%
		Accuracy = TP +TN/ Total tests = .9428 = 94%

 
Figure 7: Sensitivity versus Specificity of Algorithm for Detection of Loading Errors via Audio File Comparison

The numerical value of specificity above shows the probability of a given test to detect incorrectly loaded bearings without giving false-positive results. The specificity of the test is 100%; when conducting tests on a correctly loaded bearing, there is 100% chance that this loading will be identified as negative.  Specificity is the proportion of the true negatives correctly identified by testing. It suggests how good the test is at identifying correctly loaded bearings (normal) within the Spectra Optia machine. A positive result of an incorrectly loaded bearing test suggests the loading issue occurred.  A negative result certainly shows the possibility of the bearing being loaded correctly.  

The results above show that both sensitivity and specificity factors are equally important. Accuracy can be determined from sensitivity and specificity, and is the proportion of true positive results, in a population.  It measures the degree of reliability to detect a bearing misloading issue. This test is a good test because it has both high sensitivity and specificity.  This also meets the FDA requirement.  The FDA requires that developers and manufacturers of a new test provide target values for test results and provide evidence for the expected ranges, information on test limitations and other factors that could generate false results.  The goals of testing are to prevent critical damage to the whole system for users; taking into consideration potential failures of the program and its environments can do this.