Detecting the unknown - novelty detection of exceptional water reflectance spectra
Contents
Introduction
Synoptic monitoring of large areas in coastal waters can be performed by remote sensing using multispectral sensors on-board satellites. Many methods are in use which enable the detection and quantification of ‘standard algae’ or specific algae blooms using their known spectral response. The ‘novelty detection’ aims to find spectra outside the known range which are referred to as exceptional spectra.
This order given to the hero of a Russian fairy tale could be used to summarise the request received to detect exceptional water reflectance spectra. A recent review of the novelty detection techniques available is presented by (Markou and Singh, 2003)[1].
The present study aimed to develop a novelty detection scheme for water reflectance data which could be used operationally. The basic idea behind this approach is to use a large number of reflectance spectra from one complete year for a defined region, here the North Sea, so as to include the annual cycle of plankton species (including their blooms etc.) to teach a learning system ‘what is normal.’ Only distinctive deviations from these known spectra should be classified as novel. Thus, the algorithm should accept all reflectance spectra from the training period as being normal but should still be sensitive enough to detect novel situations.
The Algorithm
The selection of the appropriate algorithm is one central issue of novelty detection. It is intimately connected with the type and structure of the data to be tackled. The water reflectances are measured at nine different wavelengths, all of which will be used in the present algorithms. Bio-optical models often exclude several wavebands or give other wavelengths more weight. Examples of water reflectance spectra are shown in Fig. 1. Since our aim is to detect novel situations with unknown spectral signature we use all the information available. However, due to the resulting high dimensionality of the data set, nine dimensions of a point correspond to the nine wavelengths, the performance of different algorithms was carefully analysed. Different approaches of novelty detection, namely: a) a simple statistical scheme, b) an auto-associative Neural Net, c) a Neural Net classifier, and d) a tessellation scheme were analysed. Fig. 2 shows the enclosement of the different methods for a 2-dimensional case. The tessellation showed to be the optimal novelty detection scheme.
Scheme
The tessellation starts by choosing a radius R in the 9-dimensional space (dimensions for 9 wavelengths).
- The first point (spectrum) becomes the first centre.
- For each of the remaining points the smallest of the distances to all centres is compared with R. If the distance is larger than R the point is included in the set of centres.
- In the second step each point is assigned to that centre to which it is closest (Voronoi tessellation).
For each of the patches obtained by the tessellation the mean (centre of gravity) and the covariance matrix is calculated and finally for a given point (spectrum) the minimal Mahalanobis distance to all centres of the tessellation is determined. This minimal Mahalanobis distance to all centres of the tessellation is the main parameter to find new, unknown points i.e. spectra. A flowchart of the tessellation scheme is shown in Fig. 3. To minimize 'false alarms' and to optimise the discrimination power two additional conditions have to be fulfilled. The pixel must have a minimal distance to clouds of 3 pixels and only pixels within larger patches are considered.
North Sea MERIS Data
As an application for the tessellation novelty detection scheme we used MERIS data of the North Sea. The logarithms of the remote sensing reflectances in the first nine MERIS channels (centred at wavelengths λ = 413, 443, 490, 510, 560, 620, 665, 681, 708 nm) were used to span the space. For the dataset defining the ‘normal’ situation we used all MERIS scenes from June 26th 2003 to June 29th 2004 from a region of the North Sea (Fig. 4). We excluded pixels from the sun glint part as well as those which had one or more negative reflectances (bad atmospheric correction). Then ~ 1,000 pixels from each scene were randomly sampled thus ending up with 115,331 pixels for the ‘training’. We decided to declare points as representing novelty if their distance to the tessellation was above 7.5. The distance cut-off of 7.5 left only 38 pixels remaining from the training points. However, if we applied the cut to the 288 complete scenes, with 16,575,000 pixels of the time range from which the training data set was selected, we found 110,718 pixels. After applying a cloud distance cut (3 pixels) and a minimal patch size cut (19 out of 5 x 5) we reduced this number to 2,670 in 26 scenes. To remove remaining 'false alarms' in the training period we identified four groups of spectra for which we calculated the centre of gravity and the covariance matrix and appended them to the existing tessellation information. After this addition only one false alarm remained in the training set.
Finding Novelties
The test application data set comprised 223 scenes from June 30th 2004 to October 13th 2005. 10 scenes were initially marked as novelty. In the scene from November 19th 2004 novelty was signaled in the Skagerrak area, however, after further inspection this was shown to be simply a processing failure in that a significant number of water reflectances were just unity. Several scenes show 'novelties' (June 19th, 23rd, 26th, 28th and July 2nd, 18th 2005) and are located in the central and northwestern North Sea. These findings look like distinctive blooms of coccolithophores, which are in general quite frequent every year, but these spectra were classified as novelties due to their more pronounced blue reflectance as compared to those events within the training period. For further practical novelty detection, and for those not immediately interested in coccolithophore bloom alarms, one would append these spectra to the tessellation as outlined in the previous paragraph. Beside one false alarm three scenes with signaled novelties show a red tide near the island Helgoland on July 30th and on August 3rd and 5th 2004 (Fig. 5). This red tide was visible from satellite image (Fig. 6) and also at sea (Fig. 7) where observed during a measuring campaign between Cuxhaven and Helgoland on August 3rd 2004 and identified as Myrionecta rubra. Standard algorithms also detected an increase in the chlorophyll content.
Conclusion & Outlook
A novelty detection scheme which is suitable for regular monitoring purposes using multispectral remote sensing data has been constructed. It was applied to MERIS water reflectances and has a low failure (false positive) rate. It was successful in finding exceptional algae blooms, one of which was also identified as a red tide by in situ measurements. To reach this low failure rate it was necessary to use not only the spectral information but also to avoid pixels close to clouds and to require a minimum novelty patch size. The novelty detection scheme developed is flexible and adaptable to various monitoring needs. It is currently operating as standard service within InterRisk, an FP6-IST project addressing interoperable GMES services for environmental risk management in marine and coastal areas of Europe.
References
- ↑ Markou, M., & Singh, S. (2003). Novelty detection: A review – Part 1: Statistical approaches. Signal Processing, 83 (12), 2481–2497
Please note that others may also have edited the contents of this article.
|
Please note that others may also have edited the contents of this article.
|
Please note that others may also have edited the contents of this article.
|
