Daytime and nighttime aerosol optical depth implementation in CÆLIS

The University of Valladolid (UVa, Spain) has managed a calibration center of the AErosol RObotic NETwork (AERONET) since 2006. The CÆLIS software tool, developed by UVa, was created to manage the data generated by AERONET photometers for calibration, quality control and data processing purposes. This paper exploits the potential of this tool in order to obtain products like the aerosol optical depth (AOD) and Ångström exponent (AE), which are of high interest for atmospheric and climate studies, as well as to enhance the quality control of the instruments and data managed by CÆLIS. The AOD and cloud screening algorithms implemented in CÆLIS, both based on AERONET version 3, are described in detail. The obtained products are compared with the AERONET database. In general, the differences in daytime AOD between CÆLIS and AERONET are far below the expected uncertainty of the instrument, ranging in mean differences between $-\mathrm{1.3}\times {\mathrm{10}}^{-\mathrm{4}}$ at 870 nm and $\mathrm{6.2}\times {\mathrm{10}}^{-\mathrm{4}}$ at 380 nm. The standard deviations of the differences range from $\mathrm{2.8}\times {\mathrm{10}}^{-\mathrm{4}}$ at 675 nm to $\mathrm{8.1}\times {\mathrm{10}}^{-\mathrm{4}}$ at 340 nm. The AOD and AE at nighttime calculated by CÆLIS from Moon observations are also presented, showing good continuity between day and nighttime for different locations, aerosol loads and Moon phase angles. Regarding cloud screening, around 99.9 % of the observations classified as cloud-free by CÆLIS are also assumed cloud-free by AERONET; this percentage is similar for the cases considered cloud-contaminated by both databases. The obtained results point out the capability of CÆLIS as a processing system. The AOD algorithm provides the opportunity to use this tool with other instrument types and to retrieve other aerosol products in the future.

1 Introduction

Atmospheric aerosol particles contribute to climate forcing through their interactions with radiation and clouds, and its impact is still subject to large uncertainty (IPCC, 2014). Aerosol measurements are carried out worldwide in order to reduce these uncertainties using various techniques: active and passive remote sensing (from the ground and space) and in situ. Sun (and Moon) photometry is one of the most extended techniques for aerosol remote sensing; the main parameter provided by photometers is the aerosol optical depth (AOD), i.e., the extinction by aerosol particles in the entire atmospheric column. AOD is a proxy for the aerosol load in the atmosphere; its variation with wavelength, usually quantified by the Ångström exponent (AE), provides information about the size predominance of these particles (Angström, 1961).

Ground-based photometers use direct Sun (or Moon) spectral irradiance to derive AOD. It is calculated from these measurements using the Beer–Bouguer–Lambert law (Shaw, 1976). The AOD uncertainty depends on the photometer model, but it is usually small, about 0.01–0.02 in daytime. These measurements are therefore considered the “ground truth” for calibration and validation purposes.

Ground-based photometer networks provide long-term and near-real-time aerosol data that are used for aerosol property monitoring, satellite and model calibration and validation purposes, and synergy with other instruments. These are the objectives of the Aerosol Robotic Network (AERONET; Holben et al., 1998), the most extended photometer network, but similar objectives are also pursued by the GAW-PFR network (Kazadzis et al., 2018) and SKYNET (Takamura et al., 2004). The aerosol monitoring activity in the photometer networks relies on the standardization of instruments, calibration and processing (Holben et al., 1998; Wehrli, 2000). This is the case for AERONET, in which the standard instrument is the Cimel CE318 photometer. This is an automatic instrument that is able to perform direct Sun observations (and direct Moon in the latest version) and a number of sky radiance scans. Narrowband filters and two detectors (Si and InGaAs) allow spectral measurements in the range 340–1640 nm. The extinction measurements (Sun or Moon) are taken every 3–15 min and consist of three measurements per spectral channel collected within 1 min. These “triplets” are the basic measurement for evaluation of the instrument stability and the identification of cloud contamination.

The calibration needed for AOD evaluation is the extraterrestrial signal of the instrument, which is normally derived using the Langley plot technique (Shaw, 1983) for reference instruments or side-by-side comparison for field instruments. The reference instruments are calibrated at high-altitude stations like Mauna Loa and Izaña (Toledano et al., 2018). Field instruments are calibrated at intercalibration sites. In the AERONET network, calibration facilities at GSFC/NASA (Greenbelt, USA), PHOTONS/LOA (Lille, France) and GOA/UVa (Valladolid, Spain) are used for this activity. The instruments are routinely calibrated and maintained to ensure data quality. The facilities at Lille and Valladolid are also part of the Aerosol, Clouds and Trace Gases Research Infrastructure (ACTRIS, https://www.actris.eu, last access: 29 October 2020), a pan-European initiative to provide open and high-quality observations of those atmospheric constituents.

There is a need to evaluate the photometer data in real time and control large amounts of data generated by the network. Hence, in order to help in the management of the AERONET/ACTRIS calibration facility at Valladolid, a software tool called CÆLIS was developed (Fuertes et al., 2018). It provides tools for monitoring the instruments, processing the data in real time and offering the scientific community a new tool to work with the data. For this purpose, CÆLIS contains a database and a web interface to visualize raw data and metadata, provides processing of sky radiances, and supports the monitoring of the instrument performance. This tool is capable of detecting several technical problems with the network instruments through an automatic warning system based on the CÆLIS metadata and products, which allows a quick response to detect and solve operation problems.

In this framework, the calculation of the AOD is important because several checks can be applied to the data to ensure the reliability of the measurements. Moreover, CÆLIS also intends to be a framework to facilitate research activities, with the AOD being a key product in present and future investigations. Therefore, the main objective of this paper is to develop and describe the implementation of the aerosol optical depth and cloud screening algorithms in CÆLIS.

The AOD product must be robust and operational: for example, it must work for any site and instrument configuration, even with incomplete or damaged raw data files, which it should adequately flag if it is the case. CÆLIS is focused on AERONET and the Cimel photometer simply because that is the framework of our calibration activity. And this is actually the best reference point that we have in order to validate the AOD algorithm. Therefore, we will compare the results with those provided by the AERONET version 3 AOD algorithm (Giles et al., 2019), including the cloud screening, which is necessary because AOD can only be derived when the Sun or Moon is not obstructed by clouds.

We present the general framework for the AOD calculation (Sect. 2), and then the daytime (solar) and nighttime (lunar) algorithms are described in detail (Sects. 3 and 4). The latter includes a novel correction (Román et al., 2020) that considerably improves the quality of the lunar retrievals. The cloud screening is described in Sect. 5, and finally the algorithm results are compared with the AERONET database (Sect. 6).

2 General framework for AOD calculation

The calculation of aerosol optical depth in CÆLIS is intended to provide this parameter for a number of instruments, i.e., the photometers within the AERONET network that are calibrated at the University of Valladolid and routinely provide measurements to the CÆLIS system. They constitute an operational network with about 40 active sites that deliver data in near-real time. Therefore, the algorithm needs to be robust and work in a large variety of circumstances, such any site location, different instrument types, incomplete ancillary information and with defective input data.

The algorithm to calculate the aerosol optical depth is composed of two main parts. In the first part the algorithm searches in the database for all the meta-information about the photometer at the specific date and time, such as calibration coefficients, location and filters. In the second part, the raw measurement data and the meta-information are used to calculate the AOD.

Figure 1 Flux diagram of the retrieval of necessary data to be used by the AOD algorithm.

Figure 1 shows the process followed to generate the AOD, and as can be observed, it requires a lot of ancillary information. All that information is being stored in the CÆLIS database. Each photometer on any particular date and time is linked in one deployment (“installation”) with all that ancillary information. In this installation the information related to the beginning and ending dates when the photometer was deployed at one site, as well as the coordinates of the site, is stored. Once the photometer location is known, the next step is to know the specific instrument configuration. In this step all the information related to the instrument type¹, which could change from one deployment to another, is needed in addition to information about the interference filters of the photometer, e.g., the central and nominal wavelength of the spectral channels and the specific gaseous and water vapor absorption coefficients for those wavelengths.

Then, the algorithm reads the photometer calibration (extraterrestrial signal at mean Earth–Sun distance) and the temperature correction coefficients. This information is provided by the Valladolid calibration facility to the AERONET database and is therefore identical to that used in the AERONET version 3 products. The calibration is then adjusted to the Earth–Sun distance for each observation date and time.

The ancillary information needed for the processing is the local pressure and the column of absorbing gaseous species taken into account: ozone, nitrogen dioxide, carbon dioxide and methane. A detailed description of how this information is obtained by CÆLIS for the specific date, time and location is provided in Sect. 2.3. Three levels are established for the ancillary meteorological data: (1) meteorological data fields, (2) climatology and (3) standard atmosphere. This is also the hierarchy for the data usage. Therefore, a default value, as provided by a standard atmosphere model (for example, pressure), will only be used in the case that the meteorological data fields or climatology table are not available. This approach is intended to provide the necessary ancillary information in a consistent and operational way across the network, even if some sites could provide more accurate values with colocated measurements. For absorbing gaseous species, we use a monthly climatology (see Sect. 2.3.3). In the case that some stations do not have data for a certain month, a seasonal mean (or annual, if necessary) is used instead.

At this point the first main part of the algorithm flow is concluded. A series of flags have been filled in relation to the obtained meta-information, and the algorithm enters the second part, in which the raw data on direct irradiance are processed. The workflow of the computation is shown in Fig. 2.

Figure 2 Flux diagram of the AOD computation in CÆLIS.

As already mentioned, CÆLIS stores all the data generated by the photometers that are calibrated at our facility. Therefore, the AOD algorithm only needs to get the raw data from the correct table of the database and use them to calculate the AOD for each measurement. This procedure must be repeated a number of times, which depends on the instrument type. In digital and triple Cimel photometers equipped with 10 spectral channels, a total of 30 measurements are collected in each AOD observation. The three measurements per channel acquired over 1 min (triplets) constitute the basic AOD measurement for each wavelength. A temperature correction is applied to these raw data according to the internal temperature recorded at the sensor head (see Sect. 2.3.2 for details).

The total optical depth (TOD) can be computed using the Beer–Bouguer–Lambert law, as shown in Sect. 3, and then the contributions of molecules and gaseous absorption for each wavelength are subtracted from the TOD in order to obtain the AOD. The precipitable water vapor column (PWV; Sect. 3.2) is also derived from the photometer measurements using the 940 nm channel; this PWV value is used to further correct the AOD at 1020 and 1640 nm channels for (minor) water vapor absorption (Smirnov et al., 2004). The Ångström exponent is also calculated from the retrieved AOD values (Sect. 3.2). Finally, the obtained AOD values with three observations per wavelength will be screened for cloud contamination (Sect. 5).

2.1 CÆLIS database structure for AOD

CÆLIS is composed of a relational database, a processing module and a web interface (Fuertes et al., 2018). As indicated above, in this database we can find all the information required to compute the AOD. Thanks to the deployment records (installations) in the database, we can link, for a specific date, all the physical and logical information about each particular instrument and how it is (or was) configured. That means we can access the calibration coefficients of each spectral channel, the raw data and the filter specifications. All this information is stored in different tables of the database.

Similarly, after running the AOD algorithm, all the information generated will be stored in different tables of the database. Specifically, two tables are designed to store all the AOD information. One table stores the information that is common for all the spectral channels, including date and time, site, solar zenith angle, Earth–Sun distance, pressure, and algorithm version. All the information stored in the common table is used to calculate the AOD for each channel. It also stores the derived Ångström exponent and PWV values. The second table stores the specific information for each spectral channel. That means we can find in this table the exact central wavelength of the filter, the calibration coefficient and the temperature correction of the channel, the specific absorption coefficient for gases, and the calculated values for the various components (total, Rayleigh, gaseous absorption and aerosol optical depth). According to the CÆLIS database structure (Fuertes et al., 2018), the AOD is a level 1 product (“direct product”); therefore, some redundant information is included in these tables in order to facilitate data extraction by users.

2.2 Computing

Thanks to the processing chain of CÆLIS, the near-real-time provision of AOD can be achieved. Every raw data file that is received in CÆLIS activates a set of triggers. First of all, the AOD algorithm runs between the first and last measurement included in the data file. Once a first version has been calculated, the system checks whether the AOD has been generated using a pressure value obtained by meteorological analysis (Sect. 2.3.1) or if only standard pressure was available. If pressure from a meteorological analysis was not yet in the database, the system creates a new task (12 h later) to reprocess the data until analysis pressure data are available.

Each file received by the system activates the task to calculate the cloud screening. This task runs the cloud screening algorithm for the full day, between 00:00 and 23:59 local time, even if the file does not cover the entire day. Once the AOD and the cloud screening have been calculated, the AOD can be used for further calculations. For instance, a task is triggered to calculate a set of quality control flags, some of them using the calculated AOD as input.

2.3 Ancillary data

2.3.1 Global Data Assimilation System

NOAA's Air Resource Laboratory runs a series of meteorological analyses and reanalyses; one of these is the Global Data Assimilation System (GDAS; see https://www.ncdc.noaa.gov/data-access/, last access: 29 October 2020). The GDAS is run four times per day at 00:00, 06:00, 12:00 and 18:00 UTC. Model output is a grid with 1^∘ resolution (360–181^∘ latitude–longitude). This grid contains several meteorological fields at a set of pressure levels.

GDAS data are stored in CÆLIS every 6 h for the purpose of calculating the local pressure for every site in the network. The pressure at the site elevation is calculated from a set of standard geopotential heights and interpolated in time. When pressure from GDAS is not available, the algorithm uses a standard pressure calculated with the site elevation based on the US Standard Atmosphere. A flag indicates if the current AOD value is calculated with a standard pressure or using GDAS pressure.

Figure 3 (a) CÆLIS atmospheric pressure as a function of AERONET atmospheric pressure for different stations. (b) Frequency histogram of the atmospheric pressure differences between CÆLIS and AERONET databases for all stations.

This strategy to obtain pressure for all network sites is similar to the one followed by AERONET using NCEP/NCAR reanalysis data (Giles et al., 2019). Figure 3a presents the scatter plot between the pressure from CÆLIS and AERONET; the range of pressure values spans from 660 hPa at the Teide site (3570 m a.s.l.) up to 1030 hPa at sea level sites. More than 180 000 pressure values used for AOD observations are compared in this plot, showing a high correlation between the two databases. The differences between local pressure calculated by CÆLIS and AERONET are in general below 2 hPa, as shown in Fig. 3b, where we observe a mean difference of 0.07 hPa and standard deviation of about 1.1 hPa. Similar differences are found between GDAS pressure and actual pressure measurements (e.g., Abreu et al., 2012).

The use of local pressure data is expected in CÆLIS for the future and will simply add another layer on top of the abovementioned hierarchy.

2.3.2 Temperature correction

The Cimel photometers are not stabilized in temperature during operation. In turn, the sensor head is equipped with a temperature sensor that allows correcting the measured signals with respect to a reference temperature of 25 ^∘C. The correction is based on a laboratory characterization in a thermal chamber. Whenever a hardware element is changed in the photometer head (filter, detector, electronic card) a new thermal characterization is run for the instrument. The AERONET procedure for temperature characterization of the Cimel photometers is described in detail in Giles et al. (2019).

The information produced during these characterizations, i.e., the temperature correction coefficients for each wavelength above 400 nm, is stored in the corresponding table of the CÆLIS database. These are extracted by the AOD algorithm to correct raw signals according to the corresponding measurement temperature. The function to correct a signal with temperature is quadratic (i.e., two coefficients per channel). Whenever a characterization is not available for a particular instrument or channel, a default standard correction is applied, as produced by the AERONET analysis of historical filters, based on the filter manufacturer or type.

2.3.3 Climatology tables

The AOD algorithm needs to account for gaseous absorption at different wavelengths. Several gaseous species are taken into account: ozone, nitrogen dioxide, carbon dioxide and methane. The column amounts of CO₂ and CH₄ are considered constant, and a fixed value of optical depth scaled to local pressure is used to account for these absorptions in the 1640 nm channel (Giles et al., 2019). For O₃ and NO₂ CÆLIS uses climatology tables produced from satellite observations.

Figure 4 Climatology of NO₂ (Dobson units ×10³) for the month of August. Data obtained from OMI version 3 (OMNO2d gridded, level 3) data between 2005 and 2017.

These climatology tables are monthly averages assigned to the 15th day of each month. The column abundance on other days is obtained by temporal interpolation. The NO₂ climatology was obtained from OMI version 3 (OMNO2d gridded, level 3; Krotkov et al., 2017) data between 2005 and 2017. An example of global NO₂ for the month of August with this climatology can be seen in Fig. 4. For the O₃ climatology we use the multi-sensor reanalysis from GOME-2, OMI and SCIAMACHY sensors between 1978 and 2008 (van der A et al., 2010). An example, in this case the global values of O₃ for the month of May, can be seen in Fig. 5, where higher ozone values are observed in the Northern Hemisphere, as expected in spring.

The comparison between the climatology tables used in CÆLIS and AERONET for NO₂ and O₃ is shown in Fig. 6 by means of frequency distributions of the differences. For NO₂, the determination coefficient between CÆLIS and AERONET is high (R²=0.978), and the mean of all differences (−0.04 DU) highlights a small underestimation by the CÆLIS database of AERONET climatology values, with a standard deviation around 0.02 DU. In the case of O₃, the scatter plot indicates very good correlation (R²=0.995); the departure is typically within ± 5 DU, with a mean bias close to zero and a standard deviation of around 2.5 DU.

For calculation of the absorption optical depth of these species, the spectral absorption coefficients provided by Gueymard (1998) are applied, taking into account the spectral response functions of the individual filters.

3 Direct Sun algorithm

3.1 Aerosol optical depth

The basic equation for aerosol optical depth calculation is the Beer–Bouguer–Lambert law (Shaw, 1976; Cachorro et al., 1987). In practice, this equation is applied to the raw instrument signal at a given wavelength that is measured at ground level (V) and the signal that the photometer would have at the top of the atmosphere (V₀) (Eq. 1).

$$\begin{array}{}\text{(1)}& V\left(\mathit{\lambda }\right)=V_{\mathrm{0}}\left(\mathit{\lambda }\right)\cdot R^{-\mathrm{2}}\cdot e^{-\mathit{\tau }\left(\mathit{\lambda }\right)\cdot m}\end{array}$$

In this equation R is the Earth–Sun distance in astronomical units, m is the optical air mass that indicates the relation between extinction in the vertical column and that in the measurement (slant) path, thus related to the zenith angle of the target (Sun, Moon, star), and τ is the TOD. The aerosol optical depth can be then derived by subtracting the contribution to extinction by all other atmospheric components: scattering by molecules (Rayleigh scattering) and absorption by gases at a given wavelength.

The voltage signal (V) has a temperature correction following Eq. (2). C₁ and C₂ are the coefficients for the thermal characterization that are stored in the database, and T is the temperature given by the sensor head during the measurement.

$$\begin{array}{}\text{(2)}& V=V^{\prime }/\left(\mathrm{1}+C_{\mathrm{1}}(T-\mathrm{25})+C_{\mathrm{2}}(T-\mathrm{25}{)}^{\mathrm{2}}\right)\end{array}$$

An absolute calibration (given by V₀) is required for AOD retrieval. In order to obtain the top-of-atmosphere instrument signal, the Langley plot method can be applied (Shaw, 1983; Toledano et al., 2018) or the calibration can be transferred from a reference instrument by side-to-side comparison (Holben et al., 1998). This calibration is supposed to be constant over time except for the Earth–Sun distance variations. A linear interpolation between pre- and post-deployment calibration factors is applied.

Different air mass factors m are taken into account for the various species; the reason behind this is the different vertical distribution of the gases (O₃ is mainly stratospheric, CO₂ is uniformly mixed). Hence, Eq. (1) can be rewritten as

$$\begin{array}{}\text{(3)}& V\left(\mathit{\lambda }\right)=V_{\mathrm{0}}\left(\mathit{\lambda }\right)\cdot R^{-\mathrm{2}}\cdot e^{-\left[{\mathit{\tau }}_{\mathrm{a}}\right(\mathit{\lambda })\cdot m_{\mathrm{a}}+{\mathit{\tau }}_{\mathrm{R}}(\mathit{\lambda })\cdot m_{\mathrm{R}}+{\mathit{\tau }}_{\mathrm{g}}(\mathit{\lambda })\cdot m_{\mathrm{g}}]},\end{array}$$

where the “a” subscript stands for aerosol, “R” for Rayleigh and “g” for gases. Finally, the aerosol optical depth (τ_a) can be directly calculated from Eq. (3) by

$$\begin{array}{}\text{(4)}& {\mathit{\tau }}_{\mathrm{a}}\left(\mathit{\lambda }\right)=-{\displaystyle \frac{\mathrm{1}}{m_{\mathrm{a}}}}\cdot \left[\ln \left({\displaystyle \frac{V\left(\mathit{\lambda }\right)}{V_{\mathrm{0}}\left(\mathit{\lambda }\right)R^{-\mathrm{2}}}}\right)-{\mathit{\tau }}_{\mathrm{R}}\left(\mathit{\lambda }\right)\cdot m_{\mathrm{R}}-{\mathit{\tau }}_{\mathrm{g}}\left(\mathit{\lambda }\right)\cdot m_{\mathrm{g}}\right].\end{array}$$

The gaseous absorptions considered in the processing are O₃, NO₂, H₂O, CO₂ and CH₄. The air mass for molecular (Rayleigh) scattering m_R is taken from Kasten and Young (1989), whereas the Rayleigh optical depth is taken from the Bodhaine et al. (1999) formula and weighted with local pressure. The O₃ air mass is taken from Komhyr et al. (1989). For aerosol and NO₂, CÆLIS uses m_R and for water vapor (m_w) the formulation given by Kasten (1965). The CO₂ and CH₄ optical depths (1640 nm wavelength) are taken as fixed values of 0.0087 and 0.0047, respectively, corrected by local pressure (Giles et al., 2019). The solar zenith angle used in the air mass calculations is computed following Michalsky (1988).

Figure 5 Climatology of O₃ (Dobson units) for the month of May. Data obtained from the multi-sensor reanalysis from GOME-2, OMI and SCIAMACHY sensors between 1978 and 2008.

Figure 6 Frequency histogram of the differences between CÆLIS and AERONET databases for (a) NO₂ climatology and (b) O₃ climatology. Data in Dobson units (DU).

3.2 Ångström exponent and precipitable water vapor

Once the spectral AOD has been calculated, the precipitable water vapor and the Ångström exponent can be calculated. The AE is defined as the negative slope of a linear regression between the logarithm of AOD and the logarithm of the wavelength (in microns) in a defined spectral range. Two AEs are calculated: AE(440–870) for AOD between 440 and 870 nm and AE(380–500) for the range 380 to 500 nm. AE is expected to be different in the different spectral ranges, and it depends on the aerosol type (Eck et al., 1999; O'Neill et al., 2001; Vergaz et al., 2005).

The spectral channel that provides the optical depth in the 940nm water vapor absorption band is used to calculate the PWV. In this channel extinction is produced by aerosol and molecules as well as water vapor absorption. Therefore, CÆLIS first estimates the AOD at that wavelength as the extrapolation from AOD(870 nm) and AOD(675 nm) using the Ångström power law in that particular region. Then, CÆLIS follows the methodology described by Schmid et al. (1996), which requires specific characterization of the 940 nm filter function of the photometer. This is based on a series of radiative transfer simulations that provide a and b coefficients, unique for each filter, that are used to model water vapor transmittance T_w in the band for the photometer:

$$\begin{array}{}\text{(5)}& T_{\mathrm{w}}=\exp [-a(m_{\mathrm{w}}u{)}^b],\end{array}$$

where u is the water vapor abundance and m_w the corresponding air mass. Taking all this into account, u can be finally derived from the photometer signal in the 940 nm channel as

$$\begin{array}{}\text{(6)}& \begin{array}{rl}u& ={\displaystyle \frac{\mathrm{1}}{m_{\mathrm{w}}}}{\left({\displaystyle \frac{ln{T}_{\mathrm{w}}}{a}}\right)}^{\mathrm{1}/b}\\ & ={\displaystyle \frac{\mathrm{1}}{m_{\mathrm{w}}}}{\left({\displaystyle \frac{{\displaystyle \begin{array}{c}-\ln V\left(\mathrm{940}\right)+\ln V_{\mathrm{0}}\left(\mathrm{940}\right)\\ -{\mathit{\tau }}_{\mathrm{R}}\left(\mathrm{940}\right)\cdot m_{\mathrm{R}}-{\mathit{\tau }}_{\mathrm{a}}\left(\mathrm{940}\right)\cdot m_{\mathrm{a}}\end{array}}}{a}}\right)}^{\mathrm{1}/b}.\end{array}\end{array}$$

The calibration factor (extraterrestrial signal) for the 940 nm channel is also performed during the routine calibrations, together with the aerosol channels.

4 Direct Moon algorithm for AOD

The main difference between lunar and solar photometry is that the Moon reflects solar irradiance instead of emitting visible light by itself. This fact means that extraterrestrial lunar irradiance significantly changes, mainly with the Moon phase angle (MPA), even during one single night. Hence, accurate knowledge of the extraterrestrial lunar irradiance is needed for lunar photometry purposes. To this end, CÆLIS computes for each observation the extraterrestrial lunar irradiance at several wavelengths following the method of the RIMO model (ROLO Implementation for Moon Observation; Barreto et al., 2019), which is an implementation of the ROLO (RObotic Lunar Observatory) model (Kieffer and Stone, 2005), making use of the SPICE Toolkit (http://naif.jpl.nasa.gov/naif/toolkit.html, last access: 29 October 2020) (Acton, 1996; Acton et al., 2018). After that, these lunar irradiance values are multiplied by a correction factor proposed by Román et al. (2020), which depends on MPA and wavelength. Following the Beer–Bouguer–Lambert law, the AOD can be calculated as follows (Barreto et al., 2013):

$$\begin{array}{}\text{(7)}& {\mathit{\tau }}_{\mathrm{a}}\left(\mathit{\lambda }\right)={\displaystyle \frac{{\displaystyle \begin{array}{c}\ln \left[\mathit{\kappa }\left(\mathit{\lambda }\right)\right]-ln\left[V\left(\mathit{\lambda }\right)/I_{\mathrm{0}}\left(\mathit{\lambda }\right)\right]\\ -{\mathit{\tau }}_{\mathrm{g}}\left(\mathit{\lambda }\right)\cdot m_{\mathrm{g}}-{\mathit{\tau }}_{\mathrm{R}}\left(\mathit{\lambda }\right)\cdot m_{\mathrm{R}}\end{array}}}{m_{\mathrm{a}}}},\end{array}$$

where κ is the calibration coefficient for an effective λ wavelength, I₀ is the corrected extraterrestrial lunar irradiance at the same effective wavelength, V is the photometer signal at the channel of the effective λ wavelength and (m) values are the optical air masses calculated by the Kasten formula (Kasten and Young, 1989) using the Moon zenith angle (MZA) as input.

The AOD can be calculated at nighttime using Eq. (7) if the calibration coefficient κ is known. In this work κ(λ) is calculated by the so-called gain calibration method (Barreto et al., 2016). This method consists of transferring the solar calibration to the lunar channels. The detectors are the same for Sun and Moon direct irradiance measurements in the Cimel; but, in order to reach a higher signal range, the Moon signal is electronically amplified by a gain factor, G, with a nominal value of 4096 (2¹²). Taking into account the fact that the only difference between Sun and Moon measurements is in this gain factor, the Sun calibration can be transferred to the Moon channels:

$$\begin{array}{}\text{(8)}& \mathit{\kappa }\left(\mathit{\lambda }\right)={\displaystyle \frac{V_{\mathrm{0}}\left(\mathit{\lambda }\right)}{E_{\mathrm{0}}\left(\mathit{\lambda }\right)}}\cdot G,\end{array}$$

where V₀(λ) is the Sun calibration coefficient and E₀(λ) the extraterrestrial solar irradiance (Wehrli, 1985), both at the λ wavelength. The gain calibration is simpler, it is not dependent on the RIMO (or other lunar irradiance model) and it only requires the calibration of the solar channels, which is routinely provided for AERONET instruments. Hence, CÆLIS calculates AOD at nighttime using the stored V₀ values and Eqs. (7) and (8). The UV channel of 340 nm is not considered due to the low Moon signal recorded by the photometer at these channels, which implies a low signal-to-noise ratio. More details about the correction applied to the RIMO values and the methodology of AOD calculation can be found in Román et al. (2020).

5 Cloud screening

Global photometer networks like AERONET run hundreds of sites equipped with automatic instruments that measure continuously. AOD retrieval requires that the Sun is not obstructed by clouds; therefore, an automated cloud screening algorithm is required to remove cloud-contaminated AOD data, which in general are higher, present higher time variability and show lower spectral dependence than aerosol data. Many algorithms have been published in the literature, in many cases closely tied to the instruments in particular, although many common principles are frequently used: the temporal variability at different timescales, either on the raw signals or the computed AOD, and the analysis of spectral variation (Harrison et al., 1994; Smirnov et al., 2000; Wehrli, 2008; Khatri and Takamura, 2009). Recently, AERONET improved the cloud screening algorithm with several significant changes, including the addition of aureole radiance checks for detection of thin cirrus clouds (Giles et al., 2019).

A cloud screening procedure is therefore needed in CÆLIS. Given the extensive tests with large data sets performed by Giles et al. (2019) and the improvements shown with respect to the previous algorithm, we have tried to reproduce this algorithm for Cimel photometers as a first step for CÆLIS.

The first step of the algorithm is to determine whether the Sun triplet collected is a valid measurement for AOD computation. In this sense, a minimum signal must be achieved in the measurement in order to guarantee that photometer is pointing to the Sun (or Moon), i.e., more than 100 counts in the infrared channels (870 and 1020 nm). In addition, if any raw signal is lower than the extraterrestrial signal (calibration factor) divided by 1500, which means total optical depth multiplied by an air mass of about 7, then the corresponding channel is rejected. Moreover, if the variability of the triplet signal (calculated as the root mean square over the mean) is larger than 16 % in any channel, then the full observation is rejected.

The observations that qualify for AOD computation are then checked for AOD variability. Initially all observations are considered “cloud-free”. They will be flagged as cloudy if the triplet variability (maximum – minimum AOD) is larger than 0.01 (or 0.015⋅τ_a, whichever is greater) for 675, 870 and 1020 nm channels simultaneously. If all three channels exceed this threshold, then the measurement is labeled a “large triplet”. From this point on, a number of checks are done by the algorithm that can result in the remaining cloud-free triplets being flagged as cloudy. The label will indicate which check was activated. The first checks are related to quality control.

• If the air mass is larger than 7, then we apply the label airmass_range.

• We check that the Ångström exponent is within the interval [−1, 4]. Otherwise the data are not realistic and we apply the label Angstrom_range.

Then the set of measurement points within a local day (sunrise to sunset) are analyzed together. Whenever new data are received within a certain day, this part of the algorithm will run for all the data available for that day.

• All cloud-free observations of an entire day are labeled potential_measurements when the number of remaining cloud-free observations is fewer than three in the day, or 10 % of the potential measurements attempted by the photometer in that day.

• The temporal variability of AOD at 500 nm is calculated for each pair of consecutive remaining cloud-free observations; the observation with the largest measurement in the pair is assumed to be cloud-contaminated and labeled smoothness_criterion if the difference is larger than 0.01 per minute. This process is iterative and continues until no further data are classified as cloud-contaminated by this criterion or the number of data is less than 3 % or 10 % of the potential measurements, as indicated above.

• The curvature check for aureole radiance is then performed, as described by Giles et al. (2019). This is a novel approach that takes advantage of the Cimel photometer to measure solar aureole radiances, and it is intended to detect thin cirrus clouds. For this purpose, the curvature of the aureole radiance (1020 nm) vs. scattering angle is analyzed. If this flag is activated, then the triplet and all other triplets within 30 min (or within 2 min for Cimel CE318-T instruments) are flagged curvature_check.

• If an observation is distant by more than 1 h from any other cloud-free measurement and it presents an AE(440–870) value below 1.0, then this point is flagged as stand_alone.

• In the case that the standard deviation (σ) of AOD at 500 nm of the cloud-free remaining points in the day is larger than 0.015, then the observations that exceed mean ±3σ in AOD or AE are labeled 3-sigma.

A final step is done to recover observations with high spectral dependence in the case that AOD (870 nm) is larger than 0.5 and the Ångström exponent (675–1020 nm) is larger than 1.2. This prevents the removal of very high aerosol loading cases (occasionally with high temporal variability) due to biomass burning smoke and urban pollution (Giles et al., 2019; Smirnov et al., 2000). The label applied in this case is “restoration” and it is equivalent to cloud-free, although very few data in our subset fulfill this condition.

All flags mentioned above result in the measurement point not being considered cloud-free and allow us to identify in the database the reason for the rejection. Thus, we can query the database for cloud-free data at a certain site and during a certain period, but we can also analyze the cloud-screened data and discriminate for any specific check.

The full scheme as it has been described is applied to solar AOD data. For lunar observations, we have maintained the same analysis and thresholds, except for the aureole radiance check that cannot be performed at night. Further testing is needed to possibly refine the cloud screening algorithm for nighttime.

6 Validation of the AOD algorithm

The photometer data that CÆLIS is currently processing for AOD are all produced by Cimel photometers belonging to AERONET. CÆLIS uses the same raw data and calibration. Moreover, AERONET is a global reference for AOD monitoring and its data are widely used by the scientific community dealing with aerosol, satellite validation and models. Therefore, the most logical approach for validation of the CÆLIS AOD implementation is to compare it with the one produced by the AERONET version 3 direct Sun algorithm (Giles et al., 2019). This comparison is provided in Sect. 6.2. The performance of the cloud screening algorithm for this daytime AOD is given in Sect. 6.4. As for the nighttime (lunar) algorithm, Sect. 6.3 includes an analysis of the performance at several sites and Moon phases, but it is not compared to the AERONET processing because the lunar-derived AOD in AERONET is still marked as a provisional product.

Table 1 List of Sun photometers used during the validation study. The analyzed period for all sites spans from 01 January 2016 to 31 December 2017.

6.1 Data set for validation

In Table 1 we summarize the data set that has been selected for AOD validation. It comprises 2 years of data for nine sites, with about 180 000 AOD observations (triplets) collected with Cimel photometers.

The site list includes two high-mountain observatories used for Langley plot calibration of the reference instruments: Izaña and Teide (Toledano et al., 2018). The AOD is very low in these locations; therefore, they are very suitable for a detailed comparison. We have also included a rural continental site (Palencia), our calibration site at Valladolid (small city and continental climate), an urban site (Munich), a coastal site (El Arenosillo), a Caribbean site (Camagüey), and the Arctic sites Andenes and Ny-Ålesund. Thus, we have tried to cover several aerosol types and ambient conditions in order to test the robustness of the algorithm.

Another important aspect of the subset is the variety of Cimel photometer types. We have covered all generations of Cimel instruments (analog, digital and triple) and multiple versions (see Table 1). This feature involved considerable work to ensure a flexible enough algorithm and adequate database construction so that all data can be consistently processed. In turn, we expect this experience will be of help in the addition of new photometer types to CÆLIS.

Thus, we have analyzed a large amount of data to have statistical strength in the comparison and cover multiple situations. The data set is used for validation of the daytime (solar) algorithm. This data set will also be used for cloud screening comparison (Sect. 6.4), in which a variety of climate conditions is also crucial.

6.2 Daytime AOD validation

The AOD obtained with the direct Sun algorithm has been compared for the abovementioned set of Cimel data. Identical raw data, calibration coefficients and temperature correction factors are used; therefore, the differences can only be attributed to the algorithm and the ancillary data sources.

The criterion for AOD comparison between two instruments recommended by the World Meteorological Organization is the so-called U95 threshold (WMO, 2005), defined as

$$\begin{array}{}\text{(9)}& \mathrm{U}\mathrm{95}=\pm (\mathrm{0.005}+\mathrm{0.010}/m),\end{array}$$

where m is the air mass. As can be seen in the following analysis, the boundaries of U95 are in general too large for our case, in which we compare algorithms rather than instruments. But it is a good reference as a starting point because it is commonly used in this kind of study (e.g., Cuevas et al., 2019).

.

Figure 7 Differences in AOD (AERONET–CÆLIS) as a function of air mass for several channels. The red lines indicate the maximum and minimum of the differences. The orange lines indicate the boundaries of the U95 criterion of the WMO (2005)

.

Figure 8 (a) Day and nighttime AOD retrievals at different sites and Moon phases. (b) Same for the Ångström exponent (440–870 nm). The black line indicates the Moon phase angle (MPA, right axis).

The AOD comparison for the different wavelengths is shown in Fig. 7. The differences are computed as CÆLIS–AERONET and they are plotted as a function of air mass. The U95 boundary is also depicted, as is the maximum and minimum difference for each channel. The largest differences are observed for the 340 nm and the smallest for the 870 nm channel, with $\mathrm{5.1}\pm \mathrm{8.2}\times {\mathrm{10}}^{-\mathrm{4}}$ and $-\mathrm{1.3}\pm \mathrm{3.4}\times {\mathrm{10}}^{-\mathrm{4}}$, respectively. This result is expected because no gaseous corrections are needed in the 870 nm channel; therefore, the differences can only be caused by different values of the solar zenith angle and the derived air mass. This is an important result because it indicates that the ancillary data play a key role in the different processing schemes.

The AOD differences are somewhat site-dependent. Apart from the site coordinates (mainly latitude) that condition the minimum air mass values available for each site, the main relevant difference among sites is the elevation, which affects both the Rayleigh calculations with Bodhaine's formula and the correction by local pressure. The Rayleigh optical depth is larger at shorter wavelengths, and the analysis of this component indicates that it is mainly responsible for the AOD differences for all channels between 340 and 870 nm. The differences in Rayleigh optical depth and AOD clearly decrease for increasing wavelength until 870 nm. The differences in pressure for the investigated observations are shown in Fig. 3b. The mean difference is close to zero, and the standard deviation is 1 hPa. This is noticeable in short wavelengths: at 340 nm the Rayleigh optical depth is about 0.70, and 1–2 hPa would mean 0.0007 to 0.0015 optical depth. This fact accounts for half of the discrepancy. The rest can be attributed to the gaseous corrections in this channel (O₃ and NO₂).

We also noticed an increase in the AOD discrepancy for longer wavelengths (1020 and especially 1640 nm). In this case the Rayleigh correction is minor; therefore, we investigated which elements are causing this. As for 1020 nm, the water vapor absorption correction is the reason for the slightly worse agreement of the 1020 nm channel. The discrepancy is higher for the 1640 nm wavelength. The gaseous corrections are in this case responsible for the AOD differences, i.e., the water vapor absorption and the CO₂ and CH₄ absorption, which are also affected by the differences in pressure.

Overall, the mean of these differences ranges from $-\mathrm{1.3}\times {\mathrm{10}}^{-\mathrm{4}}$ at 870 nm to $\mathrm{6.2}\times {\mathrm{10}}^{-\mathrm{4}}$ at 380 nm. The standard deviation of the differences ranges from $\mathrm{2.8}\times {\mathrm{10}}^{-\mathrm{4}}$ at 675 nm to $\mathrm{8.1}\times {\mathrm{10}}^{-\mathrm{4}}$ at 340 nm. The largest discrepancies are related to the Rayleigh correction (including pressure) and the gaseous absorption corrections. The U95 criterion is fulfilled in any case, and most of the spectral AOD observations agree within 0.0015 on AOD, which is 1 order of magnitude lower that the nominal AOD uncertainty (0.01–0.02) for AERONET field instruments.

6.3 Nighttime AOD evaluation

The lunar-derived aerosol optical depth has been developed in recent years following the publication of the ROLO model (Kieffer and Stone, 2005) and the appearance of commercially available lunar photometers (Berkoff et al., 2011; Barreto et al., 2013). It is still a provisional product in AERONET. In this paper we have presented the CÆLIS implementation of the latest improvements in lunar photometry that aims at providing good continuity between solar- and lunar-derived AOD observations. As has been shown in previous works (Barreto et al., 2017, 2019), it is important to assume that nighttime AOD uncertainty is larger that the uncertainty of daytime retrievals and that it will also depend on Moon phase angle. These facts also pose additional difficulty for cloud screening, apart from the lack of an aureole radiance check to detect thin clouds.

The CÆLIS nighttime AOD retrievals at several sites and Moon phase angles have been computed, showing continuity with daytime retrievals. We have intentionally selected cases with low AOD in general because absolute errors are easier to detect in low AOD scenes. Moreover, we avoided using the Izaña site for this comparison because lunar measurements at Izaña were used to elaborate the correction proposed to improve AOD (Román et al., 2020).

The results for both the AOD and the AE are shown in Fig. 8. The upper part (Andenes site) corresponds to a first-quarter case (MPA about −80^∘). The middle part (Valladolid) is a full Moon case, in which we can see the step from negative to positive MPA during the full Moon. The lowermost part (Granada) is a third-quarter case. The other two parts (for Teide and El Arenosillo) are cases with intermediate (negative and positive) phase angles. Overall, the day–night continuity in AOD is excellent (less than 0.02 in all channels), especially if we bear in mind the AOD natural variability and the nominal uncertainty for daytime AOD of 0.01–0.02, which is larger for shorter wavelengths (Holben et al., 1998; Giles et al., 2019). The nighttime AOD has no dependence on Moon zenith angle and the AOD wavelength dependence (typical decrease with wavelength) is basically maintained.

The AE has also been included here because this parameter is very sensitive to AOD errors, especially for low AOD (Cachorro et al., 2008). The good continuity and absence of dependence on zenith angle (Sun or Moon) for the AE are reliable indicators of data quality. The continuity of this parameter is also excellent (about 0.1 or less in absolute terms) for all cases except maybe Teide because of the extremely low AOD that amplifies the differences in AE.

Note how instrumental noise is visible in Teide data, wherein AOD is extremely low, whereas for a similar phase angle at El Arenosillo, with higher AOD, such noise is not visible. The plot for El Arenosillo is the only case in which AOD at 500 nm is above 0.1. This can be the case for many field sites worldwide, and the agreement among spectral channels and with respect to daytime AOD is a clear indicator that the CÆLIS Moon-derived AOD retrieval and the associated correction (Román et al., 2020) perform as expected.

6.4 Cloud screening validation

In this section we have compared the cloud screening performance for daytime AOD data only. The AERONET level 1.5 (cloud-screened) data are used for this analysis. The procedure is very straightforward: we analyze the data with a confusion matrix in order to determine which data assumed to be cloud-free by the AERONET cloud screening algorithm are also flagged as cloud-free by CÆLIS and vice versa. The other two possibilities, i.e., that one algorithm indicates cloud but not the other one, represent the discrepancy between the two procedures.

The confusion matrix C is such that C_i,j is equal to the number of observations known to be in group i but predicted to be in group j (Pedregosa et al., 2011). Thus, in binary classification, the count of true negatives is C_0,0, false negatives is C_1,0, true positives is C_1,1 and false positives is C_0,1.

Note that level 1.0 (unscreened data) in AERONET does not include all the measurements attempted by the photometers because many observations with raw signals that are too low or too variable do not qualify for AOD level 1.0 computation (Giles et al., 2019). For those data that passed this first requirement, the AERONET cloud screening will select the cloud-free cases and include them in the level 1.5 database². According to the flagging system of our cloud screening algorithm in CÆLIS, we have compared the cloud-free or restoration flags with the AERONET level 1.5 database for all the investigated sites and time periods (Table 1).

The cloud screening comparator links all the photometer observations (full triplet) from CÆLIS with their correspondents in AERONET and stores the output in two different arrays, one for CÆLIS and another one for AERONET. The value of 1 will be stored in each database if the observation is cloud-free and 0 if it is not cloud-free. With those two arrays the confusion matrix has been generated and it can be seen in Fig. 9, where we indicate the number of observations and the corresponding relative numbers in percent. In the confusion matrix the first row represents all the values that are cloud-free in AERONET, and the second row is for the cloud-contaminated data in AERONET. In the same way, the first column represents the cloud-contaminated data in CÆLIS, and the second column includes the cloud-free data in CÆLIS.

Figure 9 Confusion matrix for comparison of the cloud screening performed by AERONET and CÆLIS. Absolute number of cases and relative values (in percent) are given.

More than 250 000 observations have been analyzed here. The results are clearly satisfactory, with more than 99.8 % agreement in the classification. The number of points outside the main diagonal of the confusion matrix is marginal. An in-depth study of these few discrepancies points out that the differences appear in cases in which minor differences in AOD and AE caused a certain threshold to be exceeded or not (triplet variability, daily standard deviation, etc.). Occasionally this also triggered other cloud screening actions, like the potential measurement criteria or 3σ threshold. We are therefore very confident that the cloud screening in CÆLIS successfully reproduces the performance of the AERONET version 3 cloud screening.

7 Conclusions

The CÆLIS software tool was primarily designed to assist in the management of the calibration facility for Cimel photometers at the University of Valladolid, associated with AERONET. It provides access to metadata information to users and intends to facilitate the daily operation of the photometers on-site, with the final aim of improving data quality. CÆLIS already provides to users processing of sky radiances and a set of flags to monitor the instrument performance in real time. The AOD product now complements this tool. Moreover, the AOD is needed for exploiting remote sensing data with the application of inversion algorithms, like GRASP (Dubovik et al., 2014; Torres et al., 2017). These are the reasons behind the development of an operational aerosol optical depth product and the necessary cloud screening algorithm.

The implemented AOD algorithm comprises a number of steps, following formulas and procedures that are well established in the literature. Comparison with the AERONET version 3 AOD product shows overall agreement better than 0.0015 optical depth (1 order of magnitude less than the nominal AOD uncertainty), with the bias and standard deviations being higher for the UV and 1640 nm channels. In the UV this is caused by different Rayleigh computation and gaseous correction, and it needs to be investigated further. Similarly, the discrepancy found for the 1640 nm channel (slightly higher than that of 1020 nm) is caused by differences in the gas absorption corrections. Such differences, even if they are low, can be significant in the case of high-altitude stations or polar sites. The AOD retrieved by CÆLIS from Moon observations has shown continuity between day and nighttime for different sites and even for low AOD values and Moon phase angles near the Moon quarters.

The cloud screening schemes in CÆLIS and AERONET agree in the identification of cloud-free and cloud-contaminated scenes in more than 99.8 % of the more than 250 000 investigated cases. For future investigations, we will need to include a site with predominant biomass burning aerosol since this aerosol type was found to be insufficiently represented in the subset of data used to compare the cloud screening algorithms.

This paper has shown the capability of CÆLIS to provide AOD values and products with a similar accuracy as AERONET. The architecture of CÆLIS is such that it can be applied to other instrument types or networks. The next planned step is to be able to assimilate and process photometers other than the Cimel. In this sense, we will be able to apply a common processing to data originating from different photometer types, each one with its own spectral channels and measurement sequence, for example. Note that AOD retrieval and cloud screening algorithms differ for the different existing networks (AERONET, GAW-PFR, SKYNET). The modular approach has proven to be successful in adding several choices to the data processing or assimilating a variety of ancillary data. This will also help incorporate into the system any future improvements such as new gas absorption coefficients and the extraterrestrial spectral irradiance of the Sun and Moon. The flagging of data allows extracting in a powerful way a subset of data according to the desired criteria.

Some of the steps in the cloud screening procedure are actually quality control flags. However, a full quality control of the AOD product is not implemented yet in CÆLIS and will need to be developed. This approach is especially important for a robust operation of the algorithm and possible near-real-time applications.