A comparison of gap-filling algorithms for  eddy covariance fluxes and their drivers

Mahabbati, Atbin; Beringer, Jason; Leopold, Matthias; McHugh, Ian; Cleverly, James; Isaac, Peter; Izady, Azizallah

doi:https://doi.org/10.5194/gi-10-123-2021

Articles | Volume 10, issue 1

https://doi.org/10.5194/gi-10-123-2021

© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/gi-10-123-2021

© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 10, issue 1

Research article

|

28 Jun 2021

Research article |

| 28 Jun 2021

A comparison of gap-filling algorithms for eddy covariance fluxes and their drivers

Atbin Mahabbati, Jason Beringer, Matthias Leopold, Ian McHugh, James Cleverly, Peter Isaac, and Azizallah Izady

Download

Final revised paper (published on 28 Jun 2021)
Supplement to the final revised paper
Preprint (discussion started on 07 Sep 2020)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC1: 'generalize beyond Australien processing chain setup', Thomas Wutzler, 02 Oct 2020
- AC1: 'The authors comments (AC) on the first referee comments (RC01)', Atbin Mahabbati, 10 Dec 2020
RC2: 'Consider refining focus of manuscript into either comparing the methods in more detail or their application to advancing EC gap-filling', Anonymous Referee #2, 20 Oct 2020
- AC2: 'The authors comments (AC) on the second referee's comments (RC02)', Atbin Mahabbati, 10 Dec 2020

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

AR by Atbin Mahabbati on behalf of the Authors (29 Jan 2021) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (02 Feb 2021) by Jean Dumoulin

RR by Thomas Wutzler (16 Feb 2021)

Suggestions for revision or reasons for rejection

General comments

The paper by Mahabbati et al. presents an updated comparison of gap-filling algorithm, which are an important tool in the analysis of data from eddy-covariance sensors and understanding the ecosystem functioning. Their methodology is oriented at the Australian version of the data processing chain taking into account information in addition to the eddy-stations from weather forecasting models and from BIOS2 model data integration environment. They corroborate previous findings of complex methods being not much better than simple methods for gap-filling of meteorological drivers. Contrary, for the carbon fluxes itself they find a better performance of the machine learning (ML) based approaches.

The current revision took into account two of my three major concerns (gaps vs. seasonality and adding MDS). However, their comparison of gap-filling still only uses a combination of drivers that is quite specific to the Australian setup (weather-forecasting model, and the Australian BIOS2 model-data-integration environment) and hard to transfer to other sites and setups. The inclusion of the ancillary datasets is (no doubt) valueable for the performance of the gapfilling. But including an additional comparison-scenario with a constraint set of drivers to my opinion would greatly help the transferability of conclusions on the choice of methods to other readers. Hence, I still encourage the authors to include such a scenario. Nevertheless, given that the usage of quite the specific set of drivers is made clear enough, the study is worth publishing without this additional scenario.

I congratulate the authors to achieve running all these various types of approaches in the same setup on the same dataset.

Specific comments

My concerns about “Australian setup” did not concern the selection of sites, but rather the selection of the set of inputs to the gap-filling which maybe not available at other sites.

To me, the current setup of “gap-filling” of environmental drivers reads more like a downscaling or interpolation/integration of various sources. The same variable from various sources is used as a driver for the prediction this variable.
The authors claim in their reply to my specific comments: “it is less likely that changing the input features makes any change in the comparative performance of the models.” I am not in a position to asses this claim.
However, this claim together with summarizing the specific drivers should be placed prominently in the discussion together with the citations given in their reply to my concern. From my previous report I repeat my suggestion of an additional scenario “using only data commonly available at eddy-covariance sites, which are the measurements themselves (Fc, Fh, Fe) together with ancillary measurements (Rg, VPD, rH, Tair, Tsoil, Ustar, precip, wind speed, and wind direction)”. Then you can also compare the very common case of using MDS of filling Tair.

Thanks for adopting the suggestion of the distribution of the consecutive gaps and the fitting to the entire data. Please, also state this also in the manuscript (section 2.3?). Currently, that way of training the model (with the data from 2012 and the data from 2013 excluding the artificial gaps, correct?), does not become clear in the current version of the manuscript.

Minor comments / Technical corrections

Tables: I found it hard to keep associating values within the same row. Please, consider adding some horizontal guiding lines.

Table 3: Please, link to the text where the data-sources are described and maybe provide a summary in the table caption.

I found it hard to always switch back to Table 2. Please, consider repeating the meaning of some acronyms at the relevant paragraphs, e.g. heading 3.1.1 “CO2 flux (FC)”. I would prefer some slightly longer acronyms, e.g. Tair, Tsoil compared to Ta and Ts.

In the version I got , some reference in parenthesis are missing, e.g. P15L339, or P16L458.

Hide

RR by Anonymous Reviewer #2 (25 Feb 2021)

Suggestions for revision or reasons for rejection

GENERAL COMMENTS

The two main changes in this version of the paper are the addition of the MDS method for gap-filling fluxes and a randomized selection of introduced gaps. These are both considerable improvements and make the results from the methods comparison more robust. However, the short records of data (only 1-2 years) was also a key concern from both reviewers and was not addressed in this revision. The paper, as it is, represents an interesting contribution showing methods were mostly comparable with the short records and the traditional MDS algorithm still performs reasonably well, and potentially the more complex methods might not lead to improvements that are worth the extra costs. However, these are not conclusive results from the paper. The paper could have been a key contribution to the literature, and although the contributions seem to be technically sound, they do not advance the state of the art in gap-filling of eddy covariance data.

SPECIFIC COMMENTS

In an answer to a reviewer comment, the authors state: "...since the main goal of the study was to compare different gap-filling algorithms, we do not believe changing the input data leads to a difference in the relative performance of the algorithm". For the comparison of these algorithms, the only factor changing their performance will be the input data. The input data is even more important for methods such as ANNs and RF, which are entirely dependent on relationships between the data variables.

Still in the answers, about ancillary datasets: "Even though this is true, the ancillary data used in the current study have been used to gap-fill the drivers’ data, and not the fluxes directly. As such, it might not be a concern." It might be good to clarify in the methods that only the measured values for the drivers were used to gap-fill fluxes. Although it is fair to assume no gap-filled driver data was used to fill the fluxes, I couldn't find this statement in the paper.

The argument that many previous research results use only single years for evaluation omits that most of these had limited access to long and uniform records. With record spanning over 20 years of data available from most regional flux networks, this is not a limitation any longer and should have been integral to the paper. Seasonal patterns can be correctly identified by many of the methods used, but only if using multi-year data. Using single year limited the results of the paper, which could have been a considerable contribution to both the eddy covariance and machine learning scientific communities.

In Moffat 2007 the RMSE values for the best performing algorithms (mainly ANN variants but also MDS) were consistently under 3.0 gC m-2 d-1. Since these were consistently higher in this manuscript, this might support the argument that there was too little data to train the runs presented in this paper. Since the year selected to perform the tests was very complete, if the short record is not an limitation, as argued by the authors, one could expect these results to be better.

The introduction of randomized gaps improves the soundness of the results. However, in the methods, it is a bit unclear how all the many realizations of the random gaps were aggregated for the final results. This could be explained in more detail. As an example, it is curious that the RMSE values for Fc at Alice Springs Mulga are so low, yet the R2 values for the site are also low, while for Tumbarumba, the RMSE values are more within the expected ranges while R2 values are also higher.

Finally, I will note that I disagree with the last recommendation in the conclusions. Ensembles are useful when there isn't a "true" value against which one can compare an estimation value. In gap-filling, artificially introducing gaps (original true values) for comparisons allow precise estimations of uncertainty. Using ensembles for gap-filling would introduce unnecessary uncertainty. However, playing to the strengths of each method one can procedurally combining them (e.g., one method for short and one for long gaps) to improve final results without mixed uncertainties.

TECHNICAL CORRECTIONS

- Net ecosystem exchange (NEE) is usually defined as the sum of CO2 turbulent fluxes (commonly represented as Fc) and CO2 storage fluxes (commonly represented as Sc); so the definition in the paper for Fc as equivalent to NEE can be misinterpreted.

- It might be good to harmonize formatting for Figures 2, 3, and 4.

- page 15, L449: missing reference "()"

- page 24, L703: "3)" -> "4)"

- From previous review, in the abstract: The acronyms RF and CLR were referenced before being defined

Hide

ED: Publish subject to minor revisions (review by editor) (02 Mar 2021) by Jean Dumoulin

AR by Atbin Mahabbati on behalf of the Authors (31 Mar 2021) Author's response Author's tracked changes Manuscript

ED: Publish as is (18 Apr 2021) by Jean Dumoulin

AR by Atbin Mahabbati on behalf of the Authors (27 Apr 2021) Manuscript

Short summary

We reviewed eight algorithms to estimate missing values of environmental drivers and three major fluxes in eddy covariance time series. Overall, machine-learning algorithms showed superiority over the rest. Among the top three models (feed-forward neural networks, eXtreme Gradient Boost, and random forest algorithms), the latter showed the most solid performance in different scenarios.