the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Benefits of using convolutional neural networks for seismic data quality analysis
Paolo Casale
Alessandro Pignatelli
Abstract. Seismic data represent an excellent source of information and can be used to investigate several phenomena such as earthquake nature, faults geometry, tomography etc. These data are affected by several types of noise that are often grouped into two main classes: anthropogenic and environmental ones. Nevertheless instrumental noise or malfunctioning stations detection is also a relevant step in terms of data quality control and in the efficiency of the seismic network. As we will show, visual inspection of seismic spectral diagrams allows us to detect problems that can compromise data quality, for example invalidating subsequent calculations, such as Magnitude or Peak Ground Acceleration (PGA). However, such visual inspection requires human experience (due to the complexity of the diagrams), time demanding and effort as there are too many stations to be checked. That’s why, in this paper, we have explored the possibility of “transferring” such human experience into an artificial intelligence system in order to automatically and quickly perform such detection. The results have been very encouraging as the automatic system we have set up shows a detection accuracy of over 90 % on a set of 840 noise spectral diagrams obtained from seismic station records.
Paolo Casale and Alessandro Pignatelli
Status: open (until 13 Jun 2023)
-
RC1: 'Comment on gi-2023-4', Anonymous Referee #1, 31 May 2023
reply
The paper discusses the applicability of AlexNet to seismic spectral plots to detect data quality problems. They combined existing diagrams from different sources into a new database containing 840 diagrams. By labeling (classifying) the diagrams using expert knowledge, they created a new training dataset for CNNs. In general, the approach was successful and the AlexNet achieved an accuracy of 90% on this dataset.
In my opinion, data quality is one of the most important issues in any domain. Therefore, the authors have addressed a very important problem. They show a possible solution to automate a pre-analysis to identify erroneous stations or stations that should be treated with caution.
1 Does the paper address relevant scientific questions within the scope of GI:
Yes. Data quality is important in any scientific field. In this case, the authors focus on seismic data, so this paper is relevant to GI.
2 Does the paper present novel concepts, ideas, tools, or data:
The authors have applied an existing and well-known concept to a new dataset. It is obvious that a CNN approach to classification will work. However, the authors noted that there are no large databases on which to train the neural networks (they “focused on the entire noise spectra to detect acceptable signals from anomalous ones”). As datasets are rare and very time consuming to create, this contribution is relevant to the scientific community if the data is made (“correctly”) publicly available (see 6).
3 Are substantial conclusions reached:
As stated above, it is obvious that the concept will work. The authors have proven this once again. Therefore, the paper does not provide any substantial new conclusions.
4 Are the scientific methods and assumptions valid and clearly outlined:
The methods are clear with strong weaknesses regarding “4. Machine learning general description”. The headline is not consistent with the following subsections. The heading should be: “4. Deep Learning” with 4.1 being: “4.1 General description”.
Some assumptions are mixed with results, such as the data splits in 5.1, 5.2, 5.3 and 5.4 and their explanations.5 Are the results sufficient to support the interpretations and conclusions:
Partially. Comparison of the four tests is not possible due to changing test sets.
6 Is the description of experiments and calculations sufficiently complete and precise to allow their reproduction by fellow scientists (traceability of results):
No. The data provided are only some diagrams without labels. What is missing:
- Source code
- Correct labelled data – from the given data, I cannot reproduce their approach. Please create 4 folders, each consisting the labels and images used during every experiment.
- The pre-trained checkpoint of the CNN
Furthermore, the images (which by themselves have no use) are published only on google drive, which is a bad decision. I suggest using an appropriate website to share your data such as https://www.kaggle.com/.
7 Do the authors give proper credit to related work and clearly indicate their own new/original contribution:
Yes
8 Does the title clearly reflect the contents of the paper:
Yes. However, in my opinion, the title suggests that a CNN is the best approach, which seems logical. However, the paper only uses a 2D CNN, and a comparison with a 1D CNN is missing. I would at least expect a section discussing the advantages and disadvantages of a 1D and 2D approach.
9 Does the abstract provide a concise and complete summary:
Partially. In my opinion, the results of the large multiclass experiment need to be added.
10 Is the overall presentation well structured and clear:
The general presentation of the introduction and theory up to section 3 is OK. Sections 4 and 5 are not. The experiments are mixed with results and discussion, which makes it difficult to follow the idea. Regarding the general structure, I suggest following common headings such as Introduction, (materials/data and) Methods, Experiments, Results and Discussion, Conclusions. See also the suggestions in 4 and the general comments below.
11 Is the language fluent and precise:
No. Some sentences are difficult to read because they are unnecessarily nested. Proofreading is needed.
There are many typing errors such as:- Missing/additional whitespaces e.g.: line 66; line 86; line 112 and many more
- Missing commas:
- line 61 end of line to 62: […], and […]
- line62: […] in some of them, […]
- line 128: […] trace statistics, [...]
- line 145: […] (McNamara Boaz, 2006), […]
- …
- spelling: line 154 (inserted new line separates the t from interest); line 156
- Please use cross-linking when referring to another section (example: line 363)
12 Are mathematical formulae, symbols, abbreviations, and units correctly defined and used:
Yes
13 Should any parts of the paper (text, formulae, figures, tables) be clarified, reduced, combined, or eliminated:
Please add a section: "Dataset" and move relevant parts to this section, as in my eyes this is an important contribution. This will make it easier to extract relevant information (see also general comments). I have major concerns with sections 4 and 5. See general comments.
14 Are the number and quality of references appropriate:
The references regarding section 4 are a little outdated in some cases or not present at all. Please add more and also recent references to section 4 as the field of deep learning is rapidly evolving. AlexNet should not be used.
15 Is the amount and quality of supplementary material appropriate:
Yes. To make it even more precise, I suggest adding a table with all four tests to make the comparison easier.
General comments:
The authors have used several data sets in the manuscript, which is confusing. I do not agree with the division into smaller datasets (of size 224, 447 and 840 plots) and one large dataset (1865 PDF spectral plots). It is obvious that more data will improve training accuracy. Removing uncertain diagrams from the training is not a good practice as results become not useful in real world applications and conclusions are not representative anymore. The authors could have just used the large dataset as it is the most general dataset in use.
Weighting strategies were not covered, for example, to address their imbalance problem faced in tests 1 and 2. Furthermore, the authors used different test sets to validate their approaches, making it impossible to compare results.
In the whole manuscript there is no information about hyperparameters such as batch size, epochs trained, duration of training and many more.
A very important topic to discuss is overfitting, which has not been considered at all. They did not provide any information about the training and validation loss, which is a common practice in applied deep learning. The lack of this information makes it very difficult to interpret the results.
Since reproducibility was not covered, the results will vary between two training runs. The authors should retrain the model several times and provide an average accuracy and standard deviation.
In conclusion: The experimental design is bad. Too many mistakes have been made.
There is no explanation as to why a 2D CNN approach was chosen in the first place. The authors claim that: “In our study we find that just images of data fulfil our goal” (line 336). Proof is lacking. In fact, they have shown that uncertain graphs are likely to be misclassified. A 1D CNN can be applied directly to the data without any loss of information due to encoding into an image. As the seismic information is stored in vectors, 1D CNNs are more efficient, easier to train and less complex. Since far fewer parameters are needed, the loss of information due to encoding the data into an image can be avoided and the only loss is introduced by subsampling the spectral data itself. Did the authors take this into account in their research? For example, the images could be used by the experts to classify the data, but the 1D data could be used for training. Why is it advantageous to use a 2D CNN? I suggest adding a section on this. I refer also to: https://doi.org/10.1016/j.ymssp.2020.107398.
Did the authors consider data augmentation? I would like to see a section discussing the possibility of data augmentation in such a case as it can improve accuracy a lot. I refer to: https://doi.org/10.1088/1741-2552/ac4430.
Further comments:
Section 4:
- Overall, this part is not well-structured and lacks some important parameters to reproduce the results. A suggestion for a new structure:
- CNNs (reduce the general description, remove ANNs and focus on the CNN),
- Training (a short overview, perhaps supported by an enumeration of important training steps),
- Metrics (accuracy, confusion matrix, F1 score (all results also class-based for multiclass problems), Precision, Recall) and
- Hyperparameters (training - validation - test split, batch size, epochs, loss function, optimizer, did the authors use dropout? did the authors use regularization techniques? did the authors use weighting to deal with class imbalances?) - Please also use F1 scores, as the training data is class unbalanced, which will bias the result towards the most common class, or at least use some weighting during training. Accuracy alone can be misleading when the classes are imbalanced because a model that predicts the majority class most of the time could still achieve high accuracy while performing poorly on the minority class. The confusion matrix, on the other hand, provides a more detailed breakdown of predictions but does not directly incorporate the trade-off between precision and recall. I refer you to the tutorials of PyTorch or tensorflow.
- AlexNet is very outdated and should not be used for such an important domain (see graph at: https://paperswithcode.com/sota/image-classification-on-imagenet). I refer to PyTorch, which provides pre-trained networks and their top-1 and top-5 accuracies on the ImageNet dataset: https://pytorch.org/vision/stable/models.html. It should be easy to replace your existing AlexNet with a newer network. Suggestion: EfficientNet V2, since it has fewer parameters and should be trainable on the used system. Please retrain on the final dataset.
- Please revise some claims that are incomplete or wrong:
- Line 262: “As well known, machine learning algorithms are divided into two groups: supervised methods and unsupervised ones.” – Reinforcement learning is missing. Please provide sources for all such claims in these sections.
- Line 300 following: “the main task of CNNs is to identify local features”. I cannot agree to that. While CNNs initially focus on local features in the first few layers, deeper layers and the latent space contain global information. However, the network's ability to determine the importance of global versus local features is achieved through the learning process by adjusting the weights based on the task's requirements.
Comment: I recommend not to go too deep into the topic, because many mistakes can be avoided. Instead, explain the general concept of CNNs and cite accordingly.
- I recommend replacing: “predictors and answers” for the more commonly used terms “inputs and ground truth”. The terms “inputs” and “ground truth” are more widely understood and commonly used in the deep learning community, making them preferable for clear and consistent communication.
- The dataset split is incorrectly explained: the dataset should be split into training, validation and test data. Training data is used to train the network. By evaluating the model's performance on the validation data, adjustments can be made to the model architecture, regularization techniques, or other hyperparameters to improve its generalization capabilities. The goal is to optimize the network for the validation data, e.g., by increasing the batch size or changing the optimizer. In the end, the final performance and generalization ability of the trained models is evaluated using the test dataset. I recommend further research. Because the authors split the test data in different random ways (In 5.1, 5.2 and 5.4: 80-20 split, in 5.3: 90-10 split), the results are not comparable. The test data set must be exactly the same for all experiments.
- Regarding the source code, I have doubts that this result can ever be reproduced, since no code was provided and reproducibility was not covered at all as well as the train/test split (see previous and question 6). Neural networks are inherently non-deterministic. However, you can achieve reproducibility by correctly initializing the random generators, which makes the result almost perfectly reproducible. I refer you to the reproducibility documentation from pytorch: https://pytorch.org/docs/stable/notes/randomness.html. Please cover this by adding a few sentences in the paper (e.g. in the suggested hyperparameters section) so that the community can reproduce the results.
Section 5:
- Section 5.1 and 5.2 and 5.3 – it makes no sense to split these sections. Training with less data gives no information. It is obvious that with such a small data set, more data will improve performance. Remove sections 5.1 and 5.2. I suggest adding a section "Dataset" explaining the two remaining training datasets (one without dubious diagrams, representing the binary classification, and one with the dubious diagrams, representing the multi-class problem). In both cases, the test dataset has to be the same.
- At the end of Section 5.3, the authors state that their data reduction is probably perfect because a human has discarded uncertain diagrams. Such an assumption is very risky and should be avoided, since errors can still occur.
- In 5.4, the authors somehow explain the problem of class imbalances without naming it. These imbalances can be addressed using weighting strategies, which the authors did not consider. Since this is one of the fundamentals of training neural networks, the authors should retrain their networks using weighting.
Section 6:
- The authors conclude that the experiment with the large data set reduced accuracy, which again needs to be verified using exactly the same test data. They also concluded that no BAD diagrams were labelled OK, which is very important and would be reflected if class-wise accuracies were shown (and precision/recall/F1). In such a multi-class scenario, the authors need to add class-wise metrics.
Citation: https://doi.org/10.5194/gi-2023-4-RC1 -
RC2: 'Comment on gi-2023-4', Anonymous Referee #2, 02 Jun 2023
reply
In this study, the authors address the topic of data quality control within seismic records, which is a time consuming process in light of rapidly growing data sets and requires human experience (experts) for the identification of malfunctioning or instrumental noise.
The authors propose to use machine learning (CNNs, AlexNet) in order to facilitate and fasten data quality control checks. This subject is definitely of importance to the seismic community and well suited for GI.
The authors put a big effort into manually labeling data (PDS). This data is used for training and testing AlexNet to predict in total three classes, which divide the PDSs into “ok”, “dubious” and “bad” PSDs/signals. Overall, the accuracy of 85 % is very promising.
However, I do feel there are some substantial shortcomings in the way the study is presented and list here some general remarks followed by some more detailed comments on the sections of the manuscript.
1. Does the paper address relevant scientific questions within the scope of GI?
It really does. Data quality is substantial and the authors propose an efficient way to account for it! Therefore, the subject is definitely suited for GI.
2. Does the paper present novel concepts, ideas, tools, or data?
The application of CNNs is not new in this context, neither is the data. The latter is not problematic. I feel it is rather meaningful to make use of existing archives here. However, methodologically – at least the way it is presented – this study seems rather basic and lacks information in the way it is presented now, e.g. on preprocessing, the manual labeling process, model hyperparameters or a comparison between different models.
3. Are substantial conclusions reached?
The application of AlexNet works and the conclusions meaningful. However, this is not a groundbreaking/new result.
4. Are the scientific methods and assumptions valid and clearly outlined?
In my opinion and as stated above, details on preprocessing, labeling and the model itself are missing. Section 4 (Machine Learning general description) is, it current state, rather superficial and overall references are not up to date.
Pros and Cons of AlexNet are not discussed, nor is this approach compared to others.
I had difficulties to follow the criteria for the four tests and wonder if it is really necessary to present all of them. Generally - the more data the better - and it is natural, that predictions become better with an increase in training data. It would be rather interesting to see how much data is really needed and when performance begins to drop. This is, however, not demonstrated based on these four tests.
5. Are the results sufficient to support the interpretations and conclusions?
In general and just looking at the outcome, it seems like it – yes. But as stated above, information are missing and I feel like this makes it hard to judge.
6. Is the description of experiments and calculations sufficiently complete and precise to allow their reproduction by fellow scientists (traceability of results)?
No. Neither data, source code nor the model are publically available. It is impossible to reproduce the results.
7. Do the authors give proper credit to related work and clearly indicate their own new/original contribution?
Partly. But there for sure is more out there when it comes to CNN application in this context, which the authors did not mention.
8. Does the title clearly reflect the contents of the paper?
Yes, it does.
9. Does the abstract provide a concise and complete summary?
Yes.
10. Is the overall presentation well structured and clear?
The structure is clear. However, the section and subsection titles sometimes sound a bit generic and bloated. This is a general remark though and also applies to parts of the main text. The presentation of methods and results is rather patchy and needs to be more precise.
11. Is the language fluent and precise?
Rather not. The authors need to carefully check spelling and writing throughout the entire manuscript again. Sentences were sometimes difficult to read, formatting or grammar was off.
12. Are mathematical formulae, symbols, abbreviations, and units correctly defined and used?
Yes.
13. Should any parts of the paper (text, formulae, figures, tables) be clarified, reduced, combined, or eliminated?
I would propose to carefully check for new/state-of-the-art papers on that matter and to include those recent studies to put this one presented here into the right light and framework. More information are needed in the methods section and results/discussion accordingly adapted.
14. Are the number and quality of references appropriate?
Not really. Partly, the authors use only one reference for most parts of a section and those studies often go back > 10 yrs. This needs to be addressed.
15. Is the amount and quality of supplementary material appropriate?
Yes.
General Remarks
- Spelling and writing needs to be carefully checked throughout the entire manuscript.
- A more explicit style of writing would be beneficial. While reading I regularly felt like it was too generic and more details supporting those generic statements are necessary.
- The references need to be updated. I am not an expert on the specific field (instrumentation or CNN application in this context), but most of the studies cited lie < 10 years in the past while both the technology and the field rapidly emerge.
- As a follow up, the study needs to be contextualized in light of the current state-of-the-art within this field. What do others use/propose to address this topic and why is your approach more suited?
- More details on preprocessing, manual labeling and model hyperparameters need to be included.
- A stress test would be particularly interesting. How much training data is needed? Is this transferable to other data sets? What are the criteria?
Chapter 2. Seismic noise and its spectral representation
- The definition of “noise” vs. “signal” is a bit old fashioned in my opinion. As seismology turned into a very broad field (earthquake seismology, environmental seismology, microseism), we always define our signals of interest, while the remaining fraction of the data is considered as noise. If this study is only related to earthquake seismology, it should be clearly stated here.
Chapter 4. Machine Learning general description
- Overall this chapter is rather sketchy. In the beginning, machine learning is almost “advertised” as being simple and easy to apply without the need of being particularly knowledgeable about it, only the amount of training data would be of importance.
- I would be very careful about those statements and can actually not support them. It is why machine learning is often still considered as “black magic” or a “black box” with some shortcoming input data and the shiny output data. This is not the case. Machine learning models can be very sensitive to the type of data they were trained on (even biased), but also to hyperparameters of the model, which can cause e.g. overfitting.
- The right terminology is sometimes missing or off. For example: the authors describe the problem of overfitting with their own words, but do not mention the specific term.
- References are missing. The section is only based on one references, which again is > 10 years old.
- What is the difference on “images of data” compared to “data”. I guess I see what is meant here, but terminology needs to be more specific to take off the guesswork.
- AlexNet is briefly introduced, but it is not discussed why this is the right choice here compared to other approaches.
5. Method application
- ll. 340-341 – how do you account for that?
- It would be nice if the authors could elaborate on the classification decision into “ok” and “broken”. It is stated, that this has been done by a human operator, but the criteria are not explicitly described. The only statement I found was “… showing trends that, according to the human eye, definitely belong to defective stations”, which in my opinion is quite subjective and loosely formulated. Overall, this is not reproducible.
- Second test: I see why the authors wanted to increase the number of “broken” labels. However, to not bias the network (now you definitely do so), the number of “ok” signals should be equally increased as well. I also doubt that “ok” signals are more homogeneous from the ML model perspective, if you consider all the potential environmental noise.
- Fouth test: looking at this data base and test, I wonder why test 1-3 are presented.
Citation: https://doi.org/10.5194/gi-2023-4-RC2
Paolo Casale and Alessandro Pignatelli
Paolo Casale and Alessandro Pignatelli
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 139 | 24 | 9 | 172 | 4 | 3 |
- HTML: 139
- PDF: 24
- XML: 9
- Total: 172
- BibTeX: 4
- EndNote: 3
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1