On Training Knowledge Graph Embedding Models
Round 1
Reviewer 1 Report
The article “On Training Knowledge Graph Embeddings Models” study the TransE, DistMult, TriModel and Complex Knowledge Graph Embedding (KGE) models and demonstrate that the choice of training components like loss functions, negative sampling strategies and hyperparameters have an impact on the model efficiency.
The article consisting of six sections is written in a very clear manner. Section 1 gives the introduction and motivation behind their research. Section 2 presents the necessary background. Section 3 discusses in detail the different loss functions in KGE model and section 4 on training hyperparameters. These two sections also describe their experimental setup and the results obtained. The authors also present a detailed discussion of their findings in section 5.
However, I have some questions and suggestions regarding their proposed approach.
Lines 58-60 talk about the availability of a number of KGE models. It is not very clear from the article, why the authors chose the aforementioned models.
Secondly, though the authors provide the necessary background in section 2 related to KGE models, they miss to give a small description of the chosen models, which makes it difficult to understand or compare these models. The readers may find some sentences here and there like in lines 25-26, lines 146-147 etc. But I think that it is important that they present a brief description of these models and their key characteristics. These characteristics should be further used in subsequent sections, especially section 5.
In Figures 4 and 5, the readers may get an impression that there are very slight differences across the different models. Even the associated text does not help understand the reasons behind these small differences. Figure 6 currently misses the y-label (accuracy).
It is interesting to read the discussion on the results obtained in section 5. This section is of high interest to the readers using these models. However, it currently misses a discussion on the limitations of their proposed approach. Also, I feel that a discussion on possible future works will further interest the readers.
Minor remarks
- Line 48-> This study is the a step towards improving -> This study is a step towards improving
- Line 362: scalability of the training process KGE models-> not clear
Author Response
The response is included in the attached file
Author Response File: 
Author Response.pdf
Reviewer 2 Report
In this paper, authors argue that the choice of training components such as loss functions, hyperparameters and negative sampling strategies can have substantial impact on the model efficiency. This area has been rather neglected by the state of the art so far, and authors have contributed to covering this gap by a thorough analysis of possible choices of training loss functions, hyperparameters and negative sampling techniques. Authors also have investigated the effects of specific choices on the scalability and accuracy of knowledge graph embedding models.
For experiments, authors have used five knowledge graph benchmarking datasets: _ NELL239; _ WN18; _ FB15k-237; _ YAGO10; _ PSE.
Not all of the benchmarking datasets can be downloaded using the following url: https://figshare.com/s/8c2f1e1f98aff44b5b71
Only NELL (50K and 239); WN (11 and 18rr); and FB15k-237!
The datasets have different structures. Possibly they are converted to some common format. It will be good to outline the preprocessing phase.
There are some typo errors in the text:
row 65: representative state-of-the-art KGE models in Section 3. We also preform an
row 186: In our experiments we use six knowledge graph benchmarking datasets:
row 329: the growth in the dataset size and the required training runtime. We the compare the
rows 434 - 437 - not finished phrase:
434 In
435 the first row of plots corresponding to the experiments done on the NELL239 dataset (the
436 smallest dataset), the results show that the changes of the number of training iteration
437 (epochs) and the embedding size have. ???
Author Response
The response is included in the attached file
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
First of all, I would like to thank the authors for considering my review comments. The authors have made the necessary changes.
In the introductory section, they have given a brief comparison between the different models, which may help the readers. Furthermore, in section 2, they have presented the necessary definitions related to embedding interaction functions and recalling some of the models mentioned in Section 1.
They have also clarified the results shown in Figure 5. Thanks to sections 5.4 and 5.4, the readers can also understand the current limitations and the possible future course of actions.
