Between Innovation and Standardization, Is There Still a Room for Scientific Reports? The Rise of a Formatting Tradition in Periodontal Research

Abstract

Everybody, regardless of their role, is aware that biomedical research is rapidly evolving, and the demand for reproducibility is increasing together with the amount of novel information. “Before reproducibility must come pre-producibility” “Checklists work to improve science”, just to quote some of the articles querying how to find a new bridge between ethics in science and the urgency for publishing. Looking for papers on anti-inflammatory compounds in periodontics, we came across a significant number of articles that could be considered a prototype of a consistent study format. The literature on the testing of active compounds on lipopolysaccharides- (LPS)-induced inflammation in gingival fibroblasts was searched to identify studies that followed a consistent format, to better understand their similarities and assess the appropriateness of their methods. Several studies were identified with a degree of similarity in their methods and formatting that was so high that it was possible to rule out that it was due to chance, and a format template common to these studies was outlined. Although this was most likely beyond the intentions of their authors, these studies may pose the basis for an in-vitro testing standard for anti-inflammatory compounds; however, the dangers of acritical uniformity are also apparent.

1. Introduction

Testing the effectiveness of a chemical compound in a biological model means to challenge that model with an experimental stimulus and assess its responses using established, accepted and, possibly, standard measures. Standardization allows for the transferability and comparison of results across different studies. Therefore, it is not surprising that the scientific community has long striven to achieve standard protocols in pre-clinical and clinical testing that could facilitate the interpretation of results and possibly streamline the development of new therapies and medications.

Dentistry is no exception to this, and there is no doubt that it shares the same need for agreed and standard protocols for testing as all the other medical fields. Periodontal diseases affect a large portion of the population [1]. Research has highlighted that the profound tissue damage that jeopardizes teeth survival arises from the inflammatory response of the organism to bacterial pathogens [2], which form complex biofilms [3] that accumulate along to the dento–gingival margin and challenge tissue defenses [4] via some bacterial products, such as lipopolysaccharides (LPS). LPS is a critical component of Gram-negative bacteria, such as Porphyromonas Gingivalis, which is able to trigger the activation of Toll-Like Receptors, and thus inflammation [5]. P. Gingivalis is one of the most extensively studied microorganisms in the pathogenesis of periodontitis and has been shown to elicit both a potent inflammation and to modulate tissue responses, thus blunting their antimicrobial effectiveness [6]. Strategies to reduce or inhibit such inflammation-driven tissue destruction are, therefore, being actively investigated world-wide.

While browsing the current literature on the effectiveness of chemical compounds in hampering the inflammatory response of human gingival fibroblasts, it became apparent that a growing number of publications are becoming aligned in terms of the reporting format. A better understanding of this format may prove important to better tailor future publications to the real clinical needs of the field.

2. The Rising Consensus

A striking finding of any, even brief, survey, of the existing literature is the emergence of a consistent protocol in the studies from several and, at least in part, unrelated research groups, who have focused on the testing of a given compound (mostly of natural origin, henceforth labelled as X compound) on lipopolysaccharide-stimulated human gingival fibroblasts to assess its anti-inflammatory properties.

This protocol can be considered a formatting template that dictates the assays that are performed, their order and the way they are plotted and presented in the manuscript, with a surprisingly high level of consistency. This formatting template appeared to date back to 2014–2015, at least, when studies with gingival fibroblasts are considered. Although we were not able to trace and identify the first publication that adopted it with certainty, we were unable to find papers in the periodontal field using this template before that date, although studies adopting this format may have first appeared in 2008 [7]. However, for the sake of clarity, we decided not to include these last works in the present commentary, as they will be better examined elsewhere.

A recurrent feature of these studies is that they usually test naturally derived compounds, such as veratric acid [8]—a benzoic acid that is isolated from vegetables and fruit [9]—acanthoic acid [10], which is isolated from a local tree [11], or tormentic acid, which is a triterpene extracted from Rosa Rugosa [12]. This is not surprising, as herbal medicine is a long-standing feature of traditional far Eastern medicine [13], and the number of research studies on the pharmacological properties of herbal extracts has exponentially increased in the last few years [14].

All of these works rely on the same in vitro model, i.e., primary human gingival fibroblasts, which are actually the preferred experimental model used in the literature, and LPS from Porphyromonas Gingivalis at the concentration of 1 μg/mL. The template is so consistent that it is possible to outline its composition and break it down, figure by figure, as expounded below.

3. The Template

This reporting format for compounds tested on human gingival fibroblasts is composed, in its most prototypical form, as follows:

3.1. Figure 1

The first figure of the template (Figure 1) is usually a viability assay, e.g., MTT [8,10,12,15,16,17,18] or, in a few cases, CCK-8 [19,20] assay that compares the effects of the X compound, alone or in combination with LPS, on HGF viability at different concentrations, plotted as a bar chart. This figure is possibly the most consistent of the whole format and it is found in virtually all the studies that follow this template. It is so consistent that it actually represents its most distinctive hallmark.

The purpose of the cell viability assay is to rule out toxic effects of the X compounds on the cell model when used alone, or in a pro-inflammatory setting caused by LPS. This graph is plotted in two colors, e.g., black/white or black/grey. One color is used to label bars corresponding to cell viability in the presence of three increasing doses of the compound (the legend says “Drugs only”), while the lighter color is typically used on the bars that correspond to the control group (i.e., no LPS, no test compound), LPS group (i.e., LPS-only) and to the groups exposed to both LPS and three increasing concentrations of the test compound. Strikingly, all the studies following this format label this color as “Drugs+LPS”, although, as we just mentioned, this very color is used also for the Control and LPS-only bars, which poses an interpretative challenge. It is also interesting to note that all these graphs have the Y-axis starting at 50% viability. Strikingly, the legend almost invariably states that:

“Effects of X compound on the cell viability of HGFs. Cells were cultured with different concentrations of X compound in the absence or in the presence of 1 μg/mL LPS for 24 h. The cell viability was determined by MTT (or CCK-8, A/N) assay. The values presented are the means ± SEM of three independent experiments”.

In one case, this legend is reproduced so precisely that it includes the term MTT, when the assay used in the study was actually CCK-8, according to the Materials and Methods Section [20] Interestingly, standard deviation was chosen as an indicator of dispersion in only two cases [15,16].

Figures similar to Figure 1 can be found even in some manuscripts that do not appear to be following this format template [21,22]. These two manuscripts, by the same research group, however, report the cell viability data (from CCK-8 assay) using four concentrations of the tested compounds, and a less controversial bar-labeling system.

Most strikingly, this peculiar format to report cell viability data can be found as Figure 1 even in a vast array of manuscripts that focus on the anti-inflammatory properties of specific compounds on LPS-induced inflammation in several unrelated cell models, such as mammary epithelial cells [23], 3T3-L1 adipocytes [24], BV2 microglial cells [25,26], RAW264.7 cells [27]—where it tends to constitute Figure 2, however—and umbilical vein endothelial cells [28]. This same model for Figure 1 is also found in two works on the testing of anti-inflammatory properties of different compounds on IL-1α -induced inflammation in chondrocytes [29,30].

Interestingly, the most recent manuscript that follows this template has split the histogram into two separate graphs, representing the effects of the X compound alone on cell viability, and the effects of the X compound in the presence of LPS [31].

3.2. Figure 2

The second figure of the template (Figure 2) is usually a multi-panel figure constituted by bar charts reporting the levels of Interleukin 8 (IL-8) and Interleukin 6 (IL-6) secreted by HGFs in the supernatant and quantitated by ELISA assay. The chart includes ILs levels in the Control group, LPS-only group and in the groups exposed to LPS and three increasing concentrations of the tested compound. The purpose of this assay is to demonstrate that the X compound is able to reduce the concentrations of these pro-inflammatory ILs in the supernatant and that there is a dose-dependent effect by the test compound.

Interestingly, this figure is most commonly composed of two graphs vertically ordered, with IL-8 on top and IL-6 at the bottom [10,12,16,19]. In one instance [15], Figure 2 includes the mRNA levels for IL-8 and IL-6 beside the data for these same ILs from the ELISA assay, side by side.

In two studies [17,18] Figure 2 is constituted of two bar charts reporting the effect of vehicle, LPS-alone or LPS and three increasing concentrations of the test compound on the release of Prostaglandin E₂ (PGE₂) in the supernatant (by ELISA), and on the production of nitrite. Most noticeably, IL-8 and IL-6 levels by ELISA assay (reported in the same way as described above) constitute Figure 3 in these two manuscripts. In one case [8], Figure 2 includes IL-8 and IL-6 supernatant levels measured by ELISA within a multi-panel figure that also comprises iNOS and COX-2 levels by Western Blot (WB). This graph panel can also be found in the study by Li et al. [32]; however, in that study it constitutes Figure 1, because the cell viability assay (CCK-8 in this study) is only described in the Results section, but data are not shown.

3.3. Figure 3

The third figure of the template is usually composed by a multi-panel figure that comprises photos of WB membranes labelled with anti phospho-p65, phospho-IkB, with α-Actin as reference and two bar charts, on the right side of the photos, obtained by the quantitation of the intensities of the WB signals (Figure 3) [8,10,12,15,19,20]. The WB lanes are constituted by the Control group, the LPS-only group and LPS and three increasing concentrations of the test compound.

In a few instances, this chart can be found in Figure 4 [16,18] or in subsequent locations [17].

3.4. Figure 4

The fourth figure of the template is usually constituted by a WB analysis of some further targets. This figure commonly comprises a photo of the WB membrane with the Control, LPS-only or LPS and tested compound groups, as above, and is quantitatively expressed with a bar chart. The actual target protein differs across studies, as they may focus on different effects of the test compound and includes PPAR-g [19], TLR4 [10], p-JNK, p-ERK and p-p38 [12], p-PI3K and p-AKT [8,15], nrf2, Lamin B and HO1 [20].

In a few cases, this final WB analysis has been moved to following figures because additional graphs have been added in preceding positions. It is the case with Zhang et al., where the WB for LXRa is in Figure 5 because a WB for iNOS and COX-2 occupies Figure 3 [16], Liu et al., where the WB for p-PI3K and p-AKT is in Figure 5 because PGE₂ and nitrites levels were reported in Figure 2 [18] or Hao et al. [17], where we find a WB for PPAR-g is shown in Figure 7.3.1.

4. Establishing a Tradition

The similarity across these reports is definitely higher than the typical average similarity between non-related and non-syngenic studies. It is true that a certain degree of similarity can be expected in studies that are designed to answer a similar question. In this case, all the studies we considered shared the underlying experimental question:

“Does X inhibit the inflammatory response triggered by LPS in human gingival fibroblasts?”

It may, therefore, not come as a surprise if, irrespective of the actual answer, the approach used to answer the question was similar. The formal similarity is arguably advantageous in science, because it may increase the efficiency of communication. It is easier for readers to find the data that matter the most to them when the surface structure of a manuscript is familiar or well known. On the other hand, the consistent adoption of a graphic format that is not optimal to the understanding of the experiments or even correct (as is the case with Figure 1 and its label) shows the dangers of excessive uniformity and reminds us that “published” (or “peer-reviewed”) is not necessarily a synonym for “best”. We have shown that Figure 1 (and its legend) is so consistently replicated that it is legitimate to wonder whether acritical acceptance of what was perceived as a winning format (presumably, because it was published) may have played a role in the authors’ decision to adopt it. We are aware that several factors, including geopolitical considerations, may have compounded the lead investigators to conform to this reporting template. We believe we are facing the generation of scientific tradition, so to speak, and the dynamics of the acceptance of this model across different research groups should be closely scrutinized.

This reporting format must be analyzed on two separate levels. The first and most apparent one is its iconographic surface, i.e., the sequence of figures that illustrate the results of the study. We have already made the case that the graphic format adopted at least for Figure 1 is far from fullproof, and it is somewhat surprising that its incongruence did not strike the attention of previous authors and peer-reviewers. It may be acceptable for independent studies to stylistically resemble previously published articles, but only if they represent an impeccable model, otherwise errors may just end up being perpetuated along with a questionable tradition of manuscripts.

The second level of analysis can be performed on a deeper, experimental level, i.e., what assays and endpoints were used to achieve the purpose of these studies, regardless of the way data were plotted.

A striking feature emerging from all these papers is that the X compound is always administered before the challenge with LPS. However, clinically, an anti-inflammatory drug is administered once the pathogenic event takes place, and not beforehand. There is no reason to take a drug to prevent the inflammatory response, which is important for tissue repair after damage. Only an excessive reaction must be controlled to avoid the spread of the injury. It is true that, as most of these compounds are of natural origin, the hypothetical setting may be a prolonged or daily consumption in patients’ diets. These studies would provide a rationale for the prevention of gingival inflammatory disorders in patients regularly consuming these compounds. However, this scenario ends up limiting the applicability of the results to clinical practice, where the real challenge is finding new drugs or active compounds that are capable of inhibiting tissue destruction in the presence of inflammation.

When the outcomes and endpoints used in these studies are assessed, many choices do appear rational and commendable. Assessing cell viability as the first outcome makes sense in this kind of study and MTT is indeed a good candidate for this endpoint because of its reliability, although newer and more time-efficient assays are now available [33].

Figure 2 of the template focuses on the effectiveness of the X compound to reduce the production of pro-inflammatory mediators, mostly Interleukins (ILs) [34]. Prostaglandins have long been recognized as important molecular players in periodontal inflammation [35,36], and it is, therefore, not surprising that PGE₂ or cyclooxygenase-2, the enzyme that synthesizes it, are included in the array of relevant endpoints in several papers that follow this format [8,16,17,18]. As LPS has been shown to increase the expression and release of a vast array of ILs; it is important to pick those that are likely to have more profound repercussions on the development of periodontal diseases. The studies that follow this reporting template consistently focus on IL-8 and IL-6, although non-aligned studies often include IL-1 and TNF-α as well [22,37,38,39]. These cytokines have actually long been known to mediate periodontal destruction [40]. TNF-α has been shown to be a consistently elevated inflammation mediator in the gingival tissue of periodontal patients [41], and it may, therefore, be an odd choice to leave it out. However, a well-known study by Dongari-Bagtzoglou et al. showed that fibroblasts from periodontally affected patients secrete higher levels of IL-6 and IL-8 than fibroblasts from healthy controls [42] and a recent review concluded that IL-8 mRNA and protein levels were higher in the tissues of chronic periodontitis patients than healthy controls [43]. This is also consistent with the pro-osteoclastogenic activity of IL-6 [44] and the overall chemoattractant effects of IL-8 [45]. Nevertheless, although certain haplotypes of the IL-8 gene have been shown to significantly associate with susceptibility to chronic periodontitis [46,47], there remain contradictory reports on IL-8 levels in the gingival crevicular fluid of healthy and chronic periodontitis patients [48,49,50,51]. Furthermore, even if IL-8 levels may be relevant markers of chronic periodontitis, this does not entail that they are the most relevant targets for an anti-inflammatory strategy, which should mostly focus on blocking the farthest-reaching mediators, those biomolecules that are likely to control important down-stream effectors. Under this perspective, it is again difficult to make the case that IL-6 and IL-8 alone are reliable indicators of drug activity.

Given the complexity of the cytokine network in inflammation, broader screening approaches are to be expected in such types of studies, to identify novel and relevant targets of action for the tested compounds. This is important because the assumption that anti-inflammatory compounds always modulate the same sets of cytokines, and even more so only two of these, i.e., IL-6 and IL-8, is questionable, to say the least.

The inclusion of phospho-p65 and phospho-IkB by Western Blot might be regarded as less controversial, as these endpoints are indicative of the activation of the NF-kB pathway and WB is an appropriate and still a state-of-the-art, albeit cumbersome, methodology to investigate protein phosphorylation. NF-kB is a family of transcription factors that are known to trigger an inflammatory response upon stimulation with several cytokines, e.g., IL-1α, TNF-α or IL-6 [52]. The activation of NF-kB entails the phosphorylation and degradation of IkB, which then frees the active subunits, e.g., p-65, which can then translocate to the nucleus [52]. NF-kB signaling has been consistently shown to be up-regulated in diseased periodontal tissue [53,54,55] and can be considered a good indicator of ongoing inflammation. However, several non-aligned studies make an effort to go beyond the assessment of the phosphorylation of p65 and IkB, e.g., investigating their nuclear localization [21], which can be a good indicator of the actual activation of the system. Taken together, these experiments show that these drugs may prevent an inflammatory response, as NF-kB is able to control inflammatory genes, rather than having an anti-phlogistic activity. What would happen, however, if the tested compounds were administered after the challenge with LPS? Would these drugs still be able to reduce inflammation markers? There are no data in literature to answer these questions.

An equally important point is that these studies, as a rule, do not use any positive control for anti-inflammatory properties, such as known anti-inflammatory molecules, e.g., glucocorticoids. This would be extremely useful because it would provide a comparison to assess the real net effect of these natural compounds. Glucocorticoids thwart NF-kB activation by inducing its inhibitory partner I-κBα. Moreover, glucocorticoids do act on some of the pathways that are investigated in these studies, but they also hinder inflammation by direct induction of genes like IL-10, annexin-1, the decoy receptor IL-1R2, SLPI, and decreasing AP-1 activity and p38 MAPK activation [56]. Interestingly, however, dexamethasone does not interfere with the activation of NF-kB in fibroblastic-like synoviocytes, while still having an anti-inflammatory effect [57]. This definitely proves that a more complex network is mobilized by glucocorticoids which can hinder the phlogistic response even after the challenge with LPS or TNF-α, and it would be fair to assess how the X compounds measure up against glucocorticoids in these pathways.

Furthermore, recent studies have reported that LPS-induced responses control other critical pathways, e.g., p53-mediated apoptosis [58], and it would be a scientifically sound approach to incorporate these newly acquired concepts into future studies, to ensure that the overall knowledge in the field can progress.

Taking all these considerations together, it can be concluded that the experimental design used in these studies, while not inappropriate, has important limitations that should be acknowledged. The core endpoints that are used in this reporting format are acceptable, and it is not our intention to suggest otherwise. Nevertheless, a broader approach to the regulation of mediators, the use of positive controls with known anti-inflammatory drugs for comparison and an experimental model closer to the clinical situation must be strongly advocated for a fairer assessment of the anti-inflammatory potential of a given drug compound on LPS-induced inflammation in human gingival fibroblasts. Even more important on a broader scale, is the realization that a whole basis of evidence that centered around an insufficiently scrutinized core of studies may serve as a platform for further research, and, when accepted because of its prestige, may serve as a model and affect future investigations, possibly impairing their efficacy. In an age, where a growing amount of attention is being dedicated to ensuring the objectivity of research and new publishing formats—e.g. pre-registered reports [59]—are being experimented, to decrease publication biases, we believe it is important to closely scrutinize any source of error, to ensure a better quality of science.

5. Conclusions

This reporting format is undoubtedly achieving a palpable success among researchers investigating anti-inflammatory compounds in periodontology, as its traces can go further than the group of studies that comprised the present review, and can be identified in several current publications. In spite of this, the particular formatting of cell viability data is questionable. Furthermore, the theoretical foundations of these models are not flawless, as no positive controls are used for comparison, and the potential of the tested compound to hamper an ongoing inflammatory response is not properly tested. Finally, as inflammation comprises a complex network of mediators, the consistent inclusion of only a limited set of endpoints, e.g., IL-6 and IL-8, does not appear to be justified, and also the set-ups of these experiments are not adequate to define a substance able to inhibit NF-kB activation as an anti-inflammatory compound.