We found a significant difference between healthy controls and early onset lung cancer patients with respect to DNA repair capacity in a separate and independent case control study applying an identical comet assay protocol [30]. These data support the hypothesis that impaired repair capacity is also detectable in our patient cohort, may be due to genetic factors.

To analyse DNA damage in the family design, the primary outcome parameter OTM was used for the analysis of basal damage, initial DNA damage and residual DNA damage after repair. The logarithm of the Olive Tail Moment (OTM) was taken to transform the data set into a normal distribution function [26,31,32,33], which was shown to be highly sensitive for the evaluation of DNA damage [26]. No difference between parents, index person, siblings, offspring or partners was detected (Figure 1). The measured basal damage (not radiation exposed) was on average −0.74 (std. dev. 0.77). Directly after irradiation lnOTM was on average 1.59 (std. dev. 0.36). DNA repair was almost completed after 60 minutes (Figure 1) as demonstrated by an average lnOTM declining to −0.56 (std. dev. 0.70). If genetic factors do influence DNA damage one may expect to observe DNA damage declining from the patient group, to the siblings, parents and children and DNA damage should be lowest in spouses due to the degree of relationship. Here, we found no differences among the investigated groups neither for the basal damage nor for the induced radiation DNA damage or for residual DNA damage (Figure 1 and Supplementary Table 3). Probably the supposed effect is too small to be resolved in this family study design. In contrast, Rajeswari et al. [25] found significant differences in the DNA damage response detected by the alkaline COMET assay among breast cancer patients, first degree relatives and controls at baseline levels as well as for induced damage and repair efficiency when human lymphocytes were treated with the mutagen N-methyl N-nitro N-nitrosoguanidine.

Legend to Figure 1: DNA damage is expressed in [lnOTM]. On X-axes the different time points are given. 0 represents the initial damage after irradiation. Residual damage was detected after 10, 30 and 60 minutes after irradiation. The non-irradiated controls are represented in the time point before irradiation (0).

To estimate if heritability was modified by other factors, we investigated the impact of age, gender and smoking on the observed DNA damage as detected by the damage parameter lnOTM. Only age was found to have a significant but weak effect on basal damage (p = 0.0439), and DNA repair after 10 minutes (p = 0.0286) and 60 minutes (p = 0.0179) while gender and smoking remained not significant. No alteration at all could be observed regarding the initial damage (see Supplementary Table 4 and Figure 2). However, the age dependent effect of the basal damage might be less a biological phenomenon, rather than an effect of residual confounding. Interestingly the group of middle aged individuals (40–60 years), which contained the majority of partners of LC cases, showed a higher basal damage compared to groups of older (mainly containing parents) and younger (mainly containing offspring) individuals. However, the age-dependent increase of basal damage could be a consequence of residual confounding, e.g., by passive smoking or sub-adequate quantification of long life smoking behaviour. More siblings and partners (enriched in middle ages) were smokers (proportion of never smokers is 27% and 25% respectively), than children (enriched in lower ages, proportion of never smokers 43%) and parents (enriched in higher ages, proportion of never smokers 47%). Therefore, we assumed that the estimation of heritability will not be noticeable biased by age, sex or smoking.

Legend to Figure 2: Lung cancer patients were disregarded to avoid bias due to disease status.

Heritability was estimated for the basal damage, the initial radiation-induced damage, and residual damage of all available two- and three-generation families. We used parent-offspring regression to estimate the nominal heritability () [34,34,35,36]. Depending on the availability of parental outcome measures we performed mid-parent-offspring regression or one parent-offspring regression. Because the COMET assay is an experiment affected by numerous noisy or confounding factors [26], we assessed the effect of the experimental design onto the estimation of heritability. Hereby we introduced a falsity-correction for considering pseudo-families generated from independent controls of a control cohort (KORA). All controls within a single trial were randomly grouped in pseudo-families, where the status as pseudo-parent or pseudo-offspring was assigned by chance. Heritability was estimated as described above in each of 3000 repetitions. The mean estimate was considered as falsity induced by study and assay design. For internal validation we investigated spouse-correlation in terms of spearman’s rank correlation of lnOTM between partners and parents, siblings and the index person respectively, separately for basal damage, induced damage and residual damage.

We found a mean heritability in pseudo-families of about = 20%, which was used as falsity-correction factor. Hence, an adjustment of the primary outcome parameter by repeated measurement of a single human blood donor (an identical human reference sample used in every experimental set) is insufficient in clearing interfering design or assay specific factors [26]. However, heritability in pseudo-families ranges from 4% to 28% (Table 2), indicating the methodological difficulties for an appropriate correction. This range might be due to experimental variability at short time points of DNA repair and the large proportion of white noise in the radiation induced damage [26].

The corrected heritability of the basal damage and DNA damage directly after irradiation were about 70% (95%-CI: 51%–88%) and 65% (95%-CI: 47%–83%), respectively. However, during the repair period (10–60 min after irradiation) we observed a nominal decreased heritability between 20% and 48%. We observed no significant spouse correlation, except to offspring, as expected (see Supplementary Table 5).

These estimates of heritability are in accordance with observations of several other studies. Already 1963 Roderick et al. estimated a heritability of 30% to 50% for survival after radiation exposure in mice. Roberts et al. [37] analysed 95 family members of radiation sensitive breast cancer patients and could demonstrate a clear evidence of heritability of 82% in the chromosomal G2 radiation sensitivity assay, assuming a single major gene effect. In a classical twin study design, the genetic and environmental variance components were estimated on the basis of the mutagen sensitivity assay [20], which is able to detect chromatid breaks after mutagen challenge. Among various mutagens, a genetic heritability of 62.5% after gamma irradiation was estimated. This heritability was highest compared to the investigated chemical mutagens in the study. The heritability of mutagen sensitivity is further supported by several other twin and sib pair studies considering, e.g., micronuclei frequency [23] as an additional endpoint. The heritability for baseline micronuclei frequency was found to be between 68% and 72% that of radiation induced micronuclei frequency between 57% and 68%. This is in agreement with our COMET assay results. Familial correlations of radiation induced apoptosis among first-degree relatives (father-mother-offspring trios) were observed for different lymphocyte populations (B- and T-cells) [24]. A heritability of 43% for the radiation sensitive phenotype of T4 effector memory cells was estimated. The large heritability of the basal damage indicates that DNA stability has a genetic component, irrespectively of experimental irradiation. This finding is especially supported by twin studies which rules out shared environmental effects [23,25].

The observed decrease of heritability of the damage after repair may be interpreted by a decreased genetic component as supported also by others [23], but it could also be due to an enhanced inherent experimental variation in determining DNA repair which results in decreased heritability estimates. We conclude that not only radiation induced damage is determined by genetic factors, already the level of basal DNA damage has a strong genetic component. This parameter is directly related to genome stability. Genomic instability on the other hand was shown to be a major factor in the chromosomal radiosensitivity [14]. Our findings support the epidemiological observations that a genetic component is involved in the etiology of early onset of lung cancer. For future studies it is important to identify genes contributing to genomic instability which are not necessarily involved in DNA repair.

Samples of fresh peripheral blood (EDTA) were taken from participants of the LUCY Study (Lung Cancer in the Young, [38], 355 LC patients–diagnosed prior to the age of 51 years–623 relatives of 175 families, and 170 population controls free of cancer (KORA Study: KOoperationsstudie im Raum Augsburg [39,40]. Blood samples of LC were taken at the time of primary diagnosis before any treatment. Results of the so called family part of the study (LC patients as index probands and their relatives) are reported here.

Independent LC cases and controls (comprised in the case-control part) were used for bias correction. Further details are described elsewhere [38,41]. In addition to these study groups, isolated peripheral blood lymphocytes out of EDTA blood samples taken from a single healthy male collected from three donations over the time period were used in all assays as human reference (HR) [26]. Isolated cell aliquots were stored in liquid nitrogen and used over the whole investigation period.

Lymphocytes were isolated from blood samples by Ficoll separation (Leucosep) and cryopreserved. For the COMET assay, lymphocytes were thawed quickly and incubated for 24 hours. Aliquots were exposed in vitro to 4 Gy of gamma radiation on ice. Irradiated aliquots were set on ice prior to embedding in agarose to avoid repair of DNA damage. DNA repair was assessed by incubation of the cell samples at 37 °C after irradiation. Aliquots were taken after 10, 30 or 60 minutes after incubation and set on ice. In parallel, sham-irradiated controls were also collected after the same time intervals to measure basal damage for control. A detailed protocol description is given elsewhere [26].

Irradiation and DNA repair are jointly denoted as exposure levels: basal damage (0 Gy)–initial irradiation-damage (4 Gy/0 min)–repair 10 min (4 Gy/10 min)–repair 30 min (4 Gy/30 min) and repair 60 min (4 Gy/60 min). Batches of aliquots of patients and their relatives were preceded together, denoted as one trial. Each sample was studied under each of the exposure levels at least once. In total, 173 trials were completed, containing 10,016 single experiments (see Supplementary Table 2) with data of about 1,750,000 single cells as base outcome measures.

The COMET assay and the equipment are described in detail in [26].

The logarithm of the Olive Tail Moment (lnOTM) was investigated as the base outcome metric per cell nucleus. The basal damage (B) corresponds to the primary metric for no irradiation (0 Gy). The initial radiation-induced damage (I) corresponds to the primary metric directly after irradiation (4 Gy after 0 minutes’ repair time) minus the basal damage: The DNA repair capacity (DRC) (secondary outcome measure) was determined for 10, 30, and 60 minutes after irradiation. DRC is defined as the non-repaired proportion of the initial radiation-induced damage I at a specific time point, corrected by the basal damage B. Additional to blood samples of cases, controls and relatives, samples of a single individual (denoted as human reference HR) were processed and measured in the same way throughout the whole investigation. We assumed, that the empirical variance of outcome measures of HR reflects the blurring effects of design-inherent endogenous factors (white noise). Comparing measure variability of cases, controls and relatives with that of HR, we recognised that a proportion of 7% to 95% can be attributed to assay variation (white noise). This differed between exposure level and study group.

After thoroughly univariate and multivariate inspection of design factors we finally used generalized additive models, which provide a high level of flexibility in modelling, to reduce white noise. Spline estimators for duration of measurement, date of survey, number of non-selected cells and position, such as the temporal progression of the long lasting investigation were performed were finally use gain adjusted measurements. More details of study groups, material preparation, performing and analysing the assay and deriving outcome measures are given elsewhere [26].

A modification of the primary outcome by age, sex and smoking was investigated by fitting semi-parametric linear models. For age and pack years a spline function with two degrees of freedom was fitted, for sex a single parameter. To avoid bias correlated to the status as lung cancer patients, indexpersons where excluded from the model fit.

Heritability was estimated for the basal damage (B), the initial radiation-induced damage (I), and all three DNA repair capacities (DRC) of all available two- and three-generation families. We used parent-offspring regression to estimate the nominal heritability [34,36,42]. Depending on the availability of parental outcome measures we performed mid-parent-offspring regression (if measures of both parents were available) or one parent-offspring regression (if measures of only one parents were available), where is the regression coefficient of a linear regression, adjusted for the offspring’s mean age and mean gender and the value of a human control to adjust for inter-assay variability. Both heritability estimates were than pooled by (weighted mean) with weights equal to the inverse of the estimated sample varianceand . The standard error of was calculated as [43].

Because the COMET assay is an experiment affected by numerous noisy or confounding factors [26], we estimated a falsity-correction for considering independent controls. All controls within a single trial were randomly grouped in pseudo-families, where the status as pseudo-parent or pseudo-offspring was assigned by chance. Heritability was estimated as described above in each of 3000 repetitions. The mean estimate was considered as falsity induced by study and assay design. Depending on the range of values for heritability spans from (instead of ) the final estimate is given by .

For internal validation we investigated spouse-correlation in terms of spearman’s rank correlation of lnOTM between partners and parents, siblings and the index person respectively, separately for B, I and DRC.