An Image-Based Approach to Measuring Human Values
Round 1
Reviewer 1 Report
I enjoyed reading the manuscript and learning about the authors' approach. I'm a psychologist; the overall writing and style appear unusual to me, but I understand that this is discipline specific. For example, I would have expected the hypotheses at the end of the introduction section.
The study is based on data from a huge sample, and the approach is original. I think there are some gaps in the Methods that need to be filled. Please find my specific comments below:
1. In the theoretical background and then throughout the article I was wondering what the authors mean by 'social values'. Focusing on Schwartz's model, I would think only values of conservation and self-transcendence are social.
2. Some of the approaches to values in the historical overview are outdated, and I wonder if the article should focus on recent and widely used approaches.
3. I like how the authors make the argument that values are related to behavior. The graph however appears too simplistic and does not reflect recent evidence. For example, there is a direct path from values to behavior (and reversed) without the need for attitudes.
4. The Oakland (1999) reference and approach is not well supported by evidence. I suggest presenting evidence from high-quality recent studies instead.
5. The authors argue that text-based measures take up too much time. What about the 21-item Portrait Values Questionnaire that takes about 10 minutes to complete? I'm wondering if the newly developed pictorial approach is meant to be projective or not?
6. The Method around how the pictures were chosen and what the pictures actually looked like could be much more detailed. Is it possible to show sample pictures? Was it photos or drawings? From Latvia?
7. The authors report that data from almost 500 respondents was excluded, but I did not fully understand why, respectively what 'outliers' means here.
8. What were participants asked to do? Choose their to 3 most important statements and pictures only? Was there any kind of rating or ranking involved?
9. The authors are aiming to make the point that respondents chose pictures that were similar to the statements in their value content. Were pictures selected to represent each of the presented text-based statements, and these are shown in the tables next to the text-based statements? I'm not sure what kind of data the authors have in the first place - just the information on which 3 pictures and which 3 statements each participant chose as their top 3? Or some rating or ranking as well? Chi-square is quite a weak statistic. If there is any other data - continuous or ordinal - this should be included as well. If not, I think it is important to show that the relationship between the pictures and the statements that should represent the same values is stronger than the relationship with the statements that represent different values. This would ideally be a correlation, but chi square could also work as a first proxy I think.
10. There are studies that show how text- and picture-based value measures converge, for example the Picture-Based Value Survey and the Portrait Values Questionnaire.
Author Response
Thank you a lot for your review; we have discovered quite a few areas where the article would benefit from an in-depth explanation. Therefore, we have uploaded a revised version which includes more detailed coverage of the areas you indicated, particularly being more detailed on Machine-Learning and methodology, where we had not been detailed enough.
Point-by-point replies are below (and see the revised text for the details):
1) Thank you for the suggestion - unless we quote, we have now replaced the terms used with 'human values' for consistency.
2) As we are proposing a novel way of measuring the values - a largely different from the former approaches - [a somewhat concentrated] review of the historical development of values research was deliberately undertaken. This help to assess the development of the thought and knowledge related to human values and explain why and what makes our approach different / explain that such an approach had not been taken before. We have encountered researchers referring to some of these older (and I agree - outdated, indeed!) concepts as topical ones where we always have to counter it - so this is done on purpose.
3) Agreed, this is a simplistic illustration which does not pretend to be universal. Therefore, we have expanded it here: L134-144 adding several sources that reveal a wider impact (including direct) of values.
4) While I might agree that Oakland is outdated, this is still one of the best-selling authors - our opponents usually quote him to emphasize that the image-based method is not the sole impactful modality. At the same time, Oakland still puts sight at the top of the senses, which is in line with NLP findings (estimating >70% of the population being visual modality driven), and we have added more recent sources (2021) to claim at least 60% of the population being driven by the impact of visual modality to support our claim on the visual impact being highly important.
5) The median test time of the pictorial approach is 4.8 minutes, and the mode of 4.15 minutes.
The newly developed approach (after this work) is not projective as questions are related to personal choice, i.e.: "Which image series do you personally associate the most with an accomplished, fulfilled life - not just in moments of leisure but in everyday life as well?"
6) Fair point - we have expanded the description of the method (see: 5.1 of the revised article), and we have included sample collages (see: Figure 2 on P.10, which is the actual screenshot from the survey).
The images are collages of four individual pictures from imagery uploaded by the tested individuals and personally associated by them with an accomplished life - i.e. they might be photos (most of them) or drawings (less frequently) upon their individual choice.
The study itself refers to the context of Latvia, and the images used are from the Latvian set. However, thus far, we have tested the approach in other markets (>20 geographic markets, taking into account the regional and cultural factors - e.g. primary language spoken in the household etc.), including the United States, United Kingdom, Lithuania, Estonia, Ukraine, and Russia. Each of the markets had been approached by in-depth interviews that included statement-based values tests and image-based tests [from sets of other markets] followed by user uploads to compile the country-specific sets and start the rotation of the images to train the ML solution.
7) Thank you, it is a precious suggestion: we have added a proper explanation in the updated article - see: L.480-483.
8) We have expanded the description of the technology - see 5.1.:
In short, the respondents were required:
8.1) in an image-based test:
- first, make a forced choice from the collages of pictures shown, choosing which of the presented collages are more associated by them personally with an accomplished life (Which image series do you personally associate the most with an accomplished, fulfilled life - not just in moments of leisure but in everyday life as well?") comparing images reflecting different value group;
- second, evaluate selected images from collages individually for the same point of reference to train the ML solution;
- finally, evaluate the overall quality of the results of the test based on the report extended to the respondent;
- they were invited to take a text-based survey [if they had not yet - we tried to switch the order of both tests and assess the impact]
8.2) in a text-based test:
- first, evaluate individual human values by their personal importance to them (grouping);
- second, select the most important values from the group of the values named as important to them (ranking);
- finally, they were asked to evaluate the report and take an image-based test [if they had not yet].
9) We added the explanation of the questionnaire construct and question series asked in the revised article.
We did not give any hint on the possible meaning of the images. We just asked respondents to select word-based value statements and select images they like in separate tests (no images and word-based values presented simultaneously). Associations among these two stimuli were determined via the Pearson chi-square test.
The choice of chi-square test was a conscious choice and recommended by other scholars, e.g. “The examination of cross-classified category data is common in evaluation and research, with Karl Pearson’s family of chi-square tests representing one of the most utilized statistical analyses for answering questions about the association or difference between categorical variables.” (The Chi-Square Test: Often Used and More Often Misinterpreted. Franke, T.M.; Ho, T.; Christie C.A. (2012). American Journal of Evaluation 2012 33: 448 (DOI: 10.1177/1098214011426594), available: https://aje.sagepub.com/content/33/3/448)
10) In the literature and from our 20+ years of experience, we have identified just The Picture-Based Value Survey for Children (PBVS-C) method as being truly image-based. However, it (a) has been used to measure children’s values; (b) the pictures deployed there are specifically developed instead of using user-sourced content such as images/photos from different real-life life pictures, scenes and setups - what we do.
The other method, the Portrait Values Questionnaire, contains text-based statements, which is different from our approach, where image-based and text-based tests are separated.
On top of that, we have revised the language to improve the style and eliminate minor mistakes, and references are revised.
Once again - we are grateful for your review, which has greatly improved the quality of our work.
Author Response File: 
Author Response.docx
Reviewer 2 Report
This is a very interesting paper with a logical structure that leads to a clear explanation of the research design and the results. Image-based research with the potential to be used in profiling consumers and employees to understand and even predict their behavior is an interesting concept and this research shows promising results. The paper is promoting a scientific and practical discussion about the method applied.
Author Response
Thank you a lot for your review; we have discovered quite a few areas where the article would benefit from an in-depth explanation. Therefore, we have uploaded a revised version which includes more detailed coverage of the areas you indicated, particularly being more detailed on Machine-Learning and methodology, where we had not been detailed enough.
On top of that, we have revised the language to improve the style and eliminate minor mistakes, and references are revised.
Once again - we are grateful for your review, which has greatly improved the quality of our work.
Author Response File: Author Response.docx
Reviewer 3 Report
Comments for the authors
The paper is quite interesting, the topic is topical and relevant. The theory behind the article is sound and clearly illustrated. However, the authors claim to have used Machine Learning (p. 2 line 53 “The instrument 53 deploys a set of machine-learning algorithms”), but I cannot understand where or how. This is a major flaw in the article; the authors should rewrite the 'Methodology' section explaining in detail: what data was analysed (the images themselves? the answers? the words?); what algorithms they used (I assume Neural Networks or Deep Neural Networks with images?); the software used. In addition, I have some other suggestions for improving the quality of the paper.
Introduction
|
|
Personal Values Behind Individual Motivation
- P. 2 line 90: “Values indirectly influence every decision an individual makes, mediated by his/her attitude towards the phenomena determined by those values.” The authors should add some reference.
Measuring Values Based on Images: Our Approach and Findings
This section is rather weak. The authors should rewrite it by answering the questions above (Did they use ML? How? On what data? With what software? etc.) and at least give an example of a question with images (single and collection) and a question with text. In addition, they should add a brief description of the continent of the images selected for each value (were they inspired by other studies?) or at least for the first three values in order of importance.
They should also add descriptive statistics for the sample (possibly in a table).
I also suggest avoiding the use of 'he/she' and prefer the pronoun 'they' when referring to persons of unknown gender.
The authors should also add a 'Discussion' section, before or within the conclusions, in which they compare their results with the existing literature on the subject.
Comments for author File: 
Comments.pdf
Author Response
Thank you a lot for your review; we have discovered quite a few areas where the article would benefit from an in-depth explanation. Therefore, we have uploaded a revised version which includes more detailed coverage of the areas you indicated, particularly being more detailed on Machine-Learning and methodology, where we had not been detailed enough.
Point-by-point replies are below (and see the revised text for the details):
1) Regarding your commment, "However, the authors claim to have used Machine Learning (p. 2 line 53 “The instrument 53 deploys a set of machine-learning algorithms”), but I cannot understand where or how. This is a major flaw in the article; the authors should rewrite the 'Methodology' section explaining in detail: what data was analysed (the images themselves? the answers? the words?); what algorithms they used (I assume Neural Networks or Deep Neural Networks with images?); the software used. ", we have:
a) added the revised article moving the description of the procedures to the suggested section (see: Chapter 5.1.);
b) updated the terminology - we used our own custom-developed iterative sequence procedures with set benchmark values, which can be referred to as a general ML algorithm. Though it might be assumed that the term ML algorithm refers to a readily available set of algorithms – NN, SVM, RF, DC and many others - they had not been used.
All our research is related to a project of the last five years to develop our own algorithms to streamline and automate the technology developed for 20+ years by us.
2) Regarding your comment "- P. 1 lines 32-40: the authors should add some reference to justify their statement.
Personal Values Behind Individual Motivation", we have:
a) revised the article;
b) provided a reference to the respective statement.
3) Regarding your comment "- P. 2 line 90: “Values indirectly influence every decision an individual makes, mediated by
his/her attitude towards the phenomena determined by those values.” The authors should
add some reference.", we have added the reference.
4) Regarding your comment, "This section is rather weak. The authors should rewrite it by answering the questions above
(Did they use ML? How? On what data? With what software? etc.) and at least give an example of a question with images (single and collection) and a question with text. In addition, they should add a brief description of the continent of the images selected for each value (were they inspired by other studies?) or at least for the first three values in order of importance.", we have:
a) significantly revised the article with an emphasis on explaining the ML approach taken by us (once again - see before: it is our own development rather than using any of OSS);
b) the sample images are added to the article as well to make it clear that the following approach is used:
- we did not give any hint on the possible meaning of the images. We just asked respondents to select word-based value statements and select images they like in separate tests (no images and word-based values presented simultaneously). Associations among these two stimuli were determined via the Pearson chi-square test.
In an image-based test:
- the images are collages of four individual pictures from imagery uploaded by the tested individuals and personally associated by them with an accomplished life - i.e. they might be photos (most of them) or drawings (less frequently) upon their individual choice. These images for the country-specific database used by ML algorithms to form collages and present them to the respondents;
- first, respondents make a forced choice from the collages of pictures shown, choosing which of the presented collages are more associated by respondents personally with an accomplished life (question: "Which image series do you personally associate the most with an accomplished, fulfilled life - not just in moments of leisure but in everyday life as well?") comparing images reflecting different value group;
- second, they evaluate selected images from collages individually for the same point of reference to train the ML solution;
- finally, evaluate the overall quality of the results of the test based on the report extended to the respondent;
- they were invited to take a text-based survey [if they had not yet - we tried to switch the order of both tests and assess the impact]
In a text-based test:
- first, evaluate individual human values by their personal importance to them (grouping);
- second, select the most important values from the group of the values named as important to them (ranking);
- finally, they were asked to evaluate the report and take an image-based test [if they had not yet].
The answers were analysed - see details in the revised article.
5) Regarding your comment, "The authors should also add a 'Discussion' section, before or within the conclusions, in
which they compare their results with the existing literature on the subject.", we highly appreciate your recommendation, and now there is a dedicated section added to the revised article.
On top of that, we have revised the language to improve the style and eliminate minor mistakes, and references are revised.
Once again - we are grateful for your review, which has greatly improved the quality of our work.
Author Response File: Author Response.docx
Round 2
Reviewer 3 Report
Thank you, now I can better understand your work, well done. The authors have improved the manuscript following all my suggestions.
