Prediction of Shampoo Formulation Phase Stability Using Large Language Models

Abstract

Predictive formulation can help reduce the number of experiments required to reach a target cosmetic product. The performance of Large Language Models from the open source Llama family is compared with that of conventional machine learning to predict the phase stability of shampoo formulations using a recently published dataset. The predictive strength is assessed for various train dataset sizes (obtained by stratified sampling of the full dataset) and for various Large Language Model sizes (3, 8, and 70B parameters). The predictive strength is found to increase on increasing the model size, and the Large-Language-Model-based approach outperforms conventional machine learning when the train dataset is small, delivering Area Under the Receiver Operating Curve above 0.7 with as few as 20 train samples. This work illustrates the potential of Large Language Models to further reduce the number of experiments required to reach a target cosmetic formulation.

1. Introduction

Predictive models hold the potential to accelerate and reduce experimental costs in the design of new formulations, which still remains, to a large extent, a trial-and-error process [1,2], with application not only in cosmetics [3,4], but also in pharmaceuticals [5,6] or oil and gas [7]. Phase stability is a primary requirement for cosmetic formulations, and for stable formulations, there are numerous other desirable physical [8], biological [9], and sensorial [10] properties. For the most part, there is no known mechanistic model that accounts for experimental results, and predictive models are thus based on statistical learning [11], also known as machine learning (ML), which requires training data and the choice of an algorithm.

Regarding training data, there is no golden rule to determine how much data is actually required to train robust ML models because the predictive strength depends on the strength of statistical associations between features (e.g., ingredients and their concentrations) and the target to be predicted (e.g., phase stability), which is not known a priori. However, all else equal, the higher the combinatorial complexity, the higher the amount of required data, and the combinatorial complexity increases with the number of ingredients mixed together in each formulation, and with the size of the library of candidate ingredients. In addition, when introducing new cosmetic ingredients (e.g., replacement of polymers derived from fossil fuels driven by the green transition [12], or replacement of controversial ingredients driven by regulatory guidance [13]), there may simply be no historical data available at all. Design-of-Experiment (DOE) approaches combined with a high level of laboratory automation can be used to generate data to train predictive models. Chitre et al. [8,14] have recently applied such an approach to generate a publicly available dataset for several hundred shampoo formulations. Even with a high level of automation, such data acquisition campaigns can prove lengthy and costly, so any modeling approach that could reduce the amount of required training data would be beneficial.

Regarding the algorithm, there is a wide range of candidates, with popular ones including Logistic Regression, Random Forest, and gradient-boosted decision trees such as the Light Gradient-Boosting Machine (LGBM) [15]. Besides such algorithms, which will collectively be referred to as conventional ML in the remainder of this article, a new approach based on Large Language Models (LLMs) has recently been proposed [16]. LLMs are a class of models underlying so-called generative AI. This new approach involves converting numerical tabular data into contextualized text, which is then fed into an LLM to make predictions on new samples. This new method has been shown to outperform conventional ML when the training data is small (typically a few tens of samples) [16,17], because it leverages not only statistical patterns in the training data, but also prior expert knowledge acquired through pre-training on massive data (typically, the whole Internet).

The present work evaluates the potential of this new LLM-based approach to predict the phase stability of cosmetic formulations using the above-mentioned shampoo formulation dataset from Chitre et al. [8,14]. State-of-the-art open source LLMs from the Llama family from Meta, with sizes of 3, 8, or 70B parameters [18], are selected. The predictive strength is assessed for various train dataset sizes (obtained by stratified sampling [19] of the full dataset) using the Receiver Operating Curve (ROC) Area Under the Curve (AUC) metric and is compared with that of conventional ML algorithms. The predictive strength is found to increase on increasing the LLM size, and the LLM-based approach outperforms conventional ML when the train dataset is small (fewer than 50 samples) and can deliver AUC above 0.7 with as few as 20 train samples. While this performance gain already illustrates the potential of this new LLM-based approach to reduce time and cost in the development of new cosmetic formulations, avenues for further improvement are discussed.

2. Materials and Methods

2.1. Data

The dataset published in [8] and further described in [14] was used in this study. It gives the composition and phase stability for 812 shampoo formulations, 294 of which were found to be stable. The original dataset also gives the viscosity and turbidity, as well as rheological properties for stable formulations, but only the phase stability was considered in the present work. Each formulation consists of a mixture of four ingredients in a base of water. The four ingredients are two surfactants (from a library of 12), one polyelectrolyte (from a library of four), and one thickener (from a library of two). The trade and INCI names of these ingredients along with their key chemical properties, as well as the full dataset, can be found on the figshare repository [20] associated with [8].

2.2. Data Sampling

One of the challenges of predictive modeling with very small datasets (i.e., in the range of a few tens of samples) is the limited ability to test trained models in a reliable manner. Standard cross-validation only uses a fraction of the data for each test round, and performance estimates are noisy because of the very small number of test cases. This limitation can be circumvented in the present case because the starting point is a much larger dataset with several hundreds of samples, so larger test sets can be used to evaluate trained models.

A generic method satisfying the following criteria was adopted in order to generate train/test splits: (i) compatibility with small train sizes in the range of 10–100 samples; (ii) use of test sizes that are larger than train sizes for reliable assessment of predictive model performance, but not as large as the complement of the full dataset in order to reduce the number of inferences and associated GPU usage when using LLMs. The adopted method consists of the following steps: First, the train size $TR$ and test size $TS$ are chosen so that the latter is a multiple of the former: $TS=k\times TR$. Second, a stratified sample [19] (stratified on the outcome to be predicted, which is phase stability) with size $TR+TS=(k+1)\times TR$ is randomly chosen from the full dataset. And third, this sample is randomly stratified into $k+1$ folds, out of which $n_{splits}$ ($n_{splits}\le k+1$) are randomly chosen as train folds, and for each such fold, the concatenation of the remaining k folds is used as a test set. This process results in an inverted cross-validation, and it was designed to work specifically with very small train sizes (smaller than test sizes). Standard scikit-learn and numpy libraries, initialized with the same random seed, were used for all random processes (sklearn.model_selection.train_test_split for initial stratified sampling of the full dataset, sklearn.model_selection.StratifiedKFold for splitting of this sample into $k+1$ folds, and numpy.random.choice for choosing $n_{splits}$ out of these). The following train and test size combinations were used: $TR=10$, $TS=50$; $TR=20$, $TS=80$; and $TR=50$, $TS=100$. For each train size /test size combination, $n_{splits}=3$ was used, and the entire process was repeated for three different random seeds, resulting in nine different train/test splits for each train size /test size combination.

2.3. Choice of LLM Models and Deployment

LLMs from the Llama family from Meta were selected. These open source models benefit from a fairly permissive license, including for commercial use [21], and can be accessed from the HuggingFace model repository [18]. The most recent models from the Llama 3 series were selected, with size 3B (Llama-3.2-3B-Instruct [22]), 8B (Llama-3.1-8B-Instruct [23]), and 70B (Llama-3.3-70B-Instruct-quantized.w8a8 [24]). Their knowledge cutoff date is December 2023, which means that during their pre-training, they were never exposed to the used dataset, which was published in July 2024. The original models from Meta were used for the 3B and 8B models, while an 8-bit quantized version of the 70B model was preferred for easier deployment and faster inference. These LLMs were deployed on Nvidia GPUs (single A40 for the 3B and 8B models, dual A100 for the 70B model) using the vLLM open source library [25,26].

2.4. LLM Prompting

There are two possible ways to use LLMs for predictions on tabular data: the first one involves using the contextualized text for train examples to fine-tune LLM weights, either all of them (full fine-tuning) or only a small subset using so-called Parameter-Efficient Fine-Tuning (PEFT) such as Low-Rank Adaptation (LoRA) [27] or (IA)³ [28]. The latter approach was adopted in the original proposal [16]. The second way, named In-Context Learning (ICL), involves using the LLM as is and including all training data in the prompt for every new prediction [29]. Fine-tuning requires fewer tokens per new prediction (a token is the basic unit into which words are split, with  0.75 word per token for most tokenizers used in current LLMs), which lowers the computational cost and time, at the expense of the prior fine-tuning step and associated hyper-parameter optimization. ICL requires more tokens per new prediction, which increases inference time, but it does not require any prior fine-tuning or hyper-parameter optimization. In the present study, ICL was used because even for the largest train size ($TR=100$), the number of tokens in the prompt remained well below the maximum context window limit of all used Llama 3 LLMs, which is 128k tokens.

2.4.1. Prompting with Full Context

The LLMs were prompted as shown in

Table 1. Example of a prompt submitted to the LLM, for $TR=10$. It consists of a series of messages: first, the system message giving the general context; then, a series of alternating user and assistant messages corresponding to the train examples; and finally a user message corresponding to the test sample, for which the LLM is invited to make a prediction.

Role	Message
System	You will be given weight concentrations of cosmetic ingredients from BASF mixed in water. Classify the phase stability of the mixture into one of the following categories: low, or high. Return only the name of the category, and nothing else. MAKE SURE your output is one of the two categories stated.
User	Plantapon ACG 50: 12.73 w/w%, Dehyton MC: 13.56 w/w%, Dehyquart CC6: 1.16 w/w%, Arlypon TT: 4.94 w/w%. Phase stability is ->
Assistant	low
User	Dehyton PK 45: 9.15 w/w%, Dehyquart A-CA: 9.38 w/w%, Salcare Super 7: 2.24 w/w%, Arlypon F: 4.14 w/w%. Phase stability is ->
Assistant	low
User	Plantapon ACG 50: 13.68 w/w%, Plantapon LC 7: 8.79 w/w%, Salcare Super 7: 1.35 w/w%, Arlypon F: 4.95 w/w%. Phase stability is ->
Assistant	high
User	Plantacare 818: 11.7 w/w%, Plantacare 2000: 8.48 w/w%, Luviquat Excellence: 1.22 w/w%, Arlypon TT: 3.95 w/w%. Phase stability is ->
Assistant	high
User	Plantacare 2000: 8.45 w/w%, Plantapon Amino KG-L: 8.76 w/w%, Luviquat Excellence: 1.7 w/w%, Arlypon TT: 3.06 w/w%. Phase stability is ->
Assistant	low
User	Plantapon ACG 50: 7.7 w/w%, Plantapon LC 7: 9.39 w/w%, Salcare Super 7: 2.71 w/w%, Arlypon TT: 2.59 w/w%. Phase stability is ->
Assistant	high
User	Dehyton MC: 11.07 w/w%, Dehyton PK 45: 12.64 w/w%, Salcare Super 7: 1.68 w/w%, Arlypon F: 4.88 w/w%. Phase stability is ->
Assistant	low
User	Texapon SB 3 KC: 4.18 w/w%, Dehyton PK 45: 10.24 w/w%, Dehyquart CC6: 1.26 w/w%, Arlypon TT: 4.09 w/w%. Phase stability is ->
Assistant	low
User	Plantapon ACG 50: 11.39 w/w%, Plantapon Amino KG-L: 9.27 w/w%, Dehyquart CC6: 2.03 w/w%, Arlypon TT: 4.14 w/w%. Phase stability is ->
Assistant	low
User	Dehyton MC: 9.13 w/w%, Dehyton ML: 9.8 w/w%, Luviquat Excellence: 3.08 w/w%, Arlypon TT: 2.59 w/w%. Phase stability is ->
Assistant	low
User	Plantapon ACG 50: 11.91 w/w%, Dehyton ML: 9.94 w/w%, Luviquat Excellence: 1.1 w/w%, Arlypon TT: 0.94w/w%. Phase stability is ->

. This prompt follows the standard chat structure for instruction-following LLMs (common abbreviated designation: instruct LLMs), which distinguishes between system, user, and assistant messages. Instruct LLMs are models that, after an initial pre-training phase which makes them strong next-token predictors, have been further trained to follow specific instructions using the chat format structure [29,30]. All messages shown in Table 1 are submitted to the LLM in a single prompt. The system message gives the general context and is followed by the training examples, each consisting of a user message giving the formulation composition followed by an assistant message giving the result of the phase stability test; the last message is a single user message giving the composition of the formulation test sample, the phase stability of which is to be predicted by the LLM. The outcome for stable (resp. unstable) formulations was encoded as “high” (resp. “low”) to avoid having to feed in the system message all experimental details of the phase stability assessment (performed after leaving samples 36 h in ambient lab conditions; see [8]).

2.4.2. Prompting Without Context

In order to differentiate between predictive strength arising from prior expert knowledge from predictive strength arising from inference of statistical patterns only, prompts were also run while keeping the same statistical patterns in the train data but removing all contextual information. This was achieved by (i) using a system message that sets the problem as a pure statistical inference task, (ii) replacing ingredient names with generic column codes, (iii) normalizing ingredient concentrations so that the average ingredient concentration over train samples is unity, and applying this normalization factor to the test data, (iv) removing all concentration units, and (v) replacing the “low” and “high” outcomes with “0” and “1”. The resulting prompt is shown in

Table 2. Prompt used when stripping away all context information, for the same example as shown in Table 1.

Role	Message
System	You will be given characteristics of a sample. Classify the sample outcome into one of the following categories: 0, or 1. Return only the name of the category, and nothing else. MAKE SURE your output is one of the two categories stated.
User	C1 is 2.21, C5 is 2.58, C13 is 1.63, C17 is 2.75. Outcome is ->
Assistant	0
User	C6 is 1.85, C11 is 3.33, C15 is 2.17, C16 is 1.93. Outcome is ->
Assistant	0
User	C1 is 2.38, C2 is 2.42, C15 is 1.31, C16 is 2.31. Outcome is ->
Assistant	1
User	C3 is 3.33, C4 is 2.5, C12 is 1.2, C17 is 2.2. Outcome is ->
Assistant	1
User	C4 is 2.5, C10 is 2.43, C12 is 1.68, C17 is 1.7. Outcome is ->
Assistant	0
User	C1 is 1.34, C2 is 2.58, C15 is 2.63, C17 is 1.44. Outcome is ->
Assistant	1
User	C5 is 2.11, C6 is 2.55, C15 is 1.63, C16 is 2.28. Outcome is ->
Assistant	0
User	C0 is 3.33, C6 is 2.07, C13 is 1.77, C17 is 2.27. Outcome is ->
Assistant	0
User	C1 is 1.98, C10 is 2.57, C13 is 2.85, C17 is 2.3. Outcome is ->
Assistant	0
User	C5 is 1.74, C7 is 3.33, C12 is 3.04, C17 is 1.44. Outcome is ->
Assistant	0
User	C1 is 2.07, C7 is 3.38, C12 is 1.09, C17 is 0.52. Outcome is ->

for the same example as shown with full context in Table 1.

2.5. Processing of the LLM Response

The response object returned by the LLM through the vLLM Application Programming Interface (API) is then parsed to extract (i) the prediction (“low” or “high” with context, “0” or “1” without context) and (ii) the corresponding $logprob$ (logarithm of its probability), which, in the most general case, is the sum of the logprobs of individual tokens making up the full returned message, but reduced to a single token for the specific outcome wording used in the present work. This $logprob$ is then exponentiated into a probability $p=exp\left(logprob\right)$, and the probability of the complement outcome is assigned as $1-p$. For all tested LLMs and all train/test folds, the returned message always exactly matched one of the two possible outcomes. This is consistent with the use of instruct LLMs (i.e., LLMs that have been trained to follow instructions) [29,30].

2.6. Benchmarking Against Conventional ML

Three popular conventional ML classification algorithms were selected: Logistic Regression (LR), Random Forest (RF), and Light Gradient-Boosting Machine (LGBM). For Logistic Regression and Random Forest, their scikit-learn implementation [31] was used, with default hyper-parameters, except for the class_weight parameter set to balanced to automatically adjust weights inversely proportional to class frequencies in the input data. For LGBM, the native implementation available as a Python package (lightgbm version 4.6.0) was used, also with default hyper-parameters. The rationale for not optimizing hyper-parameters is that this is prone to overfitting, especially when handling such small train sizes. Following [14], two data representations were tested: (i) one-hot encoding for all ingredients; and (ii) one-hot encoding for polymers only (polyectrolytes and thickeners) and featurization for surfactants, by using as features the relative amount of 13 chemical functional groups, in the two-surfactant mixtures, obtained by weighing the number of functional groups per surfactant molecule by the surfactant concentrations. LGBM failed to converge for the smallest train sizes, $TR\le 20$, for all train/test folds. The Receiver Operating Curve (ROC) Area Under the Curve (AUC) was used as predictive strength metric for both the LLM-based approach and conventional ML, and it was averaged over the nine folds (three folds per random seed times three different seeds).

3. Results

3.1. Conventional ML Benchmark

Figure 1 shows the average AUC as a function of train size for the different conventional ML algorithms (Logistic Regression, Random Forest, and LGBM) and for the different surfactant featurization schemes (one-hot encoding, or featurization of surfactants). The following can be seen:

• For each algorithm/featurization scheme combination, the AUC increases on increasing the train size, as expected.
• Regardless of the featurization scheme, the LGBM always performs worse than Logistic Regression or Random Forest, which may be explained by (i) the absence of hyper-parameter optimization and (ii) the small train sizes compared to reference benchmarks giving an advantage to LGBM over Random Forest, which have been typically reported for train sizes of thousands or tens of thousands of samples.
• Random Forest is better than Logistic Regression when $TR\le 20$, and the opposite holds when $TR\ge 50$ (which suggests a crossover for some $TR$ value between 20 and 50, and which was not further investigated).
• Overall performance is somehow comparable for the two-surfactant featurization schemes, with a slight advantage for “featurization” over “one-hot encoding” (yielding higher AUCs for 6 out of the 10 working algorithm/train size combinations).

Based on the above, LGBM is excluded from the benchmark, and Logistic Regression and Random Forest are kept, using the surfactant featurization scheme, for comparison with the LLM-based approach given in the next paragraph. This will be referred to as the conventional ML benchmark in the remainder of this article.

Figure 1. Average Area Under the Receiver Operating Curve (AUC) for conventional ML algorithms, using one-hot encoding or featurization of surfactants. LGBM failed to converge for the smallest train sizes $TR\le 20$ and is not shown.

Figure 1. Average Area Under the Receiver Operating Curve (AUC) for conventional ML algorithms, using one-hot encoding or featurization of surfactants. LGBM failed to converge for the smallest train sizes $TR\le 20$ and is not shown.

3.2. Benchmarking the LLM-Based Approach Against Conventional ML

Figure 2 shows the average AUC as a function of train size for the three tested LLM sizes (3B, 8B, and 70B) as well as for the conventional ML benchmark. Up to a train size of 50 samples, the LLM performance increases on increasing the model size, and the largest LLM (i.e., Llama 3 70B) performs better than conventional ML: Llama 3 70B with only $TR=10$ (respectively, $TR=20$) train samples approximately yields the same AUC as conventional ML with $TR=20$ (respectively, $TR=50$). This means that for these smallest train sizes, on average, the LLM-based approach requires ≈2× fewer train samples to achieve the same predictive strength as conventional ML. For the larger train sizes ($TR=50$ or $TR=100$), the LLM performance reaches a plateau (or even decreases for the smaller LLMs (Llama 3 3B and 8B)). The reason for this behavior is unclear: On the one hand, it may be hypothesized that the LLM is less capable of capturing the full prompt information for such long and repetitive prompts. On the other hand, even for $TR=100$, the prompt length remains below $8k$ tokens, which is well below the Llama 3 maximum context length (128k tokens for all three LLM sizes).

3.3. Investigating the Origin of the LLM Advantage

To determine whether the LLM advantage arises from prior expert knowledge or from a stronger ability to infer statistical patterns, the same LLM experiments were conducted with the data stripped from all context (see Section 2). Figure 3 shows the average AUC for the largest LLM (i.e., Llama 3 70B) with or without context, as well as for the conventional ML benchmark, as a function of train size. Removing the context lowers the AUC and brings it even below that of conventional ML. This suggests that the LLM advantage does not arise from being a better statistical learner, but rather from prior expert knowledge as context is key to performance.

4. Discussion

First, the question of whether the performance advantage is sufficient to justify using the LLM-based approach is discussed. Conventional ML should be preferred whenever the predictive strength is deemed sufficient for the targeted application, because (i) it is simpler and cheaper to deploy, with no need for GPU, (ii) inference is faster, and (iii) it is significantly less complex. However, the ≈2× advantage in required train size to achieve the same predictive strength may be sufficient to trigger industrial interest given the potential for formulation cost reduction and time-to-market advantage (owing to the reduced number of laboratory trials and errors to reach a target product). And there may be ways to further improve the performance of the LLM-based approach beyond this initial work. Fine-tuning could be investigated as an alternative to the In-Context Learning (ICL) used in the present work. Fine-tuning may also be used in combination with In-Context Learning using the method proposed by Chen et al. [32], which involves using the full ICL prompts (such as shown in Table 1) as training examples for fine-tuning. The ICL prompt itself might be optimized, either manually or automatically using the Declarative Self-Improving Python (DSPy) method [33]. Further extending this ICL prompt optimization path, Chain of Thought (CoT) is a method where the LLM is further trained to solve specific problems in steps, with a number of manually curated reference examples provided as the training set [34]. The LLM can then be presented with new yet-unseen questions and instructed to not just deliver the answer, but also the reasoning behind the provided answer. CoT has been shown to substantially improve the quality of the answers. While CoT might not be applicable to our present formulation classification task, its zero-shot version might: Kojima et al. [35] have shown that just prompting the LLM to not only provide an answer, but also a reasoning (but without any specific prior training on such reasoning examples), could significantly improve the LLM response.

Second, the possible origins of the prior expert knowledge are discussed, given the scarcity of published data in cosmetic formulation, especially compared to other scientific domains such as clinical science or biology. There is abundant ingredient information data from suppliers including some example formulations, and there also exist handbooks discussing formulation including phase stability [36], as well as do-it-yourself online resources, some of which include formulation examples with quantitative composition (versus mere ingredient list) [37]. Such resources may find their way into the LLM’s prior knowledge and contribute to helping make better predictions for new formulations. One way for cosmetic companies to circumvent this relative lack of knowledge about cosmetic formulation could be to develop their own LLM incorporating their proprietary know-how, by starting from an existing LLM and further training it on their entire formulation history, ideally not just including product information corresponding to successful formulations, but also all failed formulation attempts. Such modified LLMs should perform even better on the type of task described in the present work.

Third and last, a parallel is drawn between the LLM-based approach used in the present work and other pre-trained approaches, as the advantage over conventional ML lies in the abilities acquired through the pre-training process. There have been several proposals of pre-trained foundation numerical models [38,39,40,41], i.e., deep learning models that handle numbers and not text, and that have been pre-trained on a wide variety of tabular datasets, either generated synthetically [40,41] or corresponding to real-world cases [39]. On the one hand, tabular foundation models present the advantage of natively handling numbers and thus being able to better capture all statistical patterns in the data, whereas LLMs are prone to making calculation mistakes or to returning different results depending on the nature of the prompt [42,43]. On the other hand, they cannot benefit from contextual information (which can only be conveyed in text) or from a specific domain expertise, which is typically acquired through pre-training on mostly unstructured data from various sources. In the future, mixed text–quantitative models that could benefit from both advantages may emerge.