Transforming Petrochemical Safety Using a Multimodal AI Visual Analyzer ^†

Abstract

The petrochemical industry faces significant safety challenges, necessitating stringent protocols and advanced monitoring systems. Traditional methods rely on manual inspections and fixed sensors, often reacting to hazards only after they occur. Multimodal AI, integrating visual, sensor, and textual data, offers a transformative solution for real-time, proactive safety management. This paper evaluates AI models—Gemini 1.5 Pro, OPENAI GPT-4, and Copilot—in detecting workplace hazards, ensuring compliance with Process Safety Management (PSM) and DuPont safety frameworks. The study highlights the models’ potential in improving safety outcomes, reducing human error, and supporting continuous, data-driven risk management in petrochemical plants. This paper is the first of its kind to use the latest multimodal tech to identify the safety hazard; a similar model could be deployed in other manufacturing industries, especially the oil and gas (both upstream and downstream) industry, fertilizer industries, and production facilities.

1. Introduction

The petrochemical industry plays a critical role in modern economies, supplying raw materials for countless products and processes. However, this industry is fraught with significant safety risks due to the handling of hazardous chemicals, high-temperature processes, and large-scale machinery. Accidents in petrochemical plants can lead to catastrophic outcomes, including explosions, fires, environmental contamination, and loss of life. To mitigate these risks, stringent safety protocols and continuous monitoring systems are essential for preventing hazards and ensuring the safety of both personnel and equipment.

In recent years, there has been a growing emphasis on adopting robust safety management frameworks, such as the Process Safety Management (PSM) principles. Developed by organizations like OSHA and further refined by industry leaders such as DuPont, PSM focuses on preventing hazardous incidents by managing critical processes, operational practices, and safety procedures. DuPont’s safety culture, in particular, emphasizes proactive risk management, continuous safety improvements, and comprehensive audits. However, even with these principles in place, traditional safety monitoring methods in petrochemical plants have faced limitations [1].

Historically, safety monitoring has relied on manual inspections, fixed sensors, and software systems designed for task-specific hazard detection. These methods are often reactive, identifying risks only after they have manifested into serious issues. Moreover, human involvement in safety audits introduces subjectivity and inconsistencies, as different personnel may interpret safety violations differently. Given the complexity and scale of modern petrochemical operations, there is a need for a more comprehensive, real-time safety monitoring solution, capable of assessing both human behavior and equipment conditions, ensuring compliance with frameworks like DuPont’s Safety Management System and OSHA’s Process Safety Management guidelines.

Efforts to enhance safety monitoring have included the development of software systems that monitor specific aspects of plant safety, such as equipment temperature, pressure anomalies, or personal protective equipment (PPE) compliance. While these systems addressed certain safety concerns, they were often isolated and lacked the ability to offer a holistic view of plant-wide safety. Traditional machine learning (ML) models were typically trained to detect specific hazards, such as gas leaks or equipment failures, without understanding the broader context of how these incidents interact with overall plant operations [2].

The complexity of monitoring petrochemical operations presents unique challenges. A typical plant consists of numerous interconnected processes, each with its own set of safety risks. Ensuring compliance with both OSHA’s PSM principles and DuPont’s rigorous safety standards requires real-time, continuous evaluation of factors like equipment health, worker behavior, and environmental conditions.

Current monitoring systems often struggle to detect early signs of safety violations, especially when these violations arise from the interaction of multiple factors. For instance, a pressure increase in a pipeline might be manageable in isolation, but if combined with equipment fatigue or human error, it could lead to a serious safety breach. Moreover, traditional systems often fail to account for dynamic changes in plant conditions or evolving safety risks, necessitating more advanced solutions that can process complex data streams and anticipate potential hazards.

Artificial intelligence (AI) offers the potential to address these challenges by providing an integrated, intelligent approach to petrochemical safety monitoring [3]. Unlike traditional systems, AI can process vast amounts of data from diverse sources—visual feeds, sensor data, and textual reports—simultaneously, allowing for a more comprehensive understanding of safety risks. In particular, the recent advent of multimodal AI models, such as Gemini 1.5 Pro, OPENAI GPT-4, and Copilot, has paved the way for real-time proactive safety management in the petrochemical industry.

Multimodal AI combines different data types—such as video, images, text, and sensor readings—into a single, coherent framework. This allows the AI to identify patterns and correlations between data sources that traditional systems might overlook. For example, a multimodal AI system could analyze video footage of workers in conjunction with real-time sensor data to detect improper PPE use or hazardous operating conditions, flagging potential risks before they lead to incidents [4].

These AI systems also align with the PSM principles and DuPont safety monitoring protocols by ensuring that both operational and behavioral risks are managed systematically. By incorporating AI-driven safety monitoring systems, petrochemical plants can better adhere to the DuPont safety philosophy of proactive, preventive risk management. AI’s ability to continuously learn and adapt to new risks means that these systems can evolve in line with plant operations, providing an ever-improving safety infrastructure [5].

This paper is the first of its kind to explore the application of cutting-edge multimodal AI technologies to petrochemical safety monitoring, particularly in alignment with the PSM principles and DuPont’s safety monitoring frameworks. The research evaluates the performance of these AI models in identifying safety concerns, ensuring compliance with OSHA standards, and integrating seamlessly into existing safety protocols. The insights gained from this study could revolutionize the way safety is managed in the petrochemical industry, promoting a shift from reactive to proactive risk management practices.

2. Methodology

In this research, we aimed to develop and test multimodal AI systems for real-time safety monitoring in petrochemical environments. The process involved selecting state-of-the-art AI models, preparing domain-specific data aligned with OSHA standards and common high-risk tasks, defining a prompting strategy, and conducting rigorous testing to evaluate their accuracy in hazard identification. Applying large, generalist foundation models (like Gemini and GPT-4) to such a specialized, high-stakes domain requires specific adaptation techniques to ensure accuracy and relevance. The methodology therefore encompassed three key phases: (1) model selection and configuration, (2) dataset curation and domain-specific model adaptation, and (3) performance evaluation (testing/validation phase).

2.1. Selection of Multimodal AI Systems

To ensure robust results, three advanced multimodal AI systems were selected for comparison:

Gemini AI Developer Studio: Chosen for its flexibility in multimodal processing and rapid development capabilities.

OPENAI GPT-4 Visual Multimodal Technology: Known for its state-of-the-art understanding of both textual and visual inputs, which made it a strong candidate for identifying complex petrochemical hazards.

Copilot: A highly capable tunned GPT multimodal interface.

These models were accessed via their respective APIs or developer platforms. Configuration focused on utilizing their inherent multimodal capabilities without architectural modification by the team. Standardized parameters were used where applicable to ensure comparability during the evaluation phase. Figure 1 shows the architecture used for development.

2.2. Data Acquisition and Preparation

Generalist foundation models possess vast world knowledge but lack the specialized understanding of petrochemical processes, equipment, specific regulatory nuances (OSHA/DuPont), and the visual indicators of hazards unique to this industry. Bridging this gap requires adapting the models using domain-specific data. Two primary strategies exist for such adaptation: prompt engineering (including in-context learning) and fine-tuning.

Training data were gathered to simulate real-world safety scenarios in the petrochemical industry, focusing on high-risk tasks. These tasks, as shown in

Table 1. Training dataset distribution and hazard type.

Hazard Class	Number of Instances	Percentage (%)	Categorized Hazard Types Within Class
1. PPE Non-Compliance	63	25.2	Head Protection: No hard hat, damaged hard hat. Eye Protection: No safety glasses, improper eyewear. Hand Protection: No gloves, incorrect glove type. Respiratory Protection: No respirator, improperly worn respirator.
2. Confined Space Entry Hazards	38	15.2	Procedural Hazards: No entry permit displayed, lack of lockout/tagout. Atmospheric Hazards: No ventilation setup, missing gas detectors, obstructed entry/exit.
3. Working at Height Hazards	50	20	Fall Protection: No safety harness, improperly worn harness, damaged lanyard/lifeline. Equipment Hazards: Unsafe scaffolding, lack of guardrails, unstable ladder.
4. Hazardous Chemical Handling	33	13.2	Spillage/Leakage: Chemical spills, leaking containers, pooling liquids. Container Issues: Unlabeled containers, corroded drums, improper storage. Ventilation: Lack of ventilation in chemical areas.
5. Slip, Trip, and Fall (General)	29	11.6	Environmental Clutter: Cluttered walkways, tools/equipment obstructing paths. Surface Hazards: Spills on floors, wet/slippery surfaces, uneven flooring. Visibility: Poor lighting, obstructed view.
6. Hot Work/Fire Hazards	37	14.8	Ignition Sources: Open flames near flammables, sparks from welding/grinding, overheated equipment. Flammable Materials: Improper storage of flammables, combustible materials near heat sources. Emergency Access: Blocked fire exits, missing fire extinguishers.

, included the following:

• PPE incompliance;
• Confined space entry;
• Hot work;
• Working at heights;
• Handling hazardous chemicals;
• Slip, trip, and fall hazards (general).

Each AI model was trained on both OSHA safety standards and key concerns frequently encountered during these high-risk jobs.

Visual Data: A comprehensive dataset comprising 150 images and 100 video clips was compiled. A total of 80 percent of this data was selected or created to represent a wide spectrum of real-world petrochemical operations and safety scenarios, explicitly including examples of both safe practices and common hazards. Sources included publicly available industrial safety datasets, anonymized data from cooperating industry partners, and potentially staged scenarios illustrating specific risks relevant to the hazard categories in Table 1.

Textual Data: This included safety protocols, checklists, and OSHA standards related to specific tasks, offering additional context for the models to reference.

Sample Hazard Images for Training: Diagrams like the one shown in Figure 2 were used to identify hazards associated with tasks in confined spaces, such as improper use of safety harnesses or proximity to chemical leaks. Visual data were tagged to ensure the model was trained to recognize specific risk factors.

Model Adaptation Technique: The selected multimodal models were adapted using the curated training dataset. Depending on the specific model’s API capabilities and recommended practices, this adaptation likely involved supervised fine-tuning (adjusting model weights based on labeled examples) or few-shot prompting/in-context learning (providing extensive, structured examples within the prompt during inference to guide the model’s behavior). The goal was to imbue the models with domain-specific knowledge for accurately identifying petrochemical hazards based on learned patterns from the visual and textual training data.

2.3. Performance Evaluation

After training, the models underwent rigorous testing on pre-labeled datasets of both images and videos simulating safety violations in petrochemical sites. The test data were not part of the original training set, ensuring that the models could generalize beyond their training examples as shown in Figure 2 and Figure 3. Response for Figure 2 is given as a sample for different AI model in

Table 2. AI models’ hazard detection outputs in response to analyzing the input image shown in Figure 2.

Safety Concern	Gemini 1.5 Pro	GPT-4 (ChatGPT)	Copilot
Eye Hazard	Sparks and Flying Debris	Inadequate Eye Protection	PPE (General)
Respiratory Hazard	Metal Dust and Fumes	Metal Dust and Fumes	N/A
Noise Hazard	Chop Saw Noise	N/A	N/A
Fire Hazard	Sparks, Flammable Materials	Fire Hazard (Sparks)	Fire Safety Issues (General)
Ergonomic Hazard	Awkward Posture	Unsafe Work Position	Ergonomic Concerns (General)
Kickback Hazard	Kickback Hazard	N/A	N/A

.

Hazard Identification Test: A 20 percent of total sample dataset containing 150 images and 100 short/long videos depicting real-world petrochemical hazards was used. The models were required to detect the hazards and flag them in real-time.

Input Presentation: Each image or video clip from the evaluation dataset was presented to the adapted AI models.

Standardized Prompting: A consistent prompt structure was used for all models during evaluation to ensure fair comparison. The prompt requested the following:

Output Generation: The AI models generated textual outputs listing identified hazards, categories, and justifications.

Performance Assessment: The AI-generated outputs were systematically compared against the SME-defined ground truth for each test case. The evaluation focused on the following:

• Hazard Detection Accuracy (True Positives, False Positives): Did the model correctly identify hazards present in the ground truth? Did it incorrectly identify hazards that were not present?
• Hazard Miss Rate (False Negatives): Did the model fail to identify hazards that were present according to SMEs?
• Categorization Correctness: Was the identified hazard assigned to the appropriate category?
• Justification Quality: Was the model’s reasoning sound and did it align with safety standards?
• Overall Relevance: Was the output focused and free from irrelevant information?

Scoring and Reporting: The scores presented in

Table 3. Accuracy and limitation of visual analyzer model depending on hazard type.

Hazard Type	Gemini 1.5 Pro	GPT-4 (ChatGPT)	Copilot	Justification
1. PPE Non-Compliance	95	90	75	Gemini: Excellent at identifying specific PPE deficiencies (hard hat, eye protection, gloves, respiratory, boots, hi-vis vest). GPT-4: Very good, but slightly less detailed in PPE specifics across all images compared to Gemini. Copilot: Acknowledges PPE importance in its framework, but less direct in pinpointing specific PPE non-compliance in each image’s analysis. Framework is more general, less image-specific in initial responses.
2. Confined Space Entry Hazards	98	95	80	Gemini: Comprehensive understanding of confined space hazards. Identified harness, respirator (likely SAR), but also gaps like atmospheric monitoring, communication, permits, rescue plan. GPT-4: Strong on permit, ventilation, rescue, gas detection, harness—key confined space elements. Slightly less detail than Gemini. Copilot: Framework addresses “Working at Heights” more than “Confined Space” specifically in the initial output, but general safety categories are relevant. Less targeted hazard ID for confined spaces from the image directly.
3. Working at Height Hazards	90	85	70	Gemini: Good on ladder, mezzanine guardrails. Identified ladder instability, lack of fall protection on mezzanine. GPT-4: Focused on elevated platform, ladder use, overhead crane—relevant height aspects. Copilot: Category-based approach addresses “Working at Heights” as a general category, but less specific hazard identification from the warehouse image example provided. Less direct linkage to specific image elements in initial response.
4. Hazardous Chemical Handling	92	90	70	Gemini: Excellent in a Hazmat suit scenario, identified unknown hazards, drum integrity, handling procedures, decontamination. Also, linked fumes/dust in metalworking to respiratory hazards (though not “chemicals in containers” literally). GPT-4: Good on hazmat drum handling risks (manual handling, spill, contamination). Also flagged chemical/fire hazard in blacksmith image (open containers). Copilot: Framework mentions “Chemical and Biological Hazards” as a category, but less detailed in image-specific chemical hazard analysis. More about general principles than direct image interpretation.
5. Slip, Trip, and Fall (General)	88	85	75	Gemini: Identified clutter, materials on floor, uneven surfaces (uncovered pit). GPT-4: Good on trip hazards (scattered objects, debris), poor housekeeping. Copilot: Addresses “Slip, Trip, and Fall Risks” as a category, but less granular hazard identification from images. Framework helpful, but less direct image-based analysis in initial outputs.
6. Material Handling and Storage	95	90	N/A	Gemini: Excellent on stacked boxes (haphazard, mezzanine edge), unsecured pallet jack, forklift operation. GPT-4: Identified unsafe material handling, forklift safety. Copilot: Framework does not have a direct “Material Handling and Storage” category as explicitly, making direct scoring less applicable. While general framework touches on workplace organization, it is not as focused on material handling specifics in its initial responses.

are semi-quantitative metrics derived from the qualitative assessment. They represent the aggregate performance of each model for each hazard type across the entire evaluation dataset, reflecting the consistency and accuracy of hazard identification relative to the ground truth.

Figure 2 provides a specific example of a visual input presented to the AI models during the evaluation phase. Each model processed this image based on its internal representations adapted during the training phase and guided by the standardized evaluation prompt. An example of the user interface used for interacting with one such model, Gemini 1.5 Pro via its Developer Studio, is shown in Figure 3. Table 2 summarizes the resulting textual outputs generated by Gemini 1.5 Pro, GPT-4, and Copilot, detailing the specific hazards each model identified for the scenario depicted in Figure 2. The conceptual flow of how the AI model processes the visual input to arrive at these textual hazard descriptions is illustrated in the flowchart presented in Figure 4 and Figure 5. This overall comparison highlights the differences in detection capabilities, categorization, and descriptive detail among the models for a single test case.

3. Results and Discussion

This section presents the outcomes of the testing phase for multimodal AI systems and discusses their implications for petrochemical safety monitoring.

Performance Evaluation

The three AI models—OPENAI GPT-4 Visual Multimodal, Gemini 1.5 pro, and Copilot-4o—were tested on their ability to identify hazards in petrochemical environments using a predefined dataset of images and videos. The models were evaluated with the results summarized below in Table 3 and Figure 6. Limitation of the model is shown in a block diagram in Figure 7:

Analysis

• Gemini 1.5 Pro (95/100): Receives the highest score due to its exceptional accuracy, comprehensive hazard identification, nuanced understanding, and detailed analysis. It consistently demonstrated a strong ability to identify and articulate safety concerns effectively. The minor limitation of potential over-interpretation is often acceptable and even beneficial in a safety context as it errs on the side of caution.
• GPT-4 (ChatGPT) (90/100): Scores very high for accuracy and conciseness. It effectively identifies the primary safety concerns in a clear and actionable manner. While slightly less detailed than Gemini, its ability to provide focused and relevant safety information is highly valuable. It is a very strong performer in hazard identification.
• Copilot (70/100): Scores lower in direct comparison to Gemini and GPT-4 for image-based hazard identification in these specific examples. While it offers a useful and structured approach to thinking about safety through categories, it is less effective at autonomously identifying specific hazards directly from the images. Its strength lies more in providing general safety frameworks and advice, requiring more user interaction to pinpoint image-specific risks. If the goal is to achieve a quick, automated hazard assessment from an image, it is less efficient in its out-of-the-box behavior compared to the other two models based on these examples.

4. Practical Implications

While the AI models show promising hazard detection capabilities, realizing their full potential requires careful integration into existing petrochemical plant operations. This section details the key technical, operational, and organizational considerations for successful implementation.

Technical Considerations:

Integrating AI requires robust APIs and middleware to connect diverse plant systems like legacy software, sensors, and video feeds. Sufficient computing power, potentially cloud-based, and network bandwidth are crucial for real-time AI processing. System architecture should be scalable and reliable, considering cloud, edge, or hybrid deployment. Cybersecurity is paramount to protect data and system integrity. Ongoing maintenance, including model updates, is essential.

Operational Considerations:

AI alerts must integrate smoothly into existing workflows like incident response and inspections, augmenting human efforts. A timely alert system is vital, delivering prioritized hazard notifications to relevant personnel. Training is key to build human–AI collaboration and trust in AI-generated alerts. Addressing alert fatigue through refined AI sensitivity and continuous performance monitoring and calibration are needed in real-world plant conditions.

Organizational Considerations:

Successful AI integration requires change management to address resistance and highlight AI benefits. Leadership commitment is crucial to champion AI and strengthen a data-driven safety culture. Compliance with regulations like OSHA and PSM is essential, ensuring data privacy and system reliability. Resources for AI infrastructure, training, and maintenance need to be allocated, and the ethical aspects addressed to ensure responsible AI deployment.

The application of multimodal AI in petrochemical safety monitoring has the potential to achieve the following:

• Reduce accidents and near-miss incidents through real-time detection.
• Ensure compliance with OSHA standards and enhance the implementation of DuPont’s process safety guidelines.
• Improve operational efficiency by minimizing downtime related to safety incidents, as early detection can lead to quicker mitigation efforts.

Further work is needed to refine the accuracy of the models in complex scenarios and improve their integration into existing safety management systems.

5. Conclusions

The application of multimodal AI in petrochemical safety monitoring marks a significant advancement over traditional methods. The study demonstrated the superior accuracy and real-time hazard detection capabilities of OPENAI GPT-4 Visual Multimodal model and the highest for the Gemini 1.5 pro, while Copilot showed potential but requires improvement in visual hazard recognition.

By adhering to OSHA standards and DuPont Safety Principles, these systems enhance Process Safety Management (PSM), ensuring consistent procedural compliance and risk mitigation. Their ability to detect hazards in real time has the potential to significantly reduce incident rates, improve operational safety, and ensure a more robust safety culture in the petrochemical industry.

As a first-of-its-kind implementation of advanced multimodal AI in this domain, the study highlights the transformative power of AI technologies. Future development should focus on refining AI models’ performance in complex, multi-faceted industrial scenarios including fertilizer and oil and gas industries, further integrating them into comprehensive safety management systems. This will pave the way for safer and more efficient petrochemical operations, reducing risks to both personnel and the environment.