Abstract
Generative artificial intelligence (AI) is part of chemistry students’ study routines, but little is known about how such tools handle course problems or how students judge their ability to check AI output. In May 2025, free versions of ChatGPT (GPT-4o), Gemini (2.5 Flash) and DeepSeek (V3) each solved 42 inorganic chemistry problems once. On 38 problems with verified statements, 109 of 111 substantive responses were correct. Re-verification revealed naturally occurring defects: on three problems with inconsistent or incomplete statements, none of the eight substantive responses reached the intended solution or stated that the problem could not be solved as written; most noted an inconsistency or missing datum but still answered. A subsequent survey of 570 students at three universities in Southern Kazakhstan showed near-universal ChatGPT use (96.8%); 61.6% rated themselves as able to tell whether AI information in chemistry is correct. This self-rating behaved like an AI self-efficacy item: its association with reported AI use (ρ = 0.59) largely disappeared after adjustment for self-efficacy and attitudes (partial ρ = 0.12). These exploratory findings, limited to this problem set and these versions, suggest that checking the consistency of problem statements deserves attention in chemistry teaching and that students’ verification ability should be tested directly.