A Source-Record Audit of DeepSeek-R1 Responses to Romanized Sindhi Prompts

Abstract

Sindhi language models struggle with Romanized Sindhi as there is variation in the spellings of everyday words, and short prompts do not explicitly state the task or target language. In this study, the remaining source record of a small DeepSeek-R1 probe, which was based on seven canonical prompts and fourteen alleged A/B records, was audited. The screenshot captions were compared with the visible inputs and outputs, and duplication of images was checked. It was only relabeled when there was a single unambiguous canonical prompt that matched the visible input to a record. Task choice and exact target-string occurrence were then coded separately. Six records were auditable, three of which were recovered by relabeling; eight records were not auditable due to being duplicated, contaminated with a supplied answer, associated with a different phrase, or missing the user input or lost after label repair. Four of the six audited records adhered to the experimenter’s task, one provided a reasonably good English translation when the experimenter did not include the target language, and one resulted in an incompatible cross-linguistic reading. Five had the target (English/Sindhi) string and one did not. The screenshots do not demonstrate clear “standard” and “chain of thought” conditions: all audited B-labeled inputs are in the form of a simple prompt and there are no paired wrapper texts remaining. These are descriptive counts, not accuracy estimates and do not provide proof of causal benefit of prompt explicitness. The audit demonstrates the necessity of full prompt–response documentation, independently verified linguistic references, reruns, information-matched prompts, and preregistered evaluation plans in studies with low resources.

1. Introduction

Sindhi is an Indo-Aryan language, mainly spoken in the state of Sindh in Pakistan, and in parts of India. It is of cultural, administrative, and educational significance and is used in classrooms, government correspondence, and daily communication. With the emergence of smartphones and platforms like WhatsApp, Facebook, and X, the use of Sindhi has spread in informal online communication and digital cultural expression [1,2].

Still, Sindhi is under-resourced for natural language processing (NLP) as compared to high-resource languages like English and Mandarin. While mature pipelines are available for text classification, sentiment analysis, machine translation, and named-entity recognition in well-resourced languages, there are limited Sindhi pipelines [3]. The gap is not only in terms of data volume but also standardized corpora, evaluation protocols, and reusable research tools.

Another orthographic difficulty is Romanization. Many speakers prefer to write Sindhi in Latin script as it is more convenient on the regular keyboard. Informal Romanization is lossy, however, in that a single Romanized string can represent several different forms or readings in native script, and there is no standard spelling for informal usage. This diversity makes the identification of language, transliteration into Sindhi script, and translation into English difficult.

Large language models (LLMs) have a wide range of multilingual capabilities, and their responses are conditioned on the information provided in the prompt. A model can choose a task that the user does not intend if the intended operation or target language is not mentioned. This issue may be particularly severe with low-resource languages and non-standard Romanized inputs.

The meaning and function of a short Romanized Sindhi message are not always clear [4,5,6]. Clearly, one such example is the phrase “ma achan tho.” It may, by itself, prompt language identification, translation into English, translation into Sindhi script or be a request for clarification. Since the prompt does not specify a target, the lexical reading and the operation requested are both open.

Model outputs can be altered by prompt wording, without altering the model weights [7,8,9]. This does not mean that a prompt is a sign of a new reasoning ability. In this study, explicit wording changes only the task information that the model has access to, or it requests that the model explicitly state its interpretation. Figure 1 distinguishes prompt information from intrinsic inference, task selection, and output realization.

Figure 1. Prompt information, intrinsic model inference, task selection, and output realization.

1.1. Contribution of the Study

The methodological contribution of the study is the return to the source material that remains, and the differentiation between what the screenshots depict and what cannot be supported by the screenshots. This article does not show comparisons with other models, nor does it provide estimates of population-level accuracy. Specifically, the contributions are as follows:

• Canonical prompt set: P1–P7 are defined in one authoritative table, which replaces the inconsistent prompt lists in the earlier source material.
• Complete record audit: The visible input or defect is documented for each claimed record, along with the task, visible output, and provenance status and audit judgment.
• Separated outcomes: Task selection and occurrence of designated reference segment are coded separately.
• Controlled follow-up design: Future confirmatory study should involve verbatim information-matched prompts, repeated runs, independent an-notation, complete metadata, and paired analysis.

The study is designed to be small. It relies on seven canonical prompts and only assesses the source material that might be verified by explicit reference strings. The counts thus refer only to this record and not as general estimates of the performance of DeepSeek-R1 on Romanized Sindhi.

1.2. Research Objectives

We ask the following question: What information in a Romanized Sindhi prompt is related to the selected task and the generated output in the preserved DeepSeek-R1 screenshots? This study is an exploratory audit of the source record, and not an accuracy benchmark or a causal prompt word test. Below are our research objectives:

• To differentiate between intrinsic model inference and task information given by the user.
• To define a uniform set of P1–P7 prompts and identify which of the fourteen pairs of claimed prompts and model responses can be uniquely associated with the model responses observed.
• To describe separately task-selection behavior and output realization when requested operation or target is stated, omitted, ambiguous.
• To record evidence that is missing and to state testable hypotheses for a preregistered, repeated, information-matched and independently annotated confirmatory study

2. Related Material

2.1. Sindhi, Romanization, Low-Resource NLP

Research on Sindhi language technology involves corpus evaluation, optical character recognition, and example-based translation, for example, refs. [1,3,10,11], These which not only prove that computational processing is possible but also illustrate that there are still limited standardized resources. Work on other low-resource and social-media languages also shows that the degree of spelling variation and domain shift can significantly impact tagging and interpretation [12,13].

The written language is Sindhi, which is written in an extended Perso-Arabic script (Figure 2) and has 52 letters. The informal use of Romanization is understandable given the character inventory, context-sensitive forms of native scripts, and the standard Latin keyboards. There is also variation in digital Sindhi resources with respect to encoding, formatting, and corpus standardization [6].

Figure 2. Sindhi alphabet comprising 52 letters.

Romanized Sindhi is widely used in text messages. Variable spelling, incomplete sound-to-letter correspondence, and lack of a consistent informal standard are published descriptions [4,5]. Thus, a model can only operate on the operation and target information of a visible request; a short Romanized string alone may be too brief to define an NLP task.

Romanized Sindhi is also used in chat applications, websites, and informal translation in Sindhi-speaking communities [4,5]. A simple declarative sentence in English, Romanized Sindhi, and native Sindhi script is shown in Figure 3. The meaning is the same, but there are differences between the scripts (and the Romanized version has a different character inventory, and without context, there may be more than one plausible reading).

Figure 3. Cross-linguistic illustration of a declarative sentence in English, Romanized Sindhi, and native Sindhi script.

2.2. Prompt Explicitness and Reasoning-Capable Models

Previous studies have indicated that demonstrations, intermediate reasoning steps, and instruction structure can influence model outputs [7,14,15,16,17]. Studies on self-consistency, faithfulness, and reasoning without explicit rationale requests further show that internal computation cannot be interpreted as observable explanations [15,16,17]. Alternative, efficient, and safety-oriented reasoning procedures have also been recently investigated [18,19,20].

For some multi-step tasks, chain-of-thought prompting can help to obtain intermediate steps and enhance performance [16,17,18]. However, the screenshots saved for this study do not reveal the use of such a manipulation. The practical question is thus more limited: was there sufficient information in the visible request to select a defensible task?

Task selection and language-form realization should be assessed separately for a reasoning-trained system like DeepSeek-R1 [21]. One single interface response is not a stable capability and an explicit demand for interpretation is not an improvement in intrinsic reasoning.

3. Materials and Methods

3.1. Study Status and Retained Evidence

This audit is for the seven canonical prompts and fourteen claimed response records, which were previously marked with A or B. All that remains are screenshots of the previous manuscript. We added no new DeepSeek-R1 runs and did not infer or recreate any missing response.

A and B are legacy labels from the source files, which are retained. In all B-labeled auditable screenshots (Figures S9 and S11–S14 in Supplementary Materials), the user enters a bare prompt, and no step-by-step wrapper is seen. The evidence thus not only leaves the two conditions unverified but also indicates that the B-labeled requests are also bare inputs. Therefore, the study does not include a paired standard-versus-chain-of-thought comparison.

3.2. Canonical Prompt Set

The prompts P1–P7 are defined in the table below (Table 1). The “designated task” is the task that was originally planned in the study, but it is not necessarily the only sensible understanding of the wording. This is crucial for P1 and P2. The information in Figure 4 only reflects the task information explicitly stated in each prompt and not the difficulty or performance.

Table 1. Canonical Romanized Sindhi prompts and experimenter-designated tasks.

ID	Exact Base Prompt	Designated Task	Ambiguity Note
P1	ma achan tho	English translation designated by the original study	Neither an operation nor a target is stated; clarification is defensible.
P2	ma achan tho translate it	English translation designated by the original study	Translation is requested, but the target language is omitted.
P3	ma achan tho translate into Sindhi	Rendering in native Sindhi script	The target is stated, although “translate” blurs translation and transliteration.
P4	ma achan tho convert into Sindhi language	Rendering in native Sindhi script	The target is stated and “convert” signals a script transformation.
P5	ma achan tho: is a Romanized Sindhi language, translate it in English language	English translation	The source format and English target are both explicitly stated.
P6	ma achan tho convert into Sindhi	Rendering in native Sindhi script	The target is stated, but the operation remains somewhat underspecified.
P7	ma achan tho, this is a Romanized Sindhi prompt translate it into Sindhi language	Rendering in native Sindhi script	The source format and Sindhi target are explicitly stated.

Figure 4. Information explicitly presents in the canonical P1–P7 prompts.

3.3. Source-Record Verification

We examined four items of evidence for each anticipated prompt-label combination: figure caption, visible user input, visible model output, and any duplicated image. A caption was rejected if the screenshot itself denied the caption. We repaired a shifted label only if the visible input matched one canonical prompt and not any other. All remaining records were coded as not assessable. Figure 5 summarizes this rule.

Figure 5. Source-record inclusion and label-repair decision rule.

3.4. Two-Dimensional Evaluation Rubric

The audit first determines if a record is related to a prompt, and codes task selection and output realization separately (Table 2). Task selection is the operation and target that are used in the response. Output realization only requires a decision on whether the experimenter’s reference segment is present or not. Target strings in the original study were “I am coming” and “مان اچان ٿو”. The Sindhi Language Authority (SLA) has provided the word “مان اچان ٿو”. with its Romanization “maa’n aaoo’n acha’n tho” and translation “I come” [10]. This comparison demonstrates that “ma achan tho” is a short form of Romanization and that the English translation can vary depending on the context. Since the re-trained record is not annotated by a fluent speaker, the exact target-string occurrence is only described as a feature, but not as a translation-quality score.

Table 2. Operational rubric for provenance, task selection, and output realization.

Dimension	Category	Operational Rule	Interpretive Limit
Provenance	Uniquely auditable	Visible input and output identify one canonical item; no caption-only inference.	Eligible for descriptive task and output coding.
Provenance	Recoverable after relabeling	Visible input uniquely identifies an item despite a shifted source caption.	Eligible; correction recorded explicitly.
Provenance	Not assessable	Missing, duplicated, contaminated, caption-only, or about a different phrase.	Excluded from outcomes; retained among claimed records.
Task selection	Designated-task alignment	Response adopts the study’s designated operation and target.	Used only when the task is sufficiently specified.
Task selection	Reasonable alternative or clarification	Prompt permits more than one defensible target or a clarification request.	Not an error merely for differing from author intent.
Task selection	Incompatible interpretation	Response adopts a language, operation, or target incompatible with the visible request.	Descriptive category; not an accuracy estimate.
Output realization	Exact target-string occurrence	After trimming, the stated reference occurs as a standalone segment; explanation is allowed.	Not whole-response agreement or a complete quality metric.
Output realization	Non-exact visible output	Auditable response lacks the exact designated reference segment.	No broader semantic-adequacy claim.

We report counts and not prompt-level percentages. The intermediate judgments that were required to recreate the original percentages or binary item scores are not saved in the source record. Table 3 thus only adds the mutually exclusive record-level categories. Records are still not assessable, not redone as model successes or failures, but duplicated, contaminated, mismatched, incomplete.

Table 3. Record-level audit of all fourteen claimed prompt records.

ID	Record	Visible Input or Record Status	Designated Task	Selected Task and Visible Output	Provenance Decision	Audit Judgment
P1	A	Visible input: “ma achan tho”.	English translation; not uniquely recoverable from the wording.	Treats the expression as Telugu and supplies “What happened?” plus a Telugu reply.	Unique visible input-output pair.	Incompatible interpretation; non-exact output.
P1	B	Composite message contains P1, the supplied answer “I am coming”, and P2.	English translation; not uniquely recoverable from the wording.	Repeats an interpretation already supplied in the user message.	Reference-answer contamination.	Not assessable.
P2	A	The cited image duplicates the P1-A screenshot.	English translation; target omitted.	No distinct P2 output is preserved.	Duplicate image.	Not assessable.
P2	B	Visible input: “ma achan tho translate it”.	English translation; target omitted.	Selects English and returns “I am coming.”	Unique visible input-output pair.	Reasonable alternative consistent with the designated target; exact target-string occurrence.
P3	A	The screenshot concerns “Minhji Piyu Saan” (“with my father”).	Rendering in native Sindhi script.	Output concerns a different expression.	Phrase mismatch.	Not assessable.
P3	B	Visible input (Figure S11 in Supplementary Materials) matches the canonical P3 text, although the source figure was captioned P4-B.	Rendering in native Sindhi script.	Selects script rendering and includes “مان اچان ٿو”.	Recoverable after content-based relabeling.	Designated-task alignment; exact target-string occurrence.
P4	A	The screenshot concerns “Minhji Piyu Saan”.	Rendering in native Sindhi script.	Output concerns a different expression.	Phrase mismatch.	Not assessable.
P4	B	Visible input is the canonical P4 text, although the source figure was captioned P5.	Rendering in native Sindhi script.	Selects script rendering and includes مان اچان ٿو.	Recoverable after content-based relabeling.	Designated-task alignment; exact target-string occurrence.
P5	A	The screenshot concerns “Minhji Piyu Saan”.	English translation.	Output concerns a different expression.	Phrase mismatch.	Not assessable.
P5	B	Visible input is the canonical P5 text, although the source figure was captioned P6.	English translation.	Selects English and returns “I am coming.”	Recoverable after content-based relabeling.	Designated-task alignment; exact target-string occurrence.
P6	A	The cited image is duplicated and concerns a different phrase.	Rendering in native Sindhi script.	No unique P6 output is preserved.	Duplicate and phrase mismatch.	Not assessable.
P6	B	The source caption points to the image reassigned to P5-B; no P6 input is visible.	Rendering in native Sindhi script.	No unique P6 output is preserved.	Missing after label repair.	Not assessable.
P7	A	The response mentions “ma achan tho”, but the exact user input is absent and the answer shifts to an unrelated father/language discussion.	Rendering in native Sindhi script.	A script-like answer is attempted, but the exact request cannot be reconstructed.	Caption-only linkage is insufficient.	Not assessable.
P7	B	Visible input matches the complete canonical P7 prompt.	Rendering in native Sindhi script.	Selects script rendering, and includes مان اچان ٿوfollowed by explanation.	Unique visible input-output pair.	Designated-task alignment; exact target-string occurrence.

4. Item-Level Results

The audit of the fourteen claimed prompt–response records is shown in Table 3. It provides information for each record including the remaining evidence, the task assigned, the task and output displayed in the screenshot, the provenance decision, and the final audit judgment. (The full screen record is provided in Figure S1 of Supplementary Materials.)

4.1. Record Integrity

Six of the fourteen claimed records can be audited: P1-A, P2-B, P3-B, P4-B, P5-B, and P7-B. Three of these—P3-B, P4-B, and P5-B—were recovered because their visible inputs uniquely exposed shifted captions. The other eight records are not assessable because they are duplicated, the answer was supplied, they are associated with another phrase, the input is not visible or they were lost after label repair. Figure 6 shows all fourteen records, so that evidence of failure is not confused with the absence of evidence.

Figure 6. Auditability counts by retained legacy label.

4.2. Task Selection

In the six auditable records, P3-B, P4-B, P5-B and P7-B are the ones that follow the operation and target set by the experimenter. P2-B provides an English response, which aligns with the task, but the prompt does not specify the target language; it would also have been acceptable to seek clarification. However, P1-A takes an incompatible cross-linguistic reading. These findings are reported here as counts, rather than rates, in Figure 7 and Table 4.

Figure 7. Task-selection audit counts by retained legacy label.

Table 4. Count-only summary of the record-level audit in Table 3.

Record	Task Aligned	Reasonable Alternative	Incompatible	Not Assessable	Exact Target String	Non-Exact	Output Not Assessable
Record A (legacy label)	0	0	1	6	0	1	6
Record B (legacy label)	4	1	0	2	5	0	2
Total claimed records	4	1	1	8	5	1	8

4.3. Output Realization

There are five auditable B-labeled responses that include the exact English or Sindhi target string; P1-A does not. This is a deliberately limited check. It is not a measure of semantic equivalence, precision of the entire response, or linguistic accuracy. Explanatory text was permitted, each item was only observed once, and no independent linguistic annotation has survived. Figure 8 thus represents counts rather than rates.

Figure 8. Output-realization audit counts by retained legacy label.

5. Discussion

5.1. The Retained Evidence Supports

The evidence remaining is consistent with a small descriptive pattern. A prompt that specifies the operation and target is followed by the appropriate auditable response to that task. An incompatible reading can result from a bare phrase that names neither. P2-B is in between these cases: it requests translation, but does not specify the language to be translated, and an English answer is perfectly acceptable, although the language is not uniquely specified.

P1-A is helpful in understanding the dangers of the bare phrase. The model is trained with the string in Telugu. (Figures S5 and S7 in Supplementary Materials also examine the words “ma”, “achan”, and “tho” as Telugu, Kannada, or Malayalam words.) Such a pattern makes cross-linguistic segmentation a plausible source for incompatible interpretation, in addition to missing operation and target. The complete Romanization of the Sindhi language published by the Sindhi language authority is in the same direction [6].

5.2. What the Evidence Does Not Support

The one record left cannot prove that B is more correct than A. The visible B inputs are prompts, the paired wrapper texts are not present and the usable items are not identical between the two legacy labels. No item was repeated and most of the A-labeled records were missing/mismatched. Percentages, confusion matrices, or precision, recall, and F1 scores would therefore blend model behavior with missing data in the record and would deceive the experimentalist regarding control.

The study also does not suggest that the observed responses are consistent or that surface wording enhances DeepSeek-R1’s internal reasoning process, nor that the model would perform well on Romanized Sindhi in general. What evidence suggests is that there is a clear audit and a series of testable questions for a proper controlled follow-up study.

6. Controlled Confirmatory Study

6.1. Information-Matched Paired Prompts

Since the original paired prompts were not saved, Table 5 does not try to duplicate the original paired prompts. Instead, it provides prompts for a future study that are matched with information. Both versions maintain the original request and do not quietly select a missing target. The only difference that is planned is whether or not the model should say its interpretation before it provides an answer.

Table 5. Information-matched prompt pairs proposed for confirmatory testing.

ID	Direct-Answer Formulation	Interpret-Then-Answer Formulation
P1	Use only the information in this request. If the requested operation or target is unclear, ask one clarification question; otherwise return only the requested answer. Request: ma achan tho	Use only the information in this request. First state the source language, requested operation, and target language or script. If the operation or target is unclear, ask one clarification question and do not provide an answer. Otherwise provide the requested answer. Request: ma achan tho
P2	Use only the information in this request. If the requested operation or target is unclear, ask one clarification question; otherwise return only the requested answer. Request: ma achan tho translate it	Use only the information in this request. First state the source language, requested operation, and target language or script. If the operation or target is unclear, ask one clarification question and do not provide an answer. Otherwise provide the requested answer. Request: ma achan tho translate it
P3	Use only the information in this request. If the requested operation or target is unclear, ask one clarification question; otherwise return only the requested answer. Request: ma achan tho translate into Sindhi	Use only the information in this request. First state the source language, requested operation, and target language or script. If the operation or target is unclear, ask one clarification question and do not provide an answer. Otherwise provide the requested answer. Request: ma achan tho translate into Sindhi
P4	Use only the information in this request. If the requested operation or target is unclear, ask one clarification question; otherwise return only the requested answer. Request: ma achan tho convert into Sindhi language	Use only the information in this request. First state the source language, requested operation, and target language or script. If the operation or target is unclear, ask one clarification question and do not provide an answer. Otherwise provide the requested answer. Request: ma achan tho convert into Sindhi language
P5	Use only the information in this request. If the requested operation or target is unclear, ask one clarification question; otherwise return only the requested answer. Request: ma achan tho: is a Romanized Sindhi language, translate it in English language	Use only the information in this request. First state the source language, requested operation, and target language or script. If the operation or target is unclear, ask one clarification question and do not provide an answer. Otherwise provide the requested answer. Request: ma achan tho: is a Romanized Sindhi language, translate it in English language
P6	Use only the information in this request. If the requested operation or target is unclear, ask one clarification question; otherwise return only the requested answer. Request: ma achan tho convert into Sindhi	Use only the information in this request. First state the source language, requested operation, and target language or script. If the operation or target is unclear, ask one clarification question and do not provide an answer. Otherwise provide the requested answer. Request: ma achan tho convert into Sindhi
P7	Use only the information in this request. If the requested operation or target is unclear, ask one clarification question; otherwise return only the requested answer. Request: ma achan tho, this is a Romanized Sindhi prompt translate it into Sindhi language	Use only the information in this request. First state the source language, requested operation, and target language or script. If the operation or target is unclear, ask one clarification question and do not provide an answer. Otherwise provide the requested answer. Request: ma achan tho, this is a Romanized Sindhi prompt translate it into Sindhi language

6.2. Execution, Annotation, and Analysis

The minimum requirements for conducting, commenting on and reporting a confirmatory study are provided in Table 6. The proposed sequence is depicted in Figure 9, which ranges from prompt construction to preregistration, independent annotation, and paired analysis.

Table 6. Minimum design and reporting requirements for a confirmatory study.

Component	Required Specification
Dataset	Sample authentic Romanized Sindhi requests across translation, script rendering, spelling variation, and colloquial use. Determine sample size through an a priori power analysis.
Execution	Preregister prompts and outcomes; randomize condition order; use independent sessions; repeat each item; retain full prompts, responses, session IDs, dates, model builds, and decoding settings.
Annotation	Use at least two fluent Sindhi annotators. Score task selection and output realization separately, adjudicate disagreements, and report inter-annotator agreement.
Analysis	Use a paired analysis appropriate to the preregistered outcome, report confidence intervals and variability across runs, and treat chrF or another automatic metric as supplementary to human judgment.
Breadth	Replicate across reasoning-capable and general-purpose models before generalizing beyond DeepSeek-R1.

Figure 9. Proposed workflow for a controlled confirmatory study.

7. Limitations

• Incomplete provenance: Of the fourteen claimed records, eight cannot be uniquely associated with a prompt–response pair and the original wrapper texts are not available.
• Ambiguous intended task: P1 does not specify an operation or a target language, P2 requests translation without stating the target language. So, there are many possible interpretations of the two prompts, and the one that the experimenter had in mind is not necessarily the one that is correct.
• A single phrase, seen only once: All prompts are re-statements of the same expression, and all items retained were observed in just one run. Record does not demonstrate stability of response or prompt level variation.
• Unverified linguistic reference: One of the authors originally provided the strings to be designed. The Sindhi Language Authority has proposed the native-script form and provides a more complete Romanization [10] but there is no independent annotation from a fluent Sindhi speaker in the retained record. So, exact target-string occurrence cannot be used to determine linguistic quality.
• One model and partial settings: The record does not contain any information about the decoding settings, session identifiers, build, or run date of DeepSeek-R1.

8. Conclusions

The study’s defensible contribution is the transparent audit of fourteen claimed records, one consistent P1–P7 prompt set, and a clear distinction between task selection and output realization. There is no evidence to support the earlier percentages of accuracy, confusion matrices, or claims of improved or perfect reliability.

There are six records that can be audited. Four follow the experimenter-designated task, one provides a reasonable alternative (experimental target language is not available), and one uses an incompatible cross-linguistic reading. Five of them have the specified target string. These counts are based on an incomplete single-run record, and do not demonstrate that the differences observed were due to prompt explicitness. To answer that question, one needs to have a set of prerecorded, repeated, information-matched prompts, complete metadata, and independent annotation by fluent Sindhi speakers.