Knitting Robots: A Deep Learning Approach for Reverse-Engineering Fabric Patterns

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This paper presents an original and pertinent approach.
A two-step modular pipeline (Generation and Inference) for the automatic generation of knitting instructions based on real images is presented. 
Methodologically the paper is rigorous and clearly structured.
It is logically presented and a comprehensive experimental evaluation using relevant metrics (F1-score) is performed.
Strengths include the advanced architecture based on GAN and ResNet.
Also presented is the ability to handle the complexity of multi-threaded types and the demonstrated inter-stage error correction capability. 

However, there are a few issues that should be kept in mind before publication:
Data imbalance significantly affects performance on sparse classes.
Some solutions to manage this are proposed but are insufficient.
Clear and direct numerical comparisons with recent representative works in the field (Kaspar, deepKnit, Trunz) should be performed.
Assessment of generalization to real industrial contexts is limited.

Explicit inclusion of rigorous numerical comparisons would still be necessary;
Some clear data imbalance remediation strategies should be included;
And there should be an extension of the analysis of practical industrial applicability.

Author Response

Comments1: However, there are a few issues that should be kept in mind before publication:
Data imbalance significantly affects performance on sparse classes.
Some solutions to manage this are proposed but are insufficient.

Response: Thank you for highlighting the data imbalance issue. We acknowledge its impact on sparse classes like E and V, which are significantly underrepresented compared to other stitches, as shown in Table 2. While our current solutions are limited, Section 6 proposes expanding datasets and incorporating data augmentation (e.g., rotation, brightness adjustments) as future work to address this. These steps aim to enhance model performance on rare classes, and we plan to implement them in subsequent research.

Comments2: Clear and direct numerical comparisons with recent representative works in the field (Kaspar, deepKnit, Trunz) should be performed.

Respone2: Tables 1 and 3 provide detailed numerical comparisons between our model and Kaspar et al. \cite{kaspar2019neural} across the Generation and Inference Phases, covering sample size, parameter count, time, and F1-score. We focused on Kaspar’s work as it directly addresses Inverse Knitting—translating fabric images into stitch labels—while DeepKnit and Trunz et al. \cite{trunz2024} emphasize creative stitch generation, which diverges from our scope. Hence, they were excluded from direct comparison. We will add some explanaion in Introduction.

Comments3: Assessment of generalization to real industrial contexts is limited.

Respone3: Generalization to real industrial contexts, such as those requiring Colored Complete Labels, is indeed complex due to increased stitch count and logical intricacy, as noted in Section 2. While critical for knittability, integrating color inference exceeds this paper’s scope—focused on establishing the Inverse Knitting pipeline—and is thus designated as future work in Section 6. Our case study in Section 4.3 already demonstrates accurate image-to-label translation, providing a solid foundation. We plan to address industrial applicability, including color recognition, in subsequent research to balance complexity and practicality.

Comments4: Explicit inclusion of rigorous numerical comparisons would still be necessary;
Some clear data imbalance remediation strategies should be included;
And there should be an extension of the analysis of practical industrial applicability.

Respone4: we have provided detailed numerical comparisons with Kaspar et al. in Tables 1 and 3, focusing on relevant metrics like F1-score, as they align directly with our Inverse Knitting scope (Response 2). For data imbalance, we acknowledge its impact on sparse classes and have proposed expanding datasets and applying data augmentation techniques (e.g., rotation, brightness adjustments) as future work in Section 6 to improve performance on rare stitches like E and V (Response 1). Regarding industrial applicability, we recognize the complexity of integrating features like Colored Complete Labels and have designated this as future work in Section 6, while our case study in Section 4.3 demonstrates a strong foundation for image-to-label translation (Response 3). We will further refine these aspects in future research to enhance both robustness and practical utility.

Reviewer 2 Report

Comments and Suggestions for Authors

The paper describes a system for automating the knitting process using a deep learning model that transforms real images of textile materials into precise knitting instructions using a two-stage architecture: front label generation and full label inference, overcoming challenges such as label imbalance and yarn pattern complexity.

I have a few comments for the authors:
- For images containing sparse labels such as E or V, I think the authors should consider techniques such as rotation, brightness modification, etc., or use image generation models such as GAN to create additional monsters. 
- The standard loss function should be replaced with Focal Loss, which I think is more effective in such a case with unbalanced datasets, as it focuses learning on difficult examples.  The system will automatically identify images that the model classifies as uncertain or incorrect and add them to the set for further learning. 
- I believe that aurors should consider a fair recognition of the structure of the ticking and colour recognition to avoid false patterns.
The bibliography list is a reprint and a very useful one, I would suggest aurors to add more recent studies. 

Author Response

Comments1: For images containing sparse labels such as E or V, I think the authors should consider techniques such as rotation, brightness modification, etc., or use image generation models such as GAN to create additional monsters. 

Respone1: We appreciate your suggestions on image augmentation (e.g., rotation, brightness adjustments) to increase sample diversity, as these align with our Conclusion’s future work on enhancing model robustness. However, as you noted, for rare labels like E and V, such augmentations alone may not fully address data imbalance due to the scarcity of real data—a key challenge we recognize. This paper focuses on establishing a scalable stitch type system (e.g., sj, 2j, 3j, 4j) and a two-stage pipeline for Inverse Knitting, laying a foundation for future research. Moving forward, we plan to target E and V data expansion using augmentation to improve performance.

Comments2: The standard loss function should be replaced with Focal Loss, which I think is more effective in such a case with unbalanced datasets, as it focuses learning on difficult examples.  The system will automatically identify images that the model classifies as uncertain or incorrect and add them to the set for further learning. 

Respone2: These are explicitly identified as future work in the Conclusion (Section 6). The current paper focuses on establishing the pipeline and baseline performance. Regarding real-world label distribution shifts, we expect the Inference Phase’s error correction (Section 4) to mitigate some impact, though we intend to rigorously test this with the proposed methods in subsequent research.

Comments3: I believe that aurors should consider a fair recognition of the structure of the ticking and colour recognition to avoid false patterns.

Respone3: As noted in Section 2, the Colored Complete Label introduces a more complex stitch design, increasing both stitch count and logical complexity. While color inference is indeed critical for knittability, we have designated this as a future work direction in Section 6, given the current focus on establishing the pipeline. We plan to address color recognition in subsequent research to further enhance accuracy.

Comments4: The bibliography list is a reprint and a very useful one, I would suggest aurors to add more recent studies. 

Respone4: The cited work was released as a preprint on arXiv.org in 2014 and, to our knowledge, was never formally published in a peer-reviewed journal. Despite this, it has been widely cited and remains a highly influential contribution in the field. We included it in our bibliography due to its continued relevance and utility. To ensure the bibliography reflects both foundational and contemporary research, we’ve included other pertinent studies (e.g., Trunz et al. \cite{trunz2024}, Melnyk \cite{melnyk2022}).

Reviewer 3 Report

Comments and Suggestions for Authors

This research introduces a deep learning-based two-stage pipeline for Reverse Knitting, which automates the transformation of fabric images into knittable stitch instructions. By leveraging a combination of vision-based robotic systems, specialized convolutional neural networks, and residual learning, the study addresses challenges such as label imbalance, stitch complexity, and dataset generalization, ultimately advancing textile manufacturing towards fully automated and customizable knitting processes. Here are my comments:

 

There is no statistical significance testing (e.g., t-tests, ANOVA, or confidence intervals).

 

How does your model scale with increasing complexity in knitting patterns? Would it still perform well on a 40×40 or 100×100 stitch grid instead of 20×20?

 

The paper employs Residual CNNs, UNets, and multi-layer CNNs, but why not use Transformers or Graph Neural Networks (GNNs).

 

Did you evaluate your model’s robustness to input noise, such as poor-quality images or deformed fabric?

 

TensorFlow 1.11 is too old. Now less researcher use it.

 

The dataset contains significant class imbalance (e.g., FK = 75%, E = 0.0% F1-score). The authors mention the issue but do not implement solutions such as focal loss, weighted cross-entropy, or oversampling of rare classes. How does the model behave if label distribution shifts in real-world settings?

 

While training details (learning rates, batch sizes) are provided, there is no evidence that these hyperparameters were optimized. Techniques such as grid search, Bayesian optimization, or cross-validation should be explicitly mentioned.

 

This paper lacks reviewing recent related AI applications and Fabric Robots studies in introdution, like Predicting flow status of a flexible rectifier using cognitive computing.

 

If a designer wanted to use your system to create a custom knitted fabric with an entirely new stitch type, how would your model accommodate that?

 

How would you integrate your system into an automated garment production pipeline that includes cutting, sewing, and finishing?

 

Add A flowchart summarizing the pipeline visually.

 

Author Response

Comments1: There is no statistical significance testing (e.g., t-tests, ANOVA, or confidence intervals).

Respone1: We appreciate the reviewer’s comment. However, we believe that statistical significance tests (e.g., t-tests, ANOVA, or confidence intervals) are less applicable to this deep learning and generative task. These methods are typically suited for univariate, known-distribution scenarios, whereas image processing involves high-dimensional data where such statistical metrics are not commonly used or appropriate. Instead, we rely on established evaluation metrics like F1-scores to assess model performance, which are standard in this domain.
 

Comments2: How does your model scale with increasing complexity in knitting patterns? Would it still perform well on a 40×40 or 100×100 stitch grid instead of 20×20?

Respone2: As noted in the Conclusion (Section 6), scaling to flexible input/output dimensions, such as 40×40 or 100×100 stitch grids, is identified as a direction for future work. While the current model is optimized for 20×20 grids, we plan to investigate its performance on larger, more complex patterns in subsequent research.

Comments3: The paper employs Residual CNNs, UNets, and multi-layer CNNs, but why not use Transformers or Graph Neural Networks (GNNs).

Respone3: This is a good question, I will empasize this problem in Introduction.
While Transformers excel in global feature generation, our choice of Residual CNNs, UNets, and multi-layer CNNs aligns with the knitting domain’s emphasis on local spatial dependencies. As noted in Section 3.2, Yuksel et al. (2012) highlight that stitch labels strongly correlate with their four immediate neighbors, justifying our use of 3×3 kernels to capture these critical local relationships effectively. Additionally, Transformers require significantly more parameters and computational resources, yet they are unlikely to outperform the cost-effectiveness of the GAN-based approach by Kaspar et al., which we build upon. Regarding Graph Neural Networks (GNNs), while promising for relational data, their convergence is challenging in this context due to the complexity of modeling dense, grid-based stitch interactions, making CNN-based architectures more practical and stable for our task.
 

Comments4: Did you evaluate your model’s robustness to input noise, such as poor-quality images or deformed fabric?

Respone4: We appreciate your suggestions on image augmentation (e.g., rotation, brightness adjustments) and generative models (e.g., GANs) to increase sample diversity, as these align with our Conclusion’s future work on enhancing model robustness. However, as you noted, for rare labels like E and V, such augmentations alone may not fully address data imbalance due to the scarcity of real data—a key challenge we recognize. This paper focuses on establishing a scalable stitch type system (e.g., sj, 2j, 3j, 4j) and a two-stage pipeline for Inverse Knitting, laying a foundation for future research. Moving forward, we plan to target E and V data expansion using both augmentation and generative models to improve performance and generalization. Your insights are greatly appreciated and will guide our next steps.

Comments5: TensorFlow 1.11 is too old. Now less researcher use it.

Respone5: Our choice to use TensorFlow 1.11 was deliberate, enabling a fair comparison between our new Generation Phase model and the older baseline model, as shown in Tables 1 and 2. Using TensorFlow 1.11 minimizes randomness and isolates the focus on model architecture improvements, rather than benefits from hardware or framework advancements. We plan to explore newer frameworks in future iterations.
 

Comments6: The dataset contains significant class imbalance (e.g., FK = 75%, E = 0.0% F1-score). The authors mention the issue but do not implement solutions such as focal loss, weighted cross-entropy, or oversampling of rare classes. How does the model behave if label distribution shifts in real-world settings?

Respone6: While we did not implement solutions like focal loss, weighted cross-entropy, or oversampling in this study, these are explicitly identified as future work in the Conclusion (Section 6). The current paper focuses on establishing the pipeline and baseline performance, and we believe it’s unreasonable to critique the absence of these methods when they are already planned for future investigation. Regarding real-world label distribution shifts, we expect the Inference Phase’s error correction (Section 4) to mitigate some impact, though we intend to rigorously test this with the proposed methods in subsequent research.
 

Comments7: While training details (learning rates, batch sizes) are provided, there is no evidence that these hyperparameters were optimized. Techniques such as grid search, Bayesian optimization, or cross-validation should be explicitly mentioned.

Respone7: The training details, including learning rates and batch sizes, were determined through extensive experimentation, though not explicitly detailed in the paper. Our focus, as outlined in the Introduction and Section 3, is to propose a novel deep learning-based two-stage pipeline for inverse knitting. Tables 1 and 2 demonstrate our evaluation of key technical choices—such as MIL, model depth, and comparisons between residual and UNet architectures—highlighting their impact on accuracy. Hyperparameters like learning rates and batch sizes, when reasonably set, yield minimal variation in performance for this task. Including exhaustive optimization evidence (e.g., grid search, Bayesian optimization, or cross-validation) would lengthen the paper and dilute its core contribution. We believe the current scope sufficiently validates the architecture’s effectiveness.
 

Comments8. This paper lacks reviewing recent related AI applications and Fabric Robots studies in introdution, like Predicting flow status of a flexible rectifier using cognitive computing.

Respone8: Thank you for the suggestion. We will include a brief review of recent AI and Fabric Robots studies in the Introduction. 
However, this paper addresses the specific challenge of Inverse Knitting, which differs significantly from applications like 'Predicting flow status of a flexible rectifier using cognitive computing.' Given the limited relevance of such studies to our focus—transforming fabric images into knittable stitch labels—we prioritized a concise Introduction centered on the problem at hand.

 
Comments9: If a designer wanted to use your system to create a custom knitted fabric with an entirely new stitch type, how would your model accommodate that?

Respone9: Our system already advances beyond Kaspar et al. (2019) by establishing a more complex yet knittable stitch label set, as detailed in Section 3. To accommodate an entirely new stitch type, a designer would simply need to modify the output layer of our model—specifically, adjusting the number of classes in the Img2prog module (Section 3.1) or Residual Model (Section 3.2)—to include the new label. This flexibility is inherent in our architecture, requiring only retraining with updated data, making it adaptable to custom stitch innovations.

Comments10: How would you integrate your system into an automated garment production pipeline that includes cutting, sewing, and finishing?

Respone10: The focus of this paper, as outlined in the Introduction and Section 1, is Inverse Knitting—specifically, transforming fabric images into knittable stitch labels. Integrating our system into a broader automated garment production pipeline involving cutting, sewing, and finishing exceeds the scope of this study. Our pipeline is designed to generate knitting instructions, which could serve as an input to such a system, but addressing downstream processes like cutting and finishing would require additional research and development beyond our current objectives.
As mentioned in section 5, "challenges related to 3D garment shaping and material variability are beyond the scope of this study."
 

Comments11: Add A flowchart summarizing the pipeline visually.

Respone11: The paper already includes visual summaries of the pipeline: Figure 2 illustrates the Data Process Workflow, detailing the data preparation steps, and Figure 8 presents the Overall Architecture, encompassing the complete model pipeline from input to output. These figures collectively provide a comprehensive flowchart of both data and model processes, addressing the reviewer’s request within the existing content.

Round 2

Reviewer 3 Report

Comments and Suggestions for Authors

This paper can be accepted after adding a demonstration video as support material.

Please open source on github.

Comments on the Quality of English Language

Please proofread your paper using grammarly or other tools. There are  some typo.

Author Response

Comment1:  This paper can be accepted after adding a demonstration video as support material. Please open source on github.

Response1: The links to the dataset, source code, and demonstration video have been included in the Data Availability Statement.

Comment2: Please proofread your paper using grammarly or other tools. There are  some typo.

Response2: The paper has been thoroughly proofread, and several inconsistencies have been corrected. The manuscript follows American English conventions.