Testing Graded Operation–Unknown-Role Recombination in a Small Character-Level Transformer on Synthetic Grade 1 Word Problems

坂, 知樹

doi:10.51094/jxiv.5695

##article.authors##

坂, 知樹東京電機大学 https://researchmap.jp/saka214

DOI:

https://doi.org/10.51094/jxiv.5695

キーワード:

educational data mining、 arithmetic word problems、 synthetic instructional data、 compositional generalization、 Transformer、 fixed-marginal corpus intervention

抄録

We tested whether a 2.7M-parameter character-level Transformer trained from random initialization could generate a fixed role-explicit formulation for an operation-by-role conjunction absent from its synthetic instructional corpus while the operation and role remained separately observed. The fixedmarginal intervention held constant the total number of instructional units and the Add To/Take From and result/change/start-unknown marginals. Six target-absent corpora each omitted one conjunction and redistributed the remaining cells; a balanced-all corpus served as the reference. Across six corpus seeds and eight model seeds, balanced-all exact structural accuracy was approximately .99 in every cell at 2,500 iterations. Target-absent accuracy remained high for Add To/start (.977–.980), Take From/start (.891–.902), Add To/result (.916–.924), and Add To/change (.846–.868), but fell to .536–.559 for Take From/result and .538–.546 for Take From/change. Multiway bootstrap intervals retained this graded pattern. Extended training separated the difficult cells: the Take From/result contrast shrank from .435 at 2,500 iterations to .039 at 10,000, whereas Take From/change retained a .243 contrast. Closed-set sequence scoring ranked the correct exactly-one-unknown structure first for approximately .87–.91 of difficult-cell items, exceeding free-generation accuracy but supplying the legal candidate set. These results characterize behavioral recombination under this synthetic corpus, tokenizer, architecture, output linearization, and compensatory redistribution; they do not establish general semantic compositionality or human cognition. The study provides a replicable EDM framework for localizing conjunction-specific learning and decoding dependencies in digital instructional data.

利益相反に関する開示

なし

ダウンロード *前日までの集計結果を表示します

ダウンロード実績データは、公開の翌日以降に作成されます。

引用文献

BAKER, R. S. J. D. AND YACEF, K. 2009. The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining 1, 1, 3–17.

ELAZAR, Y., RAVFOGEL, S., JACOVI, A., AND GOLDBERG, Y. 2021. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics 9, 160–175.

HEWITT, J. AND LIANG, P. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, 2733–2743.

HUPKES, D., DANKERS, V., MUL, M., AND BRUNI, E. 2020. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research 67, 757–795.

JIA, Q., YOUNG, M., XIAO, Y., CUI, J., LIU, C., RASHID, P., AND GEHRINGER, E. 2022. Automated feedback generation for student project reports: A data-driven approach. Journal of Educational Data Mining 14, 3, 132–161.

KEYSERS, D., SCHÄRLI, N., SCALES, N., BUISMAN, H., FURRER, D., KASHUBIN, S., MOMCHEV, N., SINOPALNIKOV, D., STAFINIAK, L., TIHON, T., TSARKOV, D., WANG, X., VAN ZEE, M., AND BOUSQUET, O. 2020. Measuring compositional generalization: A comprehensive method on realistic data. In International Conference on Learning Representations.

KIM, K. J., POPE, D. S., WENDEL, D., AND MEIR, E. 2017. WordBytes: Exploring an intermediate constraint format for rapid classification of student answers on constructed response assessments. Journal of Educational Data Mining 9, 2, 45–71.

KIM, N. AND LINZEN, T. 2020. COGS: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 9087–9105.

LAKE, B. M. AND BARONI, M. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 80. PMLR, 2873–2882.

LEE, S., BARAL, S., LI, H., CHENG, L., ZHANG, S., THORP, C., ST. JOHN, J., THOMPSON, T., HEFFERNAN, N., AND BOTELHO, A. 2025. Developing a feedback taxonomy for math: A synergy of perspectives through data mining methods. Journal of Educational Data Mining 17, 2, 1–23.

LIU, N. F., LIN, K., HEWITT, J., PARANJAPE, A., BEVILACQUA, M., PETRONI, F., AND LIANG, P. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics 12, 157–173.

MISLEVY, R. J., BEHRENS, J. T., DICERBO, K. E., AND LEVY, R. 2012. Design and discovery in educational assessment: Evidence-centered design, psychometrics, and educational data mining. Journal of Educational Data Mining 4, 1, 11–48.

MIZUMOTO, T., OUCHI, H., ISOBE, Y., REISERT, P., NAGATA, R., SEKINE, S., AND INUI, K. 2019. Analytic score prediction and justification identification in automated short answer scoring. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics, 316–325.

PELÁNEK, R. 2015. Metrics for evaluation of student models. Journal of Educational Data Mining 7, 2, 1–19.

RUSSIN, J., JO, J., O’REILLY, R. C., AND BENGIO, Y. 2020. Compositional generalization by factorizing alignment and translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop. Association for Computational Linguistics, 313–327.

SARSA, S., LEINONEN, J., AND HELLAS, A. 2022. Empirical evaluation of deep learning models for knowledge tracing: Of hyperparameters and metrics on performance and replicability. Journal of Educational Data Mining 14, 2, 1–28.

TIAN, R., LI, Y., FU, Y., DENG, S., LUO, Q., QIAN, C., WANG, S., CONG, X., ZHANG, Z., WU, Y., LIN, Y., WANG, H., AND LIU, X. 2025. Distance between relevant information pieces causes bias in long-context language models. In Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics, 521–533.

TORNQVIST, M., MAHAMUD, M., MENDEZ GUZMAN, E., AND FARAZOULI, A. 2023. ExASAG: Explainable framework for automatic short answer grading. In Proceedings of the 18th Workshop