PENGARUH LEARNING RATE TERHADAP KINERJA DAN KONVERGENSI MODEL INDOBERT-BILSTM-CRF PADA NAMED ENTITY RECOGNITION BAHASA INDONESIA

IMPACT OF LEARNING RATE ON INDOBERT-BILSTM-CRF PERFORMANCE AND CONVERGENCE FOR INDONESIAN NAMED ENTITY RECOGNITION

Authors

  • Alfred Yulius Arthadi Putra Universitas Widya Dharma Pontianak
  • Fredrikus Suarezsaga Universitas Widya Dharma Pontianak
  • Kristina Universitas Widya Dharma Pontianak

DOI:

https://doi.org/10.52972/hoaq.vol17no2.p271-280

Keywords:

Named Entity Recognition, IndoBERT, BiLSTM-CRF, learning rate, konvergensi, Bahasa Indonesia

Abstract

Named Entity Recognition (NER) merupakan salah satu tugas penting dalam Natural Language Processing yang digunakan untuk mengidentifikasi dan mengklasifikasikan entitas dalam teks. Kinerja model NER berbasis pre-trained language model tidak hanya dipengaruhi oleh arsitektur, tetapi juga oleh konfigurasi hiperparameter, khususnya learning rate. Penelitian ini bertujuan menganalisis pengaruh learning rate terhadap kinerja dan konvergensi model IndoBERT-BiLSTM-CRF pada NER Bahasa Indonesia. Empat nilai learning rate diuji, yaitu 1×10-5, 2×10-5, 3×10-5, dan 5×10-5, dengan konfigurasi model lainnya dipertahankan tetap. Setiap konfigurasi dijalankan menggunakan tiga random seed untuk mengevaluasi kestabilan hasil. Kinerja model diukur menggunakan Token Macro-F1 dan Entity F1, sedangkan konvergensi dianalisis berdasarkan perubahan training loss dan Validation Entity F1 pada setiap epoch. Hasil menunjukkan bahwa peningkatan learning rate memberikan peningkatan kinerja hingga rentang tertentu. Entity F1 meningkat dari 0,8951 pada 1×10-5 menjadi 0,9392 pada 5×10-5, sedangkan Token Macro-F1 meningkat dari 0,5848 menjadi 0,7902. Dari sisi konvergensi, 5×10-5 mencapai Validation Entity F1 sebesar 0,90 pada epoch ke-5, lebih cepat dibandingkan 3×10-5 pada epoch ke-6 dan 2×10-5 pada epoch ke-9. Meskipun 5×10-5 memberikan rata-rata kinerja tertinggi, perbedaan Entity F1 dengan 3×10-5 sangat kecil. Temuan ini menunjukkan bahwa learning rate memberikan pengaruh yang lebih kuat terhadap kecepatan konvergensi, sementara peningkatan kinerja akhir mulai mengalami saturasi pada rentang 3×10-5 hingga 5×10-5.

 

Named Entity Recognition (NER) is an important Natural Language Processing task for identifying and classifying entities in text. The performance of NER models based on pre-trained language models is influenced not only by the model architecture but also by hyperparameter configuration, particularly the learning rate. This study investigates the effect of learning rate on the performance and convergence of an IndoBERT-BiLSTM-CRF model for Indonesian NER. Four learning rates were evaluated, namely 1×10-5, 2×10-5, 3×10-5, dan 5×10-5, while keeping the remaining model configurations unchanged. Each configuration was evaluated using three random seeds to assess performance stability. Model performance was measured using Token Macro-F1 and Entity F1, while convergence was analyzed based on training loss and Validation Entity F1 across epochs. The results show that increasing the learning rate improves model performance up to a certain range. Entity F1 increased from 0.8951 at 1×10-5 to 0.9392 at 5×10-5, while Token Macro-F1 increased from 0.5848 to 0.7902. In terms of convergence, the 5×10-5 configuration reached a Validation Entity F1 of 0.90 at epoch 5, earlier than 3×10-5 at epoch 6 and 2×10-5, at epoch 9. Although 5×10^(-5) achieved the highest average performance, its Entity F1 was only marginally higher than that of 3×10-5. These findings indicate that learning rate has a stronger effect on convergence speed, while final performance begins to saturate within the range of 3×10-5, to 5×10-5.

References

Warto, S. Rustad, G. F. Shidik, E. Noersasongko, Purwanto, Muljono, and D. R. I. M. Setiadi, “Systematic Literature Review on Named Entity Recognition: Approach, Method, and Application,” Statistics, Optimization & Information Computing, vol. 12, no. 4, pp. 907–942, Feb. 2024, doi: 10.19139/soic-2310-5070-1631.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-Training of Deep Bidirectional Transformers For Language Understanding,” in Proceedings of the 2019 Conference of the North, Stroudsburg, PA, USA: Association for Computational Linguistics, 2019, pp. 4171–4186. doi: 10.18653/v1/N19-1423.

F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” in Proceedings of the 28th International Conference on Computational Linguistics, Stroudsburg, PA, USA: International Committee on Computational Linguistics, 2020, pp. 757–770. doi: 10.18653/v1/2020.coling-main.66.

B. Wilie, K. Vincentio, G. I. Winata, S. Cahyawijaya, X. Li, Z. Y. Lim, S. Soleman, R. Mahendra, P. Fung, S. Bahar, and A. Purwarianti, “IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding,” in Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, Stroudsburg, PA, USA: Association for Computational Linguistics, 2020, pp. 843–857. doi: 10.18653/v1/2020.aacl-main.85.

S. O. Khairunnisa, A. Imankulova, and M. Komachi, “Towards a Standardized Dataset on Indonesian Named Entity Recognition,” in Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Student Research Workshop, Stroudsburg, PA, USA: Association for Computational Linguistics, 2020, pp. 64–71. doi: 10.18653/v1/2020.aacl-srw.10.

E. Dave and A. Chowanda, “IPerFEX-2023: Indonesian personal financial entity extraction using indoBERT-BiGRU-CRF model,” Journal of Big Data, vol. 11, no. 1, pp. 1–24, Sep. 2024, doi: 10.1186/s40537-024-00987-6.

Z. Zainuddin, - Mudassir, and Z. Tahir, “Entity Extraction in Indonesian Online News Using Named Entity Recognition (NER) with Hybrid Method Transformer, Word2Vec, Attention and Bi-LSTM,” JOIV?: International Journal on Informatics Visualization, vol. 9, no. 3, pp. 964–973, May 2025, doi: 10.62527/joiv.9.3.2902.

W. Zhang, C. Wang, H. Wu, C. Zhao, G. Teng, S. Huang, and Z. Liu, “Research on the Chinese Named-Entity–Relation-Extraction Method for Crop Diseases Based on BERT,” Agronomy, vol. 12, no. 9, p. 2130, Sep. 2022, doi: 10.3390/agronomy12092130.

M. B. Shishehgarkhaneh, R. C. Moehler, Y. Fang, A. A. Hijazi, and H. Aboutorab, “Transformer-Based Named Entity Recognition in Construction Supply Chain Risk Management in Australia,” IEEE Access, vol. 12, pp. 41829–41851, 2024, doi: 10.1109/ACCESS.2024.3377232.

M. Mosbach, M. Andriushchenko, and D. Klakow, “On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines,” Mar. 2021, doi: 10.48550/arXiv.2006.04884.

Warto, Muljono, Purwanto, and E. Noersasongko, “Capitalization Feature and Learning Rate for Improving NER Based on RNN BiLSTM-CRF,” in 2022 IEEE International Conference on Cybernetics and Computational Intelligence (CyberneticsCom), IEEE, Jun. 2022, pp. 398–403. doi: 10.1109/CyberneticsCom55287.2022.9865660.

M. D. S. Antariksa, A. Sugiharto, and B. Surarso, “BERT Model Fine-tuned for Scientific Document Classification and Recommendation,” Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), vol. 9, no. 4, pp. 754–764, Aug. 2025, doi: 10.29207/resti.v9i4.6789.

K. Kamdan, M. P. Anugrah, M. J. Almutaali, R. Ramdani, and I. L. Kharisma, “Performance Analysis of IndoBERT for Detection of Online Gambling Promotion in YouTube Comments,” in The 7th International Global Conference Series on ICT Integration in Technical Education & Smart Society, Basel Switzerland: MDPI, Sep. 2025, p. 66. doi: 10.3390/engproc2025107066.

A. F. Hidayatullah, M. F. D. A. Putra, A. P. Wibowo, and K. R. Nastiti, “Named Entity Recognition on Tourist Destinations Reviews in the Indonesian Language,” Jurnal Linguistik Komputasional (JLK), vol. 6, no. 1, pp. 30–35, Apr. 2023, doi: 10.26418/jlk.v6i1.89.

S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, Nov. 1997, doi: 10.1162/neco.1997.9.8.1735.

X. Ma and E. Hovy, “End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Stroudsburg, PA, USA: Association for Computational Linguistics, 2016, pp. 1064–1074. doi: 10.18653/v1/P16-1101.

G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, and C. Dyer, “Neural Architectures for Named Entity Recognition,” in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Stroudsburg, PA, USA: Association for Computational Linguistics, 2016, pp. 260–270. doi: 10.18653/v1/N16-1030.

E. F. Tjong Kim Sang and F. De Meulder, “Introduction to the CoNLL-2003 shared task,” in Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 -, Morristown, NJ, USA: Association for Computational Linguistics, 2003, pp. 142–147. doi: 10.3115/1119176.1119195.

Downloads

Published

03-11-2026

How to Cite

Putra, A. Y. A., Suarezsaga, F., & Kristina. (2026). PENGARUH LEARNING RATE TERHADAP KINERJA DAN KONVERGENSI MODEL INDOBERT-BILSTM-CRF PADA NAMED ENTITY RECOGNITION BAHASA INDONESIA: IMPACT OF LEARNING RATE ON INDOBERT-BILSTM-CRF PERFORMANCE AND CONVERGENCE FOR INDONESIAN NAMED ENTITY RECOGNITION. HOAQ (High Education of Organization Archive Quality) : Jurnal Teknologi Informasi, 17(2), 271–280. https://doi.org/10.52972/hoaq.vol17no2.p271-280

Most read articles by the same author(s)