A data-driven caption for L2 listening
DOI:
https://doi.org/10.36505/ExLing-2020/11/0042/000457Keywords:
listening difficulty, partial and synchronized caption, data-driven word selectionAbstract
Partial and Synchronized Caption (PSC) is a tool that automatically detects difficult segments for the second language (L2) listeners and displays them in the caption while omitting easy-to-recognize cases to reduce cognitive load. Given that the number of words to be shown in this caption is limited, the main challenge lies in selecting and prioritizing difficult words. Since partialization is a classifying task, we made a dataset of labeled words in TED talks (easy vs. difficult) for a target proficiency-level. A deep classifier is trained on this dataset to automate the detection of difficult words/phrases without explicitly extracting word features. This proposed data-driven PSC outperforms its feature-based versions by adopting a selection pattern that is more similar to the annotations, capturing more complicated cases, and minimizing the false positives.
References
Chang, A.C.S. 2009. Gains to L2 listeners from reading while listening vs. listening only in comprehending short stories. System, 37(4): 652–663.
Guillory, H.G. 1998. The effects of keyword captions to authentic French video on learner comprehension. Calico Journal, 15(1–3): 89–108.
Hochreiter, S., Schmidhuber, J. 1997. Long short-term memory. Neural computation, 9(8), 1735-1780.
Leveridge, A.N., Yang, J.C. 2013. Testing learner reliance on caption supports in second language listening comprehension multimedia environments. ReCALL, 25(2): 199–214.
Mayer, R.E., Moreno, R. 2003. Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1): 43–52.
Mirzaei, M.S., Meshgi, K., Kawahara, T. 2018. Exploiting automatic speech recognition errors to enhance partial and synchronized caption for facilitating second language listening. Computer Speech & Language, 49, 17-36, Elsevier.
Paivio, A. 1990. Mental representations: A dual coding approach. Oxford University Press.
Downloads
Published
Issue
Section
License
Articles are published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are properly credited.
