Difficulty Index and Cognitive Skills of English Textbook for Senior High School

This study explored the difficulty index, the terms for the category for reading text level, and cognitive levels of reading texts in the English textbook, Interlanguage, for Senior High School grade 12. The primary aim was to determine the extent to which the degree of the available different categories of reading text levels included the cognitive level of questions. Through the readability test formulas for difficulty index and a checklist based on revised Bloom Taxonomy for cognitive skills, 11 reading texts completed with 56 reading comprehension questions were analyzed. The result revealed that the Interlanguage textbook contained four different categories of reading level such as difficult, fairly difficult, plain, and fairly easy to read. The textbook had in the average category of fairly difficult to read for difficulty index and contained fewer HOTS–based questions than LOTS ones (34% < 66%). Besides, higher reading text levels included fewer HOTS-based questions than those of lower ones. This means a higher level of cognitive skill of questions may appear in any text with a different level of difficulty index. This suggests that Muslih Hambali, Soni Mirizon, Nandang Heryana 18 Indonesian Journal of EFL and Linguistics, 6(1), 2021 the English textbook should be revised to have more HOTS-based question items for developing the students’ high cognitive levels.


INTRODUCTION
A textbook has been an old but popular teaching medium in the process of teaching and learning from time to time. The provision of a textbook for secondary students in Indonesia is stated in the Ministry of Education and Culture Law (Permendikbud) number 8-year 2016. A textbook is a book that contains the materials of certain fields of study and is systematically arranged and selected based on certain goals, learning orientation, and students' development (Muslich, 2010). The English textbook usually contains some language skills and components. One of them is reading skill which is usually taught through reading texts. Richards (2001) provides the use of textbooks in different ways in language programs such as a reading textbook which is used to develop reading skills. The availability of reading texts is to particularly develop the students' mastery in reading comprehension. Students cannot avoid reading comprehension as it has become the key point of learning other subjects with a variety of disciplines.
The levels of reading text can be a problem in the teaching of English as a foreign language in Indonesia. One such problem for example is that the reading text is too difficult to read for a school level of students. The problem may cause to have the failure of many students to achieve the targeted learning outcome. This is because the reading text level is probably too difficult (complex) or too easy (simple), so it is not appropriate for the current reading level of the students. The difficulty of reading texts will determine whether or not they are readable for a certain age level of the students. The reading text is generally not readable due to its complexity of words and sentences. Satriani (2018) argues that one of the problems faced by the students in treading comprehension is, in addition to vocabulary, having difficult reading text with complex grammatical sentences. When a reading text contains many words with more syllables or many complex sentences instead of simple ones, the reading ease becomes low. This might affect students' reading comprehension to become low as the reading text is not appropriate with their reading level. Logically the lower the reading ease is the more difficult the text is probably to comprehend. In determining the complexity of reading texts the readability measures should be involved.
Another failure that the students might face is probably the answer to a reading task. A reading task is designed not only to know the students' reading comprehension but also to increase their critical thinking skills. The task for English reading text is usually constructed with a variety of cognitive levels of questions which require thinking skills critically as a part of cognitive domain. The higher the school level of the students is, the higher the cognitive levels of the question should be constructed. Cognitive levels are included in the cognitive domain of Bloom's Taxonomy. The cognitive domain "involves the development of our mental skills and the acquisition of knowledge." (Hogue, 2017, p. 47). This means cognitive level simply refers to the degree the ability of one's brain to process information.
Concerning improving students' thinking skills, a textbook certainly has an important role in which its reading texts contain the demand often called task to be done by the students. According to Hashemi et al (2012), a task is a kind of work plan that requires students to process a language pragmatically for the correct content. Richard and Rodgers (2014) as cited in Chalak (2015, p.20) mention that "a task is an activity carried out as the outcome of processing or remembering language." A task has an important role in language teaching as it helps students achieve the goals of learning a language. In most English reading texts, the task in the form of comprehension questions is commonly put at the end of the text. In some textbooks, it is also found that comprehension questions appear before the text. A task may have various types of questions such as essay, multiple-choice, true-false, etc. The text with a variety of questions usually reflects the required thinking skill level. Therefore, one of the strategies to increase students' thinking skills is to provide a variety of questions with high cognitive levels. Raphael (cited in Muzammill, 2016) states that the Question-Answer Relationships (QARs) is categorized as one of the strategies to improve students' reading comprehension to think critically by involving high-quality level of questions.
In an educational world, the kinds of thinking skills are known in the Cognitive Domain of Revised Bloom Taxonomy consisting of 6 levels, remember, understand, apply, analyze, evaluate, and create (Krathwohl, 2002). The first three levels are called lower-order thinking skills (LOTS) that require memorization, and the last three known as higher-order thinking skills (HOTS) involve understanding and applying knowledge (Watson, 2019). Text questions with HOTS commonly called high-level questions are used to improve the higher performance of the students.
Although there have been many English textbooks for both secondary schools and colleges published by different publishers, their content of cognitive skills and difficulty index of the texts need to be evaluated. This is the first choice to consider in selecting a good textbook before teaching and learning take place so that the selected textbook will give such a benefit for learners' performance. Assaly & Smadi (2015) believe that one of the basic criteria used for textbook evaluation is the cognitive levels of the questions in the textbook. It is often found that some textbooks include more LOTS-based questions than HOTS-based ones. This point of view is in line with Bloom (1956) cited in Nappi (2017) who mentioned that 95% of the assessment given to college students were in the cognitive level of recall. San (2015) in his research also found that the course book "Global Level 1" contained questions with lower-level cognitive skills than those of higher ones.
Many researches have been undertaken to evaluate English textbooks for reading difficulty. Some of which were conducted by Rahmawati and Lestari (2012), Abusa'aleek and Khataybeh (2020), and Nurhamsih (2017) for textbook readability. Rahmawati and Lestari summarized that English Today for grade X contained higher difficulty level reading materials than Developing English Competencies for grade X. In contrast, Abu and Khataybah, and Nurhamsih found the textbooks they evaluated were in the category of easy. Meanwhile, Hidayat (2016) revealed that the textbook entitled "Bahasa Inggris SMA/MA/SMK for Grade XI Semester was in the standard readability level which meant it was suitable for grade eleven.
Dealing with cognitive skills of questions in a textbook, some researches have been conducted. Assaly & Smadi (2015), foreign researchers, found that their textbook evaluated contained fewer HOTS-based questions than LOTS-based questions (40% ˂ 60%). Other researchers, Wu and Pei (2018), also a foreign researcher, and Pratiwi (2014)  In determining what a good English textbook should be selected, aspects of reading level and cognitive skills must be covered. The reading text level should be learned for their difficulty and the cognitive skills should be included in the reading texts, so these two aspects need to be considered for teaching purposes. An English teacher often selects a certain English textbook instructed by the school or the Ministry of Education without considering whether or not the textbook is appropriate for or promotes the cognitive skills of his students. This case might cause to have the failure of the students to understand the text. When this happens, the demand for learning English is not successfully achieved, so this is something that has to be considered.
The previous studies evaluated the textbooks separately for either their readability or their cognitive levels of questions, but this study did both at the same time. Therefore, this study aimed not only to find the difficulty index and cognitive level of the reading texts but also to investigate the extent to which the degree of reading text levels represented cognitive levels of thinking skills. The phrase difficulty index was adopted in this study to represent or refer to the reading text levels consisting of eight different categories ranging from very easy, easy, fairly easy, plain, fairly difficult, difficult, very difficult, to extremely difficult. The difficulty index of reading text is measured by the readability test such as the Reading Ease formula. Pikulski (2002) states that readability is the degree of reading text ease or difficulty to be understood by a certain reader for a certain purpose.

RESEARCH METHODOLOGY
The main subject of this research was the reading texts of the English textbook, Interlanguage, for Senior High School Grade 12. The book has been published by Pusat Perbukuan Departemen Pendidikan Nasional 2008 in the form of a hardcopy and e-book. The first phase of analyzing the textbook was to retrieve and retype the sample reading texts accompanied by a reading task (comprehension questions) only. The reading texts with comprehension questions were selected for cognitive levels of thinking skills analysis. Following the first phase, the difficulty index of each reading text was measured by using Automatic Readability Checker, an English writing tool for readability formulas available online. This writing tool was userfriendly and practical to use as it provided menus on the screen, so the user just followed the available instructions. Once the sample text was checked, there appeared many characters, syllables, words, and sentences with their averages. Besides, this writing tool shared scores from different readability formulas.
Once the measurement of the difficulty index of all the selected texts was completed, the final readability consensus consisting of Grade level, reading level, and Reader's age was automatically made, so the user did not have to adopt conventional formulas for readability check. This readability consensus was a primary use to decide the reading text level. The reading level was used to identify the category of reading text which was meant for the difficulty index. The category of reading text level was then interpreted based on the reading ease scores as shown in the following table.  (Anderson et al., 2001) was prepared to analyze the data. To identify which cognitive level the question item was included, the stem of the question had to be learned. A verb in the stem usually became the core of deciding the cognitive level of the question. Kennedy, D (2007) mentions that a verb is key to writing outcomes. The question stem for higher-order thinking generally begins with a question word followed by a verb. Some questions often begin with verbs such as select, list, underline for remembering (C1), discuss, describe, identify for understanding (C2), apply, predict, demonstrate for applying (C3), examine, investigate, analyze for analyzing (C4), evaluate, justify, judge for evaluating (C5), and create, design, develop for creating (C6). Each question was evaluated for its cognitive level and tallied by putting a check (√) under which cognitive level column it was included in, and then the total number of each cognitive level of the questions was calculated using percentage.
To find the extent to which the degree of each reading text level included different cognitive levels of each question, the researcher matched each reading text with the tallied item and then calculated the total tallied items. The total number of checked items of each cognitive level question was calculated using percentage.

FINDINGS
The English textbook Interlanguage contains eleven reading texts accompanied with reading comprehension questions. The following From the table, the average score of Reading Ease (RE) was 59.76, the highest score was 77, and the lowest one was 43.3. Overall, the reading level of the textbook was in the category of fairly difficult. Viewed from each text, the result of the difficulty index showed that the categories of the textbook consisted of 1 text difficult, 5 texts fairly difficult, 3 texts plain, and 2 texts fairly easy. The grade levels or estimated reading grades varied starting from grade level 8 to grade level 12. Most of the reading texts were readable for school grade 12 although one reading ease score, text number one, was theoretically for college level. Figure 1 below presents the category reading level of Interlanguage English Textbook.

Figure1. Readability Category of Reading Text of Interlanguage English Textbook
It is noticed from the figure that 46% of the reading texts are categorized as fairly difficult. This means the textbook is appropriate for school grades 10-12 which require reading ease scores between 50.0 and 60.0. The next figure also provides the level of cognitive domain of the reading comprehension questions of the textbook. The Interlanguage English textbook with eleven reading texts provided fifty-six questions consisting of forty-eight essay questions and eight multiple-choice ones. The cognitive level of questions was dominated by lower-order thinking skills (LOTS) more than higher-order thinking skills (HOTS) represented by analyzing (C4), evaluating (C5), and creating (C6). Out of the total number of questions, 56 items, there were 37 items included in LOTS-based questions and 19 items considered HOTS ones. From LOTS-based questions, it revealed that 14 items (25 %) were remembering, 23 items (41%) were understanding, and no item was included in the level of applying (C3). Among 19 items categorized as HOTS-based questions, 10 items (18%) were included in the level of analyzing, 8 items (14%) were in the level of evaluating, and 1 item (2%) was in the level of creating. This single item appeared in reading text number two entitled What is Being Done to Control Unemployment?
The following

DISCUSSION
Four different categories of reading level in table 2 above appeared in the reading texts of Interlanguage English textbook ranked from fairly easy, plain, fairly difficult, and difficult. The first two and the last two categories included in the reading levels such as very easy, easy, very difficult, and extremely difficult did not exist in the reading text. The English textbook, Interlanguage has eleven texts whose reading text category levels consist of one difficult, five fairly difficult, three plain, and two fairly easy. Among 5 fairly difficult reading texts found in this study, 2 were theoretically congruent with the school grade level 12, and other reading texts were also readable although they did not match the grade level 12. As a whole, the difficulty index of the English textbook is in the category of fairly difficult to read as the average RE is 59. It is mentioned in Flesch Kincaid's theory that a text with the readability index ranging from 50.0-60.0 is categorized fairly easy, appropriate, and readable for school grade level 12. In this case the textbook Interlanguage is generally suitable for the 12 th grade students of senior high school in Indonesia. Although contrast with Abusa'aleek and Khataybeh (2020) and Nurhamsih (2017) who found their search of textbooks easy to read and Rahmawati and Lestari (2012) who evaluated the English reading materials with higher level of difficulty, this finding is anyhow in line with the previous research on the textbook evaluation conducted by Hidayat (2016). The only difference is that the textbook he evaluated has a different grade level from the present one. The other reason to support the readability of the Interlanguage English textbook suitable for school grade level 12 in Indonesia is that 45% of the reading ease scores are below 50.0, which I believe is appropriate enough if we think of English as a foreign language for them.
Evaluating the tasks following the texts to see which category levels of reading texts include questions based on higher-order thinking skills is a great of importance for high cognitive level development of students. The questions can be the parameter to determine a higher or lower cognitive level reading text. Logically the lower the category level of reading texts, the more the higher cognitive level-based questions could be made. In fact, the investigation revealed that (1) difficult reading texts did not serve any HOTS-based question, (2) fairly difficult reading texts contained 33.3% HOTS-based questions lower than LOTS-based questions (33.3% ˂ 66.6%),  (2015), Wu & Pei (2018), and San (2019) found in their research although the materials they evaluated are different levels from the research materials of this study. From the previous studies and the present study, it shows that there are many English textbooks that still lack for the demand of the 21 th century to promote students' thinking skills.

CONCLUSION
In conclusion, the whole point of helping students is to select the reasonable difficulty of texts so that they can get access to the reading and can improve their critical thinking skills. The present study included a comprehensive result of difficulty index and cognitive skills that contributed to the English textbook entitled Interlanguage for the twelfth-grade students of senior high school in Indonesia. The first finding revealed that the textbook was in the average category of fairly difficult to read which means it was readable and appropriate for grade 12. The second one was that the Interlanguage English textbook contained fewer HOTS-based questions, which means it did not have enough contribution to developing students' critical thinking. The other finding shows that the lower readability level category of reading text does not guarantee that the higher cognitive level of questions might have. In other words, higher cognitive levels of questions do not depend on the readability level of the reading texts.