• Better learning results with Drillster

Better learning results with Drillster

An investigation into Drillster's influence on the learning result

In the spring of 2013 Educational Sciences graduate student at Utrecht University Joost van der Veen has investigated Drillster's influence on the learning result.

The core question in this research was: “Does Drillster have an effect on the user's learning result?”

In order to answer this question, the following question parts have been identified:

  • Is there a difference in learning result between those using Drillster's adaptive mechanism and those learning using traditional methods (using a book, not adaptive)?
  • Are any positive effects still significant when taking into account possible effects of “number of hours spent studying” and “motivation for the subject matter”?

The analysis of the test results has shown that:

  • Participants using Drillster as a study aid achieved 10% higher marks on a final test;
  • Participants using Drillster were able to absorb the subject matter at least 40% quicker.

Data exploration techniques have been used to determine whether there were any significant outliers. This turned out not to be the case. The differences measured are statistically significant.

Participants

The study involved 162 participants. The researcher opted for regular high school students. Participating schools were “Berlage Lyceum” in Amsterdam, Netherlands and “Bonhoeffer College” in Castricum, Netherlands. At Berlage Lyceum the participants were form 2 and 3 VWO students, and form 3 HAVO students. At Bonhoeffer College participants were taken from two form 2 classes, one VWO and one gymnasium.

The minimum number of participants required for the study was determined beforehand using a power analysis1. Power refers to the probability that the zero hypothesis — the study group does not differ from the control group — is rejected for the proper reasons. The power analysis indicated that in order to achieve an effect size of .50 with .05 significance, at least 102 participants are required in order to maintain a power of 70%. That means that for this study both the study group and the control group must consist of at least 51 participants. The study meets these criteria by a wide margin: the sample group consisted of 162 participants, with 99 persons in the study group and 62 participants in the control group.

The average age of the study group is 14.27 years (SD = .78) and varies between 13 and 16 years of age. This group consists of 46 female and 53 male participants. The average age of the control group is 13.92 years (SD = .75) and also varies between 13 and 16 years of age. The control group consists of 33 male and 30 female participants.

Study setup

The content of the targeted study material was compiled with the help of the teachers Latin, Spanish and Biology. By involving the participant's regular teachers, the researcher was able to ensure that the selected study material was completely new to the participants.

In addition to using new learning material, care was taken to ensure that the material logically followed the material used during the weeks prior to the test. This guaranteed that there were no knowledge gaps between the participants' pre-existing knowledge and the learning material used for the study, as that would have jeopardized the reliability of the test data.

The teachers were also instructed not to cover the learning material used in the test, so as to not inject a bias into the learning results.

The learning material for both the study group and the control group was exactly the same. The study group used drills, whereas the control group used PDF files with the same (non adaptive) content. Students at the selected schools all have access to iPads, which is why the control group was given the learning material in PDF format. In addition to the learning material in PDF format, the control group students were given a separate file with the correct answers to the study questions so as to allow them equal opportunity compared to the study group.

Devices

Participants in the study group used their own computer, iPad or smart phone to use the Drillster application. For iPad and smart phone this required installing the free Drillster app.

Questionnaires

As part of the study two questionnaires were prepared. The first questionnaire was administered prior to the test and was meant to collect relevant information that was later used in the analysis. Both study group and control group were given the same questionnaire.

The second questionnaire consisted of 15 statements. Out of these 15 statements, 8 statements were meant to gauge the participant's motivation for the relevant subject matter. Motivation impacts the participant's thinking, opinion and actions around the given subject.2

The statements were taken from the Checklist Study Motivation3.

Baseline and final assessment

For this study, knowledge tests were created consisting of the subject matter used. This was done by the researcher by using a random selection of learning elements taken from the subject matters given to the participants.

Prior to the test, this test was administered to all participants to establish a baseline.

A second knowledge test was administered at the end of the training period. Again, the test was given to all participants from the study group and the control group. This served as a final assessment.

Using the results of these two tests it became possible to compare the knowledge levels prior to and after the study period. All of the test questions were taken directly from the study material used in the research.

Procedure

The study was set up and executed identically at both schools. Participants took the baseline test directly after the introduction of the study.

Right after the introduction and the baseline test, the study period started. Students in each group were given four weeks of elapsed time to study the learning material.

At the end of the study period, a second knowledge test was given to all of the participants by way of a final assessment.

Data analysis

All of the data collected using the questionnaires and knowledge tests taken prior to and after the study period have been anonymized prior to further analysis.

In order to determine the reliability of the questionnaires, especially the scale used for “subject matter motivation”, the internal consistency was determined using Cronbach's Alpha4.

A T-test for independent samples was used to determine whether there are differences between the time spent studying by the Drillster group and the control group. This was done based on the number of hours reported by the participants.

Subsequently the averages and standard deviations of the percentages of correct answers on the knowledge tests were calculated for both the study group and the control group. Based on the average percentage of correct answers a T-test for independent samples4 was used to determine whether there is a difference between the average score of the baseline and the final assessment. This was done to ensure that both groups had similar scores prior to the study period so as to guarantee a meaningful comparison and the end of the study period.

The next step was to determine whether the average percentages of correct answers in the final assessment differed between the study group and the control group. This analysis was done using ANOVA4.

As a further step in the analysis was to ascertain that any possible positive effects of learning with Drillster were still present in the percentages in the final assessment if the elements of “subject matter motivation” and “number of hours reported studying” were included in the analysis as independent parameters.

This was done using ANCOVA56. This step in the analysis was taken to see whether the effect of learning with Drillster is still significant when taking into account possible other variables (covariants).

As a final step, the answers in the questionnaires used after the final assessment were analysed, both for the study group and the control group. Based on qualitative analysis the most frequently given answers were determined.

In the data exploration the researcher looked for possible outliers which could have influenced the results.

Results

The analysis of the research results has shown that:

  • Participants in the Drillster group achieved 10% betters marks on the final assessment compared to the control group
  • Participants in the Drillster group were able to absorb the learning material 40% quicker compared to the control group

The differences are significant.

Further data exploration was undertaken to look for any outliers. The conclusion was that there were none.

In order to determine the internal consistency of the scale for “subject matter motivation” as gauged in the questionnaires, Cronbach's Alfa was calculated. For values of Cronbach's Alfa higher than .70 the scale used for “subject matter motivation” can be viewed as good. Analysis of the data shows that “subject matter motivation” has an average score of 3.4 (SD = .12) with Cronbach's α = .75. This means that the items used in the scale for “subject matter motivation” are internally consistent.

References
  1. Faul, Erdfelder, Lang & Buchner, 2007
  2. Boekaerts, 2005
  3. Schouwenburg & Stevens, 1996
  4. Moore, McCabe & Craig, 2007
  5. Field, 2009
  6. Kinnear & Gray, 2011
  • Copyright © 2006–2024 Drillster BV.