Executive Summary

[[Conclude story, something like… ]] – Access data shows that
Hypothesis has been used to reach a lot of students across schools and
departments. Heavy users reach hundreds of students with their use of
Hypothesis in their classes. – Overall access has gone down in 2022, and
has been especially low in Fall 2022 – However, engagement data shows
that student annotations has remained steady. Students seem to be
engaging at a deeper level and making use of Hypothesis to engage in
lengthy/substantial/in-depth discussions around texts. – Instructors
have found that students benefit from social annotation/etc. –
Continuing to support Hypothesis would allow for interactions difficult
with other tools – Next steps would be to identify how students and
instructors are using Hypothesis effectively and better promote/support
this work with classes that have not yet used Hypothesis

Introduction

Social Annotation

Social annotation, a form of computer-supported collaborative
learning, is a genre of learning technology that enables productive
group collaboration and shared meaning-making (Kalir,
2020). Social annotation has been a tool used in classrooms for well
over a decade. With rapid shift to online and distance teaching,
instructors turned to new tools and strategies to maintain effective
teaching while engaging with their learners and connecting them to each
other. There are a number of social annotation tools on the market
including Hypothesis, Perusal, Markup, Diigo, and NowComment, among many
others.

Social annotation can be used across all disciplines and to meet a
variety of pedagogical goals benefiting teachers and learners alike. For
those readers interested in cognitive benefits of social annotation,
those include improved processing of domain-specific knowledge, increase
question asking and peer support, supporting or improving argumentation,
inquiry, and literacy skills, and practice organizing and presenting
one’s findings and ideas. For those readers interested in the
classroom-as-a-community benefits of social annotation, those include
bridging gaps in social inequalities, support of transparent
assessments, fostering conversations among peers without the limit of
time or space, and discovering new resources and interests. All of these
benefits provide the potential for higher-order thinking and incisive
learning. Social annotation tools have been credited for greater gains
in reading comprehension and meta-cognitive skills than individual work
(Johnson
et al., 2010). Finally, users of social annotation tools generally
enjoy using social annotation tools (Lin
& Tsai, 2011) and report easier access to annotations directly
linked to writing provides a more supportive environment for peer review
and collaboration (Mendenhall
& Johnson, 2010).

While threaded discussions also allow for some cognitive and social
benefits to the learners, the two environments encourage different
processes of knowledge construction, commenting, and focus. Sun
& Gao (2017) found that students participate in online
discussion differently when considering threaded discussion boards
(e.g., Canvas discussions, Piazza) and social annotation tools (e.g.,
Hypothesis, Perusal). The structure of social annotation tools leaves
highlights and comments near the relevant information, allowing readers
to review information with ease, unlike references to course learning
materials that might be externally linked or simply quoted out of
context in a discussion post. Social annotation discussions tend to be
more specific and focused, as well as encouraging greater engagement
with all parts of the learning materials. Users of threaded discussions
tend to focus on the “bigger picture” when it comes to commenting on
learning materials, while users of social annotation tools are not
limited to the bigger picture and can comment thoroughly on all specific
parts of the materials that are of interest to them.

In the following report, I review the findings of my analysis of the
use of Hypothesis, a social annotation tool that was introduced to WUSTL
in 2020.

Hypothesis

Hypothesis is a social
annotation tool used at WUSTL for collaboration among classmates and
colleagues for reading, annotating, and discussion in an interactive
web-based interface. Hypothesis can be added to Canvas to incorporate
active discussion with PDFs or webpage readings. Hypothesis allows users
to select text to annotate publicly or privately, reply to or share
annotations including links to specific notes or entire pages of text,
collaborate privately to produce group annotations, and search all
public annotations.

For more information about the use of Hypothesis as a classroom tool
at WUSTL, visit CTL’s Teaching
Resources and to interactively explore a webpage using Hypothesis
check out the Illustrated
Taxonomy of Annotation Types.

The easily-accessible Hypothesis dashboard allows us to extract
course-level data and raw-text annotations can be requested from
Hypothesis. In the following report, I will be using the following key
terms:

  • Annotation a note in the margin of a document
    with a direct reference to a specific portion of the text
  • Document any digital course material, including
    but not limited to a webpage, book chapter, scholarly journal articles,
    student work for peer review
  • Access a measure of users who have interacted
    with Hypothesis either as a teacher, teaching assistant, staff/admin, or
    student
  • Engagement some measure of participation with
    the Hypothesis tools; includes a variety of metrics such as word counts,
    comments, replies, annotations, use of questions, multimedia links,
    etc.

Access to Hypothesis

Hypothesis was introduced to Wash U in 2020, just in time for a
global pandemic to rock our socks off and force virtually all learning
to become remote. One primary drawback (among the dozens of others) to
remote learning is a lack of social interaction in the classroom. While
many teachers discourage social interaction in casual ways in the
classroom, active learning protocols require social interaction as it
facilitates peer-to-peer learning, discussion, engagement with
materials, scaffolding, and other pedagogical phenomena.

Since 2020, we have had 635 courses use Hypothesis. In 2020, 114
courses used Hypothesis. In 2021, 277 courses used Hypothesis. In 2022,
222 courses have used Hypothesis. During this time, a total of 22
demonstration or tutorial “courses” have used Hypothesis to train
instructors during a CTL workshop or for instructors to train
themselves, support staff, TAs, AIs, or graders. Demo courses were
removed from the presentation of yearly courses above.

Access by Courses

Table

Table 1. Courses Using Hypothesis
Year Season Total Number of Courses
2020 Spring 2
2020 Summer 1
2020 Fall 118
2021 Spring 134
2021 Summer 10
2021 Fall 148
2022 Spring 127
2022 Summer 5
2022 Fall 90

Plot

Access per semester

We see the highest use of Hypothesis in Spring and Fall 2021,
reaching around 1800 students. In Spring 2021, Hypothesis was used
across all schools.

Note that instructors may be undercounted due to truncated data.

Table 4. Access Summary: Semester Distribution of Access to Hypothesis
Semester Total Schools Total Departments Total Users Total Instructors Total Students
SU2020 1 1 61 2 59
FL2020 5 29 1705 151 1554
SP2021 8 43 2000 165 1835
SU2021 3 8 134 10 124
FL2021 6 30 1981 169 1812
SP2022 6 29 1671 168 1503
SU2022 3 3 36 6 30
FL2022 4 25 1291 131 1160

Schools & Departments

Taking a closer look at the data by schools and departments, we see
that Arts & Sciences dominates the use of Hypothesis, followed by
the Brown School and U College. In a later section, we will explore the
use of Hypothesis in Arts & Sciences a bit closer, by analyzing use
within departments.

All Time (Table)

Table 2. All-time Hypothesis Use Within Schools
School Name Number of Classes Percent
Architecture 20 3.15
Arts & Sciences 459 72.28
Business 7 1.10
demo 22 3.46
Engineering 5 0.79
Law 2 0.31
Medicine 18 2.83
Social Work and Public Health 51 8.03
University College 51 8.03

Semesters (Table)

Table 3. Semester Distribution of Courses Within Schools
School SU2020 FL2020 SP2021 SU2021 FL2021 SP2022 SU2022 FL2022
Architecture 0 10 2 0 6 2 0 0
Arts & Sciences 0 93 91 3 108 96 1 67
Business 0 0 3 0 0 4 0 0
demo 1 6 11 2 2 0 0 0
Engineering 0 0 1 0 3 0 0 1
Law 0 0 0 0 0 0 2 0
Medicine 0 2 4 0 0 12 0 0
Social Work and Public Health 0 0 6 0 21 7 0 17
University College 0 7 16 5 8 8 2 5

Semesters (Graph)

Instructors

Heavy users of Hypothesis typically used it consistently over the
last three years.

Note that number of courses may count sections or reading groups.

[maybe a plot of # instructors by number of semesters using
Hypothesis to show how many repeat users there are vs. new users – table
of top users show 4-5 semesters using hypothesis – how many who used it
once?]

Table 6. Primary Instructors With 10 or More Total Courses Using
Hypothesis
Dept Instructor SU2020 FL2020 SP2021 SU2021 FL2021 SP2022 SU2022 FL2022 Total
LATAM Eliza Williamson 0 3 2 0 39 11 0 22 77
CWP Colin Bassett 0 4 3 0 15 13 0 9 44
ART-ARCH Kristina Kleutghen 0 2 1 0 0 25 0 3 31
MPH Akilah Collins-Ander 0 0 5 0 21 0 0 0 26
MPH Alexis Duncan 0 0 1 0 0 2 0 17 20
SPAN Amanda Carey 0 8 2 0 3 2 0 1 16
SPAN Heidi Chambers 0 2 4 0 6 3 0 0 15
WRITING Heather McPherson 0 3 2 0 2 2 0 3 12
CWP Stefanie Boese 0 0 3 0 3 3 0 3 12
PHYSTHER Amanda Hennekes 0 0 0 0 0 12 0 0 12
PSYCH Leah Schultz 0 0 2 0 3 6 0 0 11
ELP Haley Dolosic 0 4 2 0 2 2 0 0 10

Students

Instructors can use Hypothesis to support hundreds of students in
social annotation with peers. Many of the heavy users of Hypothesis has
have engaged more than 100 students with Hypothesis in their
classes.

Classes can be further divided into smaller reading groups. The
histogram shows that on average, classes using Hypothesis had 12.72
students (SD = 9.99, range = 1–100). The majority of courses,
some of which may constitute reading groups, consist of 10 to 20
students.

Table

Table 5. Primary Instructors and Number of Students Reached (Greater
than 100)
Dept Instructor SU2020 FL2020 SP2021 SU2021 FL2021 SP2022 SU2022 FL2022 Total
LATAM Eliza Williamson 0 93 33 0 275 88 0 278 767
MPH Akilah Collins-Ander 0 0 90 0 407 0 0 0 497
CWP Colin Bassett 0 51 38 0 110 93 0 56 348
MPH Alexis Duncan 0 0 11 0 0 38 0 270 319
SPAN Amanda Carey 0 139 26 0 38 20 0 22 245
PHYSTHER Amanda Hennekes 0 0 0 0 0 212 0 0 212
SPAN Heidi Chambers 0 36 47 0 85 27 0 0 195
ARCH Megan Kidd 0 0 0 0 190 0 0 0 190
WRITING Heather McPherson 0 32 32 0 27 25 0 40 156
CWP Stefanie Boese 0 0 40 0 38 40 0 38 156
MUSIC Esther Kurtz 0 19 56 0 15 54 0 0 144
GS/IAS Rebecca Clouser 0 14 53 0 12 33 0 15 127
PSYCH Leah Schultz 0 0 28 0 56 43 0 0 127
ART-ARCH Kristina Kleutghen 0 13 30 0 0 61 0 20 124
SPAN Marisa Barragan-Peug 0 47 14 0 26 32 0 0 119
CWP Aileen Waters 0 39 38 0 41 0 0 0 118
MPH Angela Hobson 0 0 0 0 0 115 0 0 115
AFAS Karma Frierson 0 16 0 0 49 0 0 42 107
ART-ARCH Betha Whitlow 0 27 0 0 42 35 0 0 104
ELP Haley Dolosic 0 47 24 0 13 17 0 0 101

Histogram

Access Summary

Departments

Finally, we break down Hypothesis use in Arts & Sciences by
department. We find a breadth of access across all/most departments,
with greater usage by the College Writing Program, Latin American
Studies, Spanish, Master of Public Health, and Art History &
Archeology departments.

Graph

Table

Table 5. Courses Offered by Department (NA indicates ‘demo’)
Dept SU2020 FL2020 SP2021 SU2021 FL2021 SP2022 SU2022 FL2022 Total
COLLEGE WRITING PROGRAM(L59) 0 15 20 0 22 16 0 15 88
LATIN AMERICAN STUDIES(L45) 0 3 2 0 39 11 0 22 77
SPANISH(L38) 0 27 16 0 12 8 0 2 65
Master of Public Health (MPH)(S55) 0 0 6 0 21 7 0 17 51
ART HISTORY AND ARCHAEOLOGY(L01) 0 8 5 1 3 27 1 4 49
PSYCHOLOGICAL & BRAIN SCIENCES(L33) 0 1 6 0 5 11 0 2 25
NA 1 6 11 2 2 0 0 0 22
ARCHITECTURE 0 10 2 0 6 2 0 0 20
ENGLISH LANGUAGE PROGRAMS(U15) 0 4 3 0 2 4 2 2 17
WRITING(L13) 0 5 2 0 2 2 0 3 14
MUSIC(L27) 0 4 3 0 2 2 0 1 12
Physical Therapy Program-Grad(M02) 0 0 0 0 0 12 0 0 12
GLOBAL STUDIES(L97) 0 2 3 0 2 2 0 2 11
AFRICAN AND AFRICAN-AMERICAN STUDIES(L90) 0 1 2 0 4 1 0 1 9
ANTHROPOLOGY(L48) 0 1 3 0 3 0 0 2 9
FRENCH(L34) 0 3 1 0 3 1 0 1 9
FIRST-YEAR PROGRAMS(L61) 0 3 2 0 2 1 0 0 8
HISTORY(L22) 0 2 4 0 0 0 0 2 8
LATIN(L10) 0 3 1 0 3 0 0 1 8
ENGLISH LITERATURE(L14) 0 2 2 0 1 1 0 1 7
KOREAN(L51) 0 2 2 0 1 2 0 0 7
Audiology and Communication Sciences(M89) 0 2 4 0 0 0 0 0 6
WOMEN, GENDER, AND SEXUALITY STUDIES(L77) 0 1 3 0 1 0 0 1 6
ENVIRONMENTAL STUDIES(L82) 0 0 3 0 0 2 0 1 6
GERMANIC LANGUAGES AND LITERATURES(L21) 0 2 2 0 1 0 0 0 5
SPANISH(U27) 0 2 0 1 1 0 0 1 5
POLITICAL SCIENCE(L32) 0 0 2 0 0 2 0 1 5
FILM AND MEDIA STUDIES(L53) 0 2 1 0 0 0 0 1 4
BIOLOGY(U29) 0 0 2 0 1 0 0 1 4
EDUCATION(L12) 0 0 1 1 0 1 0 1 4
ENGLISH COMPOSITION(U11) 0 0 2 0 1 1 0 0 4
GENERAL ENGINEERING(E60) 0 0 1 0 2 0 0 1 4
GENERAL STUDIES(U03) 0 0 1 2 1 0 0 0 4
INTERDISCIPLINARY PROJECT IN THE HUMANITIES(L93) 0 0 1 0 0 1 0 2 4
MANAGERIAL ECONOMICS(B54/B64) 0 0 2 0 0 2 0 0 4
DATA ANALYTICS(B69/B59) 0 0 1 0 0 2 0 0 3
HISTORY(U16) 0 0 1 0 1 1 0 0 3
PSYCHOLOGICAL & BRAIN SCI (PSYCHOLOGY)(U09) 0 0 1 1 0 1 0 0 3
EAST ASIAN STUDIES(L03) 0 2 0 0 0 0 0 0 2
GREEK(L09) 0 1 0 0 0 1 0 0 2
CLASSICS(L08) 0 0 1 0 0 1 0 0 2
COMPARATIVE LITERATURE(L16) 0 0 1 0 1 0 0 0 2
MATHEMATICS AND STATISTICS(L24) 0 0 2 0 0 0 0 0 2
MLA SEMINARS(U98) 0 0 2 0 0 0 0 0 2
KOREAN(U51) 0 0 0 1 0 1 0 0 2
BIOLOGY AND BIOMEDICAL SCIENCES(L41) 0 0 0 0 1 0 0 1 2
PORTUGUESE(L37) 0 0 0 0 0 2 0 0 2
AMERICAN CULTURE STUDIES(L98) 0 1 0 0 0 0 0 0 1
DRAMA(L15) 0 1 0 0 0 0 0 0 1
INTERNATIONAL AFFAIRS(U85) 0 1 0 0 0 0 0 0 1
PHILOSOPHY(L30) 0 1 0 0 0 0 0 0 1
ART HISTORY AND ARCHAEOLOGY(U10) 0 0 1 0 0 0 0 0 1
DANCE AND SOMATIC MOVEMENT STUDIES(U31) 0 0 1 0 0 0 0 0 1
DLA SEMINARS(U96) 0 0 1 0 0 0 0 0 1
NONPROFIT MANAGEMENT(U76) 0 0 1 0 0 0 0 0 1
GENERAL STUDIES(L43) 0 0 0 1 0 0 0 0 1
BIOMEDICAL ENGINEERING(E62) 0 0 0 0 1 0 0 0 1
CLINICAL RESEARCH MANAGEMENT(U80) 0 0 0 0 1 0 0 0 1
BIOLOGICAL & PHYSICAL SCIENCES FOR PBPM(L86) 0 0 0 0 0 1 0 0 1
LAW SCHOOL(W74) 0 0 0 0 0 0 1 0 1
LAW(W77) 0 0 0 0 0 0 1 0 1
EDUCATION(U08) 0 0 0 0 0 0 0 1 1

Engagement with Hypothesis

Annotation Metrics

Blanks

For the sake of the following metrics, let’s first take a look at
“blank” annotations, or just highlights, and whether they are
shared.

We see that about 21% of all annotations are private highlights and
only 1.5% of annotations are private comments/annotations. Of the
private highlights, the majority (93%) are made by students, suggesting
students are using Hypothesis to mark up documents, in addition to
providing public annotations for the class.

highlight_only shared n percent
FALSE FALSE 2161 1.49
FALSE TRUE 111852 76.98
TRUE FALSE 30864 21.24
TRUE TRUE 432 0.30

Private highlights

user_role n percent
Instructor/Admin 2224 7.21
Student 28622 92.79

Total

How best to show change over time???

NOTE TO SELF: consistent color schemes (i.e., schools and semesters
always rep’d by same colors)

  • total and avg per student graphs to showcase annotations have stayed
    steady/increased by student

School

 

Word counts – most frequent words by school

word n
people 21824
health 9720
agree 7955
time 7940
social 6574
idea 6404
women 6370
feel 6067
question 5957
article 5717
class 5665
makes 5493
black 5229
research 5224
public 4607
society 4598
change 4593
understand 4555
power 4495
world 4494
lot 4371
art 4342
reading 4232
life 4192
culture 4148

Top words by role:

Relatinoships between number of students in class and engagement? –
engagement for different types of classes

Discussions

Annotation Structure:

  • thread
  • thread comment
  • comment reply
  • reply response
  • response discussion …
  • response discussion …
annot_level n_Instructor/Admin n_Student percent_Instructor/Admin percent_Student
thread 9085 72947 68.52 72.42
thread comment 3543 23952 26.72 23.78
comment reply 546 3405 4.12 3.38
reply response 73 386 0.55 0.38
response discussion 11 41 0.08 0.04

Documents

1,999 distinct documents have been used 2,392 times since 2020,
suggesting instructors re-use documents from semester to semester.
Instructors mainly upload documents from sources like google drive,
sharepoint, or was a direct upload from canvas.

Types

type (web, upload, demo) n
demo 1
upload 1690
web 308
type (web, upload, demo) n
demo 64
upload 130322
web 14923

Annotations per Document

Average Annots per Doc Median Annots per Doc SD Annots per Doc Min Annots per Doc Max Annots per Doc
60.51 38 70.4 1 848

Annotations

Word count

user_role totalwc maxwc minwc meanwc sdwc
Instructor/Admin 382476 500 0 28.84870 32.92942
Student 5265642 1810 0 52.27429 46.42157
NA 616 65 2 25.66667 19.81582

Types of Interactions

Questions

How many instructor and student annotations contain questions? We see
when questions are asked, instructors and students are most likely to
ask just 1-2 questions.

question Instructor/Admin Student
0 11928 90333
1 1021 7629
2 232 2040
3 62 505
4 7 148
5 0 49
6 0 12
7 7 8
8 0 6
10 1 1

About this script

R version 4.2.2 (2022-10-31)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur ... 10.16

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] scales_1.2.0       tidytext_0.3.3     RColorBrewer_1.1-3 plotly_4.10.0     
 [5] kableExtra_1.3.4   lubridate_1.8.0    forcats_0.5.1      stringr_1.4.0     
 [9] dplyr_1.0.9        purrr_0.3.4        readr_2.1.2        tidyr_1.2.0       
[13] tibble_3.1.7       ggplot2_3.4.0      tidyverse_1.3.1   

loaded via a namespace (and not attached):
 [1] fs_1.5.2          bit64_4.0.5       webshot_0.5.3     httr_1.4.3       
 [5] SnowballC_0.7.0   tools_4.2.2       backports_1.4.1   bslib_0.3.1      
 [9] utf8_1.2.2        R6_2.5.1          DBI_1.1.2         lazyeval_0.2.2   
[13] colorspace_2.0-3  withr_2.5.0       tidyselect_1.1.2  bit_4.0.4        
[17] compiler_4.2.2    cli_3.4.1         rvest_1.0.2       xml2_1.3.3       
[21] NLP_0.2-1         labeling_0.4.2    slam_0.1-50       sass_0.4.1       
[25] tm_0.7-8          systemfonts_1.0.4 digest_0.6.29     rmarkdown_2.14   
[29] svglite_2.1.0     pkgconfig_2.0.3   htmltools_0.5.2   dbplyr_2.2.0     
[33] fastmap_1.1.0     highr_0.9         htmlwidgets_1.5.4 rlang_1.0.6      
[37] readxl_1.4.0      rstudioapi_0.13   jquerylib_0.1.4   generics_0.1.2   
[41] farver_2.1.0      jsonlite_1.8.0    crosstalk_1.2.0   vroom_1.5.7      
[45] tokenizers_0.2.1  magrittr_2.0.3    Matrix_1.5-1      Rcpp_1.0.8.3     
[49] munsell_0.5.0     fansi_1.0.3       lifecycle_1.0.3   stringi_1.7.6    
[53] yaml_2.3.5        grid_4.2.2        parallel_4.2.2    crayon_1.5.1     
[57] lattice_0.20-45   haven_2.5.0       hms_1.1.1         knitr_1.39       
[61] pillar_1.7.0      reprex_2.0.1      glue_1.6.2        evaluate_0.15    
[65] data.table_1.14.2 modelr_0.1.8      vctrs_0.5.1       tzdb_0.3.0       
[69] cellranger_1.1.0  gtable_0.3.0      assertthat_0.2.1  xfun_0.31        
[73] broom_0.8.0       janeaustenr_0.1.5 viridisLite_0.4.0 ellipsis_0.3.2