TL;DR: There was a Science Jam for the first time at CHI this year. Fitting with the CHI 2018 motto “engage with CHI” the event offered a rapid learning and research prototyping experience. Teams of participants that formed on the spot developed research from concept to early outcome presentations over the course of just two days. Due to a wide array of support materials, an accompanying crash course in Research Methods in HCI, and most importantly thanks to the support of an all-star team of mentors, the participants were enabled to make the Science Jam a great success. The event attracted more than 60 participants, mostly HCI researchers who are new to the field. They reported great learning experiences and fun networking while working on more than 10 different research projects. As one participant put it:
“The ScienceJam was a fantastic introduction to my first CHI conference. I was able to learn from some of the best researchers in the field, improve my presentation skills and make new friends from all over the world.”
An announcement flyer for the Science Jam at CHI 2018.
What the Science Jam is and how it worked
You may have heard about - or participated in - hackathons or game jams. This year there was something new at CHI: a Science Jam!
In recent years, hackathons and game jams have emerged as concentrated and often spontaneous get-togethers that facilitate an outburst of rapid development and creative ideas. The best example is perhaps the Global Game Jam. In a grass-roots organized, yet global event, thousands of gamers, developers and other creative types get together for 48 hours in hundreds of locations around the globe to develop a computer of board game from a first concept to a functional prototype. Now think hackathon or game jam, but for “rapid research prototyping”. This can be great for participants who are rather new to the field to get hands-on learning experience, but it can also be great for more experienced researchers, since it offers room to try out some “unusual ideas” that otherwise may never have turned into an actual little project, or simply to peek beyond the limits of one’s day-to-day topics. As a hands-on approach to tackling learning about HCI research and exploring new avenues in a highly immersive and interactive manner, this concept was a natural fit for the 2018 conference motto “engage with CHI”. The event stretched over the full course of the two workshop days. The first day included a quick introduction, a group formation phase, research concept ideation and presentations, as well as first implementations and testing of research tools and procedures required for each project. Thanks to the wonderful support of the CHI 2018 chairs as well as the general program chairs, the Science Jam participants had access to snacks and drinks throughout both days. It was also possible to offer a taco lunch and the venue was open for extended hours. The second day commenced with a phase of running the studies, followed by gathering and analyzing outcomes. It closed with a session of short initial outcome presentations that were prepared by each team.
The outcomes of the CHI 2018 Science Jam in brief
The organizing team was extremely happy to welcome more than 60 CHI attendees to the Science Jam. This is a great turnout, since the two-day event requires a considerable investment in time and a small participation fee was required for organizational reasons. As expected, many participants were rather new to HCI research, with more than half of those who responded to a post-event survey indicating that they had little (1-2) or no prior experience with planning or conducting research studies.
Graph showing the participants' self-reported prior experience with HCI research.
Nevertheless: together, the participants managed to produce more than 10 research projects. The research approaches ranged from meta studies, over working with (paper or other) prototypes, up to interventions with functional implementations. The jam saw a variety of study topics, including augmented conference badges, the perception of terms of service under different presentation modalities, an exploration on augmented reality in museum settings, and even a meta-study on the perception of the usefulness and reliability of studies produced in the context of the Science Jam.
A selection of slides from the team presentations.
In addition to the group project presentations in the closing session of the Science Jam on Sunday evening, all groups were invited to present their projects on stage in the main exhibit area of the conference on Tuesday afternoon. This welcome addition made the outcomes visible to the larger CHI community. For most participants, presenting their work on a central stage while –in many cases– attending the 3000+ participants behemoth of a conference that is CHI for the first time, marked an exciting opportunity to gain some exposure and to put their presentation skills to a test.
Preparing for the outcome presentations in the main conference exhibit hall.
Participant feedback on the jam
In order to evaluate the perception of this new event the participants were asked to complete a survey following the public presentations on Tuesday. Overall, the Science Jam was perceived extremely well, receiving a mean rating of 8.4 (std. dev. 1.3) on a 10-point scale, with a rating of 9 being the most frequent response. Participants commended the “Extremely valuable interactive environment - especially learning and engaging with hosts and mentors!” and found the jam overall to be a “Great event and amazing organization.”
The overall rating for the Science Jam provided by respondents of the post-event survey.
Readily available materials are important to facilitate outcomes in the short time of the jam
Readily available materials that make it easier for participants to quickly setup their studies were considered an important element and given priority in preparations prior to the event. This included pre-structured study concept and informed consent forms, as well as a selection of common questionnaires and psychometric research tools, together with analysis templates in R and a template for project presentation slides. The participants found the research materials provided during the Science Jam to be mostly very good or excellent.
The participants’ rating of the quality of research materials that were provided during the Science Jam.
Structured guidance is appreciated by most participants
The materials provided were structured along a crash course on Research Methods in HCI that was offered to those participants who did not have much prior experience with planning or running HCI studies and also to those who simply wanted to hear a refresher on the topic. The crash course itself was modeled to match the natural progression of the participants’ projects through three 60 to 90-minute sessions that were dispersed over the two days of the event:
- Session 1: Research Design Basics
- Session 2: Ethics & Running a Study
- Session 3: Analysis & Presentation
The progression of topics presented in the three sessions of the crash course on Research Methods in HCI.
Participants feel that they learn about relevant topics in HCI research during the jam
While participants remarked that presentations during the jam should best be held in a separate room and could be integrated more smoothly into the overall program, as not to interrupt those teams who preferred to focus on their hands-on study implementations, most appreciated the course and materials, indicating that they agree (somewhat to strongly) that they learned how to set up a research study, with none disagreeing to this statement.
The participants’ self-reported impression of the extent to which they learned to set up a research study.
On a similar note, most participants agreed (somewhat to strongly) that they learned how to analyze research data (with only one respondent disagreeing somewhat).
The participants’ self-reported impression of the extent to which they learned to analyze research data.
The mentors simply rocked
The Science Jam organizers were excited to be able to recruit an all-star team of experienced CHI researchers who agreed to volunteer as mentors to participant teams during the event. Sharing their wisdom and expertise both in direct interactions with the teams and through brief mentor talks that were interspersed throughout the jam, the mentors contributed immensely to the success of the event. On their own terms, most mentors chose to spend the majority of the two days on site and collaborated with a number of different teams, while also taking the time – and making the effort – to get involved closely with the study planning, testing, and analysis that the teams faced.
The mentor team at the Science Jam at CHI 2018 (left to right and top to bottom): Kasper Hornbaek (University of Copenhagen), Kathrin Gerling (Uni Leuven), Scott Bateman (Univ. New Brunswick), Carman Neustaedter (Simon Fraser), Ehud Sharlin (U of Calgary), James Wallace (U of Waterloo), Carie Cunningham (DRIVE Lab / Nuance), and David Flatla (University of Guelph).
Next to comments in person during and after the event, the participants also clearly expressed their appreciation of the mentors through their survey responses. 13 out of 19 respondents found the mentor interactions and feedback to be excellent, providing the highest of 7 levels of the rating.
The participants’ rating of the interaction with - and feedback from - the mentors.
The short mentor presentations turned out to be similarly well-received, with 10 out of 19 respondents rating them as excellent and all but one rating them to be markedly positive.
The participants’ rating of the short presentations given by the mentors.
However, even more so than these structured survey responses, the free responses provided by the Science Jam participants when asked to comment on the mentors clearly speak to their contributions and to the value of the building-block of the event that was “invite great mentors”. Responses included: “The close mentorship is the biggest highlight.”, or “MENTORS. They were amazing and the life blood of the event. I was given some amazing ideas and great direction from those that came around.”. Arguably, the mentors also played an important role in providing value to some of the more experienced participants, as summarized by the following statement: “As I already took several courses about HCI research for my master’s degree, the ScienceJam for me was not about learning the foundations to conduct such research. It was more about applying, challenging and adding more specific knowledge to what I had learned before. I felt that I have fully achieved this, thanks to the wonderful mentors. Without exception, the mentors were friendly, approachable, knowledgeable and fun to work with :)”.
The Science Jam welcomes newcomers to CHI, can be a great learning experience, and provides a great networking boost
While building on a very solid core of established activities and elements, CHI as a conference is also known for being innovative and for being an event that is constantly evolving and does not shy away from considering new formats, if they seem promising. With this year’s motto “engage with CHI”, the Science Jam came naturally as a new event that implements this topic quite literally.
While a jam event is easily dismissed as a fun and immersive, but ultimately chaotic and unstructured – perhaps even unscientific - event, the outcomes and feedback resulting from the Science Jam at CHI 2018 clearly point in a different direction that indicates valuable contributions: while the projects are carried out in a very short time, the learning materials and mentors clearly highlight the kind of tradeoffs that typically occur even in more relaxed settings or “regular research work”, where researchers of all stages are constantly forced to weigh best practices and the best possible scrutiny against monetary, temporal, and many other limitations. As such, the Science Jam is a great event for newcomers to get accustomed to common research methods and best practices that play an important role at CHI, allowing participants to learn first-hand from experienced mentors. The intense and immersive setting facilitates lasting learning effects, providing especially the more novice researchers a good opportunity to pick up on methods and approaches that will allow them to produce high-quality research that also has a good chance of being accepted by the larger CHI community. At the same time, said intense and immersive – in the best sense embodied – experience of doing research in action, together with a team whose members quickly cease to be strangers and supported by mentors who are amongst the best in their fields, makes for an incredible networking opportunity that is decisively open and inviting to newcomers to HCI research. In many places after the event, groups of Science Jam team members could be observed discussing research, or attending social events. In many cases, mentors joined the groups to catch up on their conference experiences, often connecting them to other experienced researchers and practitioners in their company. The participants were well-aware of these aspects, noting – for example – that they “Learned lots of new tools and made connections with fellow jammers that will be great going forward.”, “[…] had a warmup to the conference and got a change to connect with people.”, or “[…] enjoyed working on the research, though found socializing to be just as much as the focus as the research.”.
Final 'fun' slide presented by one of the teams.
Drawn together, the Science Jam appears to be a great format for socializing and networking, while also – paradoxical as it may seem – serving as a great learning opportunity for good scientific conduct, by constantly reflecting on the limitations that are imposed by the format and weighing tradeoffs that immediately affect the ongoing projects of the ad-hoc research teams.
Feedback provides suggestions for improvements and future plans
Although the reception of the event at large was very positive, observations during the event and explicit remarks by participants provided clear opportunities for improving potential future events. For example, the optional courses and talks could be communicated more clearly with a schedule that is always visible to all participants and they could be held in a manner that allows those groups who do not want to listen in to continue working on their projects without interruption. The time-slot allotted for the final presentations of the teams should be extended a bit to allow for a more relaxed session. Furthermore, the learning materials could be made available before the event to those who are interested, and the team formation process could be adjusted to take a more comparable amount of time across groups. Lastly, the suggested common theme (which was “curiosity” for this installment) and the “diversifiers” (optional aspects to the projects that teams are invited to consider and that are meant to provide guidance or additional challenges) should be more visible throughout the event so that teams find it easier to keep them in mind.
To jam or not to jam?
Altogether, the event exceeded the expectations and hopes of the organizers and turned out to be a wonderful celebration of curiosity in and of itself. Many mentors and participants signed up to a mailing list after the event, indicating their interest – not only to hear about potential future installments, but also – to actively contribute to a potential future of the Science Jam. Accordingly, together with some of these volunteers, the organizers will explore the possibility of setting up a web-application and guide that can facilitate future Science Jams. The feedback provided by the mentors and participants speaks best for the event, should the chairs of CHI 2019 entertain the possibility of making the science jam a part of the program for next year as well.
A group picture of a large portion of the attendants of the Science Jam at CHI 2018 in Montreal.
In the words of two of the participants:
“Trying to run a whole study from defining a research question to presenting results in such a short period of time is challenging, but incredibly rewarding! I totally recommend it!”
“This was one of the best things I’ve done at a CHI! You get to meet some people, work on interesting topics, and get your ideas out there. I highly recommend [the Science Jam], especially if this is your first CHI.”
Acknowledgments:
The team of organizers of the Science Jam at CHI 2018 (left to right): Jan D. Smeddinck (ICSI / UC Berkeley), Elisa D. Mekler (University of Basel), Carl Gutwin (University of Saskatchewan), and Casper Harteveld (Northeastern University).