I have had the pleasure of interviewing Professor Carl Gutwin from the University of Saskatchewan. Carl was CHI papers chair in the past and had incredible experience writing for SIGCHI conferences.
About Branching Into a New Research Area
After he transitioned from graduate school to faculty, one of the first mistakes was when he was trying to shift into a new research area (after grad school). He underestimated the amount of work necessary to get up to speed with related literature in a new field. Thus, it was tough for him to outline how his contribution was new, different, and better from what was out there because he did not have the depth of knowledge that he had on other topics in which he had been immersed for four to five years. His advice is that if you want to do a CHI paper in an area you are not intimately familiar with, you better do your homework. Related work has much to do with whether you are contributing to the field. Reviewers, who are likely all experts in that territory, will notice right away if you missed that vital work in the field. To them, this might determine whether or not you are making a new contribution (i.e., they will find a reference that has already done what you are doing, and they will point that out to you). Doing a thorough literature search pays dividends for you with reviewers later if you are not familiar with a field. Leave some slack in your schedule so that you can avoid these mistakes.
Carl’s Style Advice
Carl says it is hard to write elegantly. He compares an elegant style and good writing to usability principles, where we try to get it right 80% of the time (referring to the Pareto principle or 80/20 rule). Professor Gutwin advises students to focus first on their paper’s workman-like aspects (i.e., all the things that reviewers will check off their lists). He means that it is of utmost importance first to get the message of a paper across. Writers have to convey what problem they are solving and why it is a crucial problem for the CHI community. While he admires eloquent style, he admits that style is not something that reviewers look for in papers. He has been in lots of committee meetings, where papers were accepted even though reviewers said: “The authors wrote this terribly, and I could barely understand what was going on, but I love the ideas behind this.”
With that in mind, what typically gets through is good research rather than good writing (i.e., substance over style; as many would argue, it should be for a scientific conference). However, I share Carl’s opinion that it is still critically important to become a good writer (and a stylist). Carl says he wants his papers to be judged by a committee on what he did (i.e., the research). The only way to accurately convey to a reviewer what you did, is by writing clear sentences. The best style for Carl is thus the style that accurately reflects the work with utmost clarity. So you would be asking yourself: What is the contribution of my work? And why is it a contribution in clear and unambiguous terms? Then, the reviewer will be judging you for what you did rather than for what they think you did (because you were not clear in how you stated it). Writing clear sentences is not trivial. It takes much clear thought and revisions to do this. Carl thinks that students can do this in two ways:
- Do not start with sentences at all when you start writing your paper. Start with an outline instead. Work out the argument first. What is the progression of ideas that you are going to convey to the reader?
- Fall in love with your favourite style guide (e.g., Strunk and White, Style, Writing Tools). Little things like grammatical errors, poor use of parallel structure, and sentences that do not agree in number or sense from one end of the sentence to the other. These obstacles provide little bumps along the way of the review where a reviewer now has to work harder to find out what you did or why your contribution is essential. Any time you make a reviewer work hard, you lose ground, and you lose marks toward their final impression (and score) on your paper. Carl says that if a reviewer is not sure, they will play it safe and mark you down at least half a point. Worse, if they cannot figure it out, you will not be able to convince them of the beautiful work you have done. Therefore clarity is the most critical style.
Focus on Introduction and Discussion
Carl is personally focused on getting the Introduction and Discussion sections of a paper right. He refers to high school essay classes, where he has learned the hourglass structure of a paper. We start broad and then we move into specifics of what we have done and then we broaden again back out at the end (focusing on three questions: What did we just learn? Why is it important? What should we do now?). The Introduction and discussion are the points in that hourglass where you are both widest in focus and then having to make the biggest change. In the introduction, you have to frame the problem, scope the research and set the stage for your reader. These are the most exciting parts of paper writing for Carl; these bits are critically important for him. He advises his students that they are always telling a story. So, he likes to call these exercises story conferences rather than paper writing sessions. All good stories have a conflict and some decision that has to happen, some fight that has to occur (see also the arch plot structure). Then, it all has to have a meaning in the overall context of the story. So, think about what your story is and what is exciting and different about your research. Set your paper up as a movie or a story with a protagonist and an antagonist (the classic conflict situation in most stories). What is the resolution of your conflict? Thinking about this is never a bad idea even if some papers do not lend themselves to this type of narrative structure.
Another important part of writing an introduction for Carl is to guide yourself by answering four important questions:
- What is the real-world problem that we are trying to solve?
- Why is it important to solve this problem?
- What is the solution that we came up with to solve it?
- How do we know that the solution is a good solution to the problem?
These questions map nicely onto different kinds of CHI research. In the HCI community, we always try to solve real-world problems that are important in some contexts because they cause lots of pain or money when they are not resolved. The evaluations that we usually carry out should show some progress on which critical problems we are working on. It is too often that Carl reads papers that are not clear about the problem they are trying to solve or that fail to argue why the problem is worth solving. You always have to convince the reader (or the reviewer) that the problem you have addressed is valuable to someone (i.e., would somebody pay you money to solve it?). Carl also thinks that reviewers make up their minds about a CHI paper after the first page. This page is where you set your first impressions, and this is where you need to convince them that your problem has value (at the minimum that it is worth their time to read about).
For reviewers, Carl mentions specifically that you always have to keep in mind their state (they are usually stressed professors doing many reviews after all their other day work is done). This is probably the 12th paper they are reading that day, and they have probably stopped paying attention by the time they get to the middle of your manuscript and if you do not give them everything they need to argue strongly for the article on the first page, it will be hard to convince them of that later.
About Writing the Abstract
Your reviewers always read the abstract first, so this is the first impression you can make on readers (and reviewers), and it has to be a good one. Everyone will likely keep their attention long enough for the entire abstract, so Carl uses the same model (the four questions mentioned above) to guide the reader through the abstract. Carl even thinks a little further than the abstract and mentions how important the title is for a CHI paper (Jofish Kaye researched CHI paper titles). Carl says: “The title is your first opportunity to summarise why this paper should be accepted.” The abstract is then the slightly longer version of this opportunity to convince the reviewer to accept this paper. The introduction is an even longer version and then the paper is the full-length version. It is an incremental increase in detail of your sales pitch for the paper. You need to look at the sections of your paper as opportunities for you to get something done. If you have developed something new and exciting, this is your chance to tell the world about it. Carl follows his four-question model precisely in his abstracts. He starts by saying: “It is a problem that…” and then “this is a critically important problem to solve, because…” and then “we have solved this important problem with…” and then “to show that this was a good solution to the problem of X, we ran a study that showed there was a 50% improvement in Y…,” so “in conclusion we have made important progress on an important problem that is going to make the world a better place.” The abstract is often neglected (and sometimes in worst cases simply copied as a paragraph from the introduction) but it has to be thought of as a nugget of information, a summary of the research project and an opportunity to convince people that what you have done is good work. The abstract also always includes the takeaway for the paper (“You need to hold the reader by the hand with your punchline and sometimes need to hit them over the head with it”). Carl puts the most important results, not just in his abstract, but even thinks they should make it into the title of the paper. So instead of writing: “An essay on selection performance on touch tablets,” write “Improving selection performance on touch tablets with [name of your method],” for example. Use the abstract as an opportunity to tell the whole story.
About the Structure of a CHI Paper
The structure of a paper is an assistance to the reader. If you can depend on a particular structure, you often have to do less work to integrate the words on a page to give meaning to you as an observer or reader of the work. So, the structure of a paper into Introduction, Related Work, Main Section, Results, and Discussion helps the reader to make sense of the manuscript. Carl would consider himself a traditionalist that adheres to the regular structure of a CHI paper because he is sensitive to the way that readers (and CHI reviewers) are expecting the paper to be structured. He believes that structure is there for a reason (“We layout buildings in a certain way so that they are easy to get around”). Section headings and other structure elements make it easy for the tired reader to find an easy way through the paper (same with the placement of the results). He quite likes different styles of articles (e.g., essay styles), but thinks that making changes to the regular CHI paper structures would increase the risk of rejection.
About his Favourite Paper
He always gives the example of “Edit wear and read wear” (Hill et al., 1992). What he remembers about the paper was a life-changing experience for him and not because of the writing style of the paper, but because of the concept that was introduced (“By graphically depicting the history of author and reader interactions with documents, these applications offer otherwise unavailable information to guide work.”). They introduced a simple concept that had a wide meaning across many application areas. The idea is that you store interaction history and then visualize that interaction history. For example, a scroll bar could keep track of where people have been. The paper clearly gets the excitement and innovation of the concepts across. Not all ideas are as revolutionary as this, but the clarity that the concept was described with is possible to achieve in our own work.
About Best Papers
Carl does not think that there is a strong correlation between excellent writing and winning best-paper awards. There are lots of criteria used for judging best papers and a revolutionary contribution might be written up poorly, but will still be strong enough to get accepted. Those discussions happen at committee meetings all the time that people are considering putting something forward that has merit but might not be packaged nicely. On the flipside, best papers are rarely given to papers because of their excellent writing. If the results are incremental, then even great writing cannot hide the fact that the contribution is not too strong to the field (or at least so is the assumption). Carl did admit that he would have to go back and read the best papers again to see if there are those rare cases, where excellent writing got them the award nomination. “Science does not really move forward on style,” Carl said (and I sadly agree, but I remain of the opinion that great style should not be an afterthought to excellent science). “The quality of the writing is there to convey the science; it’s a vehicle for the ideas.”
About Topics That Are Attractive to CHI
We all wish we had the handbook of attractive topics for CHI, but we do not. But Carl still has an anecdote to share about a paper. I am assuming it was Luis von Ahn et al.’s image-labelling paper (PDF) about the ESP game. Carl was blown away by the talk and the concept of the paper. The idea of the paper was so simple and compelling that Carl (and other people he talked to later) all thought that they could have come up with that already (practically the concept was already in their heads – applying a matchmaking word game to a simple real-world problem, but they had not actually written it down). At the time when it was published, it was considered an unbelievably cool solution to a real problem. Being ahead of the curve (as Luis von Ahn was, who is not short of great ideas and went on to win lots of awards and start Duolingo) is a definite advantage to writing papers. The important takeaway from Carl’s experience is that it is not important to chase whatever the next big thing might be but to just do something that is of real interest to you (I share Carl’s sentiment because I had a similar experience when I wrote early gamification papers together with colleagues because it was simply something that we were interested in doing [applying games to things that are not game] but neither of us was expecting the field to blow up as big as it did later) and then see if public interest follows. Carl believes in interest-driven research, where you are trying to solve the puzzle in front of you with the tools that you can think of and sometimes you come up with a brilliant solution. The papers that Carl likes best are the ones that are investigating a research problem that he can relate to or frustration that he has experienced himself.
About Good Solutions (and Good Methods)
Carl believes that you do not always need to do one particular evaluation of results, but that you need to convince your readers that the thing that you have done, has meaning and value in the domain that you are working. The evidence you provide in your research has to really outline why your solution fixes a problem. The solution most often offered is a user study, an experiment, or statistical analysis that makes such an argument but it does not have to be. Any evidence-based approach can be compelling in convincing somebody that your solution works, but this is the main point you have to be making with your research results. Learning mixed methods and statistics (big data and ANOVAs) is still an important first step for students to take that want to publish at CHI because those approaches are simply more common (and there is a danger in doing statistics incorrectly). Carl really appreciates seeing qualitative methods inside quantitative research projects and believes that CHI, in general, is a really good forum for mixed methods (see also: this interesting paper about sample sizes at CHI). This is of particular importance when trying to answer research questions about why a statistically significant difference is there (your statistics only tell you that it is there, but not necessarily why). The meaning of this difference can be explained using mixed methods more easily. This also provides “ample grist for your discussion mill” as Carl says, because it allows discussing strategies that people took to mitigate a problem (caused by a difference). You should always be happy when you learn a new method and get to apply it to a paper, which is often done via collaboration with somebody familiar with different methods than yourself (referring to a mixed models approach he recently tried). However, he also warns that there is a risk of using completely new methods within CHI because some reviewers might simply be unfamiliar with the work. Whenever a reviewer feels like they cannot evaluate the correctness of the method of a paper, they may sometimes err on the side of caution and give a lower score. We also talked about the validity of new statistical methods, some approaches, which have been published at CHI (even a SIG). In the case, where a method becomes common at CHI and you can signal to the reviewers the new method, it is easier to integrate it and work off of the new approach.
Whenever you are breaking conventions, whether it be trying new analyses or a new structure for your paper, you always have to have a good reason to deviate from the traditional approach and as long as you can make this argument to your reviewer, it can be interesting to try out new things (in small increments) at CHI.
From the Initial Idea to the Final Paper
I asked Carl about his process from having an idea towards the final paper. He said that he has two models. The first one follows the approach from his colleague Andy Cockburn, who believes you should write the paper before you write the paper. This assumes that you have pondered about the feasibility of a CHI idea (often at the conference or directly after) and do consider it feasible. Andy suggests writing the argumentation for your study into the introduction, considering what experiments you would run and how your results would look like and what takeaways you could get from them. So, you end up with a fiction piece, a paper about what your paper/study could be without having actually run any experiments. Carl is surprised how often we get to the point—when we grow our research organically—(often right before the deadline) where we wonder about having done something differently in our study and then we cannot go back and fix it. By writing the paper before you do your research, you protect yourself from two things: (a) missing some of the things that are critical when presenting CHI work and (b) it allows you to think about how much you have to do before you get a contribution (i.e., it helps you determine your scope). This approach essentially allows you to assess the risks first before you do your work.
His other model is interest-driven research, where he tinkers with things that he is just interested in. However, Carl also mentions that this is a luxury that comes with having tenure and that younger faculty and students might want to be more strategic about what they would want to end up in their CHI papers.
About his Workflow and Tools
Carl begins a paper by being focused on its story, argument, and introduction. Once he has some idea about what the project is, he thinks about the argument that he will make in the introduction (again, beginning with the problem and asking why it is important, the solution and how to say that it is a good solution). He starts by drafting these as bullet points on paper (a great activity for meetings that you have to attend but might not be interested in). This can be done wherever, in an email message, on paper, in a notebook, wherever you can stick your thoughts (and later find them again). There is no formatting, no structure, no sentences, just bullet points to see how the arguments flow. Carl recommends doing a maximum of 15 points that address his four questions. Once he has this outline written, he moves into Microsoft Word (in Windows) and does not return to paper or other editors after that. He then outlines various sections of the paper. The sections that he finds difficult to write “from whole cloth” are the related work section (which he iterates again that it needs a strong outline to become a reasonable part of the story of a paper) and the discussion section because one needs to find interesting topics or headings for it. He does all of this in Word and once he starts writing things into actual paragraphs, he likes to see what they look like in the SIGCHI Proceedings format, because sometimes paragraphs tend to look long in the format and sometimes they look short and the format helps to keep the writing consistent for him. He likes to write on a laptop so that he can move around while he is writing. He sketches it out and then revises often. In sections that are similar to ones that he has written before (like Methods or Results) in one of his papers, he goes back to his papers and follows the model that he has successfully used before, because sometimes you forget important aspects and returning to a camera-ready paper ensures that you are using a standard that is accepted (and all the necessary information is present). So, on a paragraph-by-paragraph basis, he can make sure to write everything correctly. He revises a lot (usually around 50 different versions when a paper is written collaboratively) and saves each version as a separate file (admitting that he really could be using version control software or Dropbox’s revision system) because he likes to be able to go back to previous versions if he needs to. He also (shamefully?) admitted that he is not using a bibliographic manager (which can drive his collaborators crazy). He writes everything out with square brackets with Xs in them like [X] wherever he needs a reference and then adds them in last by hand. However, with CHI not having a limit for references anymore, there might be a need for him to switch over to a database-driven reference manager in the future.
When he collaborates, he does it mainly via email and sends around the most recent version of the paper and as long as the “lock” on the file is communicated to everyone (i.e., someone decides to work on a paper for a certain amount of time), it has worked well for him (even when sections are merged later). He does not use Google Docs or other real-time collaboration tools. Once the paper reaches a medium draft stage, he likes to print it out on paper and read it over (where he is also able to mark it up and highlight sections). He finds it easier to read over the flow from section to section and check the style, mark spelling errors and problem spaces quickly. It is also a nice break from endlessly staring at a monitor.
About his Writing Environment
Carl writes his CHI papers mostly at home because writing at the office can be distracting (especially when all the colleagues and grad students are writing CHI papers as well). You learn lots of new and interesting things happening in other paper but cannot focus on your own paper. He likes to write in a space where the distractions are reduced to a minimum. Writing often takes momentum to really start (a particular level of activation) and every time something else happens to distract you, you lose that writing momentum. Whenever he sits down, he tells himself to write at least one sentence. The rationale goes that when you make it past one sentence, it can be easy to write another one and once that is happening, you are gaining momentum to write an entire section and continue to write your paper. This trick makes getting started much easier and helps you gain writing momentum.
About his own Favourite Papers (That he has Written)
Both of his favourite papers are CSCW papers. One is one from 1998 and one from 2012. The first one came out of his Ph.D. thesis but was more speculative and argued that things that are good for individuals are often bad for groups. He likes the paper because it is an exploration of an idea rather than just a regular invention study. The other one was a project that happened while I was doing my postdoc in Saskatchewan and discusses people that are just playing games and do not want to socialize in the game. This had implications for the community and sociability perspectives of game communities. Carl liked it because it conveyed an interesting idea well and we dug into some underlying sociological theories (from Georg Simmel) that Carl found really fascinating. It also won a best-paper award at the conference. The references for both papers are below:
About Tables and Figures
Leaving tables aside as an opportunity to present many numbers outside of the regular text, Carl does care about the figures in a paper and whether they are clear. Do they show the differences that you want to show? Whatever you would like your reader to recognize immediately, you would put into the figures. Again referring to the tired reviewer, the visual aids are important to them and they can be a tool for storytelling and for delivering your message as well (and reinforce your story).
His Main Advice
What Carl was amazed at when he became a Professor, was how important writing was. He calls it the “coin of our realm.” And states that “everything that we do is going to be based on what we write.” Before that, he always thought it was all going to be about research and building systems, running studies, and proper methods. “Any idea that you ever get across to anybody else in HCI is because of what you’ve written,” he continues. Learning to love writing and improving this craft is really important for successful HCI researchers. So, try to make writing as enjoyable as you can for yourself because your career does in some sense depend on it.
It was amazing speaking to Carl and I hope that I will be able to interview him again in the future. All of his advice is really applicable for HCI researchers and should help everyone to improve their writing.
