Rebecca Krosnick
About me:
I am a final year PhD student in the EECS department at the University of Michigan, advised by Professor Steve Oney. I study human-computer interaction and am interested in designing intuitive tools for programming and other creation domains.
In my PhD research I have focused on studying and designing programming tools for creating user interface automation macros. Specifically, I have explored the challenges developers faced, designed novel IDE features that integrate UI and code, and designed programming-by-demonstration approaches that enable users to create UI automation macros without writing code but instead through natural interactions (e.g., direct manipluation, natural lanuage, examples). I also have designed a programming-by-demonstration system for creating responsive user interfaces.
I have interned in the HCI and Visualization group at Autodesk Research, the EPIC group at Microsoft Research, and the UI Understanding research group at Apple AI/ML, and worked as a software engineer at MathWorks. I received my SB and MEng in Computer Science from MIT, where I was advised by Professor Rob Miller.
Papers
Conference Papers
Jason Wu, Rebecca Krosnick, Eldon Schoop, Amanda Swearngin, Jeffrey P. Bigham, Jeffrey Nichols. Never-ending Learning of User Interfaces. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2023).
[PDF]
[PDF]
Rebecca Krosnick and Steve Oney. ParamMacros: Creating UI Automation Leveraging End-User Natural Language Parameterization. In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2022).
[PDF] [Demo video]
[PDF] [Demo video]
Rebecca Krosnick and Steve Oney. Understanding the Challenges and Needs of Programmers Writing Web Automation Scripts. In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2021).
[PDF] [Supplemental materials] [Talk] [Demo video]
[PDF] [Supplemental materials] [Talk] [Demo video]
Rebecca Krosnick, Fraser Anderson, Justin Matejka, Steve Oney, Walter S. Lasecki, Tovi Grossman, George Fitzmaurice. Think-Aloud Computing: Supporting Rich and Low-Effort Knowledge Capture. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2021).
[PDF] [Supplemental materials] [Talk] [4 min video] [30 sec video]
[PDF] [Supplemental materials] [Talk] [4 min video] [30 sec video]
Steve Oney, Rebecca Krosnick, Joel Brandt, Brad Myers. Implementing Multi-Touch Gestures with Touch Groups and Cross Events. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2019). Honorable Mention.
[PDF]
[PDF]
Sang Won Lee, Rebecca Krosnick, Sun Young Park, Brandon Keelean, Sach Vaidya, Stephanie D. O'Keefe, Walter S. Lasecki. Exploring Real-Time Collaboration in Crowd-Powered Systems Through a UI Design Tool. In Proceedings of the ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2018).
[PDF] [Blog post]
[PDF] [Blog post]
Steve Oney, Alan Lundgard, Rebecca Krosnick, Michael Nebeling, Walter S. Lasecki. Arboretum and Arbility: Improving Web Accessibility Through a Shared Browsing Architecture. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2018).
[PDF]
[PDF]
Rebecca Krosnick, Sang Won Lee, Walter S. Lasecki, Steve Oney. Expresso: Building Responsive Interfaces with Keyframes. In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2018).
[PDF] [Study materials] [Demo video]
[PDF] [Study materials] [Demo video]
Rohit Singh, Rishabh Singh, Zhilei Xu, Rebecca Krosnick, Armando Solar-Lezama. Modular Synthesis of Sketches Using Models. International Conference on Verification, Model Checking and Abstract Interpretation (VMCAI). 2014.
[PDF]
[PDF]
Workshop and Demo Papers
Jason Wu, Rebecca Krosnick, Eldon Schoop, Amanda Swearngin, Jeffrey P. Bigham, Jeffrey Nichols. Towards Never-ending Learning of User Interfaces. AI & HCI Workshop at the International Conference on Machine Learning (ICML).
[PDF]
[PDF]
Rebecca Krosnick and Steve Oney. Promises and Pitfalls of Using LLMs for Scraping Web UIs. CHI 2023 Workshop on The Future of Computational Approaches for Understanding & Adapting User Interfaces.
[PDF]
[PDF]
Rebecca Krosnick and Steve Oney. UI Folding: Specifying and Visualizing Generalization in UI Automation Macros. PLATEAU: 13th Annual Workshop at the Intersection of PL and HCI.
Rebecca Krosnick and Steve Oney. Providing Feedback and UI Context to Programmers Writing Web Automation Scripts. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2021), Posters and Showpieces.
[PDF] [Demo video]
[PDF] [Demo video]
Rebecca Krosnick and Steve Oney. Toward Providing Live Feedback in Web Automation IDEs. LIVE workshop at the ACM SIGPLAN Conference on Systems, Programming, Languages, and Applications: Software for Humanity (LIVE 2020).
[PDF] [Talk] [Demo video]
[PDF] [Talk] [Demo video]
Symposia
Rebecca Krosnick. Tools for Creating UI Automation Macros. In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2022), Graduate Consortium.
[PDF]
[PDF]
Rebecca Krosnick. Designing Tools for Intuitive Creation, Generalization, and Maintenance of Web Macros. PL+HCI "Swimmer" School 2020, Doctoral Symposium.
[PDF]
[PDF]
Rebecca Krosnick. Creating Interactive User Interfaces by Demonstration using Crowdsourcing. In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2018), Graduate Consortium.
[PDF]
[PDF]
Thesis
Rebecca Krosnick. VideoDoc: Combining Videos and Lecture Notes for a Better Learning Experience. MIT Department of EECS MEng Thesis. 2015.
[PDF]
[PDF]
Projects
Web Automation IDEs
How can we build more intuitive tools to help people create web automation scripts? I'm exploring a combination of recording user page actions, presenting UI visual context, and providing live feedback and suggestions.
[Workshop paper] [Talk] [Demo video]
[Workshop paper] [Talk] [Demo video]
Expresso
A programming-by-demonstration approach for creating responsive UIs, where users create keyframes at a few representative viewport sizes and specify smooth or jump (i.e., linear interpolation) transitions between them.
[PDF] [Demo video]
[PDF] [Demo video]
VideoDoc
As online education grows in popularity, more people are learning using educational lecture videos. Videos are engaging to watch, but difficult to navigate when searching for a particular topic. To address this problem, we built VideoDoc, a lecture video interface that merges a video with the organizational features of a textbook to make educational videos easier to navigate.
[MEng Thesis]
[MEng Thesis]