people

Michael J. Johnston

Johnston, Michael J.
33 Thomas St, 26th Floor, 2617A1
New York, NY
Subject matter expert in multimodal interfaces, virtual assistants, natural language processing, natural language understanding, spoken dialog systems, web standards

My goal is to power the evolution of multimodal conversational assistants and services that enable people to be more productive, effective, and happy, wherever they live, work, and play. This involves fundamental research on the nature of multimodal interaction, determination of principles for multimodal interface design, and the development and evaluation of language processing architectures capable of parsing and understanding multimodal input and generating multimodal output. I am also actively involved in web standards and serve as editor-in-chief of the W3C Multimodal EMMA: Extensible MultiModal Annotation specification.

  • Multimodal human-computer interaction
  • Natural language processing
  • Machine learning
  • Multimedia search and visualization
  • User interface design and evaluation
  • Spoken and multimodal dialogue systems
  • Content extraction and distillation
  • Mobile computing
  • Pen and gesture-based interfaces
  • Information access in urban environments
  • Medical informatics

Publications

Johnston, M., P. Ehlen, F. G. Conrad, M. F. Schober, C. Antoun, S. Fail, A. Hupp, L. Vickers, H. Yan, and C. Zhang. 2013. Spoken Dialog Systems for Automated Survey Interviewing. Proceedings of 14th Annual SIGdial Meeting on Discourse and Dialog, Metz, France. pp. 329-333.

Ehlen, P., and M. Johnston. 2013. A Multimodal Dialogue Interface for Mobile Local Search. Proceedings of the 2013 International Conference on Intelligent User Interfaces. Santa Monica, CA. pp. 63-64.

Johnston, M., P. Ehlen, F. G. Conrad, M. F. Schober, C. Antoun, S. Fail, A. L. Hupp, L. Vickers, H. Yan, C. Zhang.  2013. Reducing Survey Error in a Mobile Speech-IVR system. Paper presented at the Annual Conference of the American Association for Public Opinion Research (AAPOR), Boston, MA.

Conrad, F.G., M. F. Schober, C. Zhang, H. Yan, L. Vickers, M. Johnston, A. L. Hupp, L. Hemingway, S. Fail, P. Ehlen, C. Antoun. 2013. Mode Choice on an iPhone Increases Survey Data Quality. Paper presented at Annual Conference of the American Association for Public Opinion Research (AAPOR), Boston, MA.

Ehlen, P. and M. Johnston. 2012. Multimodal Dialogue in Mobile Local Search. ICMI 2012: Proceedings of the 14th ACM International Conference on Multimodal Interaction, Santa Monica, CA, pp. 303-304.

Ehlen, P. and M. Johnston 2012. Multimodal Interaction Patterns in Mobile Local Search.  Proceedings of the 2012 ACM International Conference on Intelligent User Interface Conference (IUI). pp. 21-24.

Schober, M.F., F. G. Conrad, C. Antoun, D. Carroll, P. Ehlen, S. Fail, A. L. Hupp, M. Johnston, C. Kellner, K. F. Nichols, L. Percifield, L. Vickers, H. Yan, C. Zhang. 2012. Disclosure and quality of answers in text and voice interviews on iPhones. Paper presented at the Annual Conference of the American Association for Public Opinion Research (AAPOR). Orlando, FL.

Feng, J., M. Johnston, and S. Bangalore. 2011. Speech and Multimodal Interaction in Mobile Search. IEEE Signal Processing Magazine. July 2011. 28:4, pp. 40-49.

Worsley, M., M. Johnston, and P. Blikstein. 2011. OpenGesture: A Low-cost Authoring Framework for Gesture and Speech-based Application Development and Learning Analytics. IDC 2011: pp. 254-256.

Johnston, M. and G. DiFabbrizio, and S. Urbanek. 2011. mTalk: A Multimodal Browser for Mobile Services. Proceedings of Interspeech 2011, Florence, Italy, pp. 3261-3264

Johnston, M. and P. Ehlen. 2011. Speak4it and the Multimodal Semantic Interpretation System. Proceedings of Interspeech 2011, Florence, Italy, pp. 3333-3334

Ehlen, P. and M. Johnston. 2011. Multimodal Local Search in Speak4it. Intelligent User Interface Conference (IUI). pp. 435-436.

Ehlen, P. and M. Johnston. 2010. Location Grounding in Multimodal Local Search. ICMI-MLMI 2010. Beijing, China.

Johnston, M. and A. Stent. 2010. EPG: Speech Access to Program Guides for People with Disabilities. ASSETS 2010. pp. 257-258.

Ehlen, P. and M. Johnston. 2010. Speak4it: Multimodal Interaction for Local Search. ICMI-MLMI 2010. Beijing, China.

Worsley, M. and M. Johnston. 2010. Multimodal Interactive Spaces: MagicTV and MagicMAP. Proceedings of IEEE Workshop on Spoken Language Technology, Berkeley, CA.

Johnston, M. and P. Ehlen. 2010. Speak4it: Multimodal Interaction in the Wild. Proceedings of IEEE Workshop on Spoken Language Technology, Berkeley, CA.

Bangalore, S. and M. Johnston. 2009. Robust Understanding in Multimodal Interfaces. Computational Linguistics 35:3, pp. 345-397.

Johnston, M. 2009. Building Multimodal Applications with EMMA.  Proceedings of ICMI-MLMI 2009, Boston, MA. 

Johnston, M., P. Baggia, D. Burnett, J. Carter, D. Dahl, G. McCobb, D. Raggett. 2009. EMMA: Extensible Multimodal Annotation Markup Language. W3C Recommendation February 10th 2009.

Bangalore, S. and M. Johnston. 2008. Robust Gesture Processing for Multimodal Interaction. Proceedings of ICMI 2008. Chania, Crete, Greece.

Johnston, M. 2007. Automating the Survey Interview with Dynamic Multimodal Interfaces. In M. Schober and F. Conrad (eds.) Envisioning the Survey Interview of the Future. John Wiley and Sons.

Johnston, M, P. Ehlen, D. Gibbon, Z. Liu. 2007.The Multimodal Presentation Dashboard. Proceedings of the NAACL-HLT 2007 Workshop: Bridging the Gap: Academic and Industrial Research in Dialog Technologies. pp. 17-24.

Johnston, M., L-F. D'Haro, M. Levine, B. Renger. 2007.A Multimodal Interface for Access to Content in the Home. Proceedings of the Association for Computational Linguistics 45th Annual Conference. pp. 376-383.

Johnston, M. and S. Bangalore. 2006. Learning Edit Machines for Robust Multimodal Understanding. IEEE International Conference on Acoustics, Speech, and Signal Processing.

Johnston, M. and S. Bangalore. 2005. Finite-state Multimodal Integration and Understanding. Journal of Natural Language Engineering 11.2, pp. 159-187, Cambridge University Press.

Johnston, M. and S. Bangalore. 2005. Combining Stochastic and Grammar-based Language Processing with Finite-state Edit Machines. In Proceedings of IEEE Automatic Speech Recognition and Understanding Workshop.

Johnston, M. and S. Bangalore. 2004. MATCHKiosk: A Multimodal Interactive City Guide. In Proceedings of the ACL-2004 Interactive Posters/Demonstrations Session, pp. 222-225.

Johnston,M. and S. Bangalore. 2004. Multimodal Applications from Mobile to Kiosk. W3C Multimodal Workshop. Sophia Antipolis, France.

Bangalore, S. and M. Johnston. 2004. Balancing Data-driven and Rule-based Approaches in the Context of a Multimodal Conversational System. In Proceedings of Human Language Technology conference / North American chapter of the Association for Computational Linguistics annual meeting (HLT-NAACL 2004).

Bangalore, S. and M. Johnston. 2004. Robust Multimodal Understanding. Proceedings of International Conference on Acoustics, Speech, and Signal Processing (ICASSP-2004).

Walker, M., S. Whittaker, A. Stent, P. Maloor, J. Moore, M. Johnston, G. Vasireddy. 2003. Generation and Evaluation of User Tailored Responses in Multimodal Dialogue. Cognitive Science, Vol 28., Issue 5, September-October 2003, Rumelhart Prize Special Issue Honoring Aravind K. Joshi, pp. 811-840.

Johnston, M, S. Bangalore, G. Vasireddy, A. Stent, P. Ehlen, M. Walker, S. Whittaker, P. Maloor. 2002. MATCH: An Architecture for Multimodal Dialogue Systems. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics.

Johnston, M. , S. Bangalore, A. Stent, G. Vasireddy, P. Ehlen. 2002. Multimodal Language Processing for Mobile Information Access. In Proceedings of International Conference on Spoken Language Processing, Denver, Colorado.

Ehlen, P., G. Vasireddy, M. Johnston. 2002.Collecting Mobile Multimodal Data for MATCH. In Proceedings of International Conference on Spoken Language Processing, Denver, Colorado.

Hastie,H., M. Johnston, P. Ehlen. 2002. Context-sensitive Multimodal Help. In Proceedings of the 4th IEEE International Conference on Multimodal Interfaces.

Walker, M., S. Whittaker, A. Stent, P. Maloor, J. Moore, M. Johnston, G. Vasireddy. 2002. Speech plans:generating evaluative responses in spoken dialogue. In Proceedings of International Natural Language Generation Conference.

Johnston, M., S. Bangalore, G. Vasireddy. 2001. MATCH: Multimodal Access To City Help. In Proceedings of ASRU 2001 Workshop. Madonna di Campiglio, Italy.

Johnston, M. and S. Bangalore. 2001. Finite-state methods for multimodal parsing and integration. Finite-state Methods Workshop, ESSLLI Summer School on Logic Language and Information, Helsinki, Finland.

Johnston, M. and S. Bangalore. 2000. Finite-state multimodal parsing and understanding. Proceedings of COLING-2000. Saarbruecken, Germany.

Johnston, M. 2000. Deixis and conjunction in multimodal systems. In Proceedings of COLING-2000, Saarbruecken, Germany.

Bangalore, S. and M. Johnston. 2000. Integrating Multimodal Language Processing with Speech Recognition. Proceedings of International Conference on Spoken Language Processing, Beijing, China.

Johnston, M. 1998. Unification-based Multimodal Parsing. In Proceedings of the 17th International Conference on Computational Linguistics and 36tthAnnual Meeting of the Association for Computational Linguistics. (COLING-ACL 98), pp. 624-630.

Johnston, M. 1998. Multimodal Language Processing. In Proceedings of International Conference on Spoken Language Processing (ICSLP), Sydney, Australia.

Cohen, P. R., M. Johnston, D. McGee, S. L. Oviatt, J. Clow, I. Smith. 1998. The Efficiency of Multimodal Interaction: A Case Study. In Proceedings of the International Conference on Spoken Language Processing (ICSLP), Sydney, Australia.

Johnston, M., P. R. Cohen, D. McGee, S. L. Oviatt, J. Pittman, I. Smith. 1997. Unification-based Multimodal Integration. Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics. Madrid, Spain.

Cohen, P. R., M. Johnston, D. McGee, S. L. Oviatt, J. Pittman, I. Smith, L. Chen, and J. Clow. 1997. QuickSet: Multimodal interaction for distributed applications. In Proceedings of the Fifth ACM International Multimedia Conference. ACM Press: New York.

Cohen, P. R., M. Johnston, D. McGee, S. L. Oviatt, J. Pittman, I. Smith, L. Chen, and J. Clow. 1997. Multimodal interaction for distributed interactive simulation. In Proceedings of Innovative Applications of Artificial Intelligence Conference. AAAI/MIT Press: Menlo Park.

Cohen, P. R., M. Johnston, D. McGee, I. Smith, S. L. Oviatt, J. Pittman, L. Chen, and J. Clow. 1997. QuickSet: Multimodal interaction for simulation set-up and control. Proceedings of the Fifth Applied Natural Language Processing meeting. Association for Computational Linguistics: Washington, D.C.

Johnston, M. 1997. The Telic/Atelic Distinction and the Individuation of Quantificational Domains. in Empirical Issues in Formal Syntax and Semantics: Selected Papers from the Colloque de Syntaxe et de Sémantique de Paris (CSSP 95), Danièle Godard and Francis Corblin (Eds.). Peter Lang, Bern.

Cohen, P. R., L. Chen, J. Clow, M. Johnston, D. McGee, J. Pittman, and I. Smith. 1996. Quickset: A multimodal interface for distributed interactive simulation. Proceedings of the UIST'96 Demonstration Session, Seattle.

Pustejovsky, J., B. Boguraev, M. Verhagen, P. Buitelaar, M. Johnston. 1997. Semantic Indexing and Typed Hyperlinking. In Proceedings of the AAAI Spring Symposium on Natural Language Processing for the World Wide Web. American Association for Artificial Intelligence.

Johnston, M., B. Boguraev, and J. Pustejovsky.1995. The Acquisition and Interpretation of Complex Nominals. In Working Notes of AAAI Spring Symposium on the Representation and Acquisition of Lexical Knowledge. American Association for Artificial Intelligence.

Projects
Assistive Technology, At AT&T Labs - Research, we apply our speech, language and media technologies to give people with disabilities more independence, privacy and autonomy.

Connecting Your World, The need to be connected is greater than ever, and AT&T Researchers are creating new ways for people to connect with one another and with their environments, whether it's their home, office, or car.

iMIRACLE - Content Retrieval on Mobile Devices with Speech, iMIRACLE uses large vocabulary speech recognition for content retrieval with metadata words (titles, genre, channels, etc.) and content words that occur in recorded programs.

Patents

Systems, Methods, And Computer Program Products For Location Salience Modeling For Multimodal Search, April 15, 2014
Multimodal Augmented Reality For Location Mobile Information Service, April 1, 2014
Systems And Methods For Extracting Meaning From Multimodal Inputs Using Finite-State Devices, January 7, 2014
Systems And Methods For Generating Markup-Language Based Expressions From Multi-Modal And Unimodal Inputs, April 30, 2013
Systems And Methods For Extracting Meaning From Multimodal Inputs Using Finite-State Devices, January 15, 2013
Automated Demographic Analysis By Analyzing Voice Activity, October 30, 2012
Systems And Methods For Extracting Meaning From Multimodal Inputs Using Finite-State Devices, July 3, 2012
Customized Interface Based On Viewed Programming, November 8, 2011
Context-Sensitive Interface Widgets For Multi-Modal Dialog Systems, February 15, 2011
System And Method Of Identifying Web Page Semantic Structures, August 24, 2010
Learning Edit Machined For Robust Multimodal Understanding, May 11, 2010
Systems And Methods For Generating Markup-Language-Based Expressions From Multi-Modal And Unimodal Inputs, February 9, 2010
System and method of handling problematic input during context-sensitive help for multi-modal dialog systems, November 11, 2008
Systems and methods for extracting meaning from multimodal inputs using finite-state devices, November 13, 2007
Systems and methods for generating markup-language based expressions from multi-modal and unimodal inputs, August 14, 2007
System and method for accessing and annotating electronic medical records using multi-modal interface, May 29, 2007
System and method of handling problematic input during context-sensitive help for multi-modal dialog systems, February 13, 2007
System and method of context-sensitive help for multi-modal dialog systems, February 13, 2007
Systems and methods for extracting meaning from multimodal inputs using finite-state devices, June 27, 2006
Systems and methods for extracting meaning from multimodal inputs using finite-state devices, March 15, 2005

Publications

2009_Johnston_EMMA_ICMI
  2009_Johnston_EMMA_ICMI.pdf  (879k)

assets_paper_2010
  assets_paper_2010.pdf  (731k)

ehlen_johnston_2012_ICMI
  ehlen_johnston_2012_ICMI.pdf  (461k)

ehlen_johnston_icmi_2010_location_grounding
  ehlen_johnston_icmi_2010_location_grounding.pdf  (1256k)

ehlen_johnston_iui_2012
  ehlen_johnston_iui_2012.pdf  (805k)

ehlen_johnston_iui_2013
  ehlen_johnston_iui_2013.pdf  (1429k)

icmi2008_johnston_bangalore
  icmi2008_johnston_bangalore.pdf  (454k)

johnstonehlen_speak4it2011interspeech
  johnstonehlen_speak4it2011interspeech.pdf  (763k)

johnstonetal_mtalk2011
  johnstonetal_mtalk2011.pdf  (2836k)

johnstonetal_sigdial_2013
  johnstonetal_sigdial_2013.pdf  (812k)

magic_slt_2010
  magic_slt_2010.pdf  (428k)

speak4it_icmi_2010
  speak4it_icmi_2010.pdf  (861k)

speak4it_slt_2010
  speak4it_slt_2010.pdf  (576k)