AT&T Labs Research — Leading Invention, Driving Innovation

My goal is to power the evolution of multimodal conversational assistants and services that enable people to be more productive, effective, and happy, wherever they live, work, and play. This involves fundamental research on the nature of multimodal interaction, determination of principles for multimodal interface design, and the development and evaluation of language processing architectures capable of parsing and understanding multimodal input and generating multimodal output. I am also actively involved in web standards and serve as editor-in-chief of the W3C Multimodal EMMA: Extensible MultiModal Annotation specification.

• Multimodal human-computer interaction
• Natural language processing
• Machine learning
• Multimedia search and visualization
• User interface design and evaluation
• Spoken and multimodal dialogue systems
• Content extraction and distillation
• Mobile computing
• Pen and gesture-based interfaces
• Information access in urban environments
• Medical informatics

Publications

Feng, J., M. Johnston, and S. Bangalore. 2011. Speech and Multimodal Interaction in Mobile Search. IEEE Signal Processing Magazine. July 2011. 28:4, pp. 40-49.

Worsley, M., M. Johnston, and P. Blikstein. 2011. OpenGesture: A Low-cost Authoring Framework for Gesture and Speech-based Application Development and Learning Analytics. IDC 2011: pp. 254-256.

Johnston, M. and G. DiFabbrizio, and S. Urbanek. 2011. mTalk: A Multimodal Browser for Mobile Services. Proceedings of Interspeech 2011, Florence, Italy, pp. 3261-3264

Johnston, M. and P. Ehlen. 2011. Speak4it and the Multimodal Semantic Interpretation System. Proceedings of Interspeech 2011, Florence, Italy, pp. 3333-3334

Ehlen, P. and M. Johnston. 2011. Multimodal Local Search in Speak4it. Intelligent User Interface Conference (IUI). pp. 435-436.

Ehlen, P. and M. Johnston. 2010. Location Grounding in Multimodal Local Search. ICMI-MLMI 2010. Beijing, China.

Johnston, M. and A. Stent. 2010. EPG: Speech Access to Program Guides for People with Disabilities. ASSETS 2010. pp. 257-258.

Ehlen, P. and M. Johnston. 2010. Speak4it: Multimodal Interaction for Local Search. ICMI-MLMI 2010. Beijing, China.

Worsley, M. and M. Johnston. 2010. Multimodal Interactive Spaces: MagicTV and MagicMAP. Proceedings of IEEE Workshop on Spoken Language Technology, Berkeley, CA.

Johnston, M. and P. Ehlen. 2010. Speak4it: Multimodal Interaction in the Wild. Proceedings of IEEE Workshop on Spoken Language Technology, Berkeley, CA.

Bangalore, S. and M. Johnston. 2009. Robust Understanding in Multimodal Interfaces. Computational Linguistics 35:3, pp. 345-397.

Johnston, M. 2009. Building Multimodal Applications with EMMA.  Proceedings of ICMI-MLMI 2009, Boston, MA. 

Johnston, M., P. Baggia, D. Burnett, J. Carter, D. Dahl, G. McCobb, D. Raggett. 2009. EMMA: Extensible Multimodal Annotation Markup Language. W3C Recommendation February 10th 2009.

Bangalore, S. and M. Johnston. 2008. Robust Gesture Processing for Multimodal Interaction. Proceedings of ICMI 2008. Chania, Crete, Greece.

Johnston, M. 2007. Automating the Survey Interview with Dynamic Multimodal Interfaces. In M. Schober and F. Conrad (eds.) Envisioning the Survey Interview of the Future. John Wiley and Sons.

Johnston, M, P. Ehlen, D. Gibbon, Z. Liu. 2007.The Multimodal Presentation Dashboard. Proceedings of the NAACL-HLT 2007 Workshop: Bridging the Gap: Academic and Industrial Research in Dialog Technologies. pp. 17-24.

Johnston, M., L-F. D'Haro, M. Levine, B. Renger. 2007.A Multimodal Interface for Access to Content in the Home. Proceedings of the Association for Computational Linguistics 45th Annual Conference. pp. 376-383.

Johnston, M. and S. Bangalore. 2006. Learning Edit Machines for Robust Multimodal Understanding. IEEE International Conference on Acoustics, Speech, and Signal Processing.

Johnston, M. and S. Bangalore. 2005. Finite-state Multimodal Integration and Understanding. Journal of Natural Language Engineering 11.2, pp. 159-187, Cambridge University Press.

Johnston, M. and S. Bangalore. 2005. Combining Stochastic and Grammar-based Language Processing with Finite-state Edit Machines. In Proceedings of IEEE Automatic Speech Recognition and Understanding Workshop.

Johnston, M. and S. Bangalore. 2004. MATCHKiosk: A Multimodal Interactive City Guide. In Proceedings of the ACL-2004 Interactive Posters/Demonstrations Session, pp. 222-225.

Johnston,M. and S. Bangalore. 2004. Multimodal Applications from Mobile to Kiosk. W3C Multimodal Workshop. Sophia Antipolis, France.

Bangalore, S. and M. Johnston. 2004. Balancing Data-driven and Rule-based Approaches in the Context of a Multimodal Conversational System. In Proceedings of Human Language Technology conference / North American chapter of the Association for Computational Linguistics annual meeting (HLT-NAACL 2004).

Bangalore, S. and M. Johnston. 2004. Robust Multimodal Understanding. Proceedings of International Conference on Acoustics, Speech, and Signal Processing (ICASSP-2004).

Walker, M., S. Whittaker, A. Stent, P. Maloor, J. Moore, M. Johnston, G. Vasireddy. 2003. Generation and Evaluation of User Tailored Responses in Multimodal Dialogue. Cognitive Science, Vol 28., Issue 5, September-October 2003, Rumelhart Prize Special Issue Honoring Aravind K. Joshi, pp. 811-840.

Johnston, M, S. Bangalore, G. Vasireddy, A. Stent, P. Ehlen, M. Walker, S. Whittaker, P. Maloor. 2002. MATCH: An Architecture for Multimodal Dialogue Systems. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics.

Johnston, M. , S. Bangalore, A. Stent, G. Vasireddy, P. Ehlen. 2002. Multimodal Language Processing for Mobile Information Access. In Proceedings of International Conference on Spoken Language Processing, Denver, Colorado.

Ehlen, P., G. Vasireddy, M. Johnston. 2002.Collecting Mobile Multimodal Data for MATCH. In Proceedings of International Conference on Spoken Language Processing, Denver, Colorado.

Hastie,H., M. Johnston, P. Ehlen. 2002. Context-sensitive Multimodal Help. In Proceedings of the 4th IEEE International Conference on Multimodal Interfaces.

Walker, M., S. Whittaker, A. Stent, P. Maloor, J. Moore, M. Johnston, G. Vasireddy. 2002. Speech plans:generating evaluative responses in spoken dialogue. In Proceedings of International Natural Language Generation Conference.

Johnston, M., S. Bangalore, G. Vasireddy. 2001. MATCH: Multimodal Access To City Help. In Proceedings of ASRU 2001 Workshop. Madonna di Campiglio, Italy.

Johnston, M. and S. Bangalore. 2001. Finite-state methods for multimodal parsing and integration. Finite-state Methods Workshop, ESSLLI Summer School on Logic Language and Information, Helsinki, Finland.

Johnston, M. and S. Bangalore. 2000. Finite-state multimodal parsing and understanding. Proceedings of COLING-2000. Saarbruecken, Germany.

Johnston, M. 2000. Deixis and conjunction in multimodal systems. In Proceedings of COLING-2000, Saarbruecken, Germany.

Bangalore, S. and M. Johnston. 2000. Integrating Multimodal Language Processing with Speech Recognition. Proceedings of International Conference on Spoken Language Processing, Beijing, China.

Johnston, M. 1998. Unification-based Multimodal Parsing. In Proceedings of the 17th International Conference on Computational Linguistics and 36^tthAnnual Meeting of the Association for Computational Linguistics. (COLING-ACL 98), pp. 624-630.

Johnston, M. 1998. Multimodal Language Processing. In Proceedings of International Conference on Spoken Language Processing (ICSLP), Sydney, Australia.

Cohen, P. R., M. Johnston, D. McGee, S. L. Oviatt, J. Clow, I. Smith. 1998. The Efficiency of Multimodal Interaction: A Case Study. In Proceedings of the International Conference on Spoken Language Processing (ICSLP), Sydney, Australia.

Johnston, M., P. R. Cohen, D. McGee, S. L. Oviatt, J. Pittman, I. Smith. 1997. Unification-based Multimodal Integration. Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics. Madrid, Spain.

Cohen, P. R., M. Johnston, D. McGee, S. L. Oviatt, J. Pittman, I. Smith, L. Chen, and J. Clow. 1997. QuickSet: Multimodal interaction for distributed applications. In Proceedings of the Fifth ACM International Multimedia Conference. ACM Press: New York.

Cohen, P. R., M. Johnston, D. McGee, S. L. Oviatt, J. Pittman, I. Smith, L. Chen, and J. Clow. 1997. Multimodal interaction for distributed interactive simulation. In Proceedings of Innovative Applications of Artificial Intelligence Conference. AAAI/MIT Press: Menlo Park.

Cohen, P. R., M. Johnston, D. McGee, I. Smith, S. L. Oviatt, J. Pittman, L. Chen, and J. Clow. 1997. QuickSet: Multimodal interaction for simulation set-up and control. Proceedings of the Fifth Applied Natural Language Processing meeting. Association for Computational Linguistics: Washington, D.C.

Johnston, M. 1997. The Telic/Atelic Distinction and the Individuation of Quantificational Domains. in Empirical Issues in Formal Syntax and Semantics: Selected Papers from the Colloque de Syntaxe et de Sémantique de Paris (CSSP 95), Danièle Godard and Francis Corblin (Eds.). Peter Lang, Bern.

Cohen, P. R., L. Chen, J. Clow, M. Johnston, D. McGee, J. Pittman, and I. Smith. 1996. Quickset: A multimodal interface for distributed interactive simulation. Proceedings of the UIST'96 Demonstration Session, Seattle.

Pustejovsky, J., B. Boguraev, M. Verhagen, P. Buitelaar, M. Johnston. 1997. Semantic Indexing and Typed Hyperlinking. In Proceedings of the AAAI Spring Symposium on Natural Language Processing for the World Wide Web. American Association for Artificial Intelligence.

Johnston, M., B. Boguraev, and J. Pustejovsky.1995. The Acquisition and Interpretation of Complex Nominals. In Working Notes of AAAI Spring Symposium on the Representation and Acquisition of Lexical Knowledge. American Association for Artificial Intelligence.