TypeScript Extensibility

This is an umbrella issue for tracking the design of an extensibility model for TypeScript.

There have already been a number of issues opened with regards to supporting Angular2 via such a model, (especially for richer template editing support), so this will be the initial focus. This not only ensures rich support for a key TypeScript scenario, but by solving real problems, it also ensures we are not designing in a vacuum.

High level problems

This section captures an overview of the problems to be solved. These are not of equal priority, and not all features below may be implemented, but are captured here for completeness, and to be considered in the broader picture.

• Syntax highlighting: Template strings inside a TypeScript file currently show as simple string literals, yet ideally would be colored in a semantically meaninful way.
• Additional file-types and structure: Angular templates may live in a separate file, containing the HTML-like template contents. For React, TypeScript added a new file extension (.tsx) and matching grammar/parser changes for this file type. Ideally this would be an extensibility point, not baked into TypeScript.
• Syntactic and semantic diagnostics: Both the compiler and the language service should be able to surface domain specific errors in areas such as syntax, type usage, deprecation notices, etc. Such errors should map to original source locations, not intermediate artifacts.
• Rich IDE features: Statement and member completion, signature help, find all references, go to defintion, rename, etc. should work within the template contents as expected.
• Custom commands: Some plugins may provide editors additional functionality not exposed by the existing language service API (e.g. an editor visualization for the component hierarchy). A plugin should be able to expose additional commands for such usage.

Challenges

This section captures non-trivial challenges to be addressed, either with the problem domain in general or the current TypeScript architecture.

• Syntax highlighting: Several editors, (e.g. Sublime Text, VSCode, etc.), use a TextMate bundle for syntax highlighting. This is generally a static file describing the grammar for a file type, usually detected by file extension. Expanding the grammar via a plugin is a challenge. Even if statically augmenting the grammar, detecting when a string literal is an Angular template or not, or this .html file is an Angular template or regular .html file, is also challenging to do at parse time (and often depends on type information for inline template, or path resolution for external templates, to determine definitively).
• Syntax highlighting: TextMate aside, other editors (e.g. Visual Studio) do syntax highlighting via other means (such as calling the TypeScript classifier), which would need to be augmented somehow also.
• Program structure: In order to provide some of the above functionality, it is desirable to generate an intermediate representation (e.g. containing a "compiled template") and ask TypeScript questions about this code, then map the results back to the original source. However, the representation of the program that the compiler or language service has is immutable. It is not possible to receive a request about a source location, generate an intermediate representation of it, update the program with the generated code mid-request, then ask TypeScript questions about it before responding.
• Loading: TypeScript runs in environments besides Node.js (e.g. in-process in Visual Studio, or the tsc.exe command-line compiler), thus a plugin can not make assumptions such as being able to call require or take dependencies on other Node.js packages.
• Performance: As the program is immutable, a new program is created on every edit. Most language service operations require that the program is bound and type-checked, which is often done on-demand when a question is asked. For acceptable performance, (faster than 50ms response time), processing should be kept to a minimum and done entirely in memory where possible (e.g. avoid serializing/reading generated artifacts on disk such as temporary code, source map files, etc.).

Angular specific challenges

• Ambiguity: What if two different components have a templateUrl that resolves to the same file on disk? In which context should TypeScript evaluate that file?
• Determination: If a user opens an HTML file, how can the editor determine if this is an Angular template rather than just a regular HTML file without loading the full TypeScript program and resolving all the templateUrl properties?
• Path resolution: What if the path on disk doesn't match the path at runtime? Is some type of baseUrl or config object needed to map paths from the TypeScript source to the template files?
• Dynamic content: TypeScript needs to be able to statically analyze code that may be dynamically generated at runtime. For example, how to handle an inline template that reads <div>${getStart()}<foo [prop]='expr'/>${getEnd()}</div>, or analyze the component with the directives given as directives: getMyDirectives()?
• Name resolution: Within an expression in a template, scoping is quite different to normal. All instance members are available as top level names (i.e. no this. needed), and the usual global members aren't available (i.e. can't reference Date.now, parseInt, or similar). Thus resolving names and providing completion lists requires quite different logic to the usual TypeScript behavior.
• Micro-syntaxes: Within Angular templates, expressions are mostly JavaScript expressions, but not quite. For example, certain operators have different meanings, arguments are passed differently when using pipes, etc. See https://angular.io/docs/ts/latest/guide/template-syntax.html for more info.

Current work

The below is currently experimental work to spike various approaches.

• TextMate grammar to provide syntax highlighting for inline templates: https://github.com/billti/TypeScript-TmLanguage/tree/ngml
  • Note: This current uses a heuristic of determining if a template string is a value for an object literal property named template, or is preceded with a /** @html */ comment. See https://github.com/billti/TypeScript-TmLanguage/blob/ngml/TypeScript.YAML-tmLanguage#L507
• A fork of TypeScript with a rudimentary plugin model (plugin source lives within the codebase currently) and some initial feature support: https://github.com/billti/TypeScript/tree/ngml
  • TODO: Update with the latest TypeScript and Angular2 releases.
• Usage in VSCode:
  1. Replace the typescript.tmLanguage file under the install location at Contents/Resources/app/extensions/typescript/syntaxes with the version from the TypeScript-TmLanguage repo above (from the ngml branch).
  2. Fork the TypeScript repo above, checkout the ngml branch, and build.
  3. Open the VSCode global settings, and set the typescript.tsdk property to the build location in the prior step (e.g. /src/typescript/built/local).
  4. To get completions within the template on characters such as <, [, etc. open typescriptMain.ts from the Contents/Resources/app/extensions/typescript folder under the VSCode install location, and change the line that calls registerCompletionItemProvider to read vscode_1.languages.registerCompletionItemProvider(modeID, completionItemProvider, '.', '<', '(', '[', '\'');
  5. To make changes edit the local TypeScript repo, rebuild, and relaunch VSCode.
• Limitations:
  • Currently there is no dynamic plugin model. The plugin is compiled into the TypeScript service. To minimize build changes, all plugin code is lumped into one file currently (src/services/plugin-ngml.ts), as are some basic unit tests.
  • It currently uses its own rudimentary template parser. I haven't spent the time to figure out how to reuse the code from angular2/src/compiler (if possible)
  • All attribute values should be in single quotes. Unquoted or double quoted doesn't parse properly in the textMate grammar yet.
  • Interpolation inside attribute values is not wired up yet, i.e. <foo name='Hi {{name}}'></foo> won't work yet.
  • Attribute bindings (i.e. dotting off of attr) don't work yet, i.e. <td [attr.colspan]='expr'>
  • Only expressions where the first token is a member bind correctly, i.e. mem1(mem2) + mem3 will only bind mem1 to the class instance member correctly.
  • Components within components do not work yet (in fact, none of the directives values are resolved currently).
  • Special directives, such as *ngFor='#item of items', aren't implemented yet.
  • Angular specific syntax, such as pipes or the Elvis operator don't work yet, i.e. mem?.value | mypipe:"arg1"
  • Naming conversion still uses snake-case, and needs to be updated for the Beta release changes to attribute name mapping.
  • Rename isn't working yet (but is simple since the Beta change to simplify name mapping and is nearly done)
  • Still to figure out work for features such as outlets, etc.
• Current features
  • Completions and errors on HTML tags (using a basic list of HTML elements)
  • Completions on member expressions
  • Syntactic and type errors on member expressions
  • Signature help and tool-tips on expressions
  • Data and event binding using the [prop] or (event) syntax
  • Introduction of scoped template locals using the #name attribute syntax
  • Goto definition on identifers within template expressions
  • The below shows an example of the plugin in action

• TODO
  • Detail the architecture and which of the above challenges informed which choices
  • Figure out how to plug in to the compiler as well as the language service for the errors as build time as well as design time

Existing related issues:

• [typescript checking frontend template files #5151] typescript checking frontend template files
• [[tsserver] custom command (ex: to collect Angular2 @Component/selector) #5730] [tsserver] custom command (ex: to collect Angular2 @Component/selector)
• [Design Meeting Notes, 11/20/2015 #5740] Allow evaluation of a detached buffer in a specified context (angular2 templates)
• [Language Serivice support in Angular 2 inline templates #6477] [tsx] Angular 2 inline templates