Common properties and methods

Nodes, tokens, and trivia have a number of important common properties and methods:

SyntaxTree property

Returns the syntax tree to which the object belongs.

Span property

Returns the object’s position in source code (see “Finding a child by its offset”).

Kind extension method

Returns a SyntaxKind enum that classifies the node, token, or trivia into one of several hundred values (e.g., IntKeyword, CommaToken, and WhitespaceTrivia). The same SyntaxKind enum covers nodes, tokens, and trivia.

ToString method

Returns the text (source code) for the node, token, or trivia. For tokens, the Text property is equivalent.

GetDiagnostics method

Returns errors or warnings generated during parsing.

IsEquivalentTo method

Returns true if the object is identical to another node, token, or trivia instance. Whitespace differences are significant (to ignore whitespace, call NormalizeWhitespace before comparing).

The Kind extension method is a shortcut for casting the RawKind property, which is of type int, to Microsoft.CodeAnalysis.CSharp.SyntaxKind. The reason for not simply having a Kind property of type SyntaxKind is that the token and trivia types are also used in VB syntax trees, which have a different enum type for SyntaxKind.

Traversing children

SyntaxNode exposes LINQ-friendly methods to traverse its child nodes and tokens. The simplest are:

IEnumerable<SyntaxNode> ChildNodes()
IEnumerable<SyntaxToken> ChildTokens()

Following on from our previous example, our root node has a single child node of type ClassDeclarationSyntax:

var cds = (ClassDeclarationSyntax) root.ChildNodes().Single();

We can enumerate the members of cds via either its ChildNodes method or the Members property of ClassDeclarationSyntax:

foreach (MemberDeclarationSyntax member in cds.Members)
  Console.WriteLine (member.ToString());

with the following result:

static void Main() => Console.WriteLine (""Hello"");

There are also Descendant* methods which descend recursively into children. We can enumerate the tokens that make up our program as follows:

foreach (var token in root.DescendantTokens())
  Console.WriteLine ($"{token.Kind(),-30} {token.Text}");

Here’s the result:

ClassKeyword                   class
IdentifierToken                Test
OpenBraceToken                 {
StaticKeyword                  static
VoidKeyword                    void
IdentifierToken                Main
OpenParenToken                 (
CloseParenToken                )
EqualsGreaterThanToken         =>
IdentifierToken                Console
DotToken                       .
IdentifierToken                WriteLine
OpenParenToken                 (
StringLiteralToken             "Hello"
CloseParenToken                )
SemicolonToken                 ;
CloseBraceToken                }
EndOfFileToken

Notice that there’s no whitespace in the result. Replacing token.Text with token.ToFullString() would give us whitespace (and any other trivia).

The following uses the DescendantNodes method to locate the syntax node for our method declaration:

var ourMethod = root.DescendantNodes()
                    .First (m => m.Kind() == SyntaxKind.MethodDeclaration);

or alternatively:

var ourMethod = root.DescendantNodes()
                    .OfType<MethodDeclarationSyntax>()
                    .Single();

With the latter example, ourMethod is of type MethodDeclarationSyntax, which exposes useful properties specific to method declarations. For instance, if our example contained more than one method definition, and we wanted to find just the method whose name is “Main”, we could do this:

var mainMethod = root.DescendantNodes()
                     .OfType<MethodDeclarationSyntax>()
                     .Single (m => m.Identifier.Text == "Main");

Identifier is a property on MethodDeclarationSyntax that returns the token corresponding to the method’s identifier (i.e., its name). We could get the same result with more effort, as follows:

root.DescendantNodes().First (m =>
  m.Kind() == SyntaxKind.MethodDeclaration &&
  m.ChildTokens().Any (t =>
    t.Kind() == SyntaxKind.IdentifierToken && t.Text == "Main"));

SyntaxNode also has GetFirstToken and GetLastToken methods which are equivalent to calling DescendantTokens().First() and DescendantTokens().Last().

As nodes can contain both child nodes and tokens whose relative order is significant, there are also methods to enumerate both together:

ChildSyntaxList ChildNodesAndTokens()
IEnumerable<SyntaxNodeOrToken> DescendantNodesAndTokens()
IEnumerable<SyntaxNodeOrToken> DescendantNodesAndTokensAndSelf()

(ChildSyntaxList implements IEnumerable<SyntaxNodeOrToken> while also exposing a Count property and an indexer to access an element by position.)

You can traverse trivia directly from a node with the GetLeadingTrivia, GetTrailingTrivia, and DescendantTrivia methods. More commonly, though, you’d access trivia through the token to which it’s attached, via the token’s LeadingTrivia and TrailingTrivia properties. Or to convert to text, you’d use the ToFullString method, which includes trivia in the result.

Traversing parents

Nodes and tokens have a Parent property of type SyntaxNode.

For SyntaxTrivia, the “parent” is its token, accessible via the Token property.

Nodes also have methods which ascend back up the tree, which are prefixed with “Ancestor”.

Finding a child by its offset

All nodes, tokens, and trivia have a Span property of type TextSpan to indicate starting and ending offsets in the source code. Nodes and tokens also have a FullSpan property which includes leading and trailing trivia (whereas Span does not). A node’s Span does, however, include child nodes and tokens.

You can find a descendant object by position with the FindNode, FindToken, and FindTrivia methods on SyntaxNode. These methods return the descendant object with the smallest span that fully contains the span that you specify. There’s also a ChildThatContainsPosition method which searches both descendant nodes and tokens.

Should a search result in two nodes with an identical span (typically a child and grandchild), the FindNode method will return the outer (parent) node. You can change this behavior by passing true to the optional argument getInnermostNodeForTie.

The Find* methods also have an optional findInsideTrivia bool parameter. If true, this also searches for nodes or tokens within structured trivia (see “Trivia”).

CSharpSyntaxWalker

Another way to traverse a tree is by subclassing CSharpSyntaxWalker, overriding one or more of its hundreds of virtual methods. This following class counts the number of if statements:

class IfCounter : CSharpSyntaxWalker
{
  public int IfCount { get; private set; }

  public override void VisitIfStatement (IfStatementSyntax node)
  {
    IfCount++;
    // Call the base method if you want to descend into children.
    base.VisitIfStatement (node);
  }
}

Here’s how to invoke it:

var ifCounter = new IfCounter ();
ifCounter.Visit (root);
Console.WriteLine ($"I found {ifCounter.IfCount} if statements");

The result is equivalent to:

root.DescendantNodes().OfType<IfStatementSyntax>().Count()

Writing a syntax walker can be easier than using the Descendant* methods in more complex cases when you need to override multiple methods (in part, because C# has no F#-like pattern matching ability).

By default, CSharpSyntaxWalker visits just nodes. To visit tokens or trivia, you must call the base constructor with a SyntaxWalkerDepth, indicating the desired depth (node→token→trivia). Then you can override VisitToken and VisitTrivia:

class WhiteWalker : CSharpSyntaxWalker   // Counts space characters
{
  public int SpaceCount { get; private set; }

  public WhiteWalker() : base (SyntaxWalkerDepth.Trivia) { }

  public override void VisitTrivia (SyntaxTrivia trivia)
  {
    SpaceCount += trivia.ToString().Count (char.IsWhiteSpace);
    base.VisitTrivia (trivia);
  }
}

If you remove WhiteWalker’s call to the base constructor, VisitTrivia will not fire.

Preprocessor directives

It might seem odd that preprocessor directives are considered trivia, given that some directives (in particular, conditional compilation directives) have a nontrivial effect on the output.

The reason is that preprocessor directives are processed semantically by the parser itself, i.e., it’s the parser’s job to do the preprocessing. After which, there’s nothing left that the compiler need explicitly consider (except for #pragma). To illustrate, let’s examine how the parser handles conditional compilation directives:

#define FOO

#if FOO
    Console.WriteLine ("FOO is defined");
#else
    Console.WriteLine ("FOO is not defined");
#endif

Upon reading the #if FOO directive, the parser knows that FOO is defined, and so the line that follows is parsed normally (as nodes and tokens), whereas the line of code following the #else directive is parsed into DisabledTextTrivia.

Hence, with conditional compilation, it is precisely the text that can be ignored that ends up in trivia (i.e., the inactive code and the preprocessor directives themselves).

The #line directive is handled similarly, in that the parser reads and interprets the directive. The information that it harvests is used when you call GetMappedLineSpan on the syntax tree.

The #region directive is semantically empty: the only role of the parser is to check that #region directives are matched with #endregion directives. The #error and #warning directives are also processed by the parser, which generates errors and warnings that you can see by calling GetDiagnostics on the tree or node.

It can be still useful to examine the content of preprocessor directives for purposes other than producing the output assembly (syntax highlighting, for instance). This is made easier through structured trivia.

Structured trivia

There are two kinds of trivia:

Unstructured trivia

Comments, whitespace, and code that’s inactive due to conditional compilation

Structured trivia

Preprocessor directives and XML documentation

Unstructured trivia is treated purely as text, whereas structured trivia also has its content parsed into a miniature syntax tree.

The HasStructure property on SyntaxTrivia indicates whether structured trivia is present, and the GetStructure method returns the root node for the miniature syntax tree:

var tree = CSharpSyntaxTree.ParseText (@"#define FOO");

// In LINQPad:
tree.DumpSyntaxTree();  // LINQPad displays structured trivia in Visualizer

SyntaxNode root = tree.GetRoot();

var trivia = root.DescendantTrivia().First();
Console.WriteLine (trivia.HasStructure);           // True
Console.WriteLine (trivia.GetStructure().Kind());  // DefineDirectiveTrivia

In the case of preprocessor directives, you can navigate directly to the structured trivia by calling GetFirstDirective on a SyntaxNode. There’s also a ContainsDirectives property to indicate whether preprocessor trivia is present:

var tree = CSharpSyntaxTree.ParseText (@"#define FOO");
SyntaxNode root = tree.GetRoot();

Console.WriteLine (root.ContainsDirectives);      // True

// directive is the root node of the structured trivia:
var directive = root.GetFirstDirective();
Console.WriteLine (directive.Kind());             // DefineDirectiveTrivia
Console.WriteLine (directive.ToString());         // #define FOO

// If there were more directives, we could get to them as follows:
Console.WriteLine (directive.GetNextDirective());    // (null)

Once we’ve got a trivia node, we can cast it to a specific type and query its properties, just as we would with any other node:

var hashDefine = (DefineDirectiveTriviaSyntax) root.GetFirstDirective();
Console.WriteLine (hashDefine.Name.Text);     // FOO

Handling changes to the source code

If you’re writing a C# editor, for instance, you’ll need to update a syntax tree based on changes to the source code. The SyntaxTree class has a WithChangedText method which does exactly this: it partially reparses the source code based on modifications that you describe with a SourceText instance (in Microsoft.CodeAnalysis.Text).

To create a SourceText, use its static From method, giving it the complete source code. You can then use this to create a syntax tree:

SourceText sourceText = SourceText.From ("class Program {}");
var tree = CSharpSyntaxTree.ParseText (sourceText);

Alternatively, you can obtain the SourceText for an existing tree by calling GetText.

You can now “update” sourceText by calling Replace or WithChanges. For example, we could replace the first five characters (“class”) with “struct,” as follows:

var newSource = sourceText.Replace (0, 5, "struct");

Finally, we can call WithChangedText on the tree to update it:

var newTree = tree.WithChangedText (newSource);
Console.WriteLine (newTree.ToString());         // struct Program {}

Creating new nodes, tokens, and trivia with SyntaxFactory

The static methods on SyntaxFactory programmatically create nodes, tokens, and trivia, which you can use to “transform” existing syntax trees or to create new trees from scratch.

The hardest part of doing this is figuring out exactly what kind of nodes and tokens to create. The solution is to first parse a sample of the code you want, examining the result in a syntax visualizer. For instance, suppose we want to create a syntax node for the following:

using System.Text;

We can visualize the syntax tree for this in LINQPad as follows:

CSharpSyntaxTree.ParseText ("using System.Text;").DumpSyntaxTree();

(We can parse “using System.Text;” without error because it’s valid as a complete program, albeit a functionally empty one. For most other code snippets, you’ll need to wrap the snippet in a method and/or type definition so that it will parse.)

The result has the following structure, of which we are interested in the second node (i.e., UsingDirective and its descendants):

Kind                               Token Text
=================================  ==========
CompilationUnit (node)
  UsingDirective (node)
    UsingKeyword (token)           using
      WhitespaceTrivia (trailing)
    QualifiedName (node)
      IdentifierName (node)
        IdentifierToken (token)    System
      DotToken (token)             .
      IdentifierName (node)
        IdentifierToken (token)    Text
    SemiColonToken (token)         ;
  EndOfFileToken (token)

Starting from the inside, we have two IdentifierName nodes, whose parent is a QualifiedName. We can create that as follows:

QualifiedNameSyntax qualifiedName = SyntaxFactory.QualifiedName (
  SyntaxFactory.IdentifierName ("System"),
  SyntaxFactory.IdentifierName ("Text"));

We used the overload of QualifiedName that accepts two identifiers. This overload inserts the dot token for us automatically.

We now need to wrap this in a UsingDirective:

UsingDirectiveSyntax usingDirective =
  SyntaxFactory.UsingDirective (qualifiedName);

Because we didn’t specify tokens for the “using” keyword or the trailing semicolon, tokens for each were created and added automatically. However, the automatically created tokens don’t include whitespace. This wouldn’t prevent compilation, but converting the tree to a string would result in syntactically incorrect code:

Console.WriteLine (usingDirective.ToFullString());  // usingSystem.Text;

We can fix this by calling NormalizeWhitespace on the node (or one of its ancestors); doing so automatically adds whitespace trivia (for both syntactic correctness and readability). Or for more control, we could add whitespace explicitly:

usingDirective = usingDirective.WithUsingKeyword (
  usingDirective.UsingKeyword.WithTrailingTrivia (
    SyntaxFactory.Whitespace (" ")));

Console.WriteLine (usingDirective.ToFullString());  // using System.Text;

For brevity, we “harvested” the node’s existing UsingKeyword, to which we added trailing trivia. We could have created an equivalent token with more effort by calling SyntaxFactory.Token(SyntaxKind.UsingKeyword).

The final step is to add our UsingDirective node to an existing or new syntax tree (or more precisely, the root node of a tree). To do the former, we cast the existing tree’s root to a CompilationUnitSyntax and call the AddUsings method. We can then create a new tree from the transformed compilation unit:

var existingTree = CSharpSyntaxTree.ParseText ("class Program {}");
var existingUnit = (CompilationUnitSyntax) existingTree.GetRoot();

var unitWithUsing = existingUnit.AddUsings (usingDirective);

var treeWithUsing = CSharpSyntaxTree.Create (
  unitWithUsing.NormalizeWhitespace());

We called NormalizeWhitespace on our compilation unit so that calling ToString on the tree will yield syntactically correct and readable code. Alternatively, we could have added explicit new-line trivia to usingDirective as follows:

.WithTrailingTrivia (SyntaxFactory.EndOfLine("

"))

Creating a compilation unit and syntax tree from scratch is a similar process. The easiest approach is to start with an empty compilation unit and call AddUsings on the unit as we did before:

var unit = SyntaxFactory.CompilationUnit().AddUsings (usingDirective);

We can add type definitions to our compilation unit by creating them in a similar fashion, and then calling AddMembers:

// Create a simple empty class definition:
unit = unit.AddMembers (SyntaxFactory.ClassDeclaration ("Program"));

The final step is to create the tree:

var tree = CSharpSyntaxTree.Create (unit.NormalizeWhitespace());
Console.WriteLine (tree.ToString());

// Output:
using System.Text;

class Program
{
}

CSharpSyntaxRewriter

For more complex syntax tree transformations, you can subclass CSharpSyntaxRewriter.

CSharpSyntaxRewriter is similar to the CSharpSyntaxWalker class that we looked at previously (see “CSharpSyntaxWalker”), except that each Visit* method accepts and returns a syntax node. By returning something other than was passed in, you can “rewrite” the syntax tree.

For instance, the following rewriter changes method declaration names to uppercase:

class MyRewriter : CSharpSyntaxRewriter
{
  public override SyntaxNode VisitMethodDeclaration
    (MethodDeclarationSyntax node)
  {
    // "Replace" the method's identifier with an uppercase version:
    return node.WithIdentifier (
      SyntaxFactory.Identifier (
        node.Identifier.LeadingTrivia,            // Preserve old trivia
        node.Identifier.Text.ToUpperInvariant(),
        node.Identifier.TrailingTrivia));         // Preserve old trivia
  }
}

Here’s how to use it:

var tree = CSharpSyntaxTree.ParseText (@"class Program
{
  static void Main() { Test(); }
  static void Test() {         }
}");

var rewriter = new MyRewriter();
var newRoot = rewriter.Visit (tree.GetRoot());
Console.WriteLine (newRoot.ToFullString());

// Output:
class Program
{
  static void MAIN() { Test(); }
  static void TEST() {         }
}

Notice that our call to Test() in the main method did not get renamed, because we visited just member declarations and ignored invocations. To reliably rename invocations, however, we must be able to determine whether calls to Main() or Test() refer to the Program type, and not some other type. To do this, a syntax tree is not enough on its own; we also need a semantic model.

Diagnostics

A compilation may generate errors and warnings, even if the syntax trees are error-free. Examples include forgetting to import a namespace, a typo when referring to a type or member name, and type parameter inference failing. You can get the errors and warnings by calling GetDiagnostics on the compilation object. Any syntax errors will be included, too.

Symbols

In the syntax tree, names such as “System”, “Console”, and “WriteLine” are parsed as identifiers (IdentifierNameSyntax node). Identifiers have little meaning, and the syntactic parser does no work on “understanding” them other than to distinguish them from contextual keywords.

The semantic model is able to transform identifiers into symbols, which have type information (the output of the binding phase).

All symbols implement the ISymbol interface, although there are more specific interfaces for each kind of symbol. In our example, “System”, “Console”, and “WriteLine” map to symbols of the following types:

"System"      INamespaceSymbol
"Console"     INamedTypeSymbol
"WriteLine"   IMethodSymbol

Some symbol types, such as IMethodSymbol have a conceptual analog in the System.Reflection namespace (MethodInfo, in this case); whereas some other symbol types, such as INamespaceSymbol, do not. This is because the Roslyn type system exists for the benefit of the compiler, whereas the Reflection type system exists for the benefit of the CLR (after the source code has melted away).

Nonetheless, working with ISymbol types is similar in many ways to using the Reflection API we described in Chapter 19. Extending our previous example:

ISymbol symbol = model.GetSymbolInfo (writeLineNode).Symbol;

Console.WriteLine (symbol.Name);                   // WriteLine
Console.WriteLine (symbol.Kind);                   // Method
Console.WriteLine (symbol.IsStatic);               // True
Console.WriteLine (symbol.ContainingType.Name);    // Console

var method = (IMethodSymbol) symbol;
Console.WriteLine (method.ReturnType.ToString());  // void

The output of the last line illustrates a subtle difference with Reflection. Notice that “void” is in lowercase, which is C# nomenclature (Reflection is language-agnostic). Similarly, calling ToString() on the INamedTypeSymbol for System.Int32 returns “int”. Here’s something else you can’t do with Reflection:

Console.WriteLine (symbol.Language);                // C#

We can also ask the symbol where it came from:

var location = symbol.Locations.First();
Console.WriteLine (location.Kind);                     // MetadataFile
Console.WriteLine (location.MetadataModule
                   == compilation.References.Single()  // True

If the symbol was defined in our own source code (i.e., a syntax tree), the SourceTree property will return that tree, and SourceSpan will return its location in the tree:

Console.WriteLine (location.SourceTree == null);    // True
Console.WriteLine (location.SourceSpan);            // [0..0)

A partial type may have multiple definitions, in which case it will have multiple Locations.

The following query returns all the overloads of WriteLine:

symbol.ContainingType.GetMembers ("WriteLine").OfType<IMethodSymbol>()

You can also call ToDisplayParts on a symbol. This returns a collection of “parts” that make up the full name; in our case, System.Console.WriteLine(int) is comprised of four symbols interspersed with punctuation.

SymbolInfo

If you’re writing code completion for an editor, you’ll need to obtain symbols for code that’s incomplete or incorrect. For instance, consider the following incomplete code:

System.Console.Writeline(

Because the WriteLine method is overloaded, it’s impossible to match to a single ISymbol. Instead, we want to present options to the user. To deal with this, the semantic model’s GetSymbolInfo method returns an ISymbolInfo struct which has the following properties:

ISymbol Symbol
ImmutableArray<ISymbol> CandidateSymbols
CandidateReason CandidateReason

If there’s an error or ambiguity, the Symbol property returns null, and CandidateSymbols returns a collection comprising the best matches. The CandidateReason property returns an enum telling you what went wrong.

Symbol accessibility

ISymbol has a DeclaredAccessibility property that indicates whether the symbol is public, protected, internal, and so on. However, this isn’t sufficient to determine whether a given symbol is accessible at a particular position in your source code. Local variables, for instance, have a lexically limited scope, and a protected class member is accessible from source code positions within its type or a derived type. To help with this, SemanticModel has an IsAccessible method:

bool canAccess = model.IsAccessible (42, someSymbol);

This returns true if someSymbol can be accessed at offset 42 in the source code.

Declared symbols

If you call GetSymbolInfo on a type or member declaration, you’ll get no symbols back. For instance, suppose we want the symbol for our Main method:

var mainMethod = tree.GetRoot().DescendantTokens().Single (
  t => t.Text == "Main").Parent;

SymbolInfo symbolInfo = model.GetSymbolInfo (mainMethod);
Console.WriteLine (symbolInfo.Symbol == null);              // True
Console.WriteLine (symbolInfo.CandidateSymbols.Length);     // 0

To obtain the symbol, we must instead call GetDeclaredSymbol:

ISymbol symbol = model.GetDeclaredSymbol (mainMethod);

Unlike GetSymbolInfo, GetDeclaredSymbol either succeeds or it doesn’t. (If it fails, it will because it can’t find a valid declaration node.)

To give another example, suppose our Main method is as follows:

static void Main()
{
  int xyz = 123;
}

We can determine the type of xyz as follows:

SyntaxNode variableDecl = tree.GetRoot().DescendantTokens().Single (
  t => t.Text == "xyz").Parent;

var local = (ILocalSymbol) model.GetDeclaredSymbol (variableDecl);
Console.WriteLine (local.Type.ToString());             // int
Console.WriteLine (local.Type.BaseType.ToString());    // System.ValueType

TypeInfo

Sometimes you need type information about an expression or literal for which there’s no explicit symbol. Consider the following:

var now = System.DateTime.Now;
System.Console.WriteLine (now - now);

To determine the type of now - now, we call GetTypeInfo on the semantic model:

SyntaxNode binaryExpr = tree.GetRoot().DescendantTokens().Single (
  t => t.Text == "-").Parent;

TypeInfo typeInfo = model.GetTypeInfo (binaryExpr);

TypeInfo has two properties, Type and ConvertedType. The latter indicates the type after any implicit conversions:

Console.WriteLine (typeInfo.Type);             // System.TimeSpan
Console.WriteLine (typeInfo.ConvertedType);    // object

Because Console.WriteLine is overloaded to accept an object but not a TimeSpan, an implicit conversion to object took place, which manifested in typeInfo.ConvertedType.

Looking up symbols

A powerful feature of the semantic model is the ability to ask for all symbols in scope at a particular point in the source code. The result is the basis for IntelliSense listings, when the user requests a list of available symbols.

To obtain the listing, simply call LookupSymbols, with the desired source code offset. To give a complete example:

var tree = CSharpSyntaxTree.ParseText (@"class Program
{
  static void Main()
  {
    int x = 123, y = 234;

  }
}");

CSharpCompilation compilation = CSharpCompilation.Create ("test")
  .AddReferences (
    MetadataReference.CreateFromFile (typeof(int).Assembly.Location))
  .AddSyntaxTrees (tree);

SemanticModel model = compilation.GetSemanticModel (tree);

// Look for available symbols at start of 6th line:
int index = tree.GetText().Lines[5].Start;

foreach (ISymbol symbol in model.LookupSymbols (index))
  Console.WriteLine (symbol.ToString());

Here’s the result:

y
x
Program.Main()
object.ToString()
object.Equals(object)
object.Equals(object, object)
object.ReferenceEquals(object, object)
object.GetHashCode()
object.GetType()
object.~Object()
object.MemberwiseClone()
Program
Microsoft
System
Windows

(If we imported the System namespace, we’d see hundreds more symbols for types in that namespace.)