Advanced Scala Concepts
Scala is an interesting language in that it has a fairly simple specification with few keywords, yet it allows for remarkable power. This book has avoided going into detail on a number of different features of the language. After all, the goal was to teach you to program, not to be an expert in Scala. The purpose of this appendix is to give you a brief introduction to many of the features that were either glossed over or left out completely. For more details you can search for the Scala language specification. Language specifications are notoriously hard to read so you might want to consider "Programming in Scala" by Odersky et al. [9].
B.1 abstract type Members
Back in chapter 5 we briefly introduced the concept of a type declaration that can be used to give a different name to a type. At that time, the advantage was the ability to give shorter and more meaningful names to tuples that were grouping data for you. With the introduction of case classes in chapter 10, the need for meaningful names on data groups was addressed in a superior way because it allows you to name the members and have stronger type checking.
The main benefit of type declarations in Scala come about when you put abstract type declarations in a supertype. You can then use that type through the code and know that it will be bound to something meaningful in the subtypes. You can also provide type bounds on the abstract type using <: or >: after the name you are giving the type and before the type that provides the bounds. Without bounds, you can only use the type in a way that is safe with Any.
B.2 Enumerations
Many languages include a construct called an enum, short for enumeration. This is different from the java.util.Enumeration type which we discussed in chapter 19. That type is similar to an iterator. A standard enumeration is used to define a small set of values like the colors of a stop light. Scala does not provide this as a language feature. Instead, there is a class called Enumeration that you can extend to get this type of functionality.
You should create an object that extends Enumeration like the following.
object LightColors extends Enumeration {
type LightColors = Value // Not required.
val Red, Yellow, Green = Value
}
The type declaration in this is not required, but it gives you a good name to use to refer to the enumeration type. Any code that is going to use this enumeration a lot should probably include the line import LightColors._.
B.3 implicits
There have been a few points in the book where implicit conversions have been mentioned. The idea of this feature is that if you call a method on an object that is not defined on that object or if you pass an object to a method that is not of the correct type, Scala will check to see if there is an implicit conversion in scope that would make it work.
These implicit conversions provide a lot of power to Scala. Both the Array and String types are basically Java types. They do not really have methods like map and filter. Despite this, you have been able to call map and filter and both Arrays and Strings. The reason this has worked is that there are implicit conversions that are imported by default that convert those types to ArrayOps and StringOps respectively and those types have the methods we associate with Scala collections.
B.3.1 Basic implict Converstions
You can make your own implicit conversions by having a "function" in scope that takes the type you want to convert from, returns the type you want to convert to, and includes the implicit keyword before def. The "function" was put in quotes because in general this is actually a method, but it needs to be brought into scope so that no object name has to be provided.
To illustrate the creation of an implicit conversion as well as motivate their existence, consider the following class and companion object.
class Vect3D(val x:Double,val y:Double,val z:Double) {
def +(v:Vect3D) = Vect3D(x+v.x,y+v.y,z+v.z)
def -(v:Vect3D) = Vect3D(x-v.x,y-v.y,z-v.z)
def ∗(c:Double) = Vect3D(c∗x,c∗y,c∗z)
def /(c:Double) = Vect3D(c/x,c/y,c/z)
def dot(v:Vect3D) = x∗v.x+y∗v.y+z∗v.z
def cross(v:Vect3D) = Vect3D(y∗v.z-z∗v.y,z∗v.x-x∗v.z,x∗v.y-y∗v.x)
}
object Vect3D {
def apply(x:Double,y:Double,z:Double) = {
new Vect3D(x,y,z)
}
}
This class defines a class to represent vectors in 3-space with some appropriate operations. Most of the things that you would want to do are present here. There is one thing that is missing though. If v is a Vect3D, you can do v∗3 to scale the components of the vector up by 3. However, you can not do 3∗v. To see why, remember that v∗3 is seen by Scala as v.∗(3). It is calling the ∗ method on the v instance of the Vect3D class. That works because such a method has been defined. On the other hand, 3∗v is trying to call a ∗(Vect3D) method on a number. That method was not part of the standard libraries so it does not work.
The solution to this is an implicit conversion. While Double does not have the needed ∗ method, we can convert the Double to some other type that does. To make this work, we can add the following code to the companion object.
implicit def doubleToScaling(c:Double):VectScaler = new VectScaler(c)
class VectScaler(c:Double) {
def ∗(v:Vect3D):Vect3D = Vect3D(c∗v.x,c∗v.y,c∗v.z)
}
The first line of this is the implicit conversion from Double to the type VectScaler. Below that is the definition of VectScaler. It is a simple class that contains only the method that we need here for multiplication against a Vect3D. To use this, you need to import that method into the local scope. This can be done with import Vect3D._.
You should notice that what this did was to basically add a new method to the Double class. The ability to make it appear that a type has methods that was not originally part of it has been called "pimping an interface" because you are adding new features to provide functionality that is more to your liking.
B.3.2 Rules and Limits on implicits
While implicit conversions bring a lot of power, they can make code harder to read and understand if they are abused. To help prevent this, Scala has rather strict rules on when implicit conversions will be invoked. First, they must be in scope. If calling the conversion function would require any specifiers before the name, it will not be used as an implicit in that part of the code. Second, implicit conversions are not nested. If it would require two conversions to get from the declared type to a type that will work, Scala will not do that. Similarly, if there is ambiguity where two different implicit conversions could make the call work, neither will be used and you will get an error.
These various rules and limitations combine to make Scala implicits safer. They should not be invoked in ways that you are not expecting or would find overly confusing. What is more, Integrated Development Environment (IDE) plug-ins have the ability to show you what implcit conversions are being used at different points in your code so that it is even more clear.
B.3.3 implicit Parameters
It is possible to get Scala to implicitly pass extra parameters into a method as well. This can be done for multiple reasons, including passing implicit type conversions. Scala will only implicitly include the last argument list to a method or function. Typically, methods that have implicit parameters are curried. For the last parameter list to be supplied implicitly the argument list must be labeled as implicit and there must be a value matching each of the types in that parameter list which is in scope and labeled as implicit. Note that this implies that you can make implicit val declarations as well as implicit def declarations.
If you look in the Application Programming Interface (API), for example at scala.collection.Seq, you will see many examples of methods that include implicit parameters. In the collections library the implicit parameter list is almost always a single parameter of the type CanBuildFrom.
B.3.4 The Meaning of <%
When you put <% as a bound on a type parameter, Scala switches it to a normal type bound and adds a curried implicit parameter to the method. That extra parameter will be filled in with an appropriate implicit conversion from the current scope if one is available.
B.4 sealed classes
Many functional languages have a construct called algebraic types. This allows you to say that a type can be represented by a small number of different possible structures/values. If all you want are separate values, the Enumeration described above is the way to go. When the different possibilities have different structure to them, the normal object-oriented way to represent that is with inheritance. The one shortcoming of inheritance in this case is that anyone can add other subtypes at a later date, and that might be a problem. If you make a class final, no subtypes are allowed. In this case you want to be able to limit the subtypes to a specific number.
This mechanism is provided in Scala with the sealed keyword. A sealed class can be extended, but all the subtypes have to appear in the same file as the original declaration. That gives you much stronger control over what subtypes exist.
There is another benefit to a class being sealed in the form of error checking for match expressions. If you do a match on an expression whose type is a sealed class, Scala will produce syntax errors if you do not have cases that can match all subtypes. This completeness checking makes it so that you should not get MatchExceptions when a type comes in that you were not expecting.
The standard usage of a sealed class is to use a sealed abstract class that is completely empty at the top of the hierarchy, then have various case classes or case objects1 which extend the first class. The use of case types makes it easier to take full advantage of the pattern matching.
B.5 Covariant, Contravariant, and Invariant Types
Chapter 25 brought up the concept of covariant types in relation to immutable linked lists. This deals with how Scala treats type parameters when it comes to determining subtype relationships for the main type. Imagine you have the following declarations.
class Stuff[A](val a:A)
class MoreStuff[A](a:A, val b:A) extends Stuff[A](a)
You can see from this that MoreStuff is a subtype of Stuff. However, this ignores the fact that Stuff and MoreStuff are not well-defined types on their own. In reality, you have to provide a type parameter to have a well defined type. In this case we can say that MoreStuff[String] is a subtype of Stuff[String]. The question is, is MoreStuff[String] and subtype of Stuff[AnyRef]? You might be tempted to say that it is, but as written, it is not. If you try to pass an instance of MoreStuff[String] into a function/method that accepts Stuff[AnyRef], Scala will complain. If you put the types above into the REPL, you can quickly test that with the following.
scala> def hasNull(s:Stuff[AnyRef]) = s.a==null
hasNull: (s: Stuff[AnyRef])Boolean
scala> val s = new Stuff("Hi")
s: Stuff[String] = Stuff@c8d310f
scala> hasNull(s)
<console>:11: error: type mismatch;
found : Stuff[String]
required: Stuff[AnyRef]
Note: String <: AnyRef, but class Stuff is invariant in type A.
You may wish to define A as +A instead. (SLS 4.5)
hasNull(s)
^
scala> val ms = new MoreStuff("hi","mom")
ms: MoreStuff[String] = MoreStuff@35e20aca
scala> hasNull(ms)
<console>:12: error: type mismatch;
found : MoreStuff[String]
required: Stuff[AnyRef]
Note: String <: AnyRef, but class Stuff is invariant in type A.
You may wish to define A as +A instead. (SLS 4.5)
hasNull(ms)
^
This happens because the default behavior is for type parameters to be invariant. That means that types with different type parameters are always unrelated, regardless of the inheritance relationships between the main types or the type parameters.
The type MoreStuff does not even have to be involved. You might expect that Stuff[String] should be a subtype of Stuff[AnyRef]. However, if A is invariant that is not the case either, as shown here.
scala> val s = new Stuff("Hi")
s: Stuff[String] = Stuff@c8d310f
scala> hasNull(s)
<console>:11: error: type mismatch;
found : Stuff[String]
required: Stuff[AnyRef]
Note: String <: AnyRef, but class Stuff is invariant in type A.
You may wish to define A as +A instead. (SLS 4.5)
hasNull(s)
^
As the error message above indicates, this could be fixed by using the type parameter +A instead of just A. That makes the type parameter covariant. When the type parameter is covariant, the full types will be subtypes if the parameters are subtypes and the main types are the same or subtypes. In this particular example just adding a + will make this work.
class Stuff[+A](val a:A)
With this one character, everything else from above will work.
So you might wonder why invariance is the default instead of covariance. To make that clear, consider the following change to Stuff and a function that works on it.
class Stuff[A](var a:A)
def changeStuff(s:Stuff[AnyRef]) = s.a = List(1,2,3)
Now that the a member is a var, things are a bit different. Consider what happens if Scala let you pass in an instance of Stuff[String]. In that object, a has to be a String. This function would try to assign to it a List[Int], an operation that should clearly fail. The switch to a var makes covariance for this class unsafe. Scala can even tell you that.
class Stuff[+A](var a:A)
<console>:7: error: covariant type A occurs in contravariant position in type A of
value a_=
class Stuff[+A](var a:A)
^
As the error message indicates, it is the assignment method into the var that is really causing the problem. This also shows you that Scala can tell if it is safe for a type to be covariant or contravariant. It is always safe to be invariant. That is why invariant is the default. The algorithm that Scala uses to determine what is safe is beyond the scope of this appendix, but a quick rule of thumb is that when the type is passed into a method that is a contravariant position. When they are returned from methods that is a covariant position. For a type parameter to be either covariant or contravariant, it can not occur in a position of the other end. Since a public var implicitly makes one of each, any type of a public var has to be invariant.
Contravariance is the opposite of covariance. When the type parameter is -A, then Stuff[Type1] is a subtype of Stuff[Type2] only if Type2 is a subtype of Type1. This probably seems counterintuitive, but there are times when it is very useful. The most obvious example is the Function1 type in the Scala library. This type represents a function that takes one argument and returns a value.
trait Function1 [-T1, +R]
This is basically what we have been writing as (T1) => R. Here the return type is covariant and the input type is contravariant. To understand why this is, imagine a place where you say you need a function of the type (Seq[Int]) => Seq[Int]. What other types would work there? For the return type, anything that returns a subtype of Seq[Int] will work fine. For the input type, you can use any function that takes a supertype of Seq[Int] will work because you know it will be safe to pass it a Seq[Int].
B.6 Extractors
Pattern matching has been used in many places in this book and hopefully you have become aware of the power that it can provide. One thing you might have noticed that was missing was the ability for you to make your own classes capable of doing pattern matching. You can make your own types that do pattern matching by writing extractors. An extractor is an object that includes either the unapply or the unapplySeq methods. Most of the time you will do this in a companion object that also has an apply method so that the usage resembles that of case classes.
B.6.1 unapply
You should put an unapply method in an object when you know how many fields should be pulled out of the pattern. The parameter for unapply is the value that would be matched on in the pattern. It should have a type that is appropriate for what you want to be working with. The result type of unapply depends on how many fields it should give back.
- No values results in Boolean. It is true if there was a match and false if there was not.
- One value of type A results in Option[A]. The result will be None if there was a match or Some with the value of the result if there was a match.
- For two or more values the result type should be an Option of a tuple with the proper number of types. No match will result in None while a match will result in Some of a tuple filled with the proper values.
To illustrate this, we could put an unapply method in the Vect3D companion object that was used earlier to show the use of implicit conversions.
def unapply(str:String):Option[(Double,Double,Double)] = {
val s = str.trim
if(!s.startsWith("(") || !s.endsWith(")")) None
else {
val parts = s.substring(1,s.length-1).split(",")
if(parts.length!=3) None
else
Some(parts(0).trim.toDouble,parts(1).trim.toDouble,parts(2).trim.toDouble)
}
}
This example will pattern match on a String that should have three numbers, separated by commas, inside of parentheses. It gives back three values that are the numbers of the three components. A simple usage of this might look like the following.
"(1,4.5,83)" match {
case Vect3D(x,y,z) => println(x+" "+y+" "+z)
}
Note that while you can make unapply methods that take different inputs and return whatever, you can not make methods that differ only in their return type. So in this example, you can not also have a second version that creates a Vect3D instance. You have to pick between the two.
B.6.2 unapplySeq
There are also situations where the number of fields in the pattern match can vary from one match to another. Examples of this include matching on List or Array, or the matching or regular expressions where each group defines a field. To get this functionality, you need to implement the unapplySeq method in your object. The return type of this method can be an Option[Seq[A]], for whatever type A you want. It can also have an Option of a tuple where the last element of the tuple is a Seq[A]. The first version could match any number of things, but they will all have the same type. The second version can force a certain number of fields of different types at the beginning of the pattern followed by the sequence of unknown length.
B.7 lazy Member Data
val and var declarations in class, trait, and object declarations can be labeled as lazy. This changes when the value of that member data will be set. Normally member data is initialized when the object is instantiated. lazy members are not initialized until they are used.
This is particularly helpful if the value requires a significant calculation or requires significant resources such as memory. In that situation, you do not want to go through the effort of making the value unless it is actually needed. If the object can go through its life without ever needing that member, setting it to be lazy will make it so those resources are never consumed.
B.8 scala.collection.immutable.Stream
There is another type in the Scala collections that was not used in this book, the scala.collection.immutable.Stream type. Stream is a subtype of Seq and you access members of it just like you would any other Seq using an integer index. What makes Stream interesting is that it is a lazy sequence. The last section described how the lazy keyword modifies members so that they are not calculated until they are needed. This same idea can be applied to a sequence as well. The values at different indexes are not calculated until they are needed. Once calculated they are remembered so they do not have to be calculated again, but if you never use an index, it will never be calculated.
At first glance this might seem like just an interesting way to save memory. However, it can be used for more than that. The fact that the values of a Stream are not absolutely calculated and stored means that it is possible to make Streams of infinite length. The Scala API includes an example that builds an infinite Stream of prime numbers. If you look at that you will notice that it uses the method Stream.cons instead of the :: method to add elements to the front of the stream. It also adds elements using a recursive definition of everything after the head. This is required for an infinite Stream because the tail of the Stream never really exists in full.
If you have a block of code that is going to be working with Streams s lot, you should include import Stream._. One of the useful things this brings into scope is an implicit conversion from Stream to Stream.ConsWrapper. This will allow you to use the #:: and #::: operations to cons and concatenate Streams. This allows you to write the following code to define Fibonacci numbers.2
import Stream._
def fibFrom(a:BigInt, b:BigInt):Stream[BigInt] = a #:: fibFrom(b, a+b)
To test this you can do the following.
fibFrom(1,1).take(100).toList
You can use this type of approach generally for any place that you need to define something that is accurately represented by an infinite sequence. It is worth noting though that you have to be careful with these. Many of the methods in the collection API keep going until they get to the end of the collection. Needless to say, that is not a good thing to do with an infinite collection.
B.9 Nested Types
We have seen quite a few places in this book where classes are written inside of other classes. What we did not generally do is refer to these types from outside of the outer class. This was in large part because none of the code we wrote really needed to do this. In fact, in most places where we did nest types in this way, we made the nested type private so that it could not be seen from the outside. For example, the Node type in a linked list should not be known to outside code as that is an implementation detail. However, there are instances where those nested types need to be used by outside code, and there are some significant details to it that are worth mentioning.
When you put a class or trait inside of another class or trait, every instance of that type gets its own version. If you use standard "dot" notation to refer to the type, it needs to be on an object, not the class, and the type you get is specific to that object. You can also refer to the general type across all instances with the class name followed by a #. The following code demonstrates this.
class NestedClasses(x:Double,y:Double) {
val root = new Plus(new Num(x),new Num(y))
trait Node {
def eval:Double
}
class Plus(left:Node,right:Node) extends Node {
def eval = left.eval+right.eval
}
class Num(n:Double) extends Node {
def eval = n
}
}
object NestedClass {
def main(args:Array[String]) {
val a = new NestedClasses(4,5)
val b = new NestedClasses(8.7,9.3)
a.root match {
case n:a.Num => println("A number")
case n:a.Node => println("Not a number")
}
def evalNode(n:NestedClasses#Node) = n.eval
def evalNodeA(n:a.Node) = n.eval
def evalNodeB(n:b.Node) = n.eval
println(evalNode(a.root))
println(evalNodeA(a.root))
println(evalNodeB(a.root)) // This is a type mismatch.
println(evalNode(b.root))
println(evalNodeA(b.root)) // This is a type mismatch.
println(evalNodeB(b.root))
}
}
The class at the top contains types for a little expression tree. This is greatly simplified for this example. In the companion object is a main method that makes two instances of the top class called a and b. A match on a.root shows how you can refer to types inside of the object a. If you try to put something in a case that is not one of the subtypes of Node inside of a, Scala gives you an error saying it can not be that type.
After the match are three def declarations of local functions. The first is written to take a general Node from inside of any instance of NestedClasses. The other two are specific to a and b. There are six calls where these three methods are invoked on the root objects of both a and b. Two of these result in errors as a.root is a mismatch with b.Node and b.root is a mismatch with a.Node.
B.10 Self Types
There are times when you will create a trait which is intended to be mixed in with some other trait or class. Occasionally, inside of the mix-in trait, you will need to make reference to members of the type it is being mixed in to. Conceptually this makes sense as you know that this will always be safe. However, the compiler does not know that this will be safe. To get around this, you have to have a way to tell the compiler that the particular trait will only be used in situations where it is part of some other type. This can be done using a self type.
The syntax of a self type is as follows.
trait A {
this: B =>
...
}
This tells Scala that any object that includes the trait A must include the type B above it in the resolution linearization. With this in place, the body of A can include references to members of B.
One common place that this is used in Scala is in something called the Cake pattern. Interested readers can use that term as a jumping-off point for searching for more information.
B.11 Making Executable JAR Files
Scala programs compile to bytecode, not an actual executable. If you want to be able to give your program to someone else for them to run, you need to package it in a proper form. For code compiled to Java bytecode, that format is an executable JAR file. You can make these using the jar command from the Java Development Kit (JDK). Complete instructions can be found on the book’s website as the ideal set of steps is likely to change over time.
1 You use case classes when your type has arguments. If there are now arguments, you should use a case object.
2 This code is presented using Int in the Scala collections API description gives at http://www.scala-lang.org/.