So far, we've used images in the project's drawable resource folder. The next step is to read photo images from the phone and display one on our virtual screen.
Let's make a placeholder Image class. Later on, we'll build the attributes and methods. Define it as follows:
public class Image {
final static String TAG = "image";
String path;
public Image(String path) {
this.path = path;
}
public static boolean isValidImage(String path){
String extension = getExtension(path);
if(extension == null)
return false;
switch (extension){
case "jpg":
return true;
case "jpeg":
return true;
case "png":
return true;
}
return false;
}
static String getExtension(String path){
String[] split = path.split("\.");
if(split== null || split.length < 2)
return null;
return split[split.length - 1].toLowerCase();
}
}We define a constructor that takes the image's full path. We also provide a validation method that checks whether the path is actually for an image, based on the filename extension. We don't want to load and bind the image data on construction because we don't want to load all the images at once; as you'll see, we will manage these intelligently using a worker thread.
Now in MainActivity, access the photos folder on the phone and build a list of images in our app. The following getImageList helper method looks in the given folder path and instantiates a new Image object for each file found:
final List<Image> images = new ArrayList<>();
int loadImageList(String path) {
File f = new File(path);
File[] file = f.listFiles();
if (file==null)
return 0;
for (int i = 0; i < file.length; i++) {
if (Image.isValidImage(file[i].getName())) {
Image img = new Image(path + "/" + file[i].getName());
images.add(img);
}
}
return file.length;
}Use this method in the setup method, passing in the name of the camera images folder path, as follows (your path may vary):
final String imagesPath = "/storage/emulated/0/DCIM/Camera";
public void setup() {
…
loadImageList(imagesPath);
}Also, ensure that the following line is included in your AndroidManifest.xml file, giving the app the permission to read the device's external storage. Technically, you should already have this permission when using the Cardboard SDK:
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
You can add a log message to the getImageList loop and run it to verify that it is finding files. If not, you may need to discover the actual path to your photos folder.
This is the first project where we need to be really careful about permissions. Up until this point, the Cardboard SDK itself was the only thing which needed access to the filesystem, but now we need it for the app itself to function. If you are using a device with Andriod 6.0, and you don't make sure to compile the app against SDK 22, you will not be able to load the image files, and the app will either do nothing, or crash.
If you are compiling against SDK 22 and you have the permission set up correctly in the manifest but you still get an empty file list, try looking for the correct path on your device with a file browser. It could very well be that the path we provided doesn't exist or is empty. And, of course, make sure that you have actually taken a picture with that device!
If you remember, in Chapter 6, Solar System, we wrote a loadTexture method that reads a static image from the project's res/drawable folder into a memory bitmap and binds it to the texture in OpenGL. Here, we're going to do something similar but source the images from the phone's camera path and provide methods for additional processing, such as resizing and rotating its orientation.
At the top of the Image class, add a variable to hold the current texture handle:
int textureHandle;
The image's loadTexture method, given a path to an image file, will load an image file into a bitmap and then convert it to a texture. (This method will be called from MainActivity with the app's CardboardView class.) Write it as follows:
public void loadTexture(CardboardView cardboardView) {
if (textureHandle != 0)
return;
final Bitmap bitmap = BitmapFactory.decodeFile(path);
if (bitmap == null){
throw new RuntimeException("Error loading bitmap.");
}
textureHandle = bitmapToTexture(bitmap);
}We added a small (but important) optimization, checking whether the texture has already been loaded; don't do it again if not needed.
Our implementation of bitmapToTexture is shown in the following code. Given a bitmap, it binds the bitmap to an OpenGL ES texture (with some error checking). Add the following code to Image:
public static int bitmapToTexture(Bitmap bitmap){
final int[] textureHandle = new int[1];
GLES20.glGenTextures(1, textureHandle, 0);
RenderBox.checkGLError("Bitmap GenTexture");
if (textureHandle[0] != 0) {
// Bind to the texture in OpenGL
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, textureHandle[0]);
// Set filtering
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_NEAREST);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_NEAREST);
// Load the bitmap into the bound texture.
GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, bitmap, 0);
}
if (textureHandle[0] == 0){
throw new RuntimeException("Error loading texture.");
}
return textureHandle[0];
}Let's show one of our camera images in the app, say, the first one.
To show an image on the virtual screen, we can write a show method that takes the current CardboardView object and the Plane screen. It'll load and bind the image texture and pass its handle to the material. In the Image class, implement the show method as follows:
public void show(CardboardView cardboardView, Plane screen) {
loadTexture(cardboardView);
BorderMaterial material = (BorderMaterial) screen.getMaterial();
material.setTexture(textureHandle);
}Now let's use this stuff! Go to MainActivity and write a separate showImage method to load the image texture. And, temporarily, call it from setup with the first image that we find (you will need at least one image in your camera folder):
public void setup() {
setupBackground();
setupScreen();
loadImageList(imagesPath);
showImage(images.get(0));
}
void showImage(Image image) {
image.show(cardboardView, screen);
}It now also makes sense to modify setupScreen, so it creates the screen but doesn't load an image texture onto it. Remove the call to screenMaterial.setTexture in there.
Now run the app, and you will see your own image on the screen. Here's mine:
Some image file types keep track of their image orientation, particularly JPG files (.jpg or .jpeg). We can get the orientation value from the EXIF metadata included with the file written by the camera app. (For example, refer to http://sylvana.net/jpegcrop/exif_orientation.html. Note that some devices may not be compliant or contain different results.)
If the image is not JPG, we'll skip this step.
At the top of the Image class, declare a variable to hold the current image rotation:
Quaternion rotation;
The rotation value is stored as a Quaternion instance, as defined in our RenderBox math library. If you remember Chapter 5, RenderBox Engine, a quaternion represents a rotational orientation in three-dimensional space in a way that is more precise and less ambiguous than Euler angles. But Euler angles are more human-friendly, specifying an angle for each x, y, and z axes. So, we'll set the quaternion using Euler angles based on the image orientation. Ultimately, we use a Quaternion here because it is the underlying type of Transform.rotation:
void calcRotation(Plane screen){
rotation = new Quaternion();
// use Exif tags to determine orientation, only available // in jpg (and jpeg)
String ext = getExtension(path);
if (ext.equals("jpg") || ext.equals("jpeg")) {
try {
ExifInterface exif = new ExifInterface(path);
switch (exif.getAttribute(ExifInterface.TAG_ORIENTATION)) {
// Correct orientation, but flipped on the // horizontal axis
case "2":
rotation = new Quaternion().setEulerAngles(180, 0, 0);
break;
// Upside-down
case "3":
rotation = new Quaternion().setEulerAngles(0, 0, 180);
break;
// Upside-Down & Flipped along horizontal axis
case "4":
rotation = new Quaternion().setEulerAngles(180, 0, 180);
break;
// Turned 90 deg to the left and flipped
case "5":
rotation = new Quaternion().setEulerAngles(0, 180, 90);
break;
// Turned 90 deg to the left
case "6":
rotation = new Quaternion().setEulerAngles(0, 0, -90);
break;
// Turned 90 deg to the right and flipped
case "7":
rotation = new Quaternion().setEulerAngles(0, 180, 90);
break;
// Turned 90 deg to the right
case "8":
rotation = new Quaternion().setEulerAngles(0, 0, 90);
break;
//Correct orientation--do nothing
default:
break;
}
} catch (IOException e) {
e.printStackTrace();
}
}
screen.transform.setLocalRotation(rotation);
}Now we set the screen's rotation in the show method of the Image class, as follows:
public void show(CardboardView cardboardView, Plane screen) {
loadTexture(cardboardView);
BorderMaterial material = (BorderMaterial) screen.getMaterial();
material.setTexture(textureHandle);
calcRotation(screen);
}Run your project again. The image should be correctly oriented. Note that it is possible that your original image was fine all along. It will become easier to check whether your rotation code works once we get the thumbnail grid going.
Square images are easy. But usually, photos are rectangular. We can get the actual width and height of the image and scale the screen accordingly, so the display won't show up distorted.
At the top of the Image class, declare variables to hold the current image width and height:
int height, width;
Then, set them in loadTexture using bitmap options in the decodeFile method, as follows:
public void loadTexture(CardboardView cardboardView) {
if (textureHandle != 0)
return;
BitmapFactory.Options options = new BitmapFactory.Options();
final Bitmap bitmap = BitmapFactory.decodeFile(path, options);
if (bitmap == null){
throw new RuntimeException("Error loading bitmap.");
}
width = options.outWidth;
height = options.outHeight;
textureHandle = bitmapToTexture(bitmap);
}The decodeFile call returns the image's width and height (among other information) in the options (refer to http://developer.android.com/reference/android/graphics/BitmapFactory.Options.html).
Now we can set the screen size in the show method of the Image class. We'll normalize the scale so that the longer side is of size 1.0 and the shorter one is calculated as the image aspect ratio:
public void show(CardboardView cardboardView, Plane screen) {
loadTexture(cardboardView);
BorderMaterial material = (BorderMaterial) screen.getMaterial();
material.setTexture(textureHandle);
calcRotation(screen);
calcScale(screen);
}
void calcScale(Plane screen) {
if (width > 0 && width > height) {
screen.transform.setLocalScale(1, (float) height / width, 1);
} else if(height > 0) {
screen.transform.setLocalScale((float) width / height, 1, 1);
}
}If you run it now, the screen will have the correct aspect ratio for the image:
The camera in your phone is probably awesome! It's probably really mega awesome! Many-megapixel images are important when printing or doing lots of cropping. But for viewing in our app, we don't need the full resolution image. In fact, you might already be having trouble running this project if the image size generates a texture that's too big for your device's hardware.
We can accommodate this issue by constraining the maximum size and scaling our bitmaps to fit within these constraints when loading the texture. We will ask OpenGL ES to give us its current maximum texture size. We'll do this in MainActivity, so it's generally available (and/or move this into the RenderBox class in your RenderBox library project). Add the following to MainActivity:
static int MAX_TEXTURE_SIZE = 2048;
void setupMaxTextureSize() {
//get max texture size
int[] maxTextureSize = new int[1];
GLES20.glGetIntegerv(GLES20.GL_MAX_TEXTURE_SIZE, maxTextureSize, 0);
MAX_TEXTURE_SIZE = maxTextureSize[0];
Log.i(TAG, "Max texture size = " + MAX_TEXTURE_SIZE);
}We call it as the first line of the setup method of the MainActivity class.
As for scaling the image, unfortunately, Android's BitmapFactory does not let you directly request a new size of a sampled image. Instead, given an arbitrary image, you can specify the sampling rate, such as every other pixel (2), every fourth pixel (4), and so on. It must be a power of two.
Back to the Image class. First, we will add a sampleSize argument to loadTexture, which can be used as an argument to decodeFile, as follows:
public void loadTexture(CardboardView cardboardView, int sampleSize) {
if (textureHandle != 0)
return;
BitmapFactory.Options options = new BitmapFactory.Options();
options.inSampleSize = sampleSize;
final Bitmap bitmap = BitmapFactory.decodeFile(path, options);
if(bitmap == null){
throw new RuntimeException("Error loading bitmap.");
}
width = options.outWidth;
height = options.outHeight;
textureHandle = bitmapToTexture(bitmap);
}To determine an appropriate sample size for images, we need to first find out its full dimensions and then figure out what sample size will get it closest but less than the maximum texture size we're going to use. The math isn't too difficult, but instead of going through that, we'll use a procedural method to search for the best size value.
Fortunately, one of the input options of decodeFile is to only retrieve the image bounds, and not actually load the image. Write a new load texture method named loadFullTexture, as follows:
public void loadFullTexture(CardboardView cardboardView) {
// search for best size
int sampleSize = 1;
BitmapFactory.Options options = new BitmapFactory.Options();
options.inJustDecodeBounds = true;
do {
options.inSampleSize = sampleSize;
BitmapFactory.decodeFile(path, options);
sampleSize *= 2;
} while (options.outWidth > MainActivity.MAX_TEXTURE_SIZE || options.outHeight > MainActivity.MAX_TEXTURE_SIZE);
sampleSize /= 2;
loadTexture(cardboardView, sampleSize);
}We keep bumping up the sample size until we find one that produces a bitmap within the MAX_TEXTURE_SIZE bounds, and then call loadTexture.
Use loadFullTexture in the show method instead of the other loadTexture one:
public void show(CardboardView cardboardView, Plane screen) {
loadFullTexture(cardboardView);
BorderMaterial material = (BorderMaterial) screen.getMaterial();
...Run the project. It should look the same as the earlier one. But if your camera is too good, maybe it's not crashing like it was before.
This sampling will also be useful to display thumbnail versions of the images in the user interface. There's no point in loading the full-sized bitmap for a thumbnail view.