This week, I began work on my implementation of Project Familiar. The first step was finding the right API for the job. I had a pretty good idea of what I was looking for, using the following criteria:
- Free. As an undergraduate, this seems like a no-brainer. Paid AR APIs are mostly fairly expensive, so I’m not considering the paid-only ones. Some APIs, like Wikitude and Vuforia, have free trials with a watermark. Those also seem viable for Familiar, as I have no plans to monetize the app.
- Android. I have an Android phone, which I will be developing on. This mostly just rules out Apple’s ARKit, which is unfortunate due to its power.
- Unity Support. I prefer to work in Unity, and for a project that involves gamifying a game, a game engine seems to me to be the way to go.
- Image Recognition. As it stands, I intend to use image targets to detect tokens that will be used in place of miniatures. I also may want to use image targets to detect the battle mat itself, and might want to use them for other features, like Area of Effect visualization. An important note here is that I am not actually worried about image tracking. Since much of what is on the field in a game of D&D is static most of the time, being able to accurately track a target is nonessential.
- Extended Tracking. A battle mat can be large – the one I have is something like 26 inches long and 23.5 inches wide. Players will be looking around frequently – asking that they keep the center of the mat in view at all times doesn’t seem reasonable.
From these criteria, my first choice was obvious: ARCore. It has the best extended tracking available on Android, and is free. I ran into no issues while setting it up, but after doing the setup I realized I missed a vital note on their Augmented Images page: images need to take up at least 25% of the camera’s field of view to be detected. I tested this function, hoping that 25% was just a rough estimation, but alas, twas not. A target required a significant portion of the screen to be detected. This would mean that every time a player is to perform an action requiring AR, they would need to hold their token directly up to their phone, or lean in extremely close.To me, that hardly seems like a way to speed up combat, so I looked into other APIs to see what could function in tandem with ARCore – to augment the augmentor, if you will.
I stumbled across Wikitude in my search. Wikitude seems pretty straightforward – it can do image recognition/tracking and extended tracking, and has a watermarked free trial. It uses ARCore for its extended tracking, but from past experiences, though, I had noted that APIs that use ARCore for extended tracking still never seem to be quite as good. I wanted to use Wikitude, which claims to be able to recognize a target from about 8 feet away, in conjunction with ARCore. I installed them both into the same project, expecting it not to work. It didn’t. I spent about a half hour toying with it, messing with manifest files and such, but I could not find much in the way of tutorials, so I moved on and looked at other SDKs. Wikitude seemed good, but it struck me as fairly unremarkable for my uses in its free version.
I looked around a little more, and I liked what Vuforia brought to the table. I had been skeptical, as its extended tracking had always seemed questionable to me in the past, but it brings something else to the table: virtual buttons. Essentially, these allow the developer to note that certain areas of an image target, when obscured, will trigger a response in code. These could be an extremely interesting tool, and may help to alleviate the issues with recognizing small or distant images. I started setting up Vuforia just to give it a test, and it seemed to go well at first (I chuckled when the only image I had in my Google Drive that scored a 5-star target rating was a Smash Bros Vaporwave meme), but I ran into an issue where my Android build displayed a black screen. It took some googling and some tinkering for a while longer than I would have liked.
At first, the black screen appeared to be a bug in Vuforia itself on newer versions of Unity, but I’m not using the newest version of Unity. Then I thought it was a problem with shaders and materials, but couldn’t find any leads on that thread. I also wondered if it might have been a problem with my phone. In the end, I fixed it by installing the app to my phone’s internal storage rather than the SD card. I did a bit more testing, and found that Vuforia’s extended tracking, even though it uses ARCore, is still rather lacking. Rotating the device at all while not looking at the target causes noticeable drift, which isn’t ideal, but isn’t a dealbreaker, either. This can probably be ameliorated by using image targets around the battle mat, by splitting the battle mat into smaller targets, or using a large target at the center of the mat, as mentioned earlier.
Vuforia looks like a solid option going forward. It has good image recognition, and the extended tracking is imperfect but should be serviceable. Its virtual buttons offer an intriguing option to solve issues I expect to run into, and if they’re not useful, then I will still be using a perfectly good API. Over the next week, I hope to experiment with virtual buttons a bit, and see if I can simulate a full grid. I also would like to look a little bit more at Wikitude, but it’s not my top priority unless something goes horribly wrong with Vuforia.