Apple’s Swift Playgrounds Application Development – Our Agencies Experience

The challenge

For those not in the loop, Swift Playgrounds was first introduced by Apple in 2016. The goal from Apple for Swift Playgrounds was to give future programmers a way to not only learn about Swift, but more importantly, just overall good practices. While it is targeted for kids, realistically anyone can use it to get started. Initially only available on an iPad, as of February 2020 it works on iPad or Mac.

We were approached by a large global player who was interested in setting up an application for a product launch they were aiming for in 2020. This partner of ours, who is also a partner with Apple for education, needed an application custom made for Swift Playgrounds. We were elated.

We’ve done plenty of iOS applications in the past, this one was a bit more special not just for the novelty of it, but it was also treading into new grounds – an augmented reality Swift Playgrounds application. Getting kids to play with an edtech solution is cool, but getting them to play with it in AR, now that’s awesome! Plus we were doing something with a giant company with ties to Apple, so whatever we could help with, we were elated.

Our initial solution roadmap included setting up an AR environment utilizing Blender, Maya, Unreal Engine, Unity and a few complimentary tools we’ve used in AR and VR projects. We needed to know what will work and what will not. Plus, we were tasked with some high-quality expectations visually, going above and beyond Apple standards wherever possible.

During our first batch, we did the story board based on the partners needs, and started crafting the environment in Unreal Engine, hoping to reach a high-quality layout, while minimizing the footprint. Concurrently, we were tasked with setting up occlusion, as the initial goal of the application was occlusion instead of just AR.

If you are not familiar with occlusion, it’s kind of like opposite to basic AR solutions. Augmented reality usually places an object in a real world, with occlusion we were aiming to overlay an augmented reality world around a real-world object. Imagine a red ball, you want the ball to be detected by the camera/application, and “thrown into” a beach setting, or in a jungle, while the red ball is still there in the setting. This is the secondary task given to us. This is also referred to as mixed reality, either way, it requires some kind of occlusion to work right.

So, we got the occlusion working for the mixed reality, but we hit a problem with the Unreal Engine, it was too heavy for Swift Playgrounds running on an iPad. Especially on older iPad devices which barely made the cut to support augmented reality.

The limit we hit with the solution we had in mind; Swift Playground was not good at managing memory. What we were building was an application that runs in an application, an app within an app. Apple has a great way to kill off processes that are taking up too much memory, so instead of letting an app crash your device, Apple kills it before it reaches too far, this was the biggest challenge for us. Making a stable AR app within an app. But we were not giving up.

Solution that works

After testing a few things, we decided that the best solution will be to use Blender to create the environment assets, and to keep a polygon limit in mind, thus we got a polygon budget in place that would ensure we stick with high quality, and not overload the application. The code was not the problem, we just had a problem all around from the assets side. It needed to be high quality and polished, while also not barren.

To ensure we were able to do this well, we focused the Swift Playground application we were working on to have a play area in the AR environment, making a limited area someone can walk around. With this limited view, we were able to fake the rest of the environment by utilizing some amazing art by our matte painter. Mixing augmented reality and matte painting inside an application allowed us to greatly cut down the size, polygon budget, but not sacrifice the high-quality polished look we were aiming for.

Once we had this workflow down, we were able to almost fully eliminated the crashes coming from our Swift Playgrounds application. The only other issue we were still hitting was a memory leak in the way the application was too quickly cutting off some processes relating to AR, which was miraculously fixed in Playgrounds 3.3, allowing us to power through the rest of the lessons with minimal issues.

In conclusion

The application we built for this client came out amazing, especially compared to some of the other solutions that were out there. Our proudest moment was that it was featured on the official Apple website as a showpiece for utilizing Swift Playgrounds as an educational tool.

Things to keep in mind, you are building an application within an application, so there is a lot of extra precautions you must take due to limits of such an environment. We have built iOS AR applications in the past, but nothing this sensitive or requiring attention to detail.

Are you building something similar?

If you are building an application for Swift Playgrounds, whether it’s a standalone or with a physical tie in to a product, reach out to us. Ideally before you go too far down wrong paths 😊

We think that a Apple has released an amazing tool, and fully recommend to all parents to get this for their kids. Let them start exploring and at least get the logic down for what is considered best practices, as almost every job these days is touched by development in some way.

CroatiaTech Team

Our team writes on the technology services & solutions we develop for our clients. This ranges from software development, machine learning, blockchain, internet marketing, virtual reality, augmented reality, electrical & mechatronic engineering, website development and graphic design. From time to time, we post on other random things of interest.

Follow us on below channels for future posts on these topics and much more 🙂