Converting Xcode Test Runs to JUnit, the Fast Way
Testing is very important at PSPDFKit. We’re building an SDK. When you give your API to other developers, there are many more things that can go wrong. For a long time, we’ve been using either a combination of xcodebuild with xcpretty or Facebook’s xctool. No tool is perfect though. xctool is an alternative to xcodebuild but uses much of it under the hood and often breaks when Xcode is updated. At some point Facebook deprecated it, and currently it doesn’t work with Xcode 8 anymore. xcpretty is really good. It parses the very verbose output of xcodebuild and prints it into something sane that a human can read. A big shout-out to Marin Usalj, who maintains the project.
There are a few inherent problems with parsing an undocumented log output with regular expressions. Things might get lost if no regex matches, or parsing becomes extremely slow if you log too much. All of this, and much more, has happened to us. It is not really xcpretty’s fault — it tries hard. It’s more an inherent problem and changing output between Xcode versions doesn’t help either. (Oh, you think xcodebuild can’t be any more verbose? Did you know about the undocumented log level 5? Have fun!)
I was discussing this with “Mr. Fastlane” Felix on his last visit to Vienna over a (fake) Club Mate.
We were talking about potentially rewriting xcpretty. However, after some more drinking talking, we figured that Xcode Server is a thing and that there must be a way how xcodebuild could pass the test results to it. JSON maybe? Of course not, but there is a plist! We were not the first to discover this, just the first who had the idea of writing a converter that takes the plist output and converts it into JUnit format.
Why JUnit? Because that’s what Jenkins eats for breakfast. So naturally, trainer was born. PSPDFKit is a huge project (PDF is hard, and turing-complete) and we have a few thousand tests across about 10 sub-projects. So it turned out to be a good test case to get trainer up and running. Since Xcode didn’t close its log pipe (we reported this here as rdar://27447948 and it has sinced been fixed in beta 4) and our sad attempts on working around this issue were not very successful, trainer was our best way to get Jenkins green blue again.
Before it was even publicly announced, Twitter had already adopted it and saw a 10x performance increase in JUnit report generation.
It also made our CI much, much less flaky, since random weird log messages no longer confuse the test converter. You can even run this in combination with xcpretty and get the best of both worlds. A flawed, but great version a human can read and a separate converter that translates test results from a (so far) stable-looking plist format.
So far trainer seems to be the only converter, though there are other tools that try to solve this differently, such as the recently open-sourced fbxctest by Facebook.
trainer is currently provided as standalone tool, with a built-in fastlane plugin, so it’s easy to get started if you’re already using fastlane. It focuses on one thing, generating a JUnit file that is supported by all major Continuous Integration systems like Jenkins, Circle and Travis. Combining this with danger and the danger-junit plugin by @orta, you can show the exact test failures right in your pull request. This solves the common phenomenon of “PR and Run”, which happens on many major open source projects. Typically, a developer submits a pull request on GitHub (often a simple one), which causes a test failure. By the time the test failures are shown on GitHub, the author has already closed the page. After a “PR and Run” occurs, the maintainer then has to manually ping the author to fix the tests. By using danger, the failed tests will automatically be posted as a comment on the PR, triggering an email notification to be sent to the author. (See example PR)
A big thanks to Felix for building trainer! We hope to see this under the fastlane umbrella soon.
