Comparison of Test Automation Platforms

Test automation, being it web or mobile test automation should be part of every software development team. However, this is still not the case for most of the companies on the market. Whenever I talked to developers, testers or product managers e.g. on conferences the majority of people mention the lack of knowledge, infrastructure or time pressure are the biggest hurdles for them to start with automation. But starting with test automation can be much easier when using a test automation platform*. Test automation platforms usually offer e.g. the infrastructure, a list of supported test automation tools and a setup guide.

Test Automation Platforms

In this article, I want to briefly introduce you to test automation platforms with their pros and cons. For the comparison I picked the following platforms:

Depending on the own software development cycle, the used programming languages and toolchain a test automation platform must fulfill different requirements in order to be used most efficiently.

Saucelabs

Pros:

  • Provide support for many browsers, OS and mobile devices
  • Offers a testing cloud with real and emulated devices
  • Support for many open source testing frameworks (Appium, Selenium and more)
  • Visual testing support
  • Rest API access
  • Parallel test execution
  • Infrastructure hosted in the cloud
  • Test execution dashboard
  • Integration to different continuous integration systems
  • VPN Connection to own network
  • Offers lots of resources

Cons:

  • Test execution time can be slower
  • Depending on your setup the price of the hosted service can get expensive

TestProject

Pros:

  • Support for many open source testing frameworks (Appium, Selenium and more)
  • Testing on real devices, web browsers and emulators/ simulators
  • Offers a smart test recorder
  • SDK for Java and .Net Core  for test automation developers
  • Community based Add-ons
  • Rest API access
  • Test execution & report dashboards
  • Ability to easily collaborate with the entire team anywhere around the globe
  • The first test automation platform that offers a free forever plan, that includes all the small-medium size organizations test automation needs (for enterprises or highly advanced clients there’s a pro plan for only $8 per month)
  • Integration to different continuous integration systems
  • Community & team support
  • Offers lots of resources
  • Parallel test execution
  • Supports iOS on Windows

Cons:

  • Currently offers no cloud hosting of devices and browser setups

bitbar

Pros:

  • Real device testing in the cloud or with own hardware
  • Many mobile testing frameworks are supported
  • Test result dashboard
  • REST API access
  • Parallel test execution
  • Integration to different continuous integration systems
  • VPN Connection to own network
  • Lots of resources

Cons:

  • Focus “only” on mobile testing, no web browser testing capabilities
  • Depending on your setup the price of the hosted service can get expensive

BrowserStack

Pros:

  • Provide support for many browsers, OS and mobile devices
  • Many mobile testing frameworks are supported
  • Test result dashboard
  • Parallel test execution
  • Integration to different continuous integration systems
  • REST API access
  • VPN Connection to own network
  • Lots of resources

Cons:

  • Response time of endpoint sometimes slow
  • Depending on your setup the price of the hosted service can get expensive

Summary

As you can see, most of the automation platforms offer a similar feature set. It’s not easy to pick the right platform just by reading the feature specs and the support information material. Every company or team who wants to establish a test automation platform need to invest time to evaluate them. As a starting point, a company should write down their own requirements and the current setup. With this information it might be easier to find the right test automation platform that fits best into the own development infrastructure.

While using the test automation platforms for this post, I was impressed by – TestProject and their offerings, being a first of its kind end-to-end solution that’s completely free and community based. As mentioned in the pros, they offer a smart test recorder (I know what you think now ;), but I will give it a try) and the community add-on support which sounded really interesting to me. To satisfy my curiosity, I will check this platform out and will provide you with some further posts in the next weeks or months.

*Disclaimer: A minimum set of programming skills must be available in order to work with test automation platforms.