Smartwatch App Testing Ideas

In the previous posts you have read and hopefully learned something new about Tizen OS, watch OS, Pebble OS, Android Wear and what is important to know when start testing on one (or more) of those mobile platforms. Smartwatches may not be on everyone’s wrists yet, but that time can change very quickly in the next upcoming months and you must be prepared for the new way of interacting with your apps and devices. As you have seen in this series, smartwatches offer a complete new set of interactions, features and challenges for mobile testers and developers. The guys from gizmag did a great job in comparing the different smartwatch platforms from a software and hardware point of view and is a nice summary to the watches I used for this series.

However, the last post in the series is about smartwatch app testing ideas. Before I go into the smartwatch app testing ideas I want to briefly remind you about the 4 key areas of smartwatch app testing.

4 Key Areas of Smartwatch App Testing

The 4 key areas are:

Design
Usability / User Interactions
Functionality
Connectivity

As we have seen on the different smartwatch platforms the design of the apps plays a central role of everything. If the smartwatch app has a bad design, it will be very hard for the customers to use the provided functions in a fun and effective way. Therefore, it is essential to meet the design guidelines of every platform and please do not mix them.

Next to the design the usability and the user interactions are the key for a successful smartwatch app. If the app is not usable, the customers will most likely uninstall it and will move on to the next app. Keep an eye on usability even if it is a small minor thing.

The third point is functionality. This is the key area for every software out there and is very important. Your app should provide useful functions for the users and all the provided functions must work! Therefore, it is important to test the functions like for every other software system.

The last point is connectivity. Currently most of the smartwatch use cases makes only sense with a connected smartphone, tablet or Wi-Fi. With the help of the paired device, the watch is sending and receiving data for further functionality. With smartwatches the connection with other networks or devices reaches the next level and is very important to test the different types in order to provide a high quality app to the customers.

Based on the 4 key areas, I noted down some smartwatch app testing ideas that should help you to kickstart your smartwatch testing activities.

The following table will give you an overview of possible test cases or testing ideas for smartwatch apps. Please keep in mind that these test cases have been written as generic as possible to fit to all the mentioned smartwatch operating systems. It is also important to keep in mind that not all test cases/ ideas will fit to all smartwatch apps.

Depending on the feature set of the smartwatch app not all test cases will work for the app.
If you have further test case/ ideas for smartwatch apps send them to me and I will add them to this list.

#	Possible Test Case	Expected Result
1	Remove the pairing between the smartphone/ tablet and the smartwatch.	If the smartwatch app offers some “offline” functionality where a smartphone/ tablet is not needed this functions must work.
2	Remove the app from the smartphone/ tablet but not from the smartwatch.	Depending on the mobile operating system the watch should show the data that the app can show. Otherwise the app must show a user friendly text informing the user about the missing app on the smartphone/ tablet.
3	If your app requires a login, be logged out on the smartphone/ tablet.	The smartwatch must be in a logout state, too.
4	If your app requires a login, be logged in on the smartphone/ tablet.	The user must be logged in on the smartwatch, too.
5	Activate flight mode on smartphone/ tablet but not on the smartwatch.	The connection between the smartphone/ tablet and smartwatch will be lost. However, the smartwatch app must still work with the data that is available on the device. Depending on the app it may show an error that the connection has been lost.
6	Check the different data states the app can have e.g. no data, massive data.	The smartwatch app must work with no and massive app data.
7	Send all the supported push notification types to the smartphone/ tablet and have a smartwatch paired.	All supported push types must be visible on the smartwatch as well. The information must be readable. If there are call to actions in place, check that all actions are working.
8	Check that UX and design guidelines are met.	All design and UX elements must pass the platform guidelines.
9	Check the battery state during the testing activities. (Smartwatches usually have 1-2 days of battery life, except Pebble watches. Check also the battery life of your phone.)	The battery consumption must be appropriate to its usage. Note down the battery consumption every 10 minutes and compare it with other version. The same applies for the smartphone/ tablet.
10	Check the different connectivity types e.g. Bluetooth, Wi-Fi, NFC, … . Check also that the fallback connection option is used.	All used connectivity types must work as expected. A fallback option maybe Wi-Fi.
11	Test the different gestures.	All supported and used gestures must work.
12	If the smartwatch app provide features depending on user types and payments, login/ logout with different test data in order to see if the smartwatch app handles the different states and data.	All user types and payment models must be supported by the smartwatch app.
13	Leave smartphone/ tablet on a table and move away with the smartwatch. Check how the smartwatch app is handling the connection loss with the smartphone/ tablet.	The connection loss between smartwatch app and smartphone/ tablet should not influence the app in the current state. If the app relies on a permanent connection, the smartwatch app must show a proper error message to the user.
14	If the smartwatch app uses sensors that rely on the skin surface or temperature, test the app with dry, sweaty or tattooed skin.	Some smartwatch sensors may have problems with dry, sweaty or tattooed skin. If the smartwatch app under test is detecting problems with the sensors and the received data it must show a proper error message informing the user about the problem.
15	Depending on the smartwatch platform different interaction modes are available check all of them in the app.	All platform related interaction modes must be supported by the app in order to achieve a great user experience.
16	Send notifications to the smartphone/ tablet app. This notification will be forwarded to the smartwatch app when the battery level of the watch is under 10% (smartphone/ tablet >15%).	Most watches will not receive any notifications when the battery level is under e.g. 10%. Check that the app on both sides is working as expected at that battery state.
17	Test the same as in test case 16, but this time the smartphone/ tablet battery is under 15% (smartwatch over 10%) and check if the notification was send and received.	Depending on the device and your app the notifications will not be forwarded to the smartwatch. Check that the app on both sides is working as expected at that battery state.
18	Change the language of the smartphone/ tablet e.g. from English to German, Spanish or any other language.	Most smartwatches will switch to the same language as the smartphone language the same applies to the apps running on the watches. Check that the smartwatch app is handling the transition to the new language. Watch out for correct date formats, number delimiters etc.
19	Change the language of the smartphone/ tablet e.g. from English to a language that is not supported by the app.	Check that the smartwatch app is using the same language as the smartphone. Not supported languages must switch to the default app language on both sides.
20	Test the Android wear app with all supported mobile platforms e.g. Android and iOS.	Android Wear is also offering some support for iOS. If your app is not optimized for iOS you should check at least if the notifications send from the phone/ tablet to the watch are readable.
21	Test the Pebble app with all supported mobile platforms e.g. Android and iOS.	Pebble OS is also offering some support for iOS. If your app is not optimized for iOS you should check at least if the notifications send from the phone/ tablet to the watch are readable.
22	Test the Tizen app with all supported mobile platforms e.g. Android, Tizen and iOS.	Tizen is also offering some support for iOS and Android devices. If your app is not optimized for iOS or Android you should check at least if the notifications send from the phone/ tablet to the watch are readable.
23	If your app offers movement features you must walk, run or stay at the same place.	The complete movement must be measured by the app. Depending on the speed of the movement the app must distinguish between walking, running or no movement. Watch out for performance issues while walking or running.
24	If your app relies on GPS data, switch GPS off.	If the smartwatch app relies on GPS data, the app must handle the state that GPS is missing. The app must show a useful text to the user.
25	Use the smartwatch app in different weather conditions, e.g. sunny day, rainy day, snowy day, …	Sunny day: Check that you can read the information on the small screen with lots of sun. Check that sweaty fingers are able to interact with the app. Rainy day: Are you able to interact with wet fingers, too? Can you read everything? Snowy day: While wearing special smartphone/ tablet gloves are you able to interact with the smartwatch app?
26	Have an eye on the smartphone/ tablet temperature. Send lots of data to the smartwatch to increase temperature of the smartphone/ tablet. For example, send sound or videos files to increase the temperature.	Sending and receiving lots of data via Bluetooth or Wi-Fi is causing CPU and network usage. It may happen that the smartphone/ tablet is getting hotter while sending and receiving the date to the smartwatch. If the device is getting too hot it may damage the hardware. Therefore, the apps should not consume too many hardware resources.
27	Have an eye on the smartwatch temperature: Receive lots of data on the smartwatch to increase temperature of the watch. For example, send sound or videos files to increase the temperature.	Sending and receiving lots of data is causing CPU, network and memory usage and may slow down the device and have an impact to the smartwatch app. Smartwatches still have limited hardware resources compared to smartphones/ tablets. While checking this idea, watch out for app crashes, data or connection loss between the devices.
28	Test all supported sensors.	All supported sensors must work as expected.
29	Have an eye on the smartphone/ tablet performance. Send lots of data to the smartwatch to increase hardware usage of the smartphone/ tablet. For example, send sound or videos files to increase the CPU, memory or network usage.	The smartphone/ tablet may slow down because of the high amount of data. Use performance-profiling tools to measure bottlenecks. It is also useful to compare the different app versions using a blink test (http://www.thefriendlytester.co.uk/2015/02/blink-testing-in-mobile-context.html).
30	Have an eye on the smartwatch performance. Receive lots of data on the smartwatch to increase hardware usage of the watch. For example, send sound or videos files to increase the CPU, memory or network usage.	The smartwatch app may slow down because of hardware limitations that can cause performance issues. Use the blink test to compare the different app versions on the watch.
31	Send data from smartwatch to smartphone/ tablet with the different functions like keyboard, voice or predefined text/ sound.	The data must be send to the smartphone/ tablet and must be the same as on the smartwatch.
32	Test the different input methods e.g. bezel, buttons, touch screen.	All supported input methods must be supported.
33	Try to send data from smartwatch when phone is in airplane mode.	Useful error message must be displayed on the watch. The message should offer a call to action to retry or retry later.
34	Activate flight mode on smartwatch but not on the smartphone/ tablet.	Both sides of the app (smartphone/ tablet, smartwatch) must show a meaningful message to the user.
35	Test the smartwatch app with left and right hand.	The smartwatch app must be usable by left and right handers.
36	Test the smartwatch app on different screen resolutions e.g. on Android Wear with different hardware manufacturer.	The smartwatch app must support different screen resolutions e.g. 128×128px, 144×168px, 220×176px, 272×340px, 312×390px, 320×290px, and 320×320px.
37	Close the smartwatch app to send it into background. Start the app from multi task again.	The app must start from multi task from the last position. If the app relies on data, check that the data e.g. time, list, weather gets updated.
38	While testing the smartwatch app call the smartphone that is paired with the watch.	Incoming calls must pause the running watch app. When the phone call is over, the watch app must be in the same state as before the call.
39	Most smartwatches offer a “Do not disturb mode”, turn it on and check functions of the app.	Depending on the use case of the app, functions may not work in this mode. In that case the app must display a meaningful message to the user.
40	Check that the text fits to all UI elements.	All supported languages must fit into the defined UI elements.
41	Test install, uninstall and update process.	During this test no errors or warnings should be shown to the users. Check also for data loss.
42	Check that own watch face is not…	… consuming to much battery. Is showing the data correct. Getting updated from time to time.
43	Check install of own app via companion app.	Installation of app must work.
44	Configure watch app via companion app.	The watch must be configured via the companion app.
45	Remove companion app on paired device.	Own app must be able to handle to uninstallation of companion app.
46	Change the timezone on the paired device e.g. smartphone or tablet. (submitted by Ron Werner)	The watch and the app must reflect the new time. Use case: frequent traveller
47	Sync the smartwatch with multiple smartphones or tablets one after each other (if possible). (submitted by Ron Werner)	Previous user data must be erased and/or impossible to access. Use case: User with multiple devices or shared smartwatches.
48	Use special characters & UTF-8 within the app either on the paired device or on the watch. (submitted by Ron Werner)	All characters in text must be shown equally correct on watch as on the companion app.

As a final word for this series I would like to remind you to test your smartwatch apps in the wild as much as possible. Using smartwatch emulators or simulators is not a good idea because of the connectivity with other devices or networks and may not show the expected behavior. Maybe the mobile test pyramid will help you to understand why in the wild testing is so important.

I really hope you enjoyed my series about smartwatch app testing. If you have any further ideas or feedback, please don’t hesitate to contact me via the known channels.

If you want to get this series as eBook, you can get it on leanpub.