I’m Cezary Statkiewicz and I’m Senior Software Engineer at Bitbar.com. One of our offerings is Android CTS integration with CI environment, so today I’d like to make a short introduction to CTS usage.
Update: If you are interested in hearing more about running automated Android CTS runs with Testdroid Server against your own devices, please contact us and we will get back to you with details.
One of the first things that one can see when starting using CTS is a complete lack of documentation. There is no mention in official on-line documentation about it, and if one would google a bit, he’d find only a couple of blog posts and some questions on android mailing lists. This is also one of motivators for me to write about it.
After writing this entry I’ve found that actually there IS quite good documentation (very broad introduction, compatibility definition document and CTS docs) on http://source.android.com/compatibility/index.html, which probably was released few days ago.
First of all, CTS is not a different story than normal testing in Android, which is based on JUnit and Instrumentation testing. It is just an extension of this process, which automates test execution, test results gathering and aggregation. CTS is made of two main elements:
cts cli command and tests, that are mostly unit tests and reference applications from rest of Android platform code. Some tests are provided expilictly by this test suite. What CTS does is grouping test packages (which are actually Android apps) into test plans (there are several test plans pre-created and you can define your own plans), executes those plans on device, gets the results and creates test result file (aggregated from junit test results). It uses
adb command internally.
Since Android 2.0 CTS is available in public source repository, so anyone can come and play. I assume you are:
- using Android platform from git repository,
- using unix-like host system,
- operating in in directory with android,
- have built platform (
make) and sdk (
If not, see instructions on how to get the code. To build CTS you have to execute:
This will build CTS and install it in
$host_type is your host operating system, i.e.
linux-x86. Under this directory you can find directories and files:
android-cts/repository/plans/with test plans defined as xml files,
android-cts/repository/results/with test results,
android-cts/repository/testcases/with test cases for CTS.
android-cts/repository/host_config.xmlwith configuration of CTS.
For convenience add to your
$PATH path to
out/host/linux-x86/bin/. To run cts, first you will have to connect a device to your host machine or run emulator with some AVD (see http://developer.android.com/guide/developing/tools/avd.html for more details on creating Android Virtual Device and http://developer.android.com/guide/developing/tools/emulator.html for information on using Android’s emulator). Then run
adb start-server to be able to connect debugging tools with the device (ADB is used by CTS internally, so if you’re interested in more details about it, read http://developer.android.com/guide/developing/tools/adb.html). We’ll be running AVD named “test_case” in console on screen without video nor audio output (since this is all about automated testing, we want to run it on remote, headless machine):
$ screen emulator @test_case -no-window -noaudio &
$ adb start-server
Working with CTS
Now you should have working cts environment. To get help, type ‘help’. You will get:
cts_host > help
Usage: command options
Avaiable commands and options:
help: show this message
exit: exit cts command line
ls --plan: list available plans
ls --plan plan_name: list contents of the plan with specified name
add --plan plan_name: add a new plan with specified name
add --derivedplan plan_name -s/--session session_id -r/--result result_type: derive a plan from the given session
rm --plan plan_name/all: remove a plan or all plans from repository
start --plan test_plan_name: run a test plan
start --plan test_plan_name -d/--device device_ID: run a test plan using the specified device
start --plan test_plan_name -t/--test test_name: run a specific test
start --plan test_plan_name -p/--package java_package_name: run a specific java package
start --plan test_plan_name -t/--test test_name -d/--device device_ID: run a specific test using the specified device
start --plan test_plan_name -p/--package java_package_name -d/--device device_ID: run a specific java package using the specified device
ls -p/--package: list available packages
ls -p/--package package_name: list contents of the package with specified name
add -p/--package root: add packages from root to repository
rm -p/--package package_name/all: remove a package or all packages from repository
ls -r/--result: list all result of sessions
ls -r/--result -s/--session session_id: list detail case result of a specified session
ls -r/--result [pass/fail/notExecuted/timeout] -s/--session session_id: list detail cases of a specified session by the specified result.
history/h: list all commands in command history
history/h count: list the latest count records in command history
history/h -e num: run the command designated by 'num' in command history
ls -d/--device: list available devices
As you can see, you can list test plans, list test packages (and add/remove those), execute test plans (or just some packages from them) on specified device and see some short information about test results (sessions). One note here: test plans are executed in sessions, which in general means that until test plan is not finished (i.e. interrupted with Ctrl+C), it is considered as an ‘open session’. If you would run into a situation when you have many open sessions, just stop cts, remove results from
out/host/linux-x86/cts/android-cts/repository/results/ and start cts again.
To see a list of sessions, execute ls -r in cts console:
Android CTS version 2.1_r1
cts_host > ls -r
cts_host > List of all results:
Session Test result Start time End time Test plan name
Pass Fail Timeout NotExecuted
1 5 0 0 0 2010.05.20 10:15:19 2010.05.20 10:18:39 AppSecurity
3 1 0 0 0 2010.05.20 10:51:22 2010.05.20 10:54:44 Signature
5 34 0 0 14728 2010.05.20 10:10:03 2010.05.20 10:09:50 Java
Standard CTS installation has 8 plans available:
cts_host > ls --plan
List of plans (8 in total):
and have over 60 test packages to use:
cts_host > ls --package
Available packages (55 in total):
Note that some packages used in CTS are not visible in this list. Total count of test cases is over 23000, so full run of all tests takes several hours to complete.
You can run test plan by running cts shell, or non-interactively:
$ cts start --plan $PLAN_NAME
To be more exact:
$ cts start --plan Signature
Android CTS version 2.1_r1
cts_host > start test plan Signature
CTS_INFO >>> Checking API...
CTS_INFO >>> This might take several minutes, please be patient...
API Check PASS.
Test summary: pass=1 fail=0 timeOut=0 notExecuted=0 Total=1
This will produce results in
$session_date can be
2010.05.20_10.42.58. Contents of this directory:
Most importaint file is testResult.xml. It contains information on host machine, test device, test plan and tests execution status:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="cts_result.xsl"?>
<TestResult endtime="Thu May 20 10:46:21 CDT 2010" starttime="Thu May 20 10:42:58 CDT 2010" testPlan="Signature" version="1.2">
<BuildInfo Xdpi="160.0" Ydpi="160.0" androidPlatformVersion="7" buildID="ECLAIR" buildName="generic" buildVersion="2.1-update1" build_board="unknown" build_brand="generic" build_device="generic" build_fingerprint="generic/generic/generic/:2.1-update1/ECLAIR/eng.shaker.20100519.082949:eng/test-keys" build_model="generic" build_type="eng" deviceID="emulator-5554" imei="000000000000000" imsi="310260000000000" keypad="qwerty" locales="en_US" navigation="trackball" network="Android" touch="finger"/>
<Os arch="amd64" name="Linux" version="2.6.26-2-amd64"/>
<Java name="Sun Microsystems Inc." version="1.5.0_17"/>
<IntValue name="maxTestCount" value="600"/>
<IntValue name="maxTestsInBatchMode" value="100"/>
<IntValue name="testStatusTimeoutMs" value="300000"/>
<IntValue name="batchStartTimeoutMs" value="1800000"/>
<IntValue name="individualStartTimeoutMs" value="300000"/>
<IntValue name="signatureTestTimeoutMs" value="600000"/>
<IntValue name="packageInstallTimeoutMs" value="120000"/>
<IntValue name="postInstallWaitMs" value="10000"/>
<Summary failed="0" notExecuted="0" pass="1" timeout="0"/>
<TestPackage appPackageName="android.tests.sigtest" digest="" name="SignatureTest" signatureCheck="true">
<TestCase name="SignatureTest" priority="">
<Test endtime="Thu May 20 10:46:21 CDT 2010" name="signatureTest" result="pass" starttime="Wed Dec 31 18:00:00 CST 1969"/>
This file is viewable under some web browsers (Firefox is recommended) – it will use appended css and display transfored XML. Note, that those files also available in
There are some notes that should be made about CTS:
- CTS can be run only in one instance on one machine. Executing many instances will result in
CTS_ERROR >>> Error: CTS is being used at the moment. No more than one CTS instance is allowed simultaneouslymessage.
- Tests are executed in context of device, so their execution speed depends on device.
- Test results are aggregated from JUnit data, and during conversion from JUnit there is some information loss.