• Android
• Our Code

Firebase Test Lab on CircleCI

Circle CI

We decided to take a look at CircleCI as it looked like it would be flexible enough for all our needs. Unfortunately, this flexibility leads to a bit more complexity when setting up your project. While Buddybuild has a nice UI for configuring your project and has built in support for Firebase test lab, CircleCI leaves you to do it yourself.

This post aims to explain how to get your project setup to run your UI tests using Firebase Test Lab.

Step by step

1. Create a Firebase project

You'll need a Firebase project to use Test Lab. A free account on the Spark plan allows 10 tests per day. If you upgrade to the Blaze plan, tests cost $1/device/hour.

2. Create a service account

We'll be using the Google Cloud SDK to run the tests and this requires that you be authenticated. Instead of storing your Google email and password on the CI server, you can create a service account which gives you a json file that contains a private key. You then use this json file to authenticate.

You can create a service account on the Google Cloud Platform console. From this page, select the Firebase project that you created in the previous step then click Create Service Account at the top of the page. Enter a Service Account Name, I chose CircleCI. Select a role of Project > Editor (this is required according to the Firebase docs). Now tick “Furnish a new private key” and ensure JSON is selected as the key type. Clicking Create will finish the process and your JSON key will be downloaded.

3. Enable APIs

The next step in the Firebase docs is to enable a couple of APIs. You do this in the Google Developers Console. The APIs that need to be enabled are Google Cloud Testing API and Cloud Tool Results API - search for these in the list and click Enable if they’re not already enabled.

4. Create a CircleCI project

If you haven’t already added your project to CircleCI, click the Add Projects button, find it in the list and click Set Up Project. Before clicking Start building, we’ll load in our config file.

5. Config file

You’ll probably need to customise this to fit your needs with various jobs and workflows but this is a basic config file for running the tests in Firebase.

The first 4 steps - checkout, restore_cache, run and save_cache - come from the CircleCI template. They improve build times by saving a cache of the gradle dependencies.

Next we run unit tests - this isn’t necessary for what we’re trying to achieve but you’ll probably want to run these anyway.

The store_artifacts step saves the HTML report from the unit test step for future reference and the store_test_results step gives you a test summary on the CircleCI build page.

The next steps all relate to main purpose of this post - running UI tests. First you have to create the APKs for uploading to Firebase - there are 2 of them, the main app APK and the test APK.

The next 3 run steps are for configuring the gcloud SDK tool. We first export the JSON private key into a file on the build server (this relies on an environment variable that we’ll create in the next step). Then we activate the service account using the JSON file. Finally, we tell gcloud which Firebase project to use.

The penultimate step is about finding the APKs that we built earlier. We use the find command with some path patterns to look for the files and put the paths into environment variables. What we’re actually doing is adding 2 lines to $BASH_ENV which is a bash file that CircleCI sources before each step. By adding these lines, the 2 export commands will be run before the next step and our environment variables will be defined.

The final step actually runs the tests. You can adjust the --device parameter to match your requirements. The no_output_timeout setting was necessary because the gcloud command doesn’t print much output while the tests are running and CircleCI was killing our builds after 10 mins of no output.

Having committed this file to your repo, you can go back to CircleCI and click Start building. The build will fail on the activate-service-account step because we haven’t defined our environment variables yet, but I don’t think there’s a way of defining them before starting your first build.

6. Set environment variables

As mentioned in the previous step, we need to create a couple of environment variables in the project settings. This keeps our secrets secret rather than committing them to git.

PROJECT_ID - This is your Firebase Project ID

GOOGLE_AUTH - This is the contents of the JSON file that you downloaded in step 2

With all this in place, re-run your build. Somewhere in the output of the final step you should see “Test results will be streamed to [URL HERE]”. This gives you the URL to see the test results. It would be nice if we could pull these in to the CircleCI build page like Buddybuild does. We’ll post details in the future if we achieve this or find someone else has.

Alternatives

You might want to consider bitrise if your requirements are simple. It has good UI and beta support for running tests using Test Lab. Unfortunately, we have projects with multiple modules and nothing with built-in support for Test Lab seems to support this, whereas CircleCI gives us the control to run multiple gcloud commands, 1 per module.