February 2, 2021

10 Tips for Writing Your Tugboat YAML Config

Matt Westgate

The primary reason we don’t take the time to set up automated review environments in our development workflow is that the process of getting to automation is front-loaded and hard. It requires you to sit down and codify every decision and failsafe necessary to deploy your infrastructure. However, the benefit of doing this work is developers can now use infrastructure like they use code, and they can spin up what they need when they need it.

By investing in automating your ephemeral infrastructure, you are literally removing all technical barriers for collaboration between development and the rest of the team. And that is crucial. You are making pathways for engineers to share their work the moment it is ready, in a clean and fully isolated environment, they (and you!) didn’t have to spin up or wait in line to access. In short, you’ve made the development process visual and accessible and removed the conflict and mistakes that resulted from not being able to see the work as if it were already in production.

In this article, we’ll walk you through the shortcuts of how to codify your review environments to give you fully isolated and clean deployment Previews for any code push. That will pave the way to eventually leverage Tugboat for automated visual regression, accessibility audits, SEO reviews, and performance analysis, all within the testing cycle of your feature branches.

As we go through creating the Tugboat config file, if you have questions or concerns, reach out to us on Twitter or join our community in Slack!

For reference and a quick jump, here are the 10 tips we’re going to cover:

  1. Install the Tugboat CLI
  2. Don’t copy/paste the starter configs
  3. Use Tugboat’s container images
  4. Start with a minimal config file
  5. Validate your config
  6. Make a lightweight build
  7. Test additional commands in the Tugboat Shell
  8. Leverage Base Previews
  9. Save your work!
  10. You’re not alone

Assumptions

To get started, we’re going to assume three things:

  1. You have a Tugboat account. If you don’t, grab a free one. No credit card needed. Free accounts will give you 2GB of build-time RAM per service and 512MB of run-time RAM for each service.
  2. You have a Git repository to work in, and have logged into Tugboat, created a project, and connected the repository to a new project.
  3. At the root of your Git repository, you have created an empty .tugboat/config.yml file. Don’t forget the “dot” in front of the “tugboat” folder. Note: the “.tugboat” folder will be invisible by default when you view it locally, so you’ll need to show hidden files and folders in your text editor or IDE.

Tip 1: Install the Tugboat CLI

As a developer, it can be more enjoyable to work locally with your preferred apps and setup. If you want the full range of what Tugboat is capable of, download the CLI, and get all the power of Tugboat right from your terminal. For Mac users, Tugboat can be installed using homebrew. After you’ve got the CLI, you’ll need to set an access token.

To see what the CLI can do, run tugboat from the command line. Once you’ve browsed the available commands, adding the --help parameter to any command will give you details. Say, for instance, you want to learn about the validate command. Type tugboat validate --help to see what it can do.

Here’s the help output of the high-level tugboat command:


$ tugboat
Usage: tugboat [options] [command]

Commands:


api   [args...]           Execute a Tugboat API call directly
 cancel                              Cancel a preview that is currently building
 clone  [args...]                    Clone a Tugboat Preview
 create  [name\ [args...]          Create a Tugboat resource
 delete|rm  [args...]                Delete a Tugboat resource
 find                                Find a Tugboat resource
 grant  [args...]                    Grant access to a Tugboat resource
 list|ls  [args...]                  List a Tugboat resource
 log [options]  [args...]            View the log for a Tugboat resource
 output                              View the output of a Tugboat Service
 rebuild  [args...]                  Rebuild a Tugboat Preview
 refresh  [args...]                  Refresh a Tugboat Preview
 rekey                               Rekey a Tugboat Preview
 reset                               Reset a Tugboat Preview or Service
 revoke  [args...]                   Revoke access from a Tugboat resource
 shell  [args...]                    Start a shell session in a Tugboat resource
 start                               Start a Tugboat Preview
 statistics|stats   [args...]  Get historical statistics
 stop                                Stop a Tugboat Preview
 suspend                             Suspend a Tugboat Preview
 update  [args...]                   Update a Tugboat resource
 validate  [ref]            Validate a tugboat config.yml
 version                                 Show the Tugboat client/server versions
 help [cmd]                              display help for [cmd]

With the Tugboat CLI downloaded and installed, it’s time for tip number 2!

Tip 2: Don’t copy/paste the starter configs

There are many starter configs for Tugboat, but if you copy and paste the whole config, you are bound to get errors since they need to be customized. You’ll need to specify data sources, for example. It’s better to test the starter config lines one by one using the tugboat shell. It will be faster this way so errors can be isolated.

Tip 3: Use Tugboat’s container images

You can bring your own containers to Tugboat, but when you’re getting set up, start with things that are known to work. All of Tugboat’s container images are extensions of official Docker images and have been tailored to work in the Tugboat environment. Start with them. Later, when you have a working build, you can dive into expert mode and make Tugboat images that mirror production. Our friends at Chromatic have a great article about using custom Docker images within Tugboat.

Tip 4: Start with a minimal config file

Your initial config file should be simple and straightforward: install the image you need and make it the default service. That’s it.

Say you’re installing PHP, for example; your config file might look like this:


services:
 # What to call the service hosting the site.
 php:
   # Use PHP 7.x with Apache; this syntax pulls in the latest version of PHP 7
   image: tugboatqa/php:7-apache

   # Set this as the default service. This does a few things
   #   1. Clones the git repository into the service container
   #   2. Exposes port 80 to the Tugboat HTTP proxy
   #   3. Routes requests to the preview URL to this service
   default: true

The line default: true is not required, but it’s a good practice to have it. If you have a single service, it will be the default service with or without that line. But as soon as you have two services defined, you’ll need to specify which one is the default service to route requests to.

The above config file will give you a PHP service with your Git repository checked out at /var/lib/tugboat in that service container. Now before you commit that code to your repo, let’s validate it and do a build.

Tip 5: Validate your config

With the CLI, you can validate your config locally before you push it to Git, which then Tugboat will grab and build. This will save you time, so you don’t wait for a build to potentially error out because of a typo, or worse, tabs rather than spaces. YAML files don’t allow tabs. Validating your config can only be done through the tugboat CLI and not the Tugboat web interface.

In our PHP example above, we run this validation command from the root of our checked-out repository on our local:


$ tugboat validate .tugboat/config.yml

And we’d receive the following output:


{
  "services": {
    "php": {
      "image": "tugboatqa/php:7-apache",
      "default": true,
      "checkout": true,
      "expose": 80,
      "https": true
    }
  }
}

Notice the structured content? That is how Tugboat is parsing your YAML config. If fully structured content was returned, then you’re all set! If Tugboat could not parse your YAML config, you would not see the structured output, and instead, you’d see an error message, and in most cases, the location of the error within your YAML file.

For example, if we changed the name we gave our service from php to php yippee, Tugboat will not be happy and throw an error. Service names double as hostnames, and a host name cannot have a space character. This is the error you will see:

Invalid Preview configuration (Error 1056): service name must be a valid host name: php yippee

Some of the other standard error 1056 validation errors are:

  • No default service defined.
  • Multiple default services defined.
  • Service name is longer than 39 characters.
  • Service name must be a valid host name.
  • Exposed service must have at least one of “http” or “https” enabled.
  • No image provided for service.

Tip 6: Make a lightweight build

With a clean validation, let’s create a Tugboat Preview and see how it goes. Since we’re not sure if the build will work yet, we can pass the config file to Tugboat without pushing a commit to our repository. First, we’ll grab the Tugboat Repository ID using the tugboat list command, and then we’ll create our first Preview with tugboat create.

Grab the Repository ID with this:


$ tugboat list repos

And if you want the full details of the repository, you can call tugboat list repos --verbose

The raw output will look something like this:


Project: 23udbh293hd8238hd892347w - TugboatDemo

 Repository ID             Size      Previews  Provider   Name
 ------------------------  --------  --------  ---------  -----------------
 238239dj505fj23223ehd334    2.85GB         3  github     TugboatDemo/foo

Now let’s build a Preview from the main branch of our repository using our Repository ID and pass the path to our work-in-progress Tugboat config.yml file as a parameter as well.


$ tugboat create preview main repo=238239dj505fj23223ehd334 
config=.tugboat/config.yml

Note: if you want to build something other than a branch, you can also build a tag, commit, or a pull/merge request. If you want to do one of those, you’ll need to pass the type option (i.e. type=tag, type=commit, type=pullrequest, or type=mergerequest). If you don’t pass in type, Tugboat attempts to guess the type by searching for a ref that matches the provided preview name. In our case, that was main.

And here is the abbreviated output from our successful build:


   2021-01-22T18:40:43.151Z - Building preview: 2348jwjwfbfcwwke2302340f (yml-tips)
   2021-01-22T18:41:05.067Z - 2348jwjwfbfcwwke2302340f (yml-tips) is ready
   
      Project: 23udbh293hd8238hd892347w - TugboatDemo
         Repository: 238239dj505fj23223ehd334 - TugboatDemo/drupal
   
            Preview ID                 OPTS     Status       Size      Name
            ------------------------  --------  -----------  --------  ------------
            2348jwjwfbfcwwke2302340f            ready        145.19MB  main
              https://main-2qqjitkerdje83.tugboat.qa

Make a note of your Preview ID. It will come in handy later to shell into the build. Our Preview ID is 2348jwjwfbfcwwke2302340f.

Just a note that you won’t see anything if you visit the Preview URL. We haven’t told Tugboat where the document root is, so there is nothing to display. If your web folder was ./web in your repos, you could add the line - ln -snf “${TUGBOAT_ROOT}/web” “${DOCROOT}” in your config to serve those pages. See here for details on setting up your document root.

Tip 7: Test additional commands in the Tugboat Shell

Now that you have build up and running, you can shell into the service container to test the subsequent build steps directly in the environment. The terminal for a service can be accessed in your browser via the Preview landing page, or through the command line using tugboat shell.

The command looks like this:


 $ tugboat shell 

Running that command will drop you into a shell on your default service. If you need to test a command on a different service than your default, use tugboat ls services to get the corresponding Service ID, and you’d pass that instead with tugboat shell [service-id].

So you should now be in your container and able to navigate around! By default, you start at /var/lib/tugboat. If you ls that directory, you’ll see your git repository. And this will be true for any service where checkout: true is set, or you’re in the default service.

As you gear up to test your build steps, now is an excellent time to familiarize yourself with the environment variablesTugboat makes available. $TUGBOAT_ROOT, for example, points to the filesystem location where the git repository is cloned, usually /var/lib/tugboat.

You can also set custom environment variables in the Tugboat UI on the Repository Settings page. Custom environment variables come in handy when you don’t want secrets or snippets committed to your git repository to be used.

One by one, start testing out your commands. Since most of Tugboat’s images are Debian-based, a common starting place is apt-get update && apt install <foo> to install additional packages. Or you could go back to one of our starter configs and begin testing out the lines, one by one.</foo>

One thing to be mindful of: when using the shell to test your commands, the shell will remember what the current working directory is. However, the lines executed in the Tugboat are run one by one. That means doing something like this in your config.yml won’t work:


 - cd /tmp
 - curl -o db-dump.sql https://example.com/db-dump.sql

Instead, you can change that to a one-liner:


  - cd /tmp && curl -o db-dump.sql https://example.com/db-dump.sql

When you have a working line that does what you want it to do, add it to your local Tugboat config and then validate it again. Now is a good time to commit our work back into our Git repos. Once your updated config file is pushed into your repository, let’s delete the existing Preview and then rebuild it.


$ tugboat delete 
$ tugboat create preview main repo=238239dj505fj23223ehd334

If, for some reason, you’ve made a mess of your environment and can’t remember which config file Tugboat is using, you can ask Tugboat to show you, though you won’t get the raw version of the config file. You’ll receive the config file as processed by Tugboat and broken down by service. Here’s the command:


$ tugboat ls services preview= -j

If you also have jq, the command-line JSON processor installed, you can take this one step further.


$ tugboat ls services preview=[previewid] -j | jq '.[] | { (.name): .config }'

And if you want to reset your environment back to the way it was before you started issuing shell commands to test build steps, you can issue a reset.


$ tugboat reset 

The Preview is reset back to the original YAML config you passed in via the command line. It’s a better way to start again without the overhead of waiting for a rebuild.

Before you move on to tip number 8, make sure you have a working config file checked into your repository, and you’ve asked Tugboat to rebuild (not reset) the Preview.

Tip 8: Leverage Base Previews

Now that you’ve got a good pattern for validating your config and testing your script lines, we can do one more thing to make the setup happen as quickly as possible. We can take a snapshot of your build and then rebuild using the snapshot as the starting place. We’ll do that with Base Previews. This is what makes Tugboat the fastest and most efficient Preview Deployment platform on the market.

Within the config file, the build script happens in different “steps.” Each service can have init, update, and buildsteps. There are other steps, but we don’t need to cover those now.

Overall, the structure looks like this:


services:
 php:
   image: tugboatqa/php:7-apache
   default: true
   commands:
     init:
       # Put the commands you know work, here. 
       - docker-php-ext-install opcache
       - a2enmod headers rewrite
       - composer self-update
       - composer install --optimize-autoloader

     build:
       # After you make a base preview, test your new commands here so you don’t 
       # have to start a build from scratch each time. Instead, kick off a rebuild
       # command. 

While there’s a flowchart on the build process for Tugboat, what you need to know is that when you promote a Preview to a Base Preview, Tugboat will then use the Base Preview as a starting point for any new environments, and only run the build steps of each service, saving loads of time.

Before you make a Base Preview, commit your working build script to your repository! By default, the Base Preview will pull in new changes from your repository nightly, so you want to keep things in sync. Usually, your Base Preview is the branch you have in production.

To make a Base Preview through the UI, click on “Settings” for the Preview to promote it a Base Preview. Using the command line, we need to “update” the Preview and define an anchor point (see what we did there, how nautical of us).

To promote a Preview to a Base Preview:


$ tugboat update  anchor=true

Now, subsequent builds will start from this newly defined Base Preview.

At this point, make a new branch to continue working on your Tugboat build script. That branch will inherit the Base Preview you created as the starting point. And anytime you make a build or a rebuild from that new branch, you’ll notice a couple of things from the CLI output:

  1. You’ll only see the “Running BUILD commands” line and not the “Running INIT commands” or “Running UPDATE commands” line in the log files.
  2. The resulting Preview table will have an anchor emoji next to it (or a capital “a” if you’re using Windows), signifying it’s using a Base Preview.

The extra effort of setting up a Base Preview can end up shaving hours off of build times when working with huge file assets, such as large databases and the like. All that data has been saved in the Base Preview and doesn’t need to be reimported. Time is money, friends, and waiting for builds is utterly mind-numbing.

Tip 9: Save your work!

Remember, these are ephemeral environments. If the commands are not in the YAML config, your data will not be remembered. When you have working lines, add them to your config and commit them back to your repository. When you delete a Preview, everything you worked on will be gone unless scripted. This is an essential best practice for automation and sometimes an easy one to forget.

Tip 10: You’re not alone

Look, the most challenging part of automation is getting started. But putting in the effort now means everyone on your team has consistent and fully isolated environments to work and test in. It means you can make the entire development process visual and collaborative, and your whole team benefits. It’s worth it, you’ll get there, and we are here to help!

You don’t have to do it alone. Sometimes it’s hard to ask for help. We have a small community of engineers and customers who help each other get our automation scripts in place. Stop on by and say ahoy!

Need More Info on Tugboat?

Tell us about your project, schedule a demo, or consult with a Tugboat Technical Account Executive.

Your request has been sent.
Oops! Something went wrong while submitting the form.