If you’re like most Linux users, you’re probably familiar with the command line utilities set and pipefail. These two commands are essential for scripting and automation on Linux. In this article, we’ll show you how to use set and pipefail in Bash scripts to create powerful automation tools. First, let’s take a look at what set does. set is used to manipulate the environment variables of a running process or shell session. This includes setting the value of variables, setting default values for variables, and clearing variable values. For example, the following command sets the value of the MYVAR variable to “Hello world!”: set MYVAR=“Hello world!” Next, let’s take a look at what pipefail does. pipefail is used to control whether or not a command succeeds or fails based on input from another command or script. For example, if we wanted to create a script that would print “If successful…,” followed by an error message if something went wrong while running our script, we could use pipefail like so: #!/bin/bash if [ $# -eq 0 ]; then echo “If successful…”; else echo “Error: missing argument” fi ..
RELATED: The Beginner’s Guide to Shell Scripting: The Basics
Bash Scripts and Error Conditions
Bash shell scripts are great. They’re quick to write and they don’t need compiling. Any repetitive or multi-stage action that you need to perform can be wrapped in a convenient script. And because scripts can call any of the standard Linux utilities, you’re not limited to the capabilities of the shell language itself.
But problems can arise when you call an external utility or program. If it fails, the external utility will close down and send a return code to the shell, and it might even print an error message to the terminal. But your script will carry on processing. Perhaps that’s not what you wanted. If an error occurs early in the execution of the script it might lead to worse issues if the rest of the script is allowed to run.
You could check the return code from each external process as they complete, but that becomes difficult when processes are piped into other processes. The return code will be from the process at the end of the pipe, not the one in the middle that failed. Of course, errors can occur inside your script too, such as trying to access an uninitialized variable.
The set and pipefile commands let you decide what happens when errors like these occur. They also let you detect errors even when they happen in the middle of a pipe chain.
Here’s how to use them.
Demonstrating the Problem
Here’s a trivial Bash script. It echoes two lines of text to the terminal. You can run this script if you copy the text into an editor and save it as “script-1.sh.”
To make it executable you’ll need to use chmod:
You’ll need to run that command on each script if you want to run them on your computer. Let’s run the script:
The two lines of text are sent to the terminal window as expected.
Let’s modify the script slightly. We’ll ask ls to list the details of a file that doesn’t exist. This will fail. We saved this as “script-2.sh” and made it executable.
When we run this script we see the error message from ls .
Although the ls command failed, the script continued to run. And even though there was an error during the script’s execution, the return code from the script to the shell is zero, which indicates success. We can check this using echo and the $? variable which holds the last return code sent to the shell.
The zero that gets reported is the return code from the second echo in the script. So there are two issues with this scenario. The first is the script had an error but it carried on running. That can lead to other problems if the rest of the script expects or depends on the action that failed actually succeeded. And the second is that if another script or process needs to check the success or failure of this script, it’ll get a false reading.
The set -e Option
The set -e (exit) option causes a script to exit if any of the processes it calls generate a non-zero return code. Anything non-zero is taken to be a failure.
By adding the set -e option to the start of the script, we can change its behavior. This is “script-3.sh.”
If we run this script we’ll see the effect of set -e.
The script is halted and the return code sent to the shell is a non-zero value.
Dealing With Failures in Pipes
Piping adds more complexity to the problem. The return code that comes out of a piped sequence of commands is the return code from the last command in the chain. If there’s a failure with a command in the middle of the chain we’re back to square one. That return code is lost, and the script will carry on processing.
We can see the effects of piping commands with different return codes using the true and false shell built-ins. These two commands do no more than generate a return code of zero or one, respectively.
If we pipe false into true —with false representing a failing process—we get true‘s return code of zero.
Bash does have an array variable called PIPESTATUS, and this captures all of the return codes from each program in the pipe chain.
PIPESTATUS only holds the return codes until the next program runs, and trying to determine which return code goes with which program can get very messy very quickly.
This is where set -o (options) and pipefail come in. This is “script-4.sh.” This will try to pipe the contents of a file that doesn’t exist into wc.
This fails, as we’d expect.
The first zero is the output from wc, telling us it didn’t read any lines for the missing file. The second zero is the return code from the second echo command.
We’ll add in the -o pipefail , save it as “script-5.sh”, and make it executable.
Let’s run that and check the return code.
The script halts and the second echo command isn’t executed. The return code sent to the shell is one, correctly indicating a failure.
RELATED: How to Use the Echo Command on Linux
Catching Uninitialized Variables
Uninitialized variables can be difficult to spot in a real-world script. If we try to echo the value of an uninitialized variable, echo simply prints a blank line. It doesn’t raise an error message. The rest of the script will continue to execute.
This is script-6.sh.
We’ll run it and observe its behavior.
The script steps over the uninitialized variable, and continues to execute. The return code is zero. Trying to find an error like this in a very long and complicated script can be very difficult.
We can trap this type of error using the set -u (unset) option. We’ll add that to our growing collection of set options at the top of the script, save it as “script-7.sh”, and make it executable.
Let’s run the script:
The uninitialized variable is detected, the script halts, and the return code is set to one.
The -u (unset) option is intelligent enough not to be triggered by situations where you can legitimately interact with an uninitialized variable.
In “script-8.sh”, the script checks whether the variable New_Var is initialized or not. You don’t want the script to stop here, in a real-world script you’ll perform further processing and deal with the situation yourself.
Note that we’ve added the -u option as the second option in the set statement. The -o pipefail option must come last.
In “script-9.sh”, the uninitialized variable is tested and if it is uninitialized, a default value is provided instead.
The scripts are allowed to run through to their completion.
Sealed With a x
Another handy option to use is the set -x (execute and print) option. When you’re writing scripts, this is can be a lifesaver. it prints the commands and their parameters as they are executed.
It gives you a quick “rough and ready” form of execution trace. Isolating logic flaws and spotting bugs becomes much, much easier.
We’ll add the set -x option to “script-8.sh”, save it as “script-10.sh”, and make it executable.
Run it to see the trace lines.
Spotting bugs in these trivial example scripts is easy. When you start to write more involved scripts, these options will prove their worth.
