Today I Learned

Right sed Fred

Thought I’d do a quick summary of the sed command in Linux as I’ve found myself using it today. There’s a lot to it, so much in fact it’s a whole chapter in the Linux in a Nutshell book I found the other day.

Overview

sed is a stream editor. Inputs or streams flow into it, are processed and then flow out of it again. It’s primarily used for editing one or more files. I tend to use it for performing find and replace operations on text files.

Examples

I wanted to think of some real world examples rather than using the usual contrived ones you see everywhere else. Let’s say you have a mysqldump of a wordpress database. It contains a bunch of URL’s that reference your test server (bowett.test) but you want to move the site to your live server (bowett.dev). Here’s what the database file looks like before:

$ cat database_backup.sql 
INSERT INTO 
   `wp_options` 
VALUES 
   (1,'siteurl','https://bowett.test','yes'),
   (2,'home','https://bowett.test','yes');

And here’s the command to replace bowett.test with bowett.dev

$ sed 's/bowett.test/bowett.dev/g' database_backup.sql

Let’s break this down:

  • The first part is the command – sed.
  • The second part is the option: ‘s/bowett.test/bowett.dev/g’
    • The ‘s’ specifies the substitution operation.
    • The ‘/’ are delimiters:
      • The first part is the search pattern
      • The second part is the replacement string.
    • The ‘g’ (global) specifies we want to replace all occurrences.
  • The third part is the input file. In this case database_backup.sql

So here’s the full command and output:

$ sed 's/bowett.test/bowett.dev/g' database_backup.sql 
INSERT INTO 
   `wp_options` 
VALUES 
   (1,'siteurl','https://bowett.dev','yes'),
   (2,'home','https://bowett.dev','yes');

To only replace occurances on a line 5:

$ sed '5 s/bowett.test/bowett.dev/' database_backup.sql 
INSERT INTO 
   `wp_options` 
VALUES 
   (1,'siteurl','https://bowett.test','yes'),
   (2,'home','https://bowett.dev','yes');

Obviously, most of the time you don’t want to just see the output you want it in a new file. This being Linux you can just pipe the output:

$ sed 's/bowett.test/bowett.dev/g' 
> database_backup.sql > new_backup.sql

Advantages

Obviously you could just open the backup file in a text editor and do a ‘find and replace’ to achieve the above. But…

  1. What if you are on a server? Do you really want to ship the file down to your machine, do the operation and ship it back?
  2. What if the file is large? I’ve had database backups in the hundreds of megabtyes (even gigabytes!). Try opening those with a text editor! Because sed is a ‘stream’ editor it will happily handle files of any size.

Summary

sed is REALLY useful! I have only scratched the surface of what it can do here. There are loads and loads of options for doing lots of varied manipulations of files – you can even use Regex’s 🙂 For a complete overview head on over to gnu.org.

  • By
  • On May 12, 2020