Home Delete lines in a text file that contain a specific string
Reply: 13

Delete lines in a text file that contain a specific string

A Clockwork Orange
1#
A Clockwork Orange Published in 2011-03-23 19:46:07Z

How would I use sed to delete all lines in a text file that contain a specific string?

admdrew
2#
admdrew Reply to 2017-07-13 14:12:49Z

To remove the line and print the output to standard out:

sed '/pattern to match/d' ./infile

To directly modify the file:

sed -i '/pattern to match/d' ./infile

To directly modify the file (and create a backup):

sed -i.bak '/pattern to match/d' ./infile

For Mac OS X users:

sed -i '' '/pattern/d' ./infile
thanasisp
3#
thanasisp Reply to 2017-11-17 10:28:26Z

there are many other ways to delete lines with specific string besides sed

awk

awk '!/pattern/' file > temp && mv temp file

Ruby (1.9+)

ruby -i.bak -ne 'print if not /test/' file

Perl

perl -ni.bak -e "print unless /pattern/" file

Shell (bash3.2+)

while read -r line
do
  [[ ! $line =~ pattern ]] && echo "$line"
done <file > o 
mv o file

GNU grep

grep -v "pattern" file > temp && mv temp file

and of course sed (printing the inverse is faster than actual deletion. )

sed -n '/pattern/!p' file 
DomainsFeatured
4#
DomainsFeatured Reply to 2015-08-19 15:07:10Z

You can use sed to replace lines in place in a file. However, it seems to be much slower than using grep for the inverse into a second file and then moving the second file over the original.

e.g.

sed -i '/pattern/d' filename      

or

grep -v "pattern" filename > filename2; mv filename2 filename

The first command takes 3 times longer on my machine anyway.

Kjetil S.
5#
Kjetil S. Reply to 2018-01-08 12:07:39Z
perl -i    -nle'/regexp/||print' file1 file2 file3
perl -i.bk -nle'/regexp/||print' file1 file2 file3

The first command edits the file(s) inplace (-i).

The second command does the same thing but keeps a copy or backup of the original file(s) by adding .bk to the file names (.bk can be changed to anything).

codeforester
6#
codeforester Reply to 2017-12-11 23:50:54Z

The easy way to do it, with GNU sed:

sed --in-place '/some string here/d' yourfile
Bhuvanesh
7#
Bhuvanesh Reply to 2015-03-28 07:11:29Z

You can use this also

 grep -v 'pattern' filename

here -v will print only other than your pattern(that means Invert match)

Jahid
8#
Jahid Reply to 2015-06-13 19:24:59Z

To get a inplace like result with grep you can do this:

echo "$(grep -v "pattern" filename)" >filename
Community
9#
Community Reply to 2017-04-13 12:36:30Z

You may consider using ex (which a standard UNIX command-based editor):

ex +g/match/d -cwq file

where:

  • + executes given Ex command (man ex), same as -c which executes wq (write and quit)
  • g/match/d - Ex command to delete lines with given match, see: Power of g

Above example is POSIX-compliant method for in-place editing a file as per this post at Unix.SE and POSIX specifications for ex.


The difference with sed is that:

sed is a Stream EDitor, not a file editor.BashFAQ

unless you enjoy unportable code, I/O overhead and some other bad side effects. So basically some parameters (such as in-place/-i) are non-standard FreeBSD extensions and may not be available on other operating systems.

Community
10#
Community Reply to 2017-05-23 12:10:48Z

I was struggling with this on Mac. Plus, I needed to do it using variable replacement. So I used:

sed -i '' "/$pattern/d" $file

where $file is file where deletion is needed and $pattern is the pattern to be matched for deletion. Picked the '' from this comment. The thing to note here is use of double quotes in "/$pattern/d". Variable won't work when we use single quote.

JakeGould
11#
JakeGould Reply to 2016-09-20 13:28:37Z

SED:

  • '/James\|John/d'
  • -n '/James\|John/!p'

AWK:

  • '!/James|John/'
  • /James|John/ {next;} {print}

GREP:

  • -v 'James\|John'
Shizzmo
12#
Shizzmo Reply to 2016-09-16 23:51:31Z

echo -e "/thing_to_delete\ndd\033:x\n" | vim file_to_edit.txt

DragonRider
13#
DragonRider Reply to 2017-03-02 11:06:35Z

Just in case someone wants to do it for exact matches of strings, you can use -w flag in grep, w for whole. That is, for example if you want to delete the lines that have number 11 but keep the lines with number 111:

-bash-4.1$ head file
1
11
111

-bash-4.1$ grep -v "11" file
1

-bash-4.1$ grep -w -v "11" file
1
111

Also works with -f flag if you want to exclude several exact patterns at once. If "blacklist" is a file with several patterns on each line that you want to delete from "file":

grep -w -v -f blacklist file
Jadzia
14#
Jadzia Reply to 2017-03-19 12:45:18Z

I have made a small benchmark with a file which contains approximately 345 000 lines. The way with grep seems to be around 15 times faster than the sed method in this case.

I have tried both with and without the setting LC_ALL=C, it does not seem change the timings significantly. The search string (CDGA_00004.pdbqt.gz.tar) is somewhere in the middle of the file.

Here are the commands and the timings:

time sed -i "/CDGA_00004.pdbqt.gz.tar/d" /tmp/input.txt

real    0m0.711s
user    0m0.179s
sys     0m0.530s

time perl -ni -e 'print unless /CDGA_00004.pdbqt.gz.tar/' /tmp/input.txt

real    0m0.105s
user    0m0.088s
sys     0m0.016s

time (grep -v CDGA_00004.pdbqt.gz.tar /tmp/input.txt > /tmp/input.tmp; mv /tmp/input.tmp /tmp/input.txt )

real    0m0.046s
user    0m0.014s
sys     0m0.019s
You need to login account before you can post.

About| Privacy statement| Terms of Service| Advertising| Contact us| Help| Sitemap|
Processed in 0.3323 second(s) , Gzip On .

© 2016 Powered by mzan.com design MATCHINFO