Home Looping through the content of a file in Bash

Looping through the content of a file in Bash

Peter Mortensen
1#
Peter Mortensen Published in 2009-10-05 17:52:54Z
 How do I iterate through each line of a text file with Bash? With this script: echo "Start!" for p in (peptides.txt) do echo "${p}" done  I get this output on the screen: Start! ./runPep.sh: line 3: syntax error near unexpected token (' ./runPep.sh: line 3: for p in (peptides.txt)'  (Later I want to do something more complicated with$p than just output to the screen.) The environment variable SHELL is (from env): SHELL=/bin/bash  /bin/bash --version output: GNU bash, version 3.1.17(1)-release (x86_64-suse-linux-gnu) Copyright (C) 2005 Free Software Foundation, Inc.  cat /proc/version output: Linux version 2.6.18.2-34-default (geeko@buildhost) (gcc version 4.1.2 20061115 (prerelease) (SUSE Linux)) #1 SMP Mon Nov 27 11:46:27 UTC 2006  The file peptides.txt contains: RKEKNVQ IPKKLLQK QYFHQLEKMNVK IPKKLLQK GDLSTALEVAIDCYEK QYFHQLEKMNVKIPENIYR RKEKNVQ VLAKHGKLQDAIN ILGFMK LEDVALQILL 
rogerdpack
2#
rogerdpack Reply to 2017-05-03 17:39:21Z
 One way to do it is: while read p; do echo $p done Warren Young 3# Warren Young Reply to 2009-10-05 17:54:38Z  cat peptides.txt | while read line do # do something with$line here done 
mtk
4#
mtk Reply to 2014-10-07 09:13:27Z
 Option 1a: While loop: Single line at a time: Input redirection #!/bin/bash filename='peptides.txt' echo Start while read p; do echo $p done <$filename  Option 1b: While loop: Single line at a time: Open the file, read from a file descriptor (in this case file descriptor #4). #!/bin/bash filename='peptides.txt' exec 4<$filename echo Start while read -u4 p ; do echo$p done  Option 2: For loop: Read file into single variable and parse. This syntax will parse "lines" based on any white space between the tokens. This still works because the given input file lines are single work tokens. If there were more than one token per line, then this method would not work as well. Also, reading the full file into a single variable is not a good strategy for large files. #!/bin/bash filename='peptides.txt' filelines=cat $filename echo Start for line in$filelines ; do echo $line done  mightypile 5# mightypile Reply to 2013-12-22 15:47:48Z  This is no better than other answers, but is one more way to get the job done in a file without spaces (see comments). I find that I often need one-liners to dig through lists in text files without the extra step of using separate script files. for word in$(cat peptides.txt); do echo $word; done  This format allows me to put it all in one command-line. Change the "echo$word" portion to whatever you want and you can issue multiple commands separated by semicolons. The following example uses the file's contents as arguments into two other scripts you may have written. for word in $(cat peptides.txt); do cmd_a.sh$word; cmd_b.py $word; done  Or if you intend to use this like a stream editor (learn sed) you can dump the output to another file as follows. for word in$(cat peptides.txt); do cmd_a.sh $word; cmd_b.py$word; done > outfile.txt  I've used these as written above because I have used text files where I've created them with one word per line. (See comments) If you have spaces that you don't want splitting your words/lines, it gets a little uglier, but the same command still works as follows: OLDIFS=$IFS; IFS=$'\n'; for line in $(cat peptides.txt); do cmd_a.sh$line; cmd_b.py $line; done > outfile.txt; IFS=$OLDIFS  This just tells the shell to split on newlines only, not spaces, then returns the environment back to what it was previously. At this point, you may want to consider putting it all into a shell script rather than squeezing it all into a single line, though. Best of luck!
0zkr PM
6#
0zkr PM Reply to 2014-03-24 17:57:03Z
 #!/bin/bash # # Change the file name from "test" to desired input file # (The comments in bash are prefixed with #'s) for x in $(cat test.txt) do echo$x done 
Jahid
7#
Jahid Reply to 2017-03-29 00:10:24Z
 Use a while loop, like this: while IFS= read -r line; do echo "$line" done Whome 8# Whome Reply to 2015-06-30 08:15:45Z  Here is my real life example how to loop lines of another program output, check for substrings, drop double quotes from variable, use that variable outside of the loop. I guess quite many is asking these questions sooner or later. ##Parse FPS from first video stream, drop quotes from fps variable ## streams.stream.0.codec_type="video" ## streams.stream.0.r_frame_rate="24000/1001" ## streams.stream.0.avg_frame_rate="24000/1001" FPS=unknown while read -r line; do if [[$FPS == "unknown" ]] && [[ $line == *".codec_type=\"video\""* ]]; then echo ParseFPS$line FPS=parse fi if [[ $FPS == "parse" ]] && [[$line == *".r_frame_rate="* ]]; then echo ParseFPS $line FPS=${line##*=} FPS="${FPS%\"}" FPS="${FPS#\"}" fi done <<< "$(ffprobe -v quiet -print_format flat -show_format -show_streams -i "$input")" if [ "$FPS" == "unknown" ] || [ "$FPS" == "parse" ]; then echo ParseFPS Unknown frame rate fi echo Found $FPS  Declare variable outside of the loop, set value and use it outside of loop requires done <<< "$(...)" syntax. Application need to be run within a context of current console. Quotes around the command keeps newlines of output stream. Loop match for substrings then reads name=value pair, splits right-side part of last = character, drops first quote, drops last quote, we have a clean value to be used elsewhere.
Alan Jebakumar
9#
Alan Jebakumar Reply to 2015-08-30 05:00:05Z
 @Peter: This could work out for you- echo "Start!";for p in $(cat ./pep); do echo$p done  This would return the output- Start! RKEKNVQ IPKKLLQK QYFHQLEKMNVK IPKKLLQK GDLSTALEVAIDCYEK QYFHQLEKMNVKIPENIYR RKEKNVQ VLAKHGKLQDAIN ILGFMK LEDVALQILL 
dawg
10#
dawg Reply to 2017-05-02 17:17:53Z
 Suppose you have this file: $cat /tmp/test.txt Line 1 Line 2 has leading space Line 3 followed by blank line Line 5 (follows a blank line) and has trailing space Line 6 has no ending CR  There are four elements that will alter the meaning of the file output read by many Bash solutions: The blank line 4; Leading or trailing spaces on two lines; Maintaining the meaning of individual lines (i.e., each line is a record); The line 6 not terminated with a CR. If you want the text file line by line including blank lines and terminating lines without CR, you must use a while loop and you must have an alternate test for the final line. Here are the methods that may change the file (in comparison to what cat returns): 1) Lose the last line and leading and trailing spaces: $ while read -r p; do printf "%s\n" "'$p'"; done Anjul Sharma 11# Anjul Sharma Reply to 2016-03-08 16:10:51Z  If you don't want your read to be broken by newline character, use - #!/bin/bash while IFS='' read -r line || [[ -n "$line" ]]; do echo "$line" done < "$1"  Then run the script with file name as parameter.
codeforester
12#
codeforester Reply to 2018-02-06 18:43:41Z

A few more things not covered by other answers:

Reading from a delimited file

# ':' is the delimiter here, and there are three fields on each line in the file
# IFS set below is restricted to the context of read, it doesn't affect any other code
while IFS=: read -r field1 field2 field3; do
# process the fields
# if the line has less than three fields, the missing fields will be set to an empty string
# if the line has more than three fields, field3 will get all the values, including the third field plus the delimiter(s)
done < input.txt


Reading from a null delimited input, for example find ... -print0

while read -r -d '' line; do
# logic
# use a second 'read ... <<< "$line"' if we need to tokenize the line done < <(find /path/to/dir -print0)  Reading from more than one file at a time while read -u 3 -r line1 && read -u 4 -r line2; do # process the lines # note that the loop will end when we reach EOF on either of the files, because of the && done 3< input1.txt 4< input2.txt  Reading a whole file into an array (Bash versions earlier to 4) while IFS= read -r line; do my_array+=("$line")
done < my_file


Reading a whole file into an array (Bash versions 4x and later)

readarray -t my_array < my_file


or

mapfile -t my_array < my_file


And then

for line in "\${my_array[@]}"; do
# process the lines
done

• More about the shell builtins read and readarray commands - GNU

• More about IFS - Wikipedia

• BashFAQ/001 - How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?

Related posts:

• Creating an array from a text file in Bash
 You need to login account before you can post.
Processed in 0.351533 second(s) , Gzip On .