Posts Tagged ‘find’

Move/Copy many files in bash

Monday, March 17th, 2014

I tried to move a bunch (>30,000) of files starting with the same prefix to a new directory:

mv color-*.png ../some-other-dir/

But I got the error:

-bash: /bin/mv: Argument list too long

Instead, I used:

find . -name "color-*.png" -exec mv {} ../some-other-dir/ \;

Note that the {} will get replaced with the individual file names and the mv command is executed separately for each.


List text files sorted by line count

Sunday, January 5th, 2014

Here’s a bash snippet to list the text files (only text files, no binaries) in the current directory sorted by line count:

find . -depth 1 -type f -exec grep -Il "" {} \; | xargs wc -l | sort

This outputs something like:

    6 ./alexa.h
   35 ./Astrid.h
   70 ./
   74 ./dagmar.h
  133 ./Adrea.cpp
  168 ./max.cxx
  216 ./Sabina.cpp
 2339 ./Thorsten.cpp
39991 ./

Update: For the particular case of wc (and maybe) others its faster to do this:

wc -l `grep -Il "" *` | sort

ls output sorted by “date added”

Sunday, August 11th, 2013

Mac keeps track of a useful bit of metadata: the date a file is added to its parent folder. Finder lets you sort a folders contents by “date added”. I wanted to do this with my ls output in the Terminal, too.

find . -depth 1 -exec mdls -name kMDItemFSName -name kMDItemDateAdded "{}" \; | sed 'N;s/\n//' | grep -v '(null)' | awk '{print $3 " " $4 " " substr($0,index($0,$7))}'

which lists files in the current directory prefixed by their date added timestamps:

2013-07-21 17:07:26 DGPDEC.pdf
2013-08-11 10:10:49 erlenmeyer-flask-difficult-inside-outside.pdf
2013-07-21 15:10:05 first-run
2013-03-13 23:10:43 new
2013-03-16 20:48:32 NPR Music Austin 100 ZIP (71 Tracks)
2013-04-25 09:04:26 particle.gif
2013-04-25 09:05:34 particle.html

I saved this as an alias by appending this to my .profile:

alias lsadded='find . -depth 1 -exec mdls -name kMDItemFSName -name kMDItemDateAdded "{}" \; | sed -e "s/^kMDItemFSName    = \"\(.*\)\"/ \1/g" | sed "N;s/\n//" | sed -e "s/(null)/0000-00-00 00:00:00 +0000/g" | '"awk '{print \$3 \" \" \$4 \" \" substr(\$0,index(\$0,\$6))}'"

This is especially useful when combined with sort and sort -r:

lsadded | sort


Find minimum (or maximum, etc.) non-zero entry in a sparse matrix in MATLAB

Wednesday, November 2nd, 2011

Actually this should also work for dense matrices, that is it will give you the smallest entry in the matrix not equal to zero.


Find files from the future and touch them

Friday, July 29th, 2011

Moving files off my old laptop I somehow managed to corrupt the Last Modified dates. Many had their dates set to Epoch, but others were set to May 2021. This was very annoying. When I would sort by Last Modified I would get all these bogus files coming up on time. So, here’s a simple bash script to find all files with last modified date’s greater than the current time and touch them so that their last modified date is rest to the current time.


USAGE="Usage: find_and_touch_future_files [directory]"
if test -z "$1" 
  echo "$USAGE"
  exit 1

if [ ! -d $1 ]; then
  echo "find_and_touch_future_files: $1 is not a directory"
  echo "$USAGE"
  exit 1

# make a temporary file (timestamped with right now)
touch $tmp
# find all files modified after temp file (right now) and touch them 
find $1 -newer $tmp -exec touch {} \;
# clean up temp file
rm $tmp

List of installed applications on mac

Wednesday, February 23rd, 2011

Seems like there are several hacky ways to achieve this. And it depends on what you mean by “installed”. If you just mean, list all application bundles (files ending with .app) then you can achieve it the following command in your terminal:

find / -name "*.app"

That command returns a list where each line is the full path to the application. If you just want the name of the application then use:

find / -name "*.app" | sed -e "s/\(.*\)\/\([^\/]*\).app/\2/g"

If you’re really a stickler and want to see all applications even ones the current user does not have permissions to see then use:

sudo find / -name "*.app" | sed -e "s/\(.*\)\/\([^\/]*\).app/\2/g"

On my computer this takes around ∞ minutes.

But maybe you don’t mean to find every application anywhere on the computer, but say just the ones in the /Applications folder. Then you can slightly change the above command to:

find /Applications -name "*.app" | sed -e "s/\(.*\)\/\([^\/]*\).app/\2/g"

On my computer this takes around 3 minutes.

Notice that this is still taking a considerable amount of time and its returning a bunch of weird applications. These are Utility applications or example applications buried deep in the sub folders of the /Applications folder. You can speed up the previous command by providing a maximum depth for the search:

find /Applications -name -maxdepth 1 "*.app" | sed -e "s/\(.*\)\/\([^\/]*\).app/\2/g"

This only finds .app files directly in the /Applications folder. So it doesn’t find the Microsoft Office or Adobe CS5 apps I have because those are grouped into a folder. But it only takes 0.050 seconds.

I settled on using -maxdepth 2:

find /Applications -name -maxdepth 2 "*.app" | sed -e "s/\(.*\)\/\([^\/]*\).app/\2/g"

This finds all of my meaningful applications and in only 0.060 seconds. Higher maxdepths found a few more apps but they were weirdo utility/example apps and made the command take over a second.

If you need this list as an applescript list just use the following:

set list_of_installed_apps to paragraphs of (do shell script "find /Applications/ -name \"*.app\" -maxdepth 2| sed -e \"s/\\(.*\\)\\/\\([^\\/]*\\).app/\\2/g\"")

Replace string in file names, bash one-liner

Saturday, December 5th, 2009

After running lame on a bunch of wav files I ended up with tons of files named:


Now, obviously I could have avoided this in this case with the proper options and arguments to lame, but it’s all in hindsight.

Here’s the bash line I use to strip out .wav from the middle of all these files in the current directory:

for filename in *.mp3; do newname=`echo $filename | sed 's/\.wav\.mp3$/.mp3/g'`; mv $filename $newname; done

It’s long for one line but I think it’s still understandable and manageable. The result is:


Note: I notice that I’m being overly carefully with my regexes and wildcards in the example above but it should make it more clear how to adapt it to your case.

Find and delete all .DS_Store files recursively

Friday, November 6th, 2009

This is posted all over the web, but sanity’s sake I’ll post it again here. This use of the unix command line program find will locate and remove all the .DS_Store files that Finder populates throughout your directories. The following removes all files by the name .DS_Store in the current directory recursively:

find . "-name" ".DS_Store" -exec rm {} \;

Remove all duplicate songs/mp3s iTunes creates using bash script

Tuesday, September 22nd, 2009

I gave up using iTunes to play music about year ago, but I haven’t found a free alternative to iTunes’s exceptional file management based on mp3 ID3 tags (If you know of one — I mean better than iTunes one — let me know). So occasionally I let iTunes organize my music library. I drop in folders containing new music and let iTunes go at it. The problem is that if I have folders containing an mp3 and an m3u playlist I get duplicates. If I don’t notice this right away the duplicates build up.

Here’s a bash script to delete all true duplicates. The files must be exactly the same and have almost the same name (the difference being the number iTunes appends on a copy: “song.mp3” becomes “song 1.mp3”).

Verbose version:

find "$1" -regex ".* [0-9][0-9]*\.[^.]*" -print0 | while read -d $'\0' copy
  original=`echo "$copy" | sed "s/\(.*\) [0-9][0-9]*\(\.[^.]*\)/\1\2/"`
  # check that  the hypothetical original exists
  if [ -e "$original" ];then
    # check that the original is exactly the same file as the copy
    if diff "$original" "$copy" >/dev/null ;then
      rm "$copy"
      echo "$copy deleted..."
      echo "$copy is different..."
      echo "  $copy not deleted..."
    echo "$original does not exist..."
    echo "  $copy not deleted..."

Quiet Version

find "$1" -regex ".* [0-9][0-9]*\.[^.]*" -print0 | while read -d $'\0' copy
  original=`echo "$copy" | sed "s/\(.*\) [0-9][0-9]*\(\.[^.]*\)/\1\2/"`
  # check that  the hypothetical original exists
  if [ -e "$original" ];then
    # check that the original is exactly the same file as the copy
    if diff "$original" "$copy" >/dev/null ;then
      rm "$copy"

Inverse regex find results

Thursday, July 23rd, 2009

This might seem obvious and there might even be a better way to do it, but to find all files in a directory that don’t match a specified regex just find all files and grep out those that don’t match.

So if you want to find all files in dir/ which don’t match ^.*\.mp3$ issue:

find dir/ -type f | grep -v "^.*\.mp3$"

Notice that the -type f forces find to only return files (no directories) and the -v inverts the results of grep.

One thing to be careful of is that now the regex is applied to the entire paths returned by find. Not just the name of each file.