Get-ChildItem and the–Include and –Filter parameters

I saw a good question the other day in the PowerShell.Com Learn PowerShell Forum which related to using –Include when calling Get-ChildItem (or DIR, or LS!). The OP had a bunch of files in a folder (C:\Data) and wanted to get at just the *.txt files as follows:

Get-ChildItem –Path C:\Data –Include *.Txt

But it did not work – it returned no files at all (even though there were some in the folder. The reason is clear if you read the great help text closely: the Include switch is only active if you are also using the –Recurse parameter! Another small to make is that the –include property specifies a globbed string (I.e. a file name specified with Wild cards) and not a regular expression.

The simplest way to just get the text files form a single folder would be:

Get-ChildItem –Path C:\Data\*.Txt

Another way to get just the Text files in a given folder would be to use the –Filter parameter. The –Filter parameter  is sent to the provider and is used to qualify the –Path value. You can call it like:

Get-ChildItem –Path C:\Data\ –Filter *.Txt

So you have two ways to get a subset of files in a folder using a form of early filtering. And if you use –Recurse (thus are getting all the files in a folder and it’s child folders), you can use either the –Include or –Filter parameters. They give the same result. Well almost. If you use –Include *.txt, you only get the files which have an extension of .txt. Using  -Filter, you get a slightly different result as shown here:

Psh[Cookham8:C:\foo]>Get-ChildItem -path c:\DATA  -filter *.txt -recurse  -ea Silentlycontinue
    Directory: C:\DATA
Mode                LastWriteTime     Length Name                                                        
----                -------------     ------ ----                                      
-a---          2/5/2012   3:48 PM          0 copy.txtfoo                                                        
-a---          2/5/2012   3:48 PM          0 one.txt
-a---          2/5/2012   3:48 PM          0 three.txt
-a---          2/5/2012   3:48 PM          0 two.txt

As you can see, *.txt also copies *.txtfoo. I’m sure this is ‘by design’ but it doesn’t seem to map to my expectations. Still, in most cases, extensions do not overlap like this (of course PowerShell is an exception – with .PS1 and .PS1XML extenstions!).

As an alternative to using either –Filter or –Include, you could always get ALL the child items (e.g. all the file and folder objects in C:\Data), and pipe the output to Where-Object for further (later) filtering. This works, but as most IT admins know, early filtering tends to be more efficient. But the filtering done by –Filter may be surprising – If I use –Filter *.ps1, then PowerShell returns me all the PowerShell scripts in

The –Filter parameter also turns out to have an additional benefit. The value of Filter you specify is used by the provider to qualify the path value. By comparison, the –Include specifies a filter for PowerShell to apply. Thus, the –Filter parameter generates early, early filtering, whereas-Include is later early filtering! The performance difference between the two approaches turns out to be significant! I wrote a little script to calculate the costs of using the three methods, as follows:

# Test-Filtering.PS1
$start = get-date
$f = Get-ChildItem -path c:\windows  -include *ps1 -recurse -ea Silentlycontinue
$end = Get-Date
$time1=$end-$start
"Using -Include    : {0,4} files in {1,-6:n2} seconds" -f $f.count,$time1.totalseconds
$f| ft name,length

$start = get-date
$f=Get-ChildItem -path c:\windows  -filter *ps1 -recurse  -ea Silentlycontinue
$end = Get-Date
$time2=$end-$start
"Using -Filter     : {0,4} files in {1,-6:n2} seconds" -f $f.count,$time2.totalseconds
$f | ft name,length

$start = get-date
$f=Get-ChildItem -path c:\windows  -recurse  -ea Silentlycontinue | where {$_.extension -eq '.ps1'}
$end = Get-Date
$time3=$end-$start
"Using Where clause: {0,4} files in {1,-6:n2} seconds" -f $f.count,$time3.totalseconds

The results are here:

I have to say, the numbers were not what I was expecting! I was surprised how much faster –Filter turned out to be – around 3 times quicker than either –Include, or using late filtering. And compared to using –Include, late filtering is really not all that much slower.

Technorati Tags: Filter,Include,PowerShell,Get-Child Item

 

Performance with PowerShell

Over the weekend, at the most recent PowerShell PowerCamp, I got to discussing the performance of PowerShell. The point I was making was that PowerShell made doing some things very easy, even though they were not performant. Two examples are the early filtering  of WMI data using –Filter (vs using Where-Object after you retrieve all the data from a remote machine) and the two variants of ForEach.

In the case of WMI, where you early filter properties/occurrences on the target machine, PowerShell has less data to serialize and transmit across the network. Also, late filtering requires more local memory, and additional processing. Thus I’d expect early filtering to be faster. We are thus comparing two statements which might look something like this:

Get-WMIObject win32_share -computer  Cookham1 -filter "Description='remote admin'"

versus

Get-WMIObject win32_share -computer  Cookham1  | Where {$_.description -eq 'remote admin'}

In the first example, only one share is returned from Cookham1, whereas in the second example multiple shares are returned and are then filtered locally (aka late filtering). If I wrap both of these commands in a Measure-Command, and do the operation a number of times, the code and results look like this:

 

Psh[Cookham8:fmt:\]>"Early Filter:"
" {0} ms"  -f  ((Measure-command {1..100 | foreach {
Get-WMIObject win32_share -computer  Cookham1 -filter "Description='remote admin'"}}).totalmilliseconds).tostring("f")

"Late filter:"
" {0} ms"  -f  ((Measure-command {1..100 | foreach {
Get-WMIObject win32_share -computer  Cookham1  | Where {$_.description -eq 'remote admin'}}}).totalmilliseconds).tostring("f")
Early Filter:
1948.91 ms
Late filter:
2715.44 ms

So the difference between late filter and early filter is around 28%, although if I run this test a few times, the numbers do vary a bit, but almost always early filtering is in the region of 20% faster.

But a much bigger difference was observed by Anita Boorboom, a Dutch SharePoint guru, in the second case, i.e. using For-Each-object (vs using ForEach in a pipeline).

When you use the foreach operator in a pipeline, PowerShell is able to optimise the creation of objects at one stage of a pipeline and their consumption in the next. Using Foreach-Object, you need to first persist all the objects you wish to iterate across, then perform the iteration. The latter clearly requires a bit more processing and it is likely to require more memory (which can be a bad thing if the collection of objects is large! I knew this, but Anita’s results were a little more than I was expecting, so I duplicated her scripts, well nearly, and found here results were indeed correct, like this:

$items = 1..10000
Write-Host "ForEach-Object: "
" {0} ms"  -f ((Measure-Command { $items | ForEach-Object { "Item: $_" } }).totalmilliseconds).tostring("f") 
Write-Host "Foreach: "
" {0} ms" -f ((Measure-Command {Foreach ($item in $items) { "Item: $item" }}).totalmilliseconds).tostring("f")
ForEach-Object:
  629.73 ms
Foreach:
31.84 ms

Thus the pipelined foreach is nearly 20 times faster for this experiment. I ran this code several times, and the multipler was consistently in the 20-30 times as fast range. That floored me. The For-Each Object does require PowerShell to instantiate every object in memory, then to iterate over it, vs iterating as it instantiates. But I did not expect a 20-30 fold difference in performance!

So it’s obvious that some language constructs will be a little more efficient, You also need to consider the time it takes to write the code, and how often it will be run.  In the first case above, I managed to save just over 750ms by using early WMI filtering. But it probably took me more than that just to write the code for early binding. And for a lot of admins that don’t know WMI very well, filtering using Where-Object is familiar and uses PowerShell Syntac (the –filter clause on Get-WMIObject used WQL which is different). In the second case, the difference was staggering. Of course, when the processing you want to apply to the collection members is non-trivial (i.e. more than a couple of lines of code), you often find the improvement in readability of the resulting script block to be worth considering. By using task oriented variable names, the resulting code is easier to read then when you use $_. And for some production orient5ed scripts, that improvement in readability may be worthwhile.

In summary, there always a lot of different ways to achieve the same result in PowerShell. I advocate using what is easiest for you to remember. At the same time, PowerShell can provide some big performance differences between the approaches – and it pays to know more!

Technorati Tags: PowerShell,Performance