Search file system like "Everything" app?

BarryG · Post by **BarryG** » Sun May 01, 2022 3:48 am

There's a really fast PC file searching tool called "Everything" (https://www.voidtools.com) and apparently it finds files by querying the NTFS index. Is that something we can do in PureBasic? I searched these forums but didn't find any code or examples.

Currently, to search files, I have to use ExamineDirectory() and then While/Wend to get all files and test for matches as I go, which is extremely slow. I'd love to be able to speed up the search like "Everything" can (searches 100,000 files in 1 second!).

AZJIO · Post by **AZJIO** » Sun May 01, 2022 4:26 am

https://www.purebasic.fr/english/viewto ... le#p573636

Bitblazer · Post by **Bitblazer** » Sun May 01, 2022 10:15 am

BarryG wrote: ↑Sun May 01, 2022 3:48 amCurrently, to search files, I have to use ExamineDirectory() and then While/Wend to get all files and test for matches as I go, which is extremely slow. I'd love to be able to speed up the search like "Everything" can (searches 100,000 files in 1 second!).

You probably want to parse the windows.edb file.

I have written even more lowlevel system tools in the past and can definately confirm that you could write them in purebasic too

acreis · Post by **acreis** » Sun May 01, 2022 4:47 pm

"Everything" has a SDK and C examples, so it can indeed be convertd to Purebasic.

BarryG · Post by **BarryG** » Mon May 02, 2022 2:30 am

Thanks for the replies.

@acreis: I don't understand enough about C to use the SDK and convert "Everything"'s C examples, so that's out.

@AZJIO: I don't see how that example with "fsutil" helps? It's just getting basic drive info, and not parsing the file tree of a given folder?

@Bitblazer: I can't open or access the "Windows.ebd" file at all, even as admin (see screenshots below). But I doubt "Windows.ebd" would contain all the file info (name, date created, size, etc) of all attached drives anyway, right? Like if I stick a 32 GB USB stick of 10,000 files into my PC, it's not going to know the contents of it, because it's never parsed it before. So I don't think this is a usable answer.

"HxD" when run as admin:

Windows Explorer when run as admin:

AZJIO · Post by **AZJIO** » Mon May 02, 2022 6:20 am

BarryG wrote: ↑Mon May 02, 2022 2:30 am I don't see how that example with "fsutil" helps?

I gave links there in which direction to look.
The matter is that "Everything" receives files and the sizes from MFT. There's even a command line to get the size of a folder. But it doesn't access the files, it reads the sizes from the MFT. The only drawback is that it does not indicate the size of the file, but the space it occupies, that is, taking into account the unfilled cluster.

Win8.1 and Win10.

Code: Select all

dfp /elapsed /b /top 25 /study {largest} C:\ProgramData

BarryG · Post by **BarryG** » Mon May 02, 2022 6:38 am

Actually, I just realized "Everything" doesn't let you search by file size or date, so it's not really suitable for my needs after all. Sorry!

Lord · Post by **Lord** » Mon May 02, 2022 6:49 am

Help may help:

attrib:<Attribute> Suche nach Dateien und Ordnern mit den angegebenen Attributen.
attributes:<Attribute> Suche nach Dateien und Ordnern mit den angegebenen Attributen.
child:<Dateiname> Suche nach Ordnern welche einen untergeordneten Ordner mit angegebenem Dateinamen enthalten.
dateaccessed:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Zugriffsdatum.
datecreated:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Erstellungsdatum.
datemodified:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Änderungsdatum.
daterun:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Ausführungsdatum.
da:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Zugriffsdatum.
dc:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Erstellungsdatum.
dm:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Änderungsdatum.
dr:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Ausführungsdatum.
dupe: Suche nach doppelten Dateinamen.
empty: Suche nach leeren Ordnern.
ext:<Liste> Suche nach Dateien mit passenden (durch Semikolon getrennten) Erweiterungen in der angegebenen Liste.
fsi:<Index> Suche nach Dateien und Ordnern im angegebenen Null-basierten internen Dateisystem-Index.
len:<Länge> Suche nach Dateien und Ordnern mit der angegebenen Dateinamen-Länge.
parents:<Anzahl> Suche nach Dateien und Ordnern mit der angegebenen Anzahl von übergeordneten Ordnern.
rc:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Änderungsdatum.
recentchange:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Änderungsdatum.
root: Suche nach Dateien und Ordnern ohne übergeordnete Ordner.
runcount:<Anzahl> Suche nach Dateien und Ordnern mit der angegebenen Ausführungs-Anzahl.
size:<Größe> Suche nach Dateien und Ordnern mit der angegebenen Größe in Bytes.
type:<Typ> Suche nach Dateien und Ordnern mit dem angegebenen Typ.

AZJIO · Post by **AZJIO** » Mon May 02, 2022 6:53 am

Bitblazer · Post by **Bitblazer** » Mon May 02, 2022 8:25 am

BarryG wrote: ↑Mon May 02, 2022 2:30 am... even as admin ...

There are at least two levels in windows above the "admin". This one is pretty public, the other one is a bit tricky and less public.

BarryG wrote: ↑Mon May 02, 2022 2:30 am@Bitblazer: I can't open or access the "Windows.ebd" file at all, even as admin (see screenshots below). But I doubt "Windows.ebd" would contain all the file info (name, date created, size, etc) of all attached drives anyway, right? Like if I stick a 32 GB USB stick of 10,000 files into my PC, it's not going to know the contents of it, because it's never parsed it before. So I don't think this is a usable answer.

I knew it would not be a simple 3 line solution due to the locking and update problem, but i am sure you can solve both with google and the windows API and maybe a little trick. For a starting point, check this page that i got from a quick google search.

This sounds pretty interesting too. Or this.

Marc56us · Post by **Marc56us** » Mon May 02, 2022 9:28 am

Actually, I just realized "Everything" doesn't let you search by file size or date, so it's not really suitable for my needs after all. Sorry!

If you don't know how to use API C, you can use the command line version (es.exe)
- Install Everything (as service)
- Install ES.EXE (http://www.voidtools.com/support/everyt ... interface/)
Samples (date format by country, so here in Fr (DD/MM/YYYY)

Code: Select all

Files *.pb modified from april 25 to may 1 
C:\> es *.pb dm:25/04/2022..1/05/2022

Files *.pb size between 5 to 20 kb
C:\> es *.pb size:5kb..20kb

Everything is much faster and has more options than the MS internal search.
However, it does not index the content of known files (e.g. *.doc, *.xls) so it is also faster and less cumbersome (and uses less SSD). So you can disable the MS search if you don't use the internal indexing.
More and more programs can also use it to speed up the search functions (example: Total Commander)
If you don't know how to form the command line, just run the advanced search in the GUI and copy the generated line.

BarryG · Post by **BarryG** » Mon May 02, 2022 10:31 am

Hi Marc56us! I didn't even notice the Advanced search had a scrollbar with further options down lower. Thanks; I'll explore the method you said.

Bitblazer · Post by **Bitblazer** » Tue May 03, 2022 7:35 am

BarryG wrote: ↑Mon May 02, 2022 6:38 am Actually, I just realized "Everything" doesn't let you search by file size or date, so it's not really suitable for my needs after all. Sorry!

Everything can do that.

For the technical implementation check swiftsearch or ntfssearch.

Yes you could do the same in purebasic. Parsing "file entry 0 - $MFT" is the key. It is a NTFS metafile.

Windows Internals by Mark Russinovich is the must read book for many topics like that. Or Inside the Windows NT File System.

BarryG · Post by **BarryG** » Tue May 03, 2022 9:37 am

Okay, I missed that - I only looked in the standard search at first. Thanks!

jassing · Post by **jassing** » Sun Jun 18, 2023 12:43 am

Just stumbled on this old thread... in case anyone else looks for it; you can do it using the SDK easily.
1) You need to have everything running. (service is best)
2) you also have to download the SDK
2) copy the DLL to your exe's folder

Here's a simple example (more in-depth below)

Code: Select all

If OpenLibrary(0,"everything64.dll")
  CallFunction(0,"Everything_SetRequestFlags",$1|$2|$10)
  CallFunction(0,"Everything_SortResultsByPath")
  ;CallFunction(0, "Everything_SetSearchW",@"*.exe")
  If CallFunction(0,"Everything_QueryW", 1)
    t = CallFunction(0,"Everything_GetNumResults")
    Define path.s, file.s, size.large_integer
    t-1
    For i = 0 To t
      path = PeekS(CallFunction(0,"Everything_GetResultPathW",i))
      file = PeekS(CallFunction(0,"Everything_GetResultFileNameW",i))
      CallFunction(0,"Everything_GetResultSize",i,@size)
      Debug path+"\"+file+" ["+str(size\QuadPart)+"]"
    Next
    Debug "Total files: "+Str(t+2)
  Else
    Debug "EV Error# "+Str( CallFunction(0,"Everything_GetLastError"))
  EndIf  
  CallFunction(0,"Everything_CleanUp")
Else
  Debug "no dll"
EndIf

Code: Select all

EnableExplicit

EnumerationBinary ;- Search Flags
  #EVERYTHING_REQUEST_FILE_NAME                            
  #EVERYTHING_REQUEST_PATH                                 
  #EVERYTHING_REQUEST_FULL_PATH_AND_FILE_NAME              
  #EVERYTHING_REQUEST_EXTENSION                            
  #EVERYTHING_REQUEST_SIZE                                 
  #EVERYTHING_REQUEST_DATE_CREATED                         
  #EVERYTHING_REQUEST_DATE_MODIFIED                        
  #EVERYTHING_REQUEST_DATE_ACCESSED                        
  #EVERYTHING_REQUEST_ATTRIBUTES                           
  #EVERYTHING_REQUEST_FILE_LIST_FILE_NAME                  
  #EVERYTHING_REQUEST_RUN_COUNT                            
  #EVERYTHING_REQUEST_DATE_RUN                             
  #EVERYTHING_REQUEST_DATE_RECENTLY_CHANGED                
  #EVERYTHING_REQUEST_HIGHLIGHTED_FILE_NAME                
  #EVERYTHING_REQUEST_HIGHLIGHTED_PATH                     
  #EVERYTHING_REQUEST_HIGHLIGHTED_FULL_PATH_AND_FILE_NAME  
EndEnumeration
Enumeration       ;- Errors
  #EVERYTHING_OK    
  #EVERYTHING_ERROR_MEMORY    
  #EVERYTHING_ERROR_IPC                ; Everything search client is Not running
  #EVERYTHING_ERROR_REGISTERCLASSEX    ; unable To register window class.
  #EVERYTHING_ERROR_CREATEWINDOW       ; unable To create listening window
  #EVERYTHING_ERROR_CREATETHREAD       ; unable To create listening thread
  #EVERYTHING_ERROR_INVALIDINDEX 
  #EVERYTHING_ERROR_INVALIDCALL  
  #EVERYTHING_ERROR_INVALIDREQUEST     ; invalid request Data, request Data first.
  #EVERYTHING_ERROR_INVALIDPARAMETER    
EndEnumeration
Enumeration 1     ;- Sort options
  #EVERYTHING_SORT_NAME_ASCENDING    
  #EVERYTHING_SORT_NAME_DESCENDING    
  #EVERYTHING_SORT_PATH_ASCENDING    
  #EVERYTHING_SORT_PATH_DESCENDING    
  #EVERYTHING_SORT_SIZE_ASCENDING    
  #EVERYTHING_SORT_SIZE_DESCENDING    
  #EVERYTHING_SORT_EXTENSION_ASCENDING    
  #EVERYTHING_SORT_EXTENSION_DESCENDING   
  #EVERYTHING_SORT_TYPE_NAME_ASCENDING    
  #EVERYTHING_SORT_TYPE_NAME_DESCENDING   
  #EVERYTHING_SORT_DATE_CREATED_ASCENDING 
  #EVERYTHING_SORT_DATE_CREATED_DESCENDING
  #EVERYTHING_SORT_DATE_MODIFIED_ASCENDING
  #EVERYTHING_SORT_DATE_MODIFIED_DESCENDING
  #EVERYTHING_SORT_ATTRIBUTES_ASCENDING   
  #EVERYTHING_SORT_ATTRIBUTES_DESCENDING  
  #EVERYTHING_SORT_FILE_LIST_FILENAME_ASCENDING  
  #EVERYTHING_SORT_FILE_LIST_FILENAME_DESCENDING 
  #EVERYTHING_SORT_RUN_COUNT_ASCENDING    
  #EVERYTHING_SORT_RUN_COUNT_DESCENDING   
  #EVERYTHING_SORT_DATE_RECENTLY_CHANGED_ASCENDING  
  #EVERYTHING_SORT_DATE_RECENTLY_CHANGED_DESCENDING 
  #EVERYTHING_SORT_DATE_ACCESSED_ASCENDING   
  #EVERYTHING_SORT_DATE_ACCESSED_DESCENDING  
  #EVERYTHING_SORT_DATE_RUN_ASCENDING    
  #EVERYTHING_SORT_DATE_RUN_DESCENDING    
EndEnumeration
Procedure.s EverythingErrorString( evErrorNum )
  Protected evErrorString.s 
  Select evErrorNum
    Case #EVERYTHING_OK    
      evErrorString = "OK"
    Case #EVERYTHING_ERROR_MEMORY    
      evErrorString = "Out of memory"
    Case #EVERYTHING_ERROR_IPC                ; Everything search client is Not running
      evErrorString = "IPC Server not running"
    Case #EVERYTHING_ERROR_REGISTERCLASSEX    ; unable To register window class.
      evErrorString = "Could not register window class"
    Case #EVERYTHING_ERROR_CREATEWINDOW       ; unable To create listening window
      evErrorString = "Failed to create window"
    Case #EVERYTHING_ERROR_CREATETHREAD       ; unable To create listening thread
      evErrorString = "Failed to create listen thread"
    Case #EVERYTHING_ERROR_INVALIDINDEX 
      evErrorString = "Invalid file index"
    Case #EVERYTHING_ERROR_INVALIDCALL  
      evErrorString = "Invalid call"
    Case #EVERYTHING_ERROR_INVALIDREQUEST     ; invalid request Data, request Data first.
      evErrorString = "Invalid request"
    Case #EVERYTHING_ERROR_INVALIDPARAMETER
      evErrorString = "Parameter incorrect"
    Default
      evErrorString = "<Error #"+Str(evErrorNum)+">"
  EndSelect
  
  ProcedureReturn evErrorString
EndProcedure
Procedure   FileTimeToDate( *fileTime.FILETIME ) ; Convert FILETIME to pb date value
  Protected.SYSTEMTIME UTCtime, sysTime
  FileTimeToSystemTime_( *fileTime, @UTCtime ) 
  SystemTimeToTzSpecificLocalTime_( #NUL, @UTCtime, @sysTime )  
  With sysTime
    ProcedureReturn Date( \wYear, \wMonth, \wDay, \wHour, \wMinute, \wSecond )  
  EndWith 
EndProcedure ;() 

Import "everything64.lib"
  Everything_SetRequestFlags( flags.l )
  Everything_SortResultsByPath()
  Everything_SetSort( sort.l )
  Everything_SetMatchCase( bool.b )
  Everything_SetRegex( bool.b )
  Everything_SetSearchW( *searchString )
  Everything_QueryW( wait.b )
  Everything_GetNumResults()
  Everything_GetResultPathW( index.i )
  Everything_GetResultFileNameW( index.i )
  Everything_GetResultSize( index.i, *size.LARGE_INTEGER )
  Everything_GetResultDateModified( index.i, *date.FILETIME )
  Everything_GetLastError()
  Everything_CleanUp()
EndImport

Everything_SetRequestFlags( #EVERYTHING_REQUEST_FILE_NAME|#EVERYTHING_REQUEST_PATH|#EVERYTHING_REQUEST_SIZE|#EVERYTHING_REQUEST_DATE_MODIFIED )
;Everything_SortResultsByPath()
;Everything_SetSort( #EVERYTHING_SORT_PATH_ASCENDING )
;Everything_SetMatchCase( #False ) ; this is Default
;Everything_SetRegex( #True )
Everything_SetSearchW( @"c:\*.log" ) 
If Everything_QueryW( #True )
  Define total = Everything_GetNumResults() 
  If total > 0 
    total-1
    Define i, size.LARGE_INTEGER, moddate.FILETIME, path.s, file.s
    For i = 0 To total
      path = PeekS( Everything_GetResultPathW( i ) )
      file = PeekS( Everything_GetResultFileNameW( i ) )
      Everything_GetResultSize( i, @size )
      Everything_GetResultDateModified(i, @moddate )
      If size\QuadPart = -1
        Debug path+"\"+file+" <DIR> "+FormatDate("%yyyy-%mm-%dd %hh:%ii:%ss",FileTimeToDate(@moddate))
      Else 
        Debug path+"\"+file+" ["+Str(size\QuadPart)+"] "+FormatDate("%yyyy-%mm-%dd %hh:%ii:%ss",FileTimeToDate(@moddate))
      EndIf 
    Next : total+1
  Else
    Debug "No files discovered"
  EndIf 
  Debug "Total files found "+Str(total)
Else
  Debug "EV Error: "+EverythingErrorString( Everything_GetLastError() )
EndIf  
Everything_CleanUp()

PureBasic Forums - English

Search file system like "Everything" app?

Search file system like "Everything" app?

Re: Search file system like "Everything" app?

Re: Search file system like "Everything" app?

Re: Search file system like "Everything" app?

Re: Search file system like "Everything" app?

Re: Search file system like "Everything" app?

Re: Search file system like "Everything" app?

Re: Search file system like "Everything" app?

Re: Search file system like "Everything" app?

Re: Search file system like "Everything" app?

Re: Search file system like "Everything" app?

Re: Search file system like "Everything" app?

Re: Search file system like "Everything" app?

Re: Search file system like "Everything" app?

Re: Search file system like "Everything" app?