Search file system like "Everything" app?
Search file system like "Everything" app?
There's a really fast PC file searching tool called "Everything" (https://www.voidtools.com) and apparently it finds files by querying the NTFS index. Is that something we can do in PureBasic? I searched these forums but didn't find any code or examples.
Currently, to search files, I have to use ExamineDirectory() and then While/Wend to get all files and test for matches as I go, which is extremely slow. I'd love to be able to speed up the search like "Everything" can (searches 100,000 files in 1 second!).
Currently, to search files, I have to use ExamineDirectory() and then While/Wend to get all files and test for matches as I go, which is extremely slow. I'd love to be able to speed up the search like "Everything" can (searches 100,000 files in 1 second!).
Re: Search file system like "Everything" app?
You probably want to parse the windows.edb file.
I have written even more lowlevel system tools in the past and can definately confirm that you could write them in purebasic too
Re: Search file system like "Everything" app?
"Everything" has a SDK and C examples, so it can indeed be convertd to Purebasic.
Re: Search file system like "Everything" app?
Thanks for the replies.
@acreis: I don't understand enough about C to use the SDK and convert "Everything"'s C examples, so that's out.
@AZJIO: I don't see how that example with "fsutil" helps? It's just getting basic drive info, and not parsing the file tree of a given folder?
@Bitblazer: I can't open or access the "Windows.ebd" file at all, even as admin (see screenshots below). But I doubt "Windows.ebd" would contain all the file info (name, date created, size, etc) of all attached drives anyway, right? Like if I stick a 32 GB USB stick of 10,000 files into my PC, it's not going to know the contents of it, because it's never parsed it before. So I don't think this is a usable answer.
"HxD" when run as admin:
Windows Explorer when run as admin:
@acreis: I don't understand enough about C to use the SDK and convert "Everything"'s C examples, so that's out.
@AZJIO: I don't see how that example with "fsutil" helps? It's just getting basic drive info, and not parsing the file tree of a given folder?
@Bitblazer: I can't open or access the "Windows.ebd" file at all, even as admin (see screenshots below). But I doubt "Windows.ebd" would contain all the file info (name, date created, size, etc) of all attached drives anyway, right? Like if I stick a 32 GB USB stick of 10,000 files into my PC, it's not going to know the contents of it, because it's never parsed it before. So I don't think this is a usable answer.
"HxD" when run as admin:
Windows Explorer when run as admin:
Re: Search file system like "Everything" app?
I gave links there in which direction to look.
The matter is that "Everything" receives files and the sizes from MFT. There's even a command line to get the size of a folder. But it doesn't access the files, it reads the sizes from the MFT. The only drawback is that it does not indicate the size of the file, but the space it occupies, that is, taking into account the unfilled cluster.
Win8.1 and Win10.
Code: Select all
dfp /elapsed /b /top 25 /study {largest} C:\ProgramData
Re: Search file system like "Everything" app?
Actually, I just realized "Everything" doesn't let you search by file size or date, so it's not really suitable for my needs after all. Sorry!
Re: Search file system like "Everything" app?
Help may help:
attrib:<Attribute> Suche nach Dateien und Ordnern mit den angegebenen Attributen.
attributes:<Attribute> Suche nach Dateien und Ordnern mit den angegebenen Attributen.
child:<Dateiname> Suche nach Ordnern welche einen untergeordneten Ordner mit angegebenem Dateinamen enthalten.
dateaccessed:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Zugriffsdatum.
datecreated:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Erstellungsdatum.
datemodified:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Änderungsdatum.
daterun:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Ausführungsdatum.
da:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Zugriffsdatum.
dc:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Erstellungsdatum.
dm:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Änderungsdatum.
dr:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Ausführungsdatum.
dupe: Suche nach doppelten Dateinamen.
empty: Suche nach leeren Ordnern.
ext:<Liste> Suche nach Dateien mit passenden (durch Semikolon getrennten) Erweiterungen in der angegebenen Liste.
fsi:<Index> Suche nach Dateien und Ordnern im angegebenen Null-basierten internen Dateisystem-Index.
len:<Länge> Suche nach Dateien und Ordnern mit der angegebenen Dateinamen-Länge.
parents:<Anzahl> Suche nach Dateien und Ordnern mit der angegebenen Anzahl von übergeordneten Ordnern.
rc:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Änderungsdatum.
recentchange:<Datum> Suche nach Dateien und Ordnern mit dem angegebenen Änderungsdatum.
root: Suche nach Dateien und Ordnern ohne übergeordnete Ordner.
runcount:<Anzahl> Suche nach Dateien und Ordnern mit der angegebenen Ausführungs-Anzahl.
size:<Größe> Suche nach Dateien und Ordnern mit der angegebenen Größe in Bytes.
type:<Typ> Suche nach Dateien und Ordnern mit dem angegebenen Typ.
Re: Search file system like "Everything" app?
There are at least two levels in windows above the "admin". This one is pretty public, the other one is a bit tricky and less public.
I knew it would not be a simple 3 line solution due to the locking and update problem, but i am sure you can solve both with google and the windows API and maybe a little trick. For a starting point, check this page that i got from a quick google search.BarryG wrote: ↑Mon May 02, 2022 2:30 am@Bitblazer: I can't open or access the "Windows.ebd" file at all, even as admin (see screenshots below). But I doubt "Windows.ebd" would contain all the file info (name, date created, size, etc) of all attached drives anyway, right? Like if I stick a 32 GB USB stick of 10,000 files into my PC, it's not going to know the contents of it, because it's never parsed it before. So I don't think this is a usable answer.
This sounds pretty interesting too. Or this.
Last edited by Bitblazer on Mon May 02, 2022 8:45 am, edited 1 time in total.
Re: Search file system like "Everything" app?
If you don't know how to use API C, you can use the command line version (es.exe)Actually, I just realized "Everything" doesn't let you search by file size or date, so it's not really suitable for my needs after all. Sorry!
- Install Everything (as service)
- Install ES.EXE (http://www.voidtools.com/support/everyt ... interface/)
Samples (date format by country, so here in Fr (DD/MM/YYYY)
Code: Select all
Files *.pb modified from april 25 to may 1
C:\> es *.pb dm:25/04/2022..1/05/2022
Files *.pb size between 5 to 20 kb
C:\> es *.pb size:5kb..20kb
However, it does not index the content of known files (e.g. *.doc, *.xls) so it is also faster and less cumbersome (and uses less SSD). So you can disable the MS search if you don't use the internal indexing.
More and more programs can also use it to speed up the search functions (example: Total Commander)
If you don't know how to form the command line, just run the advanced search in the GUI and copy the generated line.
Re: Search file system like "Everything" app?
Hi Marc56us! I didn't even notice the Advanced search had a scrollbar with further options down lower. Thanks; I'll explore the method you said.
Re: Search file system like "Everything" app?
Everything can do that.
For the technical implementation check swiftsearch or ntfssearch.
Yes you could do the same in purebasic. Parsing "file entry 0 - $MFT" is the key. It is a NTFS metafile.
Windows Internals by Mark Russinovich is the must read book for many topics like that. Or Inside the Windows NT File System.
Re: Search file system like "Everything" app?
Okay, I missed that - I only looked in the standard search at first. Thanks!
Re: Search file system like "Everything" app?
Just stumbled on this old thread... in case anyone else looks for it; you can do it using the SDK easily.
1) You need to have everything running. (service is best)
2) you also have to download the SDK
2) copy the DLL to your exe's folder
Here's a simple example (more in-depth below)
1) You need to have everything running. (service is best)
2) you also have to download the SDK
2) copy the DLL to your exe's folder
Here's a simple example (more in-depth below)
Code: Select all
If OpenLibrary(0,"everything64.dll")
CallFunction(0,"Everything_SetRequestFlags",$1|$2|$10)
CallFunction(0,"Everything_SortResultsByPath")
;CallFunction(0, "Everything_SetSearchW",@"*.exe")
If CallFunction(0,"Everything_QueryW", 1)
t = CallFunction(0,"Everything_GetNumResults")
Define path.s, file.s, size.large_integer
t-1
For i = 0 To t
path = PeekS(CallFunction(0,"Everything_GetResultPathW",i))
file = PeekS(CallFunction(0,"Everything_GetResultFileNameW",i))
CallFunction(0,"Everything_GetResultSize",i,@size)
Debug path+"\"+file+" ["+str(size\QuadPart)+"]"
Next
Debug "Total files: "+Str(t+2)
Else
Debug "EV Error# "+Str( CallFunction(0,"Everything_GetLastError"))
EndIf
CallFunction(0,"Everything_CleanUp")
Else
Debug "no dll"
EndIf
Code: Select all
EnableExplicit
EnumerationBinary ;- Search Flags
#EVERYTHING_REQUEST_FILE_NAME
#EVERYTHING_REQUEST_PATH
#EVERYTHING_REQUEST_FULL_PATH_AND_FILE_NAME
#EVERYTHING_REQUEST_EXTENSION
#EVERYTHING_REQUEST_SIZE
#EVERYTHING_REQUEST_DATE_CREATED
#EVERYTHING_REQUEST_DATE_MODIFIED
#EVERYTHING_REQUEST_DATE_ACCESSED
#EVERYTHING_REQUEST_ATTRIBUTES
#EVERYTHING_REQUEST_FILE_LIST_FILE_NAME
#EVERYTHING_REQUEST_RUN_COUNT
#EVERYTHING_REQUEST_DATE_RUN
#EVERYTHING_REQUEST_DATE_RECENTLY_CHANGED
#EVERYTHING_REQUEST_HIGHLIGHTED_FILE_NAME
#EVERYTHING_REQUEST_HIGHLIGHTED_PATH
#EVERYTHING_REQUEST_HIGHLIGHTED_FULL_PATH_AND_FILE_NAME
EndEnumeration
Enumeration ;- Errors
#EVERYTHING_OK
#EVERYTHING_ERROR_MEMORY
#EVERYTHING_ERROR_IPC ; Everything search client is Not running
#EVERYTHING_ERROR_REGISTERCLASSEX ; unable To register window class.
#EVERYTHING_ERROR_CREATEWINDOW ; unable To create listening window
#EVERYTHING_ERROR_CREATETHREAD ; unable To create listening thread
#EVERYTHING_ERROR_INVALIDINDEX
#EVERYTHING_ERROR_INVALIDCALL
#EVERYTHING_ERROR_INVALIDREQUEST ; invalid request Data, request Data first.
#EVERYTHING_ERROR_INVALIDPARAMETER
EndEnumeration
Enumeration 1 ;- Sort options
#EVERYTHING_SORT_NAME_ASCENDING
#EVERYTHING_SORT_NAME_DESCENDING
#EVERYTHING_SORT_PATH_ASCENDING
#EVERYTHING_SORT_PATH_DESCENDING
#EVERYTHING_SORT_SIZE_ASCENDING
#EVERYTHING_SORT_SIZE_DESCENDING
#EVERYTHING_SORT_EXTENSION_ASCENDING
#EVERYTHING_SORT_EXTENSION_DESCENDING
#EVERYTHING_SORT_TYPE_NAME_ASCENDING
#EVERYTHING_SORT_TYPE_NAME_DESCENDING
#EVERYTHING_SORT_DATE_CREATED_ASCENDING
#EVERYTHING_SORT_DATE_CREATED_DESCENDING
#EVERYTHING_SORT_DATE_MODIFIED_ASCENDING
#EVERYTHING_SORT_DATE_MODIFIED_DESCENDING
#EVERYTHING_SORT_ATTRIBUTES_ASCENDING
#EVERYTHING_SORT_ATTRIBUTES_DESCENDING
#EVERYTHING_SORT_FILE_LIST_FILENAME_ASCENDING
#EVERYTHING_SORT_FILE_LIST_FILENAME_DESCENDING
#EVERYTHING_SORT_RUN_COUNT_ASCENDING
#EVERYTHING_SORT_RUN_COUNT_DESCENDING
#EVERYTHING_SORT_DATE_RECENTLY_CHANGED_ASCENDING
#EVERYTHING_SORT_DATE_RECENTLY_CHANGED_DESCENDING
#EVERYTHING_SORT_DATE_ACCESSED_ASCENDING
#EVERYTHING_SORT_DATE_ACCESSED_DESCENDING
#EVERYTHING_SORT_DATE_RUN_ASCENDING
#EVERYTHING_SORT_DATE_RUN_DESCENDING
EndEnumeration
Procedure.s EverythingErrorString( evErrorNum )
Protected evErrorString.s
Select evErrorNum
Case #EVERYTHING_OK
evErrorString = "OK"
Case #EVERYTHING_ERROR_MEMORY
evErrorString = "Out of memory"
Case #EVERYTHING_ERROR_IPC ; Everything search client is Not running
evErrorString = "IPC Server not running"
Case #EVERYTHING_ERROR_REGISTERCLASSEX ; unable To register window class.
evErrorString = "Could not register window class"
Case #EVERYTHING_ERROR_CREATEWINDOW ; unable To create listening window
evErrorString = "Failed to create window"
Case #EVERYTHING_ERROR_CREATETHREAD ; unable To create listening thread
evErrorString = "Failed to create listen thread"
Case #EVERYTHING_ERROR_INVALIDINDEX
evErrorString = "Invalid file index"
Case #EVERYTHING_ERROR_INVALIDCALL
evErrorString = "Invalid call"
Case #EVERYTHING_ERROR_INVALIDREQUEST ; invalid request Data, request Data first.
evErrorString = "Invalid request"
Case #EVERYTHING_ERROR_INVALIDPARAMETER
evErrorString = "Parameter incorrect"
Default
evErrorString = "<Error #"+Str(evErrorNum)+">"
EndSelect
ProcedureReturn evErrorString
EndProcedure
Procedure FileTimeToDate( *fileTime.FILETIME ) ; Convert FILETIME to pb date value
Protected.SYSTEMTIME UTCtime, sysTime
FileTimeToSystemTime_( *fileTime, @UTCtime )
SystemTimeToTzSpecificLocalTime_( #NUL, @UTCtime, @sysTime )
With sysTime
ProcedureReturn Date( \wYear, \wMonth, \wDay, \wHour, \wMinute, \wSecond )
EndWith
EndProcedure ;()
Import "everything64.lib"
Everything_SetRequestFlags( flags.l )
Everything_SortResultsByPath()
Everything_SetSort( sort.l )
Everything_SetMatchCase( bool.b )
Everything_SetRegex( bool.b )
Everything_SetSearchW( *searchString )
Everything_QueryW( wait.b )
Everything_GetNumResults()
Everything_GetResultPathW( index.i )
Everything_GetResultFileNameW( index.i )
Everything_GetResultSize( index.i, *size.LARGE_INTEGER )
Everything_GetResultDateModified( index.i, *date.FILETIME )
Everything_GetLastError()
Everything_CleanUp()
EndImport
Everything_SetRequestFlags( #EVERYTHING_REQUEST_FILE_NAME|#EVERYTHING_REQUEST_PATH|#EVERYTHING_REQUEST_SIZE|#EVERYTHING_REQUEST_DATE_MODIFIED )
;Everything_SortResultsByPath()
;Everything_SetSort( #EVERYTHING_SORT_PATH_ASCENDING )
;Everything_SetMatchCase( #False ) ; this is Default
;Everything_SetRegex( #True )
Everything_SetSearchW( @"c:\*.log" )
If Everything_QueryW( #True )
Define total = Everything_GetNumResults()
If total > 0
total-1
Define i, size.LARGE_INTEGER, moddate.FILETIME, path.s, file.s
For i = 0 To total
path = PeekS( Everything_GetResultPathW( i ) )
file = PeekS( Everything_GetResultFileNameW( i ) )
Everything_GetResultSize( i, @size )
Everything_GetResultDateModified(i, @moddate )
If size\QuadPart = -1
Debug path+"\"+file+" <DIR> "+FormatDate("%yyyy-%mm-%dd %hh:%ii:%ss",FileTimeToDate(@moddate))
Else
Debug path+"\"+file+" ["+Str(size\QuadPart)+"] "+FormatDate("%yyyy-%mm-%dd %hh:%ii:%ss",FileTimeToDate(@moddate))
EndIf
Next : total+1
Else
Debug "No files discovered"
EndIf
Debug "Total files found "+Str(total)
Else
Debug "EV Error: "+EverythingErrorString( Everything_GetLastError() )
EndIf
Everything_CleanUp()