Readspeed Test v2.10 Test HD/OS and chunksize speeds.

Applications, Games, Tools, User libs and useful stuff coded in PureBasic
User avatar
Rescator
Addict
Addict
Posts: 1769
Joined: Sat Feb 19, 2005 5:05 pm
Location: Norway

Readspeed Test v2.10 Test HD/OS and chunksize speeds.

Post by Rescator »

This "tool" was born from the following thread: http://www.purebasic.fr/english/viewtopic.php?t=37438

Curious which chunksize is fastest?,
and indirectly what FilebufferSize() might make the most sense?
Or want to compare Hard Drive speeds?
Or which OS or Operating Filesystem has best speeds?
Or test USB memory stick speeds or SSD speeds?

Then this tool might be of interest! (Hopefully)

Make sure you compile the tool without the debugger. Or just create an executable and run that (advised).

v2.6 was mostly rewritten and the whole program is now native (PureBasic) code no conditional Window/Linux stuff like with v2.4 etc.
One requirement though is you must use one of the more recent PureBasic released with the improved accuracy on ElapsedMilliseconds().

When you run the test tool it open a file requester will appear, simply select 1 or more files to test. I suggest selecting a assortment of files both very small and large, do not that the larger the file (and the more there is of them) the slower the test will go, especially the random part of it. (Both random seek and a sequential read is performed). When done a messagebox will show the result of the test, after you click the OK button it will also try to launch your browser with a special url for Google Charts API, if all works as intended you should get a nice looking chart representing the info in the message box. The slowest chunksize is used as the base line and will show as 100% in the Google graph, in the message requester it's shown as 1.0x instead.

Please feel free to post your results in this thread.

This way I (and anyone else interested) can simply compare results.

The source is Public Domain, although I request that if you intend to post results in this thread that the way the speed measurement is done etc. is not changed, this is just to avoid the pain of mixed ways to measure things (change the name etc. if you do that).

Please note that the test starts slow (due to the small chunksizes) and will speed up as it starts increasing the chunksizes.
On my system 40 files with a total if 1.92 GB took about 0.317 minutes (19 seconds) as can be seen as the first example below.
On large files it will look like the test has gotten stuck, it has not it just takes a long time to update the progress/percentage as this is updated once per filechange, I did ponder updating the percentage during the filereading itself but that could have affected the timing a bit (updating the console title could have caused a delay/lag in the stats).

It is highly advised you select a larger amount of files of varying sizes, your download directory should do nicely as it'll have different sized files and both old and new files and most likely quite some fragmentation, making for a nice "real" test.

Do note that a lot can influence the results and you may find that testing with the same files again may cause one test to cause a particular chunksize to be very slow or fast, if you see such anomalies you may want to re-run your test until you get consistent results.

Anomalies are unfortunate and do happen but are impossible to optimize against. If you want to know a good size to use then pick the fastest one from multiple tests on multiple drives and different file collections.
Myself I've found my system seems to favor 128KB and 256K with 128KB being the more consistent one.

Also note that you will probably need to use PureBasic's URLEncoder() to make the Google chart url useable on the forum here as the forum has issues with certain urls that are not urlencoded.

Readspeed Test v2.6 on a HDD (SATA2)

Code: Select all

Readspeed Test v2.6
---------------------------
The baseline is the slowest chunksize, and shown as 1.0x
Data based on 40 files.
Amount of total data processed is ~35493.81 MB.
The test took 0.317 minutes to complete.

16384 chunksize speed = 1.00x
32768 chunksize speed = 1.14x
65536 chunksize speed = 1.29x
131072 chunksize speed = 1.38x
262144 chunksize speed = 1.39x
524288 chunksize speed = 1.31x
1048576 chunksize speed = 1.34x
2097152 chunksize speed = 1.31x
4194304 chunksize speed = 1.07x
Readspeed Test v2.6 on a USB stick (USB 2.0 connector/stick)

Code: Select all

Readspeed Test v2.6
---------------------------
The baseline is the slowest chunksize, and shown as 1.0x
Data based on 2 files.
Amount of total data processed is ~22555.12 MB.
The test took 2.200 minutes to complete.

16384 chunksize speed = 1.00x
32768 chunksize speed = 82.24x
65536 chunksize speed = 90.61x
131072 chunksize speed = 98.10x
262144 chunksize speed = 97.62x
524288 chunksize speed = 94.57x
1048576 chunksize speed = 95.17x
2097152 chunksize speed = 92.97x
4194304 chunksize speed = 76.23x
Readspeed Test v2.6 on a USB HDD (USB 2.0 connector/cable/drive)

Code: Select all

Readspeed Test v2.6
---------------------------
The baseline is the slowest chunksize, and shown as 1.0x
Data based on 6 files.
Amount of total data processed is ~35209.25 MB.
The test took 15.517 minutes to complete.

16384 chunksize speed = 1.00x
32768 chunksize speed = 397.83x
65536 chunksize speed = 437.27x
131072 chunksize speed = 461.82x
262144 chunksize speed = 459.72x
524288 chunksize speed = 446.68x
1048576 chunksize speed = 451.31x
2097152 chunksize speed = 442.14x
Readspeed Test v2.6 on another HDD (SATA2)

Code: Select all

Readspeed Test v2.6
---------------------------
The baseline is the slowest chunksize, and shown as 1.0x
Data based on 19 files.
Amount of total data processed is ~7831.46 MB.
The test took 2.117 minutes to complete.

16384 chunksize speed = 1.00x
32768 chunksize speed = 215.75x
65536 chunksize speed = 255.92x
131072 chunksize speed = 283.83x
262144 chunksize speed = 272.45x
524288 chunksize speed = 271.85x
1048576 chunksize speed = 276.14x
2097152 chunksize speed = 267.10x
4194304 chunksize speed = 219.23x
And here's the source, have fun.

Code: Select all

;Test Readspeed v2.10, Public Domain.
EnableExplicit

DisableDebugger
#Program_Title = "Readspeed Test v2.10"

Enumeration 1
	#Window_Main
EndEnumeration

Enumeration 1
	#Gadget_Progress
	#Gadget_Button1
	#Gadget_Button2
EndEnumeration

Global result$, url$

Procedure.i ReadspeedTest()
	Protected *mem
	Protected text$
	Protected size.q, totalsize.q, flen.q, filesizesum.q
	Protected fh.i, index.i
	Protected base.d, sum.d, maxvalue.d, filecount.d, filecountmax.d
	Protected start.l, stop.l, datestart.l, datestop.l, chunksize.l
	Protected NewList speedtest.q()
	Protected Newlist file.s()
	
	chunksize = 4194304
	*mem=AllocateMemory(4194304)
	If *mem=#Null
		ProcedureReturn #False
	EndIf
	
	For index = 1 To 9
		AddElement(speedtest())
		speedtest() = 0
	Next
	
	text$ = OpenFileRequester("Choose file(s) for readspeed test:", "", "All (*.*)|*.*", 0, #PB_Requester_MultiSelection)
	While text$
		size = FileSize(text$)
		If size > 0
			AddElement(file())
			file() = text$
			filesizesum + size
			filecount + 1
		EndIf
		text$ = NextSelectedFileName()
	Wend
	If ListSize(file()) < 1
		If *mem
			FreeMemory(*mem)
		EndIf
		ProcedureReturn #False
	EndIf
	
	filecountmax = filecount * 2 * ListSize(speedtest())
	filecount = 0.0
	chunksize = 16384
	datestart = Date()
	ForEach speedtest()
		ConsoleTitle(#Program_Title + " " + StrD((filecount / filecountmax) * 100.0, 2) + "% Random " + Str(chunksize) + "B chunksize.")
		ForEach file()
			fh = ReadFile(#PB_Any, file())
			If fh
				size = Lof(fh)
				PrintN(GetFilePart(file()) + " " + Str(size) + " bytes.")
				FileBuffersSize(fh, 0)
				flen = size
				totalsize + size
				start = ElapsedMilliseconds()
				Repeat
					If flen > chunksize
						FileSeek(fh, Random(flen - chunksize))
					EndIf
					size - ReadData(fh, *mem, chunksize)
				Until size <= 0
				stop = ElapsedMilliseconds()
				speedtest() + (stop - start)
				CloseFile(fh)
			EndIf
			filecount + 1
		Next
		ConsoleTitle(#Program_Title + " " + StrD((filecount / filecountmax) * 100.0, 2) + "% Sequential " + Str(chunksize) + "B chunksize.")
		ForEach file()
			fh = ReadFile(#PB_Any, file())
			If fh
				FileBuffersSize(fh, 0)
				size = Lof(fh)
				PrintN(GetFilePart(file()) + " " + Str(size) + " bytes.")
				totalsize + size
				start = ElapsedMilliseconds()
				Repeat
					size - ReadData(fh, *mem, chunksize)
				Until size <= 0
				stop = ElapsedMilliseconds()
				speedtest() + (stop - start)
				CloseFile(fh)
			EndIf
			filecount + 1
		Next
		chunksize << 1
	Next
	datestop = Date()
	ConsoleTitle(#Program_Title + "100%")
	
	If *mem
		FreeMemory(*mem)
	EndIf
	
	base = 0
	ForEach speedtest()
		If base < speedtest()
			base = speedtest()
		EndIf
	Next

	text$ = ""
	text$ + "The baseline is the slowest chunksize, and shown as 100.00 %" + #LF$
	size = ListSize(file())
	If size > 1
		text$ + "Based on " + Str(size) + " files and " + StrD(filesizesum / 1048576.0,2) + " MB of data (processed " + StrD(totalsize / 1048576.0,2) + ")." + #LF$
	Else
		text$ + "Based on 1 file" + " and " + StrD(filesizesum / 1048576.0,2) + " MB of data (rocessed " + StrD(totalsize / 1048576.0,2) + ")." + #LF$
	EndIf
	text$ + "Test took " + StrD((datestop - datestart) / 60.0, 3) + " minutes, speed was " + StrD(totalsize / 1048576.0 / (datestop - datestart), 2) + " MB/s (" + StrD(totalsize / 131072.0 / (datestop - datestart), 2) + " mbit/s)." + #LF$

	url$ = "http://chart.apis.google.com/chart?chtt=" + #Program_Title
	url$ + "&chs=388x773&chm=N*f*%,000000,0,-1,11&cht=bvg&chbh=a"
	url$ + "&chco=00007F|7F7F7F|7F0000|007F00|7F7F00|007F7F|7F007F|FF7F7F|7F7FFF"
	url$ + "|7F7F00|007F7F|7F007F|FF7F7F|7F7FFF&chdl="
	chunksize = 16384
	ForEach speedtest()
		index + 1
		If chunksize < 1024
			url$ + Str(chunksize) + "B"
		ElseIf chunksize < 1048576
			url$ + Str(chunksize / 1024) + "KB"
		Else
			url$ + Str(chunksize / 1048576) + "MB"
		EndIf
		If ListIndex(speedtest()) <> (ListSize(speedtest()) - 1)
			url$ + "|"
		EndIf
		chunksize << 1
	Next
	url$ + "&chd=t:"

	chunksize = 16384
	ForEach speedtest()
		sum = speedtest()
		If sum = 0
			sum = 1.0
		EndIf
		sum = base / sum
		If sum > maxvalue
			maxvalue = sum
		EndIf
		text$ + #LF$ + Str(chunksize) + " chunksize speed = " + StrD(sum *100.0, 2) + " %"
		chunksize << 1
 		url$ + Str((sum * 100))
		If ListIndex(speedtest()) <> (ListSize(speedtest()) - 1)
 			url$ + ","
 		EndIf
	Next
	
	result$ = text$
 	url$ + "&chds=99," + Str((maxvalue * 100) + 1)
	
	ProcedureReturn #True
EndProcedure

Define result.i

If OpenConsole() = #False
	;Failed to open window.
	End
EndIf
ConsoleTitle(#Program_Title)

result = ReadspeedTest()

CloseConsole()
If result
MessageRequester(#Program_Title, result$)
RunProgram(URLEncoder(url$))
EndIf

End
Last edited by Rescator on Sat Dec 20, 2014 7:43 am, edited 25 times in total.
User avatar
Rescator
Addict
Addict
Posts: 1769
Joined: Sat Feb 19, 2005 5:05 pm
Location: Norway

Here we go again!

Post by Rescator »

Larger chunksizes use less CPU than the smaller chunksizes, this is due to less OS/filesystem and HD overhead with multiple reads, larger chunksizes let the CPU breathe a little during the reading from the disk. (modern systems and HD's use DMA, Direct Memory Access so the cpu is barely used to read stuff into the memory, it's the read loop code itself that eats most of the CPU)

PureBasic's FileBufferSizes defaults to 4096 bytes, as seen in my results I definitely benefit from setting those higher. (I've used 64KB a lot in my programs) but looking at my test results I might have to consider using 128KB as my new prefered buffer size when reading data, FileBufferSizes might also show similar improvement with ReadByte and ReadLong, and ReadString and it's other variants.

I am interested to see what results the rest of you folks get, as I'm guessing that some systems out there will have wildly different behavior than this.

Vista SP1, x64 OS and x86 compile.

Code: Select all

1048576 chunksize is the baseline, and = 1.0x
Data based on 6 files and 10 loops per chunksize.
Amount of total data processed is ~250643.46 MB.
The test took 5 minutes to complete.

524288 chunksize speed = 1.15x
262144 chunksize speed = 1.19x
131072 chunksize speed = 1.21x
65536 chunksize speed = 1.17x
32768 chunksize speed = 1.11x
16384 chunksize speed = 1.22x
8192 chunksize speed = 1.07x
4096 chunksize speed = 0.81x
2048 chunksize speed = 0.54x
1024 chunksize speed = 0.32x
512 chunksize speed = 0.18x
User avatar
Rescator
Addict
Addict
Posts: 1769
Joined: Sat Feb 19, 2005 5:05 pm
Location: Norway

Post by Rescator »

Vista SP1, x64, x64 compile.

Code: Select all

1048576 chunksize is the baseline, and = 1.0x
Data based on 6 files and 10 loops per chunksize.
Amount of total data processed is ~250643.46 MB.
The test took 5 minutes to complete.

524288 chunksize speed = 1.26x
262144 chunksize speed = 1.32x
131072 chunksize speed = 1.36x
65536 chunksize speed = 1.31x
32768 chunksize speed = 1.28x
16384 chunksize speed = 1.36x
8192 chunksize speed = 1.19x
4096 chunksize speed = 0.91x
2048 chunksize speed = 0.61x
1024 chunksize speed = 0.37x
512 chunksize speed = 0.21x
User avatar
Demivec
Addict
Addict
Posts: 4086
Joined: Mon Jul 25, 2005 3:51 pm
Location: Utah, USA

Post by Demivec »

XP SP3 x86

Three separate tests were performed. The first two had other computer activity occurring at the same time. The last one was run in more isolation.

I did not modify the test code in any way. I thought the baseline was supposed to be 1.0x but the figures don't seem to match up with yours (none are below 1.0x). In case you are wondering why the output header is displayed differently it is because you provided no convenient way to copy the output, I had to type the figures and I summarized the the header data myself from the file's. :(

My results indicate a very close finish between 128K,64K,32K on my system. With these results I think 32K would be sufficient because the performance increase for doubling the size would only be a 1% increase.

Code: Select all

1048576 chunksize is the baseline, and = 1.0x
Data based on 10 loops per chunksize. 
8 files totaling ~2878279680KB (~2.67GB) (average size ~350MB)

524288 chunksize speed = 3.90x
262144 chunksize speed = 5.07x
131072 chunksize speed = 6.49x
65536 chunksize speed = 6.45x
32768 chunksize speed = 6.44x
16384 chunksize speed = 6.24x
8192 chunksize speed = 5.66x
4092 chunksize speed = 4.83x
2048 chunksize speed = 3.75x
1024 chunksize speed = 2.57x
512 chunksize speed = 1.57x


1048576 chunksize is the baseline, and = 1.0x
Data based on 10 loops per chunksize. 
1 file totaling ~354628KB (~350MB)

524288 chunksize speed = 3.61x
262144 chunksize speed = 4.99x
131072 chunksize speed = 6.16x
65536 chunksize speed = 6.14x
32768 chunksize speed = 6.14x
16384 chunksize speed = 5.88x
8192 chunksize speed = 5.41x
4092 chunksize speed = 4.61x
2048 chunksize speed = 3.56x
1024 chunksize speed = 2.43x
512 chunksize speed = 1.49x


1048576 chunksize is the baseline, and = 1.0x
Data based on 10 loops per chunksize. 
1 file totaling ~1084874KB (~1GB)

524288 chunksize speed = 2.97x
262144 chunksize speed = 3.96x
131072 chunksize speed = 4.84x
65536 chunksize speed = 4.92x
32768 chunksize speed = 4.90x
16384 chunksize speed = 4.76x
8192 chunksize speed = 4.39x
4092 chunksize speed = 3.80x
2048 chunksize speed = 2.98x
1024 chunksize speed = 2.07x
512 chunksize speed = 1.31x
User avatar
Rescator
Addict
Addict
Posts: 1769
Joined: Sat Feb 19, 2005 5:05 pm
Location: Norway

Post by Rescator »

You had to manually type it? Ugh!
I did point this out above I think but, I'll teach you a mind boggling thing I learned a few years ago, and cursed myself for not knowing since it's so convenient.

Click your mouse (just in case) on the messagebox text, then press Ctrl+C
now Ctrl+V the text into the forum :P

I'm pretty sure this works on XP too as I've done it there too, should work on W2K as well.
Unfortunately I have no idea about Mac or Linux if you can do the copy text thing there too.

As to the 1.0x thing, by the looks of it 1MB chunksize (or aka buffer) seems to be slower than even 512Bytes, surprising maybe but seeing as 512KB seems to be among the slower it does seem correct.

Most of us doing these tests will see something like a bell curve:
http://en.wikipedia.org/wiki/Normal_distribution
Look at the image in the top right.

Here are some results I did, I selected all the files in my download directory. A mix of different sized files ranging from 21Bytes up to 4+GB files.

Results still look consistent with my previous test results,
but I would not be surprised if some poor bastard with a odd system setup ends up with results completely reverse of ours so far, but that would most likely be just one out of a hundred.

Code: Select all

1048576 chunksize is the baseline, and = 1.0x
Data based on 144 files and 10 loops per chunksize.
Amount of total data processed is ~457489.16 MB.
The test took 10 minutes to complete.

524288 chunksize speed = 1.69x
262144 chunksize speed = 3.10x
131072 chunksize speed = 3.24x
65536 chunksize speed = 3.13x
32768 chunksize speed = 2.97x
16384 chunksize speed = 3.29x
8192 chunksize speed = 2.86x
4096 chunksize speed = 2.15x
2048 chunksize speed = 1.42x
1024 chunksize speed = 0.85x
512 chunksize speed = 0.47x
Now if more people are getting results similar to ours,
it may just turn out that 16KB, 32KB, 64KB and 128KB are the "sweet spot" of chunk/buffersizes for disk reading.

Currently PureBasic uses 4KB, but if we get lots of nice results up here we might convince the PB team to use 16KB for the default filebuffers,
the speed for (ReadData being the exception) the ReadXXXX functions could seem to be twice as fast as currently.

ReadData itself is less affected by the filebuffer size of PureBasic, it's set to 0 (disabled) in these tests, but if turned on the results should be very similar still.

Incidentally, the highest scoring sizes should also be the ones using the least amount of CPU in the tests, a rather nice Win-Win don't you think? :)
cxAlex
User
User
Posts: 88
Joined: Fri Oct 24, 2008 11:29 pm
Location: Austria
Contact:

Post by cxAlex »

Results:

Tested: 6 Files, ~ 4,6 GB Size.

Code: Select all

1048576 chunksize is the baseline, and = 1.0x
Data based on 6 files and 10 loops per chunksize.
The test took 5 minutes to complete.

524288 chunksize speed = 5.59x
262144 chunksize speed = 5.58x
131072 chunksize speed = 5.53x
65536 chunksize speed = 5.45x
32768 chunksize speed = 5.26x
16384 chunksize speed = 4.96x
8192 chunksize speed = 4.55x
4096 chunksize speed = 3.79x
2048 chunksize speed = 2.87x
1024 chunksize speed = 1.91x
512 chunksize speed = 1.16x
User avatar
Rescator
Addict
Addict
Posts: 1769
Joined: Sat Feb 19, 2005 5:05 pm
Location: Norway

Post by Rescator »

Sweet, thanks :)


Oh and folks, I improved the code a little.
Everything is now dynamic, so it's possible to change the maxchunksize (was previously hardcoded with the loops).

Also did some other minor improvements, like the total data processed etc.
The maxchunksize is set to 16MB as default now, so this will probably max out the built in cache of most hardrives on larger files.
The loops is set to 1 as default now, I instead advise to choose a large selection of different files and sizes, as that will be closer to normal use.

Changing loops may still have it's uses though if you are testing only 1 or a few files, and adjusting the maxchunksize is a nice way for you to test possible buffersizes for your own software (image or video or audio or log processing software maybe?)

As a result the bell curve peaks is closer to the middle of the results here now as expected, and since 16MB is so "slow" all values are above 1.0x so 16MB "should" be the slowest for almost everybody unless you got a weird drive.

Here are my results as an example:

Code: Select all

Readspeed Test v2.1
---------------------------
16777216 chunksize is the baseline, and = 1.0x
Data based on 144 files and 1 loops per chunksize.
Amount of total data processed is ~60998.55 MB.
The test took 2 minutes to complete.

16777216 chunksize speed = 1.00x
8388608 chunksize speed = 1.56x
4194304 chunksize speed = 13.33x
2097152 chunksize speed = 15.03x
1048576 chunksize speed = 18.91x
524288 chunksize speed = 20.65x
262144 chunksize speed = 21.30x
131072 chunksize speed = 21.70x
65536 chunksize speed = 21.04x
32768 chunksize speed = 19.89x
16384 chunksize speed = 21.56x
8192 chunksize speed = 18.62x
4096 chunksize speed = 13.86x
2048 chunksize speed = 9.42x
1024 chunksize speed = 5.72x
512 chunksize speed = 3.16x
User avatar
Rescator
Addict
Addict
Posts: 1769
Joined: Sat Feb 19, 2005 5:05 pm
Location: Norway

Post by Rescator »

As it may be a bit hard to see the difference with the numbers above,
here's Google to the rescue (this is so freakin' cool :P )

Unfortunately this forum is unable to show the chart inline so use this link instead: Chart of the above data.

http://code.google.com/apis/chart/ for more info on how to make these charts, cool huh? :wink:

Edit: removed the % chart.
PS! It may just be a characteristic of my HD but it seems that the first few tests are exponentially better but after 16KB performance improvement is no longer linear but bellcurved.

To see your own test data in it's graph glory, simply edit the end of this insanely long url.
The graph dataset is the latter part chd=t:3.16,5.72,9.42,13.86,18.62,21.56,19.89,21.04,21.70,21.30,20.65,18.91,15.03,13.33,1.56,1.0
Starting with the 512B one which is 3.16 n this case and ending with the 16MB one which is the 1.0
You may also need to edit the chds=0.0,23.0 the 23.0 her is the top of the scale so if you end up with higher values then I did then increase that value to your highest or higher so the graph looks correct.
User avatar
Rescator
Addict
Addict
Posts: 1769
Joined: Sat Feb 19, 2005 5:05 pm
Location: Norway

Post by Rescator »

@Demivec Here is your 1st graph.
@Demivec Here is your 2nd graph.
@Demivec Here is your 3rd graph.

@cxAlex Here is your graph.

*looks at the graphs*
Heh! Now we're getting somewhere, cool, and informative.
Fred
Administrator
Administrator
Posts: 16619
Joined: Fri May 17, 2002 4:39 pm
Location: France
Contact:

Post by Fred »

What about random read of very small chunk (let's say 10 bytes, randomly over a 1Gb file) ? In this case, large buffer will be a big drawback, as you fetch much more infos than needed, and flush it right after.
User avatar
Rescator
Addict
Addict
Posts: 1769
Joined: Sat Feb 19, 2005 5:05 pm
Location: Norway

Post by Rescator »

Very good point, and as you made me curious I'll just have to do some tests on that as well. :wink:

In the meantime enjoy this "patch" for the test tool.

Locate this code:

Code: Select all

(see v2.3 in first post)
The test tool should now also open the Google Chart in your browser when the results are shown. :twisted:

*heads back into the IDE to do some random related coding.*

EDIT: Fixed a ending comma that could mess up the graph if included.
Last edited by Rescator on Sun May 24, 2009 10:58 pm, edited 1 time in total.
Fred
Administrator
Administrator
Posts: 16619
Joined: Fri May 17, 2002 4:39 pm
Location: France
Contact:

Post by Fred »

Why don't you edit your first post instead of post a patch ?
User avatar
Rescator
Addict
Addict
Posts: 1769
Joined: Sat Feb 19, 2005 5:05 pm
Location: Norway

Post by Rescator »

Well, because of:

v2.3 is now done. (see first post for new source and info).

:wink:

It was only a temp, which is a good thing as the chart values was in reverse order. v2.3 is correct obviously.

About the random read however, it's coded to read 100 times randomly from throughout file, it will repeat this for each chunksize.

This basically simulates reading 1-512 bytes of data randomly from a file (since the smallest test chunk size is 512bytes),
using 512B to 16MB read buffers. I can assume this satisfies your small random read criteria?
After all this is just like PureBasic's filebuffers does it if it wish to read say 400 bytes. (by reading 4096 bytes but only needing 400 of them).

So. 100 random reads per chunk, and 16 chunk sizes = 1600 reads per file.

If you look at my first post I've put up two new chart urls of my test there from my download directory again, with files as small as 21 bytes up to 4GB+,
some files are recent and some files are a year+ old, and many are probably somewhat fragmented as well.

So the data should be pretty "imperical". Obviously that test data is biased to my system/OS/HD and whatever caching is in effect, which is why I hope to see plenty lovely charts ;)
Fred
Administrator
Administrator
Posts: 16619
Joined: Fri May 17, 2002 4:39 pm
Location: France
Contact:

Post by Fred »

512 bytes is too big ti be realistic. When your read record all around a file, it can be much smaller (16 bytes) ? Indeed, you should disactivate the PB buffering when testing such small read to see how big is their impact.
User avatar
Rescator
Addict
Addict
Posts: 1769
Joined: Sat Feb 19, 2005 5:05 pm
Location: Norway

Post by Rescator »

*points to the line with the FileBuffersSize(#File1,0) * :wink:

Since I did not know whether PureBasic's filebuffer has a read ahead buffer or sliding window I decided to disable it to remove one unknown factor.
If I had used PureBasic's buffers then that would almost be a double buffering effect.

Of course there is still the issue of the filesystem/OS cache, on very small files on the 2nd read one could end up with 0ms on all reads (since the OS filesystem has the entire file in memory) but not much I can do about that.

I'm working on a v2.4 that I hope will be the last for now,
nothing dramatic, just switching files between chunksize changes, as it's possible the smaller chunksizes (for the smaller sized files in tests) they they are cached by the OS on the 2nd read through.

The other change I'm working on is mostly for the Google charts,
as 0ms results are not really counted (0 is nothing to meassure as it's, um 0) so I'm gonna artificially give 0ms results a value that is x times better than the slowest. so if the slowest read test was 500ms then any 0ms results would be displayed as being 500 times faster.
The graphs will mostly look the same, just that the really fast ones will get an even bigger size in the graph than currently.
Post Reply