Feature Request 'SaveMemory' to Disk Drive

Got an idea for enhancing PureBasic? New command(s) you'd like to see?
User avatar
Caronte3D
Addict
Addict
Posts: 1027
Joined: Fri Jan 22, 2016 5:33 pm
Location: Some Universe

Re: Feature Request 'SaveMemory' to Disk Drive

Post by Caronte3D »

My test was with 1GB too (PB 6 32)
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: Feature Request 'SaveMemory' to Disk Drive

Post by NicTheQuick »

Just write the data in chunks:

Code: Select all

EnableExplicit

; The maximum value is: 2 GiB - 1 Byte = (2 * 1024 * 1024 * 1024 - 1) Bytes = (2^31 - 1) Bytes = 2147483647 Bytes
#MAX_CHUNK_SIZE = 2048 * 1024 * 1024 - 1
Procedure SaveMemory(*Memory, FileName.s, Size = #PB_Ignore)
	Protected result.i, file.i
	Protected chunkSize.i, writtenBytes.i
	
	file = CreateFile(#PB_Any, FileName)
	If Not file
		DebuggerError("Could not open file for writing: " + FileName)
		ProcedureReturn #False
	EndIf
	
	If Size = #PB_Ignore
		Size = MemorySize(*Memory)
	EndIf
	
	While Size
		If Size > #MAX_CHUNK_SIZE
			chunkSize = #MAX_CHUNK_SIZE
		Else
			chunkSize = Size
		EndIf
			
		writtenBytes = WriteData(file, *Memory, chunkSize)
		Debug writtenBytes
		If Not writtenBytes
			result = #False
			DebuggerError("Could not write " + chunkSize + " bytes into file.")
			Break
		EndIf
		
		Size - writtenBytes
		*Memory + writtenBytes
	Wend
	
	CloseFile(file)
	
	ProcedureReturn result
EndProcedure

Define *mem = AllocateMemory(4 * 1024 * 1024 * 1024)
SaveMemory(*mem, "/root/nicolas/tmp/purebasic_memory_test.dat")
FreeMemory(*mem)
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
User avatar
ChrisR
Addict
Addict
Posts: 1127
Joined: Sun Jan 08, 2017 10:27 pm
Location: France

Re: Feature Request 'SaveMemory' to Disk Drive

Post by ChrisR »

NicTheQuick wrote: Tue Jun 28, 2022 6:21 pm Just write the data in chunks:
Yes, It worked perfectly now with #MAX_CHUNK_SIZE = 2048 * 1024 * 1024 - 1
With 4Gb allocated

Code: Select all

2147483647
2147483647
2
User avatar
Caronte3D
Addict
Addict
Posts: 1027
Joined: Fri Jan 22, 2016 5:33 pm
Location: Some Universe

Re: Feature Request 'SaveMemory' to Disk Drive

Post by Caronte3D »

NicTheQuick wrote: Tue Jun 28, 2022 6:21 pm Just write the data in chunks:
Candidate to Tricks 'n' Tips forum? :D
Simo_na
Enthusiast
Enthusiast
Posts: 177
Joined: Sun Mar 03, 2013 9:01 am

Re: Feature Request 'SaveMemory' to Disk Drive

Post by Simo_na »

NicTheQuick wrote: Tue Jun 28, 2022 6:21 pm Just write the data in chunks:
Yes, try to change the chunks size and monitoring the disk efficiency with this app:
https://www.sysgauge.com
you will see the results changing according to the block size
But with 'SaveMemory' command ...is not more comfortable ? 8)
User avatar
Bisonte
Addict
Addict
Posts: 1226
Joined: Tue Oct 09, 2007 2:15 am

Re: Feature Request 'SaveMemory' to Disk Drive

Post by Bisonte »

However, I now see that WriteData() has a 2GB limit.
It looks like the function works internally with longs (so a 32 bit limit).

That is maybe a bug report for the x64 version of pb ?
PureBasic 6.04 LTS (Windows x86/x64) | Windows10 Pro x64 | Asus TUF X570 Gaming Plus | R9 5900X | 64GB RAM | GeForce RTX 3080 TI iChill X4 | HAF XF Evo | build by vannicom​​
English is not my native language... (I often use DeepL to translate my texts.)
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: Feature Request 'SaveMemory' to Disk Drive

Post by NicTheQuick »

Bisonte wrote: Wed Jun 29, 2022 7:10 am However, I now see that WriteData() has a 2GB limit.
It looks like the function works internally with longs (so a 32 bit limit).

That is maybe a bug report for the x64 version of pb ?
I don't think it can be changed easily because the underlying functions by itself uses 32 bit parameters. See fwrite which uses `size_t` which in turn usually is `unsigned int` and `int` usually only has 32 bits.
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
Simo_na
Enthusiast
Enthusiast
Posts: 177
Joined: Sun Mar 03, 2013 9:01 am

Re: Feature Request 'SaveMemory' to Disk Drive

Post by Simo_na »

NicTheQuick wrote: Wed Jun 29, 2022 9:43 am I don't think it can be changed easily because the underlying functions by itself uses 32 bit parameters. See
it's a 'bit' limiting/obsolete for today's times ... :?

can be do with the Assembler ???
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: Feature Request 'SaveMemory' to Disk Drive

Post by NicTheQuick »

Simo_na wrote: Wed Jun 29, 2022 11:16 am it's a 'bit' limiting/obsolete for today's times ... :?

can be do with the Assembler ???
But why? It's not limiting. It's quite unusual to write such big memory chunks into a file at once. Just split it up in small chunks and all is good.
And no, Assembler is not a solution for everything. You must use the functions the kernel offers you to write into files and other things. They can not be replaced with your own user space functions.
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
Simo_na
Enthusiast
Enthusiast
Posts: 177
Joined: Sun Mar 03, 2013 9:01 am

Re: Feature Request 'SaveMemory' to Disk Drive

Post by Simo_na »

NicTheQuick wrote: Wed Jun 29, 2022 11:28 am But why? It's not limiting. It's quite unusual to write such big memory chunks into a file at once. Just split it up in small chunks and all is good.
:(
:)
Thank you
User avatar
jacdelad
Addict
Addict
Posts: 1431
Joined: Wed Feb 03, 2021 12:46 pm
Location: Planet Riesa
Contact:

Re: Feature Request 'SaveMemory' to Disk Drive

Post by jacdelad »

Simo_na wrote: Wed Jun 29, 2022 11:16 am
NicTheQuick wrote: Wed Jun 29, 2022 9:43 am I don't think it can be changed easily because the underlying functions by itself uses 32 bit parameters. See
it's a 'bit' limiting/obsolete for today's times ... :?

can be do with the Assembler ???
No, no, not assembler. This will not work with the C compiler (at least for now). Also I don't see a problem to chunk it. The result is the same.
PureBasic 6.04/XProfan X4a/Embarcadero RAD Studio 11/Perl 5.2/Python 3.10
Windows 11/Ryzen 5800X/32GB RAM/Radeon 7770 OC/3TB SSD/11TB HDD
Synology DS1821+/36GB RAM/130TB
Synology DS920+/20GB RAM/54TB
Synology DS916+ii/8GB RAM/12TB
Simo_na
Enthusiast
Enthusiast
Posts: 177
Joined: Sun Mar 03, 2013 9:01 am

Re: Feature Request 'SaveMemory' to Disk Drive

Post by Simo_na »

did some test with chunk dimension...

the best (fastest in write) size is 2097120 for my system.

Thank you
User avatar
jacdelad
Addict
Addict
Posts: 1431
Joined: Wed Feb 03, 2021 12:46 pm
Location: Planet Riesa
Contact:

Re: Feature Request 'SaveMemory' to Disk Drive

Post by jacdelad »

That's suspiciously close to two 2MB and somehow I doubt it, because this way the loop is run 1000 times for 2 GB, instead of once.
PureBasic 6.04/XProfan X4a/Embarcadero RAD Studio 11/Perl 5.2/Python 3.10
Windows 11/Ryzen 5800X/32GB RAM/Radeon 7770 OC/3TB SSD/11TB HDD
Synology DS1821+/36GB RAM/130TB
Synology DS920+/20GB RAM/54TB
Synology DS916+ii/8GB RAM/12TB
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: Feature Request 'SaveMemory' to Disk Drive

Post by NicTheQuick »

jacdelad wrote: Wed Jun 29, 2022 4:30 pm That's suspiciously close to two 2MB and somehow I doubt it, because this way the loop is run 1000 times for 2 GB, instead of once.
Keep in mind that your HDD/SSD is way slower than your program code. Also in the background data gets not transferred to disk in the same moment you call WriteData(). Instead it goes into a memory cache and gets flushed to disk when your kernel thinks it is the time to do so.
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
User avatar
jacdelad
Addict
Addict
Posts: 1431
Joined: Wed Feb 03, 2021 12:46 pm
Location: Planet Riesa
Contact:

Re: Feature Request 'SaveMemory' to Disk Drive

Post by jacdelad »

@Nic: Exactly, that sounds like another argument to make the chunks as big as possible, because the data transfer to the cache is faster.
PureBasic 6.04/XProfan X4a/Embarcadero RAD Studio 11/Perl 5.2/Python 3.10
Windows 11/Ryzen 5800X/32GB RAM/Radeon 7770 OC/3TB SSD/11TB HDD
Synology DS1821+/36GB RAM/130TB
Synology DS920+/20GB RAM/54TB
Synology DS916+ii/8GB RAM/12TB
Post Reply