Feature Request 'SaveMemory' to Disk Drive
Re: Feature Request 'SaveMemory' to Disk Drive
My test was with 1GB too (PB 6 32)
- NicTheQuick
- Addict
- Posts: 1227
- Joined: Sun Jun 22, 2003 7:43 pm
- Location: Germany, Saarbrücken
- Contact:
Re: Feature Request 'SaveMemory' to Disk Drive
Just write the data in chunks:
Code: Select all
EnableExplicit
; The maximum value is: 2 GiB - 1 Byte = (2 * 1024 * 1024 * 1024 - 1) Bytes = (2^31 - 1) Bytes = 2147483647 Bytes
#MAX_CHUNK_SIZE = 2048 * 1024 * 1024 - 1
Procedure SaveMemory(*Memory, FileName.s, Size = #PB_Ignore)
Protected result.i, file.i
Protected chunkSize.i, writtenBytes.i
file = CreateFile(#PB_Any, FileName)
If Not file
DebuggerError("Could not open file for writing: " + FileName)
ProcedureReturn #False
EndIf
If Size = #PB_Ignore
Size = MemorySize(*Memory)
EndIf
While Size
If Size > #MAX_CHUNK_SIZE
chunkSize = #MAX_CHUNK_SIZE
Else
chunkSize = Size
EndIf
writtenBytes = WriteData(file, *Memory, chunkSize)
Debug writtenBytes
If Not writtenBytes
result = #False
DebuggerError("Could not write " + chunkSize + " bytes into file.")
Break
EndIf
Size - writtenBytes
*Memory + writtenBytes
Wend
CloseFile(file)
ProcedureReturn result
EndProcedure
Define *mem = AllocateMemory(4 * 1024 * 1024 * 1024)
SaveMemory(*mem, "/root/nicolas/tmp/purebasic_memory_test.dat")
FreeMemory(*mem)
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
Re: Feature Request 'SaveMemory' to Disk Drive
Yes, It worked perfectly now with #MAX_CHUNK_SIZE = 2048 * 1024 * 1024 - 1
With 4Gb allocated
Code: Select all
2147483647
2147483647
2
Re: Feature Request 'SaveMemory' to Disk Drive
Candidate to Tricks 'n' Tips forum?
Re: Feature Request 'SaveMemory' to Disk Drive
Yes, try to change the chunks size and monitoring the disk efficiency with this app:
https://www.sysgauge.com
you will see the results changing according to the block size
But with 'SaveMemory' command ...is not more comfortable ?
Re: Feature Request 'SaveMemory' to Disk Drive
However, I now see that WriteData() has a 2GB limit.
It looks like the function works internally with longs (so a 32 bit limit).
That is maybe a bug report for the x64 version of pb ?
It looks like the function works internally with longs (so a 32 bit limit).
That is maybe a bug report for the x64 version of pb ?
- NicTheQuick
- Addict
- Posts: 1227
- Joined: Sun Jun 22, 2003 7:43 pm
- Location: Germany, Saarbrücken
- Contact:
Re: Feature Request 'SaveMemory' to Disk Drive
I don't think it can be changed easily because the underlying functions by itself uses 32 bit parameters. See fwrite which uses `size_t` which in turn usually is `unsigned int` and `int` usually only has 32 bits.
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
Re: Feature Request 'SaveMemory' to Disk Drive
it's a 'bit' limiting/obsolete for today's times ...NicTheQuick wrote: ↑Wed Jun 29, 2022 9:43 am I don't think it can be changed easily because the underlying functions by itself uses 32 bit parameters. See
can be do with the Assembler ???
- NicTheQuick
- Addict
- Posts: 1227
- Joined: Sun Jun 22, 2003 7:43 pm
- Location: Germany, Saarbrücken
- Contact:
Re: Feature Request 'SaveMemory' to Disk Drive
But why? It's not limiting. It's quite unusual to write such big memory chunks into a file at once. Just split it up in small chunks and all is good.
And no, Assembler is not a solution for everything. You must use the functions the kernel offers you to write into files and other things. They can not be replaced with your own user space functions.
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
Re: Feature Request 'SaveMemory' to Disk Drive
NicTheQuick wrote: ↑Wed Jun 29, 2022 11:28 am But why? It's not limiting. It's quite unusual to write such big memory chunks into a file at once. Just split it up in small chunks and all is good.
Thank you
Re: Feature Request 'SaveMemory' to Disk Drive
No, no, not assembler. This will not work with the C compiler (at least for now). Also I don't see a problem to chunk it. The result is the same.Simo_na wrote: ↑Wed Jun 29, 2022 11:16 amit's a 'bit' limiting/obsolete for today's times ...NicTheQuick wrote: ↑Wed Jun 29, 2022 9:43 am I don't think it can be changed easily because the underlying functions by itself uses 32 bit parameters. See
can be do with the Assembler ???
PureBasic 6.04/XProfan X4a/Embarcadero RAD Studio 11/Perl 5.2/Python 3.10
Windows 11/Ryzen 5800X/32GB RAM/Radeon 7770 OC/3TB SSD/11TB HDD
Synology DS1821+/36GB RAM/130TB
Synology DS920+/20GB RAM/54TB
Synology DS916+ii/8GB RAM/12TB
Windows 11/Ryzen 5800X/32GB RAM/Radeon 7770 OC/3TB SSD/11TB HDD
Synology DS1821+/36GB RAM/130TB
Synology DS920+/20GB RAM/54TB
Synology DS916+ii/8GB RAM/12TB
Re: Feature Request 'SaveMemory' to Disk Drive
did some test with chunk dimension...
the best (fastest in write) size is 2097120 for my system.
Thank you
the best (fastest in write) size is 2097120 for my system.
Thank you
Re: Feature Request 'SaveMemory' to Disk Drive
That's suspiciously close to two 2MB and somehow I doubt it, because this way the loop is run 1000 times for 2 GB, instead of once.
PureBasic 6.04/XProfan X4a/Embarcadero RAD Studio 11/Perl 5.2/Python 3.10
Windows 11/Ryzen 5800X/32GB RAM/Radeon 7770 OC/3TB SSD/11TB HDD
Synology DS1821+/36GB RAM/130TB
Synology DS920+/20GB RAM/54TB
Synology DS916+ii/8GB RAM/12TB
Windows 11/Ryzen 5800X/32GB RAM/Radeon 7770 OC/3TB SSD/11TB HDD
Synology DS1821+/36GB RAM/130TB
Synology DS920+/20GB RAM/54TB
Synology DS916+ii/8GB RAM/12TB
- NicTheQuick
- Addict
- Posts: 1227
- Joined: Sun Jun 22, 2003 7:43 pm
- Location: Germany, Saarbrücken
- Contact:
Re: Feature Request 'SaveMemory' to Disk Drive
Keep in mind that your HDD/SSD is way slower than your program code. Also in the background data gets not transferred to disk in the same moment you call WriteData(). Instead it goes into a memory cache and gets flushed to disk when your kernel thinks it is the time to do so.
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
Re: Feature Request 'SaveMemory' to Disk Drive
@Nic: Exactly, that sounds like another argument to make the chunks as big as possible, because the data transfer to the cache is faster.
PureBasic 6.04/XProfan X4a/Embarcadero RAD Studio 11/Perl 5.2/Python 3.10
Windows 11/Ryzen 5800X/32GB RAM/Radeon 7770 OC/3TB SSD/11TB HDD
Synology DS1821+/36GB RAM/130TB
Synology DS920+/20GB RAM/54TB
Synology DS916+ii/8GB RAM/12TB
Windows 11/Ryzen 5800X/32GB RAM/Radeon 7770 OC/3TB SSD/11TB HDD
Synology DS1821+/36GB RAM/130TB
Synology DS920+/20GB RAM/54TB
Synology DS916+ii/8GB RAM/12TB