Fast string

Got an idea for enhancing PureBasic? New command(s) you'd like to see?
User avatar
pdwyer
Addict
Addict
Posts: 2813
Joined: Tue May 08, 2007 1:27 pm
Location: Chiba, Japan

Re: Fast string

Post by pdwyer »

In the past I often worked around this with loops inside loops with a mod(loop,100) or something to build smaller strings first which was reasonably effective but recently, after a little thought I tried something else.

In many cases, these sorts of long string concatenation loops are for preparing data for disk (not always of course). Many times I've hit this is a loop to prepare lines of text or csv for a data dump, export or log dump. In fact I had this come up just the other day and was seeing significant slowdown as the string length went over 100's of kb and was just about to add the mod() trick and thought, "why not just stream straight to disk?" and then had that auto-thought (because disk is slower than memory!).

So I tried it, Openfile(), loop the strings being built to hold a single line of CSV, WriteString(), end loop, CloseFile().

All delays went away and it scales a lot larger.

I would guess that if your target was a network destination and streaming data the situation would be similar.

Perhaps if you are building huge strings you should ask what your ultimate goal of the built string is as there may be a way to stream it. If strings of data get really huge then streaming will be a better plan in most cases anyway.

I'm sure there will always be cases where fast string cat of large strings is needed and I'm sure people here are hitting them, but perhaps there are less than I first imagined.
Paul Dwyer

“In nature, it’s not the strongest nor the most intelligent who survives. It’s the most adaptable to change” - Charles Darwin
“If you can't explain it to a six-year old you really don't understand it yourself.” - Albert Einstein
User avatar
HeX0R
Addict
Addict
Posts: 980
Joined: Mon Sep 20, 2004 7:12 am
Location: Hell

Re: Fast string

Post by HeX0R »

No one thought about CopyMemoryString()?

Code: Select all

DisableDebugger


Str.s   = ""
Text.s  = "1234567890"
Time    = ElapsedMilliseconds()
*Buffer  = AllocateMemory(2000002)
*Pointer = *Buffer
CopyMemoryString(Text, @*Pointer)
For i=1 To 99999
  CopyMemoryString(Text)
Next i

MessageRequester("", StrF((ElapsedMilliseconds()-Time)/1000, 3))
EnableDebugger
Debug PeekS(*Buffer)
BarryG
Addict
Addict
Posts: 3292
Joined: Thu Apr 18, 2019 8:17 am

Re: Fast string

Post by BarryG »

HeX0R wrote:No one thought about CopyMemoryString()?
Nice, but all the tricks like this shouldn't be needed.
pdwyer wrote:if you are building huge strings you should ask what your ultimate goal of the built string is as there may be a way to stream it
Nah, PureBasic should just build them faster, as HeX0R just showed.
Quin
Enthusiast
Enthusiast
Posts: 279
Joined: Thu Mar 31, 2022 7:03 pm
Location: United States
Contact:

Re: Fast string

Post by Quin »

+1
PB v5.40/6.10, Windows 10 64-bit.
16-core AMD Ryzen 9 5950X, 128 GB DDR5.
ShadowStorm
Enthusiast
Enthusiast
Posts: 237
Joined: Tue Feb 14, 2017 12:07 pm

Re: Fast string

Post by ShadowStorm »

Hi, not bad at all, I propose this humble code, it can certainly be improved :)

Code: Select all

; -----------------------------------------------------------------------------------------------

; By HeX0R the 11/02/2020

; DisableDebugger
; 
; Str.s   = ""
; Text.s  = "1234567890"
; Time    = ElapsedMilliseconds()
; *Buffer  = AllocateMemory(25000000)
; *Pointer = *Buffer
; CopyMemoryString(Text, @*Pointer)
; For i=1 To 999999
;   CopyMemoryString(Text)
; Next i
; 
; MessageRequester("", StrF((ElapsedMilliseconds()-Time)/1000, 3))
; EnableDebugger
; Debug PeekS(*Buffer)

; -----------------------------------------------------------------------------------------------

; By ShadowStorm the 17/11/2023, from HeX0R Code.

; This is just a single buffer.
; To have several independent buffers, see my StringBuilder 2 And 3 that I created on the PureBasic forum.

; Ceci est juste une mémoire tampon unique.
; Pour avoir plusieur mémoire tampon indépendente, voir mon StringBuilder 2 et 3 que j'ai créér sur le forum Français de PureBasic.

Global *SimplyStringBuilder_Buffer
Global *Pointer

; Create and initialize the StringBuilder buffer.
; Créer et initialiser le tampon StringBuilder.
Procedure.b SimplyStringBuilder_Initialize()
  
  ; If a buffer already exists, it will be deleted. Only one buffer is allowed, To avoid memory leaks when successive calls are made To this function.
  ; Si un tampon existe déjà, il sera supprimé. Un seul tampon est autorisé, afin d'éviter les fuites de mémoire lors d'appels successifs à cette fonction.
  If *SimplyStringBuilder_Buffer
    FreeMemory(*SimplyStringBuilder_Buffer)
  EndIf
  
  ; This is meant To be simple, so super large memory buffer For speed, no checking Or reallocation.
  ; Ceci est censé être simple, donc un tampon de mémoire super grand pour la vitesse, pas de vérification ou de réallocation.
  *SimplyStringBuilder_Buffer = AllocateMemory(999999999)
  *Pointer = *SimplyStringBuilder_Buffer
  CopyMemoryString("", @*Pointer)
  
  If *SimplyStringBuilder_Buffer
    ProcedureReturn 1
  Else
    ProcedureReturn 0
  EndIf
  
EndProcedure

; Clears the StringBuilder buffer.
; Efface la mémoire tampon du StringBuilder.
Procedure.b SimplyStringBuilder_Clear()
  ; Clears the buffer but does not delete it.
  ; Efface la mémoire tampon mais ne la supprime pas.
  If *SimplyStringBuilder_Buffer
    FillMemory(*SimplyStringBuilder_Buffer, MemorySize(*SimplyStringBuilder_Buffer))
    ProcedureReturn 1
  Else
    ProcedureReturn 0
  EndIf
EndProcedure

; Add a string to the end of the StringBuilder buffer.
; Ajoute une chaîne a la fin du tampon StringBuilder.
Procedure.b SimplyStringBuilder_AddString(String.s)
  If *SimplyStringBuilder_Buffer And String.s > ""
    CopyMemoryString(String.s, @*Pointer)
    ProcedureReturn 1
  Else
    ProcedureReturn 0
  EndIf
EndProcedure

; Retrieves Data from the StringBuilder buffer.
; Récupère les données de la mémoire tampon du StringBuilder.
Procedure.s SimplyStringBuilder_GetString()
  If *SimplyStringBuilder_Buffer
    ProcedureReturn PeekS(*SimplyStringBuilder_Buffer)
  Else
    ProcedureReturn ""
  EndIf
EndProcedure

; ------------------------------------- EXEMPLE ------------------------------------------------------------------------------------

; Create and initialize buffer.
; Créer et Initialise la mémoire tampon.
SimplyStringBuilder_Initialize()

; Character string to be added to the buffer.
; Chaîne de caractère à ajouter à la mémoire tampon.
Text.s  = "1234567890"

Time    = ElapsedMilliseconds()

For i = 1 To 999999
  ; Adds string to buffer 9 999 999 times.
  ; Ajoute 9 999 999 fois la chaîne dans la mémoire tampon.
  SimplyStringBuilder_AddString(Text)
Next i

MessageRequester("", Str((ElapsedMilliseconds()-Time)) + " Millisecondes.")

; Retrieves Data from the buffer.
; Récupère les données de la mémoire tampon.
Debug SimplyStringBuilder_GetString()

Debug Chr(13) + "Longueur: " + Len(SimplyStringBuilder_GetString()) + " Octets."
Debug Chr(13) + "Taille approximative en mémoire en Mega Octets: " + StrF(StringByteLength(SimplyStringBuilder_GetString()) / 1024 / 1000, 3) + " Mo."
Last edited by ShadowStorm on Sat Nov 18, 2023 4:17 pm, edited 3 times in total.
I am French, I do not speak English.
My apologies for the mistakes.

I have sometimes problems of expression
I am sometimes quite clumsy, please excuse me and let me know.
User avatar
Demivec
Addict
Addict
Posts: 4086
Joined: Mon Jul 25, 2005 3:51 pm
Location: Utah, USA

Re: Fast string

Post by Demivec »

HeX0R wrote: Tue Feb 11, 2020 10:06 pm No one thought about CopyMemoryString()?
It was in the first pages in the thread, even from 2014.
ShadowStorm
Enthusiast
Enthusiast
Posts: 237
Joined: Tue Feb 14, 2017 12:07 pm

Re: Fast string

Post by ShadowStorm »

My new simplistic version here is the fastest of all my StringBuilders.
I am French, I do not speak English.
My apologies for the mistakes.

I have sometimes problems of expression
I am sometimes quite clumsy, please excuse me and let me know.
User avatar
HeX0R
Addict
Addict
Posts: 980
Joined: Mon Sep 20, 2004 7:12 am
Location: Hell

Re: Fast string

Post by HeX0R »

There are plenty of such threads now, here e.g.:
viewtopic.php?t=80831
Post Reply