PureBasic Forums - English

http://www.purebasic.com
http://forums.purebasic.com/english/

is it possible to Speed up stringfield()?

http://forums.purebasic.com/english/viewtopic.php?t=77107

Page 1 of 1

is it possible to Speed up stringfield()?

Posted: Mon Apr 19, 2021 6:24 am
by deathmx
I was wondering if it is possible to update the stringfield() function and speed it up.

I came up with a quick alternative but would be nice if we had something similar native in purebasic.

Thank you very much :)

I believe I also saw someone requesting case sensitivity options. Sorry I didn't include them in this.

This would allow the user to be able to cache and/or do their own indexing.

Here is my speed comparison test and code.

x is summing all the value strings making sure the results between the original and new are the same.

Generating String for speed test

Testing Original
Test Complete: Results: x =501000500, time =35797 ms

x has been reset: x = 0
Starting New test:
Test Completex: Results: x =501000500, time =171 ms

reposted : fixed a bug

Code: Select all

; Sorry for the messiness in advance ;)


OpenConsole()

Structure StringField
  *position
  *StringAddress.UNICODE
  *Seperator.UNICODE
  size.i
  delsize.i
  complete.i
EndStructure
Macro SetupStringField(stringfieldobject,String,delimiter,start=0)
  ;stringfieldobject#\position = 0
  stringfieldobject#\seperator = @delimiter#
  stringfieldobject#\StringAddress = @String#
  stringfieldobject#\position = start#
  stringfieldobject#\complete = 0
  stringfieldobject#\delsize = Len(delimiter#)*2
EndMacro
Macro NextStringField(stringfieldobject)
  stringfieldobject#\position = stringfieldobject#\stringaddress
  Repeat
    
    If stringfieldobject#\stringaddress\u = stringfieldobject#\seperator\u
      If stringfieldobject#\delsize > 2
        If CompareMemory(stringfieldobject#\stringaddress,stringfieldobject#\seperator,stringfieldobject#\delsize)
          StringFieldobject#\size = StringFieldobject#\StringAddress - stringfieldobject#\position
          StringFieldobject#\StringAddress + stringfieldobject#\delsize
          
          ;StringFieldobject#\StringAddress + 2
        Break  
        EndIf  
        Else
        StringFieldobject#\size = StringFieldobject#\StringAddress - stringfieldobject#\position  
        StringFieldobject#\StringAddress + 2 
        Break
      EndIf
    ElseIf stringfieldobject#\stringaddress\u = 0
      StringFieldobject#\size = StringFieldobject#\StringAddress - stringfieldobject#\position
      stringfieldobject#\complete = stringfieldobject#\stringaddress
      Break
    EndIf
    StringFieldobject#\StringAddress + 2 
  ForEver 
  
EndMacro
Macro GetStringField(StringFieldobject)
  PeekS(StringFieldobject#\position,StringFieldObject#\size/2)
EndMacro
Macro GetStringFieldAddress(StringFieldobject)
  StringFieldObject#\position
EndMacro
Macro GetStringFieldEnd(StringFieldobject)
StringFieldObject#\complete  
EndMacro
Macro GetStringFieldSize(StringFieldobject)
  StringFieldobject#\size
EndMacro
Macro starttimer()
  timer = GetTickCount_()
EndMacro
Macro checktimer()
  GetTickCount_() - timer
EndMacro

Macro debugx(info)
  PrintN(info#)
EndMacro
DisableDebugger


debugx("Generating String for speed test")
For i = 0 To 1000
  string$ + Str(i) + "abc"
Next
debugx("")
debugx("Testing Original")
count = CountString(string$,"abc")
starttimer()
Repeatamount = 1000
For therepeat = 0 To Repeatamount
  For i = 1 To count
    x+Val(StringField(string$,i,"abc"))
  Next
Next
old = checktimer()
debugx("Test Complete: Results: x =" + Str(x) + ", time =" + Str(old) + " ms")
debugx("")
x = 0
Debugx("x has been reset: x = "+Str(x))
debugx("Starting New test:")

del$ = "abc"


starttimer()
For therepeat = 0 To Repeatamount
  SetupStringField(this.StringField,string$,del$)
  While GetStringFieldEnd(this) = 0
    NextStringField(this)
    x+Val(GetStringField(this))
  Wend
Next
newx = checktimer()
debugx("Test Completex: Results: x =" + Str(x) + ", time =" + Str(newx) + " ms")
debugx("")
debugx("Press Enter to Close")
Input()

Re: is it possible to Speed up stringfield()?

Posted: Mon Apr 19, 2021 8:46 am
by BarryG
StringField() works fastest with one-character strings. So if you replace the delimiter string with something like Chr(1), then it's way faster. For example, your code takes 12 seconds for me for the first test ("Testing original"), but only 2 seconds when replacing the delimiter with Chr(1). See my amended code below. You can always replace the original delimiter after the StringField() loop is done, if necessary.

BTW, your code is buggy in that your fast version doesn't do a correct StringField(). It returns strings like "100ab", "101ab", "102ab", etc (that is, it incorrectly includes part of the delimiter). Now this doesn't matter when using Val() on them, but yeah, just be aware that it's not a valid StringField() replacement when numbers aren't involved.

Code: Select all

del$="abc"

OpenConsole()
Macro debugx(info)
  PrintN(info#)
EndMacro
Macro starttimer()
  timer = GetTickCount_()
EndMacro

Macro checktimer()
  GetTickCount_() - timer
EndMacro
Structure StringField
  *position
  *StringAddress.UNICODE
  *Seperator.UNICODE
  delsize.i
  complete.b
EndStructure

Macro SetupStringField(stringfieldobject,String,delimiter,start=0)
  ;stringfieldobject#\position = 0
  stringfieldobject#\seperator = @delimiter#
  stringfieldobject#\StringAddress = @String#
  stringfieldobject#\position = start#
  stringfieldobject#\complete = 0
  stringfieldobject#\delsize = Len(delimiter#)*2
EndMacro

Macro NextStringField(stringfieldobject)
  stringfieldobject#\position = stringfieldobject#\stringaddress
  Repeat
    
    If stringfieldobject#\stringaddress\u = stringfieldobject#\seperator\u
      If stringfieldobject#\delsize > 1
        If CompareMemory(stringfieldobject#\stringaddress,stringfieldobject#\seperator,stringfieldobject#\delsize)
          StringFieldobject#\StringAddress + stringfieldobject#\delsize 
          Break  
        EndIf  
      Else
        
        StringFieldobject#\StringAddress + 2 
        Break
      EndIf
    ElseIf stringfieldobject#\stringaddress\u = 0
      stringfieldobject#\complete = 1
      Break
    EndIf
    StringFieldobject#\StringAddress + 2 
  ForEver 
  
EndMacro

Macro getstring(StringFieldobject)
  PeekS(StringFieldobject#\position,(StringFieldobject#\StringAddress - StringFieldobject#\position-2)/2)
EndMacro


DisableDebugger
debugx("Generating String for speed test")
For i = 0 To 1000
  string$ + Str(i) + del$
Next

; These two lines speed up the code immensely!
string$=ReplaceString(string$,del$,Chr(1))
del$=Chr(1)

debugx("")
debugx("Testing Original")
count = CountString(string$,del$)
starttimer()
Repeatamount = 1000
For therepeat = 0 To Repeatamount
  For i = 1 To count
    x+Val(StringField(string$,i,del$))
  Next
Next
old = checktimer()
debugx("Test Complete: Results: x =" + Str(x) + ", time =" + Str(old) + " ms")
debugx("")
x = 0
Debugx("x has been reset: x = "+Str(x))
debugx("Starting New test:")

starttimer()
For therepeat = 0 To Repeatamount
  SetupStringField(this.StringField,string$,del$)
  While this\complete =0
    NextStringField(this)
    x+Val(getstring(this))
  Wend
Next
newx = checktimer()
debugx("Test Completex: Results: x =" + Str(x) + ", time =" + Str(newx) + " ms")
debugx("")
debugx("Press Enter to Close")
Input()

Re: is it possible to Speed up stringfield()?

Posted: Mon Apr 19, 2021 9:04 am
by deathmx
Yes, I noticed that sorry, already fixed and reposted it :)

Re: is it possible to Speed up stringfield()?

Posted: Mon Apr 19, 2021 9:20 am
by deathmx
I tried with a single delimiter and ended up with these results: My laptop is running in slow mode apparently. 0.79ghz lol
Generating String for speed test

Testing Original
Test Complete: Results: x =501000500, time =10437 ms

x has been reset: x = 0
Starting New test:
Test Completex: Results: x =501000500, time =157 ms

Code: Select all

OpenConsole()

Structure StringField
  *position
  *StringAddress.UNICODE
  *Seperator.UNICODE
  size.i
  delsize.i
  complete.i
EndStructure
Macro SetupStringField(stringfieldobject,String,delimiter,start=0)
  ;stringfieldobject#\position = 0
  stringfieldobject#\seperator = @delimiter#
  stringfieldobject#\StringAddress = @String#
  stringfieldobject#\position = start#
  stringfieldobject#\complete = 0
  stringfieldobject#\delsize = Len(delimiter#)*2
EndMacro
Macro NextStringField(stringfieldobject)
  stringfieldobject#\position = stringfieldobject#\stringaddress
  Repeat
    
    If stringfieldobject#\stringaddress\u = stringfieldobject#\seperator\u
      If stringfieldobject#\delsize > 2
        If CompareMemory(stringfieldobject#\stringaddress,stringfieldobject#\seperator,stringfieldobject#\delsize)
          StringFieldobject#\size = StringFieldobject#\StringAddress - stringfieldobject#\position
          StringFieldobject#\StringAddress + stringfieldobject#\delsize
          
          ;StringFieldobject#\StringAddress + 2
        Break  
        EndIf  
        Else
        StringFieldobject#\size = StringFieldobject#\StringAddress - stringfieldobject#\position  
        StringFieldobject#\StringAddress + 2 
        Break
      EndIf
    ElseIf stringfieldobject#\stringaddress\u = 0
      StringFieldobject#\size = StringFieldobject#\StringAddress - stringfieldobject#\position
      stringfieldobject#\complete = stringfieldobject#\stringaddress
      Break
    EndIf
    StringFieldobject#\StringAddress + 2 
  ForEver 
  
EndMacro
Macro GetStringField(StringFieldobject)
  PeekS(StringFieldobject#\position,StringFieldObject#\size/2)
EndMacro
Macro GetStringFieldAddress(StringFieldobject)
  StringFieldObject#\position
EndMacro
Macro GetStringFieldEnd(StringFieldobject)
StringFieldObject#\complete  
EndMacro
Macro GetStringFieldSize(StringFieldobject)
  StringFieldobject#\size
EndMacro
Macro starttimer()
  timer = GetTickCount_()
EndMacro
Macro checktimer()
  GetTickCount_() - timer
EndMacro

Macro debugx(info)
  PrintN(info#)
EndMacro
DisableDebugger


Delimieter$ = " "

debugx("Generating String for speed test")
For i = 0 To 1000
  string$ + Str(i) + Delimieter$
Next
debugx("")
debugx("Testing Original")
count = CountString(string$,Delimieter$)
starttimer()
Repeatamount = 1000
For therepeat = 0 To Repeatamount
  For i = 1 To count
    x+Val(StringField(string$,i,Delimieter$))
  Next
Next
old = checktimer()
debugx("Test Complete: Results: x =" + Str(x) + ", time =" + Str(old) + " ms")
debugx("")
x = 0
Debugx("x has been reset: x = "+Str(x))
debugx("Starting New test:")

del$ = Delimieter$


starttimer()
For therepeat = 0 To Repeatamount
  SetupStringField(this.StringField,string$,del$)
  While GetStringFieldEnd(this) = 0
    NextStringField(this)
    x+Val(GetStringField(this))
  Wend
Next
newx = checktimer()
debugx("Test Completex: Results: x =" + Str(x) + ", time =" + Str(newx) + " ms")
debugx("")
debugx("Press Enter to Close")
Input()

Re: is it possible to Speed up stringfield()?

Posted: Mon Apr 19, 2021 9:34 am
by Keya
have you tested to ensure that it's not CACHING?

Re: is it possible to Speed up stringfield()?

Posted: Mon Apr 19, 2021 9:45 am
by deathmx
caching within my code on purebasic or the cpu itself?
Shouldn't be caching within my code since it sets up every time in the repeated loop.

Re: is it possible to Speed up stringfield()?

Posted: Mon Apr 19, 2021 1:53 pm
by skywalk
StringField() should only be used for very small strings and not when speed is required.

Fast Split() functions.

Re: is it possible to Speed up stringfield()?

Posted: Mon Apr 19, 2021 3:26 pm
by deathmx
I agree that's how it should be used the way it is now.

However, modifying or Adding some additional features can be really beneficial.
It feels like one of Purebasic's least efficient string functions.
There are many string functions that don't have this efficiency issue.

Maybe not many people are looking to change this?
So far the only response has been "oh it's meant to be slow, or it's not that slow" lol

I do appreciate your opinions :) Hopefully, I am not requesting something most people care little about lol.

Re: is it possible to Speed up stringfield()?

Posted: Mon Apr 19, 2021 5:20 pm
by Sicro
deathmx wrote: ↑Mon Apr 19, 2021 3:26 pm Hopefully, I am not requesting something most people care little about lol.
No, look here: String length should be stored for string variables

Re: is it possible to Speed up stringfield()?

Posted: Sat Oct 15, 2022 9:28 am
by AZJIO

Code: Select all

StringField(String$ , Index , Delimiter$, @String$)
If the 4th parameter contains a pointer to the position where the previous call ended, then the function will work quickly. Do as in the CopyMemoryString function.
All times are UTC+01:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited