is it possible to Speed up stringfield()?

Got an idea for enhancing PureBasic? New command(s) you'd like to see?
deathmx
User
User
Posts: 27
Joined: Mon Feb 26, 2018 3:14 am

is it possible to Speed up stringfield()?

Post by deathmx »

I was wondering if it is possible to update the stringfield() function and speed it up.

I came up with a quick alternative but would be nice if we had something similar native in purebasic.

Thank you very much :)

I believe I also saw someone requesting case sensitivity options. Sorry I didn't include them in this.

This would allow the user to be able to cache and/or do their own indexing.

Here is my speed comparison test and code.

x is summing all the value strings making sure the results between the original and new are the same.

Generating String for speed test

Testing Original
Test Complete: Results: x =501000500, time =35797 ms

x has been reset: x = 0
Starting New test:
Test Completex: Results: x =501000500, time =171 ms

reposted : fixed a bug

Code: Select all

; Sorry for the messiness in advance ;)


OpenConsole()

Structure StringField
  *position
  *StringAddress.UNICODE
  *Seperator.UNICODE
  size.i
  delsize.i
  complete.i
EndStructure
Macro SetupStringField(stringfieldobject,String,delimiter,start=0)
  ;stringfieldobject#\position = 0
  stringfieldobject#\seperator = @delimiter#
  stringfieldobject#\StringAddress = @String#
  stringfieldobject#\position = start#
  stringfieldobject#\complete = 0
  stringfieldobject#\delsize = Len(delimiter#)*2
EndMacro
Macro NextStringField(stringfieldobject)
  stringfieldobject#\position = stringfieldobject#\stringaddress
  Repeat
    
    If stringfieldobject#\stringaddress\u = stringfieldobject#\seperator\u
      If stringfieldobject#\delsize > 2
        If CompareMemory(stringfieldobject#\stringaddress,stringfieldobject#\seperator,stringfieldobject#\delsize)
          StringFieldobject#\size = StringFieldobject#\StringAddress - stringfieldobject#\position
          StringFieldobject#\StringAddress + stringfieldobject#\delsize
          
          ;StringFieldobject#\StringAddress + 2
        Break  
        EndIf  
        Else
        StringFieldobject#\size = StringFieldobject#\StringAddress - stringfieldobject#\position  
        StringFieldobject#\StringAddress + 2 
        Break
      EndIf
    ElseIf stringfieldobject#\stringaddress\u = 0
      StringFieldobject#\size = StringFieldobject#\StringAddress - stringfieldobject#\position
      stringfieldobject#\complete = stringfieldobject#\stringaddress
      Break
    EndIf
    StringFieldobject#\StringAddress + 2 
  ForEver 
  
EndMacro
Macro GetStringField(StringFieldobject)
  PeekS(StringFieldobject#\position,StringFieldObject#\size/2)
EndMacro
Macro GetStringFieldAddress(StringFieldobject)
  StringFieldObject#\position
EndMacro
Macro GetStringFieldEnd(StringFieldobject)
StringFieldObject#\complete  
EndMacro
Macro GetStringFieldSize(StringFieldobject)
  StringFieldobject#\size
EndMacro
Macro starttimer()
  timer = GetTickCount_()
EndMacro
Macro checktimer()
  GetTickCount_() - timer
EndMacro

Macro debugx(info)
  PrintN(info#)
EndMacro
DisableDebugger


debugx("Generating String for speed test")
For i = 0 To 1000
  string$ + Str(i) + "abc"
Next
debugx("")
debugx("Testing Original")
count = CountString(string$,"abc")
starttimer()
Repeatamount = 1000
For therepeat = 0 To Repeatamount
  For i = 1 To count
    x+Val(StringField(string$,i,"abc"))
  Next
Next
old = checktimer()
debugx("Test Complete: Results: x =" + Str(x) + ", time =" + Str(old) + " ms")
debugx("")
x = 0
Debugx("x has been reset: x = "+Str(x))
debugx("Starting New test:")

del$ = "abc"


starttimer()
For therepeat = 0 To Repeatamount
  SetupStringField(this.StringField,string$,del$)
  While GetStringFieldEnd(this) = 0
    NextStringField(this)
    x+Val(GetStringField(this))
  Wend
Next
newx = checktimer()
debugx("Test Completex: Results: x =" + Str(x) + ", time =" + Str(newx) + " ms")
debugx("")
debugx("Press Enter to Close")
Input()

Last edited by deathmx on Mon Apr 19, 2021 9:02 am, edited 1 time in total.
BarryG
Addict
Addict
Posts: 3292
Joined: Thu Apr 18, 2019 8:17 am

Re: is it possible to Speed up stringfield()?

Post by BarryG »

StringField() works fastest with one-character strings. So if you replace the delimiter string with something like Chr(1), then it's way faster. For example, your code takes 12 seconds for me for the first test ("Testing original"), but only 2 seconds when replacing the delimiter with Chr(1). See my amended code below. You can always replace the original delimiter after the StringField() loop is done, if necessary.

BTW, your code is buggy in that your fast version doesn't do a correct StringField(). It returns strings like "100ab", "101ab", "102ab", etc (that is, it incorrectly includes part of the delimiter). Now this doesn't matter when using Val() on them, but yeah, just be aware that it's not a valid StringField() replacement when numbers aren't involved.

Code: Select all

del$="abc"

OpenConsole()
Macro debugx(info)
  PrintN(info#)
EndMacro
Macro starttimer()
  timer = GetTickCount_()
EndMacro

Macro checktimer()
  GetTickCount_() - timer
EndMacro
Structure StringField
  *position
  *StringAddress.UNICODE
  *Seperator.UNICODE
  delsize.i
  complete.b
EndStructure

Macro SetupStringField(stringfieldobject,String,delimiter,start=0)
  ;stringfieldobject#\position = 0
  stringfieldobject#\seperator = @delimiter#
  stringfieldobject#\StringAddress = @String#
  stringfieldobject#\position = start#
  stringfieldobject#\complete = 0
  stringfieldobject#\delsize = Len(delimiter#)*2
EndMacro

Macro NextStringField(stringfieldobject)
  stringfieldobject#\position = stringfieldobject#\stringaddress
  Repeat
    
    If stringfieldobject#\stringaddress\u = stringfieldobject#\seperator\u
      If stringfieldobject#\delsize > 1
        If CompareMemory(stringfieldobject#\stringaddress,stringfieldobject#\seperator,stringfieldobject#\delsize)
          StringFieldobject#\StringAddress + stringfieldobject#\delsize 
          Break  
        EndIf  
      Else
        
        StringFieldobject#\StringAddress + 2 
        Break
      EndIf
    ElseIf stringfieldobject#\stringaddress\u = 0
      stringfieldobject#\complete = 1
      Break
    EndIf
    StringFieldobject#\StringAddress + 2 
  ForEver 
  
EndMacro

Macro getstring(StringFieldobject)
  PeekS(StringFieldobject#\position,(StringFieldobject#\StringAddress - StringFieldobject#\position-2)/2)
EndMacro


DisableDebugger
debugx("Generating String for speed test")
For i = 0 To 1000
  string$ + Str(i) + del$
Next

; These two lines speed up the code immensely!
string$=ReplaceString(string$,del$,Chr(1))
del$=Chr(1)

debugx("")
debugx("Testing Original")
count = CountString(string$,del$)
starttimer()
Repeatamount = 1000
For therepeat = 0 To Repeatamount
  For i = 1 To count
    x+Val(StringField(string$,i,del$))
  Next
Next
old = checktimer()
debugx("Test Complete: Results: x =" + Str(x) + ", time =" + Str(old) + " ms")
debugx("")
x = 0
Debugx("x has been reset: x = "+Str(x))
debugx("Starting New test:")

starttimer()
For therepeat = 0 To Repeatamount
  SetupStringField(this.StringField,string$,del$)
  While this\complete =0
    NextStringField(this)
    x+Val(getstring(this))
  Wend
Next
newx = checktimer()
debugx("Test Completex: Results: x =" + Str(x) + ", time =" + Str(newx) + " ms")
debugx("")
debugx("Press Enter to Close")
Input()
deathmx
User
User
Posts: 27
Joined: Mon Feb 26, 2018 3:14 am

Re: is it possible to Speed up stringfield()?

Post by deathmx »

Yes, I noticed that sorry, already fixed and reposted it :)
deathmx
User
User
Posts: 27
Joined: Mon Feb 26, 2018 3:14 am

Re: is it possible to Speed up stringfield()?

Post by deathmx »

I tried with a single delimiter and ended up with these results: My laptop is running in slow mode apparently. 0.79ghz lol
Generating String for speed test

Testing Original
Test Complete: Results: x =501000500, time =10437 ms

x has been reset: x = 0
Starting New test:
Test Completex: Results: x =501000500, time =157 ms

Code: Select all

OpenConsole()

Structure StringField
  *position
  *StringAddress.UNICODE
  *Seperator.UNICODE
  size.i
  delsize.i
  complete.i
EndStructure
Macro SetupStringField(stringfieldobject,String,delimiter,start=0)
  ;stringfieldobject#\position = 0
  stringfieldobject#\seperator = @delimiter#
  stringfieldobject#\StringAddress = @String#
  stringfieldobject#\position = start#
  stringfieldobject#\complete = 0
  stringfieldobject#\delsize = Len(delimiter#)*2
EndMacro
Macro NextStringField(stringfieldobject)
  stringfieldobject#\position = stringfieldobject#\stringaddress
  Repeat
    
    If stringfieldobject#\stringaddress\u = stringfieldobject#\seperator\u
      If stringfieldobject#\delsize > 2
        If CompareMemory(stringfieldobject#\stringaddress,stringfieldobject#\seperator,stringfieldobject#\delsize)
          StringFieldobject#\size = StringFieldobject#\StringAddress - stringfieldobject#\position
          StringFieldobject#\StringAddress + stringfieldobject#\delsize
          
          ;StringFieldobject#\StringAddress + 2
        Break  
        EndIf  
        Else
        StringFieldobject#\size = StringFieldobject#\StringAddress - stringfieldobject#\position  
        StringFieldobject#\StringAddress + 2 
        Break
      EndIf
    ElseIf stringfieldobject#\stringaddress\u = 0
      StringFieldobject#\size = StringFieldobject#\StringAddress - stringfieldobject#\position
      stringfieldobject#\complete = stringfieldobject#\stringaddress
      Break
    EndIf
    StringFieldobject#\StringAddress + 2 
  ForEver 
  
EndMacro
Macro GetStringField(StringFieldobject)
  PeekS(StringFieldobject#\position,StringFieldObject#\size/2)
EndMacro
Macro GetStringFieldAddress(StringFieldobject)
  StringFieldObject#\position
EndMacro
Macro GetStringFieldEnd(StringFieldobject)
StringFieldObject#\complete  
EndMacro
Macro GetStringFieldSize(StringFieldobject)
  StringFieldobject#\size
EndMacro
Macro starttimer()
  timer = GetTickCount_()
EndMacro
Macro checktimer()
  GetTickCount_() - timer
EndMacro

Macro debugx(info)
  PrintN(info#)
EndMacro
DisableDebugger


Delimieter$ = " "

debugx("Generating String for speed test")
For i = 0 To 1000
  string$ + Str(i) + Delimieter$
Next
debugx("")
debugx("Testing Original")
count = CountString(string$,Delimieter$)
starttimer()
Repeatamount = 1000
For therepeat = 0 To Repeatamount
  For i = 1 To count
    x+Val(StringField(string$,i,Delimieter$))
  Next
Next
old = checktimer()
debugx("Test Complete: Results: x =" + Str(x) + ", time =" + Str(old) + " ms")
debugx("")
x = 0
Debugx("x has been reset: x = "+Str(x))
debugx("Starting New test:")

del$ = Delimieter$


starttimer()
For therepeat = 0 To Repeatamount
  SetupStringField(this.StringField,string$,del$)
  While GetStringFieldEnd(this) = 0
    NextStringField(this)
    x+Val(GetStringField(this))
  Wend
Next
newx = checktimer()
debugx("Test Completex: Results: x =" + Str(x) + ", time =" + Str(newx) + " ms")
debugx("")
debugx("Press Enter to Close")
Input()
User avatar
Keya
Addict
Addict
Posts: 1891
Joined: Thu Jun 04, 2015 7:10 am

Re: is it possible to Speed up stringfield()?

Post by Keya »

have you tested to ensure that it's not CACHING?
deathmx
User
User
Posts: 27
Joined: Mon Feb 26, 2018 3:14 am

Re: is it possible to Speed up stringfield()?

Post by deathmx »

caching within my code on purebasic or the cpu itself?
Shouldn't be caching within my code since it sets up every time in the repeated loop.
User avatar
skywalk
Addict
Addict
Posts: 3972
Joined: Wed Dec 23, 2009 10:14 pm
Location: Boston, MA

Re: is it possible to Speed up stringfield()?

Post by skywalk »

StringField() should only be used for very small strings and not when speed is required.

Fast Split() functions.
The nice thing about standards is there are so many to choose from. ~ Andrew Tanenbaum
deathmx
User
User
Posts: 27
Joined: Mon Feb 26, 2018 3:14 am

Re: is it possible to Speed up stringfield()?

Post by deathmx »

I agree that's how it should be used the way it is now.

However, modifying or Adding some additional features can be really beneficial.
It feels like one of Purebasic's least efficient string functions.
There are many string functions that don't have this efficiency issue.

Maybe not many people are looking to change this?
So far the only response has been "oh it's meant to be slow, or it's not that slow" lol

I do appreciate your opinions :) Hopefully, I am not requesting something most people care little about lol.
User avatar
Sicro
Enthusiast
Enthusiast
Posts: 538
Joined: Wed Jun 25, 2014 5:25 pm
Location: Germany
Contact:

Re: is it possible to Speed up stringfield()?

Post by Sicro »

deathmx wrote: Mon Apr 19, 2021 3:26 pm Hopefully, I am not requesting something most people care little about lol.
No, look here: String length should be stored for string variables
Image
Why OpenSource should have a license :: PB-CodeArchiv-Rebirth :: Pleasant-Dark (syntax color scheme) :: RegEx-Engine (compiles RegExes to NFA/DFA)
Manjaro Xfce x64 (Main system) :: Windows 10 Home (VirtualBox) :: Newest PureBasic version
AZJIO
Addict
Addict
Posts: 1312
Joined: Sun May 14, 2017 1:48 am

Re: is it possible to Speed up stringfield()?

Post by AZJIO »

Code: Select all

StringField(String$ , Index , Delimiter$, @String$)
If the 4th parameter contains a pointer to the position where the previous call ended, then the function will work quickly. Do as in the CopyMemoryString function.
Post Reply