CreateThread Procedure corrupting strings

Just starting out? Need help? Post your questions and find answers here.
sc4pb
User
User
Posts: 26
Joined: Tue Mar 07, 2023 5:33 pm

CreateThread Procedure corrupting strings

Post by sc4pb »

I have a bit of confusion about strings, arrays, memory allocation, and threads. I can't post actual project code here, the situation is complex, so I'll try to simplify. I'm not looking for a magic fix, just wanted community thoughts about what I might be misunderstanding.

On program startup I create an array of structures. I wanted it global, but don't know at startup how many records I'll need so did it like this:

Code: Select all

Structure tFoo
  String1.s
  String2.s
EndStructure

Global Dim xInfo.tFoo(0) ; intially just one element
  ; ...
Redim xInfo(lMax) ; later in program, redim to the # of items I need and assign contents
  ; ...
xInfo(x)\String1="hello"
xInfo(x)\String2="world"
I am assuming that the Redim statement allocates enough memory upfront for all of the structures in the array. And, as I assign string values to all of them, that the string space is allocated as well. Are the String1 and String2 items in the record actually pointers to the externally allocated string buffers? I would think so. At any rate, I assume at this point that all of my array records, strings, etc, are all in safely allocated purebasic-managed memory spaces.

Later in the program I do some work, during which some new maps are created and used, etc. I want to pass it a single record from the array, so I've been doing it like this:

Code: Select all

DoSomeWork(@xInfo(n))	; which I assume passes the address of the Nth record in array

 ; ...and in the procedure I have the parameter passing declared as:

Procedure DoSomeWork(*OneRecord.tFoo)
   ...
EndProcedure
...my assumption above is that this is a good way to pass a the contents of the Nth record in the array to the procedure DoSomeWork(). DoSomeWork does not modify the contents of the record in any way. When I get back to main program after the call, all the record contents should be unchanged.

THE QUESTION: Now, DoSomeWork can take some time, so I then tried having it run on a separate thread. I call it like this:

Code: Select all

lThreadHandle=CreateThread(@BackgroundWork(),n)

Procedure Backgroundwork(*Value)

  Define n.l ;at this point should already be executing on new thread
  n=*Value

  DoSomeWork(@xInfo(n))

EndProcedure
When I invoke DoSomeWork on a new thread as above, the string contents of the xInfo() records get corrupted.

Yes the new thread for DoSomeWork allocates a number of new objects in memory, like NewMaps and map elements. But in my mind nothing being allocated should ever be able to overwrite contents of the original array or its strings as all of that was allocated ahead of time.

DoSomeWork reads from the xInfo record it is handed, but doesn't write to it. Yet the contents get corrupted, only when threading used. If I drop the CreateThread and call DoSomeWork directly, no problems.

Finally, DoSomeWork() uses a global map during processing. I wanted each worker thread to have its own copy of the map, so used 'threaded' declare. But discovered:

Code: Select all

Threaded NewMap SomeMap() ; when declared like this, the xInfo corruption DOES occur.
Global NewMap SomeMap()   ; when declared like this, corruption DOES NOT occur.
It seems clear that objects being allocated on the new thread by DoSomeWork (like SomeMap() above) are overwriting the string records that existed previously in the xInfo() records. But why?? Shouldn't PB know that the memory for both the xInfo array and its asociated strings are already in use?

(Thanks in advance to anyone who read this far!)
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: CreateThread Procedure corrupting strings

Post by NicTheQuick »

Did you enable thread-safe executables in the compiler options? If not that's the reason.
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
sc4pb
User
User
Posts: 26
Joined: Tue Mar 07, 2023 5:33 pm

Re: CreateThread Procedure corrupting strings

Post by sc4pb »

Yes, checked that a number of times. Not in code, but in IDE Compiler Options, checked the box.
jassing
Addict
Addict
Posts: 1745
Joined: Wed Feb 17, 2010 12:00 am

Re: CreateThread Procedure corrupting strings

Post by jassing »

FYI Maps and Lists are NOT thread-safe
You need to wrap add/updates/deletes with a mutex to prevent problems if multiple threads are accessing the same list, map, array
sc4pb
User
User
Posts: 26
Joined: Tue Mar 07, 2023 5:33 pm

Re: CreateThread Procedure corrupting strings

Post by sc4pb »

I have not included a mutex because maps are never being shared for writing by more than one thread. I assumed:

a) multiple threads could *read* from a global map as long as no writing (changes) occurred.

b) By using the "threaded" keyword I was giving each thread its own instance of the map, so only one thread user per map.

Am I wrong about these?

(And as mentioned before, it is the inclusion of the "threaded" keyword that causes the corruption of the global strings. The creation of new map objects when threads created appears to be part of the cause.)
User avatar
Demivec
Addict
Addict
Posts: 4086
Joined: Mon Jul 25, 2005 3:51 pm
Location: Utah, USA

Re: CreateThread Procedure corrupting strings

Post by Demivec »

sc4pb wrote: Tue Jun 06, 2023 8:21 pma) multiple threads could *read* from a global map as long as no writing (changes) occurred.
Maps only have one active element pointer. This means there will be problems with reads of different elements by different threads.
jassing
Addict
Addict
Posts: 1745
Joined: Wed Feb 17, 2010 12:00 am

Re: CreateThread Procedure corrupting strings

Post by jassing »

did you try adding mutexes?
sc4pb
User
User
Posts: 26
Joined: Tue Mar 07, 2023 5:33 pm

Re: CreateThread Procedure corrupting strings

Post by sc4pb »

not yet. But must point out that in my testing I have only had one worker process, so only one thread accessing maps anyway. Putting threading in there was preparation for the future. The maps themselves are not getting messed up, rather the data in a totally unrelated array of structures (defined at the beginning of the program) is getting lost or overwritten. Finally I don't think I need any mutex for a map declared as:

Threaded NewMap SomeMap()

...because every thread would have its own "private" SomeMap, right? Unless I misunderstand the purpose of "threaded"

Thanks
jassing
Addict
Addict
Posts: 1745
Joined: Wed Feb 17, 2010 12:00 am

Re: CreateThread Procedure corrupting strings

Post by jassing »

Nope,that's the point of 'threaded'
would you post some code to replicate?

Code: Select all

Threaded NewList test.s()

Procedure threadTest(n)
  For x = 1 To 100
    AddElement( test() ) : test() = Str(n)+":"+Str(x)
    Delay(n)
  Next
  
  Debug Str(n)+": "+Str(ListSize(test()))
  SelectElement(test(),50)
  Debug Str(n)+": "+test()
EndProcedure

h1 = CreateThread(@threadTest(),100)
h2 = CreateThread(@threadTest(), 75)
h3 = CreateThread(@threadTest(),150)
h4 = CreateThread(@threadTest(), 50)

WaitThread(h1) : WaitThread(h2) : WaitThread(h3) : WaitThread(h4)
produced:
program output wrote: [16:35:02] 50: 100
[16:35:02] 50: 50:51
[16:35:05] 75: 100
[16:35:05] 75: 75:51
[16:35:07] 100: 100
[16:35:07] 100: 100:51
[16:35:17] 200: 100
[16:35:17] 200: 200:51
So no data corruption...
User avatar
mk-soft
Always Here
Always Here
Posts: 5335
Joined: Fri May 12, 2006 6:51 pm
Location: Germany

Re: CreateThread Procedure corrupting strings

Post by mk-soft »

A "Threaded NewMap xyz()" is only valid for the duration of the thread. It is created when the thread is started and destroyed when the thread is terminated.
To work with strings and threads, the Threadsafe option must always be activated. Threadsafe also affects other functions.
Access to shared variables, lists and maps must be protected with mutex.

So that one does not forget the threadsafe option.

Code: Select all


CompilerIf Not #PB_Compiler_Thread
  CompilerError "Set Compiler Option ThreadSafe!"
CompilerEndIf
My Projects ThreadToGUI / OOP-BaseClass / EventDesigner V3
PB v3.30 / v5.75 - OS Mac Mini OSX 10.xx - VM Window Pro / Linux Ubuntu
Downloads on my Webspace / OneDrive
User avatar
NicTheQuick
Addict
Addict
Posts: 1224
Joined: Sun Jun 22, 2003 7:43 pm
Location: Germany, Saarbrücken
Contact:

Re: CreateThread Procedure corrupting strings

Post by NicTheQuick »

mk-soft wrote: Wed Jun 07, 2023 8:42 am A "Threaded NewMap xyz()" is only valid for the duration of the thread. It is created when the thread is started and destroyed when the thread is terminated.
To work with strings and threads, the Threadsafe option must always be activated. Threadsafe also affects other functions.
Access to shared variables, lists and maps must be protected with mutex.

So that one does not forget the threadsafe option.

Code: Select all


CompilerIf Not #PB_Compiler_Thread
  CompilerError "Set Compiler Option ThreadSafe!"
CompilerEndIf
He already wrote that he chcked that box.
The english grammar is freeware, you can use it freely - But it's not Open Source, i.e. you can not change it or publish it in altered way.
User avatar
spikey
Enthusiast
Enthusiast
Posts: 581
Joined: Wed Sep 22, 2010 1:17 pm
Location: United Kingdom

Re: CreateThread Procedure corrupting strings

Post by spikey »

There are four ways a string array can be corrupted, it might be helpful to determine which of them applies to your case.
1) The pointer to the base of the array can be invalidated. This is unlikely but it's not impossible. You can check this by preserving the array's base address after you redim and finialise it and asserting that this is still ok in the thread and afterwards.

Code: Select all

*Array = @xinfo()
;...
If *Array <> @xinfo()
  Debug "Array pointer assertion failed."
EndIf
2) As you mentioned in your first post a string array is an array of pointers to heap allocated strings, or in your case pointers to pointers to strings because the strings are in a structure. It's possible that these pointers are being invalidated. You can assert this by making a copy of the pointer data and asserting that they are still ok.

Code: Select all

*First = @xinfo(0)
*First1 = @xinfo(0)\String1
*First2 = @xinfo(0)\String2
*Last = @xinfo(n)
*Last1 = @xinfo(n)\String1
*Last2 = @xinfo(n)\String2
; ...
If *First <> @xinfo(0)
  Debug "First member pointer assertion failed."
EndIf
If *First1 <> @xinfo(0)\String1
  Debug "First String1 pointer assertion failed."
EndIf
If *First2 <> @xinfo(0)\String2
  Debug "First String2 pointer assertion failed."
EndIf
If *Last <> @xinfo(n)
  Debug "Last member pointer assertion failed."
EndIf
If *Last1 <> @xinfo(n)\String1
  Debug "Last String1 pointer assertion failed."
EndIf
If *Last2 <> @xinfo(n)\String2
  Debug "Last String2 pointer assertion failed."
EndIf
; ...
3) The pointers are all fine but the string content is being invalidated. This is harder to assert fully because it would require making a copy not only of the pointer array but the allocated strings too. If you can set known values in the array for test purposes you could assert that this is ok.
4) More than one of the above simultaneously.
There's usually a significant distance in memory between the array and the string allocations so corrupting both together is unlikely but not impossible in the case of a large overrun type error.

My initial expectation would be to see either case 2 or case 3.

I mocked up the code you described and threw some random multi-threading activity at it and it behaved fine (on Windows 11, X64, 6.02, ASM backend) this tells me that there's something a bit more complex going on than just 'Maps break Threads'. Can you cause this code to fail in the way you're experiencing - if so how? That might tell us something useful.

Code: Select all

Structure tFoo
  String1.s
  String2.s
EndStructure

Global Dim xInfo.tFoo(0) ; intially just one element

Threaded NewMap SomeMap() ; when declared like this, the xInfo corruption DOES occur.

ReDim xInfo(4) ; later in program, redim to the # of items I need and assign contents
  ; ...
xInfo(0)\String1="First-One"
xInfo(0)\String2="First-Two"
xInfo(1)\String1="Second-One"
xInfo(1)\String2="Second-Two"
xInfo(2)\String1="Third-One"
xInfo(2)\String2="Third-Two"
xInfo(3)\String1="Fourth-One"
xInfo(3)\String2="Fourth-Two"
xInfo(4)\String1="Fifth-One"
xInfo(4)\String2="Fifth-Two"

Debug "Initial"
Debug "------"
Debug @xInfo()
Debug @xInfo(0)
Debug @xInfo(0)\String1
Debug xInfo(0)\String1
Debug @xInfo(0)\String2
Debug xInfo(0)\String2
Debug @xInfo(1)
Debug @xInfo(1)\String1
Debug xInfo(1)\String1
Debug @xInfo(1)\String2
Debug xInfo(1)\String2
Debug @xInfo(2)
Debug @xInfo(2)\String1
Debug xInfo(2)\String1
Debug @xInfo(2)\String2
Debug xInfo(2)\String2
Debug @xInfo(3)
Debug @xInfo(3)\String1
Debug xInfo(3)\String1
Debug @xInfo(3)\String2
Debug xInfo(3)\String2
Debug @xInfo(4)
Debug @xInfo(4)\String1
Debug xInfo(4)\String1
Debug @xInfo(4)\String2
Debug xInfo(4)\String2

 ; ...and in the procedure I have the parameter passing declared as:

Procedure DoSomeWork(*OneRecord.tFoo)
  Debug "In DoSomeWork " + StrU(*OneRecord)
  SomeMap("a") = 1
  SomeMap("b") = 2
  SomeMap("c") = 3
  SomeMap("c") = 4
  SomeMap("b") = 5
  SomeMap("a") = 6
EndProcedure

Procedure Backgroundwork(*Value)
  
  
  Define n.l ;at this point should already be executing on new thread
  n=*Value

  DoSomeWork(@xInfo(n))

EndProcedure

Debug #Empty$
Debug "Working"
Debug "-------"

lThreadHandle1=CreateThread(@BackgroundWork(),0)
lThreadHandle2=CreateThread(@BackgroundWork(),1)
lThreadHandle3=CreateThread(@BackgroundWork(),2)
lThreadHandle4=CreateThread(@BackgroundWork(),3)
lThreadHandle5=CreateThread(@BackgroundWork(),4)
lThreadHandle6=CreateThread(@BackgroundWork(),0)
lThreadHandle7=CreateThread(@BackgroundWork(),1)
lThreadHandle8=CreateThread(@BackgroundWork(),2)
lThreadHandle9=CreateThread(@BackgroundWork(),3)
lThreadHandle10=CreateThread(@BackgroundWork(),4)
lThreadHandle11=CreateThread(@BackgroundWork(),4)
lThreadHandle12=CreateThread(@BackgroundWork(),3)
lThreadHandle13=CreateThread(@BackgroundWork(),2)
lThreadHandle14=CreateThread(@BackgroundWork(),1)
lThreadHandle15=CreateThread(@BackgroundWork(),0)

WaitThread(lThreadHandle15)
Delay(100)

Debug #Empty$
Debug "Final"
Debug "-----"
Debug @xInfo()
Debug @xInfo(0)
Debug @xInfo(0)\String1
Debug xInfo(0)\String1
Debug @xInfo(0)\String2
Debug xInfo(0)\String2
Debug @xInfo(1)
Debug @xInfo(1)\String1
Debug xInfo(1)\String1
Debug @xInfo(1)\String2
Debug xInfo(1)\String2
Debug @xInfo(2)
Debug @xInfo(2)\String1
Debug xInfo(2)\String1
Debug @xInfo(2)\String2
Debug xInfo(2)\String2
Debug @xInfo(3)
Debug @xInfo(3)\String1
Debug xInfo(3)\String1
Debug @xInfo(3)\String2
Debug xInfo(3)\String2
Debug @xInfo(4)
Debug @xInfo(4)\String1
Debug xInfo(4)\String1
Debug @xInfo(4)\String2
Debug xInfo(4)\String2
sc4pb
User
User
Posts: 26
Joined: Tue Mar 07, 2023 5:33 pm

Re: CreateThread Procedure corrupting strings

Post by sc4pb »

To everyone, thanks for the feedback. Yes I've constructed a few similar tests and in my case also they pass. I can't simply post the project, too much customer specific code in it. The "DoSomeWork" as I was calling it was doing a lot of things, utilizing the network, database, crypto libraries. So that muddies the waters.

The odd thing was that when I discovered the problem, in the main program xInfo(x)\String1 had a value in it. This was passed to a threaded procedure call as @xInfo(x), and it was caught on the other side and accessed as *info\String1.

When the procedure started the value was there. As I stepped through the first five statements of the threaded procedure, the value was fine until the fourth step, where after the procedure call returned, the contents of String1 were lost.

But the procedure being called at that point was not accessing xInfo array in any way. It was performing an unrelated lookup of information from a database. As part of the process it was using a Threaded map. Since I'm testing with only one thread for now, I changed it to a global map, meaning the only difference would be that the map would not be instantiated and destroyed for the thread. And that one change stopped the value from being lost.

I've been away from this for a few days, but wanted to log in here and say thanks. Since posting this I have added mutexes around even read-only access to global maps, as it was pointed out there is only one "cursor." I'm still working on a test app to recreate the problem (rather like you were above) but so far no success. So clearly there are still mysteries, I'll keep at it.

Thanks again for the brainstorming
sc4pb
User
User
Posts: 26
Joined: Tue Mar 07, 2023 5:33 pm

Re: CreateThread Procedure corrupting strings

Post by sc4pb »

Awww nuts, stupid question time.

For PureBasic built in library procedures that return "handles" to things, you know databases, images, memory blocks whatever, what data type should I be using? I just realized as I was looking over the docs that I was mistaken about something.

While I'm at it I'll complain about this a bit too.

Many examples in docs utilize variables that are not declared, just generated on the fly. And if no type is specified, for some reason I thought the default type was "long" (4 bytes). Now I see the default type is "integer", which can be either 4 or 8 bytes depending on 32 or 64 bit processor.

Well ok, for all of my calls with handles, like AllocateMemory and OpenDatabase, I've been using longs. So...

Code: Select all

Define handle.l
handle=AllocateMemory(1000)
...but this is wrong, right? I should have been using integers? I bring this up because of another big gripe of mine:

I won't write a line of code without EnableExplicit. But in my mind EnableExplicit should require both a variable declaration AND a type declaration. So when explicit:

Code: Select all

Define foo ; this should be REJECTED by EnableExplicit, there is no type
Define foo.s ;this is fine, it's a string.
I've had a number of bugs that have been hard to track down because I failed to add a type to a variable declaration.

Is there way to require a type? Or in other words, to disable the default behavior of PureBasic assuming variables are integers?
jassing
Addict
Addict
Posts: 1745
Joined: Wed Feb 17, 2010 12:00 am

Re: CreateThread Procedure corrupting strings

Post by jassing »

(misinformation removed)
Last edited by jassing on Thu Jun 08, 2023 2:58 am, edited 1 time in total.
Post Reply