[SOLVED] Finding cause of a crashing application

Just starting out? Need help? Post your questions and find answers here.
Oso
Enthusiast
Enthusiast
Posts: 595
Joined: Wed Jul 20, 2022 10:09 am

[SOLVED] Finding cause of a crashing application

Post by Oso »

Hi all, I'm looking to find out if OnErrorCall() can help trap an elusive crash. A little background first. I have a server-side application communicating via SendNetworkString() and ReceiveNetworkData() with client-side processes. The server sends data to the client within threads, then ends when the data is sent. It works reliably and quickly except for one aspect. When stress-testing, mid-way through continuous output of a long data stream, sent from server to client, I close the client-side abruptly, to simulate a user ending the process in this way. I can see that the server process recognises this, because it triggers a #PB_NetworkEvent_Disconnect.

Despite this, it sometimes causes the server-side to crash. I use a proper method of checking the number of bytes sent, delaying and re-sending, shown in the line of code below. If further data cannot be sent from the server to the client, then I stop sending data and instead end the thread gracefully. The mechanism used for this is at viewtopic.php?p=328229#p328229

Code: Select all

result.i = SendNetworkString(clientid.i, senddata.s, #PB_Unicode) 
As you may have noticed, my method of stopping the sending of data to the client, is not triggered by #PB_NetworkEvent_Disconnect, because the sending of data is performed inside a separate thread, which doesn't check for that condition. Instead, the thread triggers on timeout, and stops sending further data.

The crash never happens when the debugger is being used, making it difficult for me to find the cause. It also doesn't appear to happen if I add a 1ms delay to the sending of data segments, but when I compile the server-side code and run it normally, then I sometimes see this problem. I see only the following.
"A problem caused the program to stop working correctly. Windows will close the program and notify you if a solution is available."

Should I be able to make use of OnErrorCall(@ErrorHandler()) that I saw on the forum a few days ago? Looking at the documentation, I'm not sure what kind of error it is intended to trap, as it only says "Changes the action taken if an error occurs ". Is it referring to this type of crash? Any suggestions appreciated.
Last edited by Oso on Fri Feb 03, 2023 10:27 am, edited 1 time in total.
infratec
Always Here
Always Here
Posts: 6817
Joined: Sun Sep 07, 2008 12:45 pm
Location: Germany

Re: Finding cause of a crashing application

Post by infratec »

First enable the Purifier an use

Code: Select all

PurifierGranularity(1, 1, 1, 1)
Maybe you have a memory problem.
Oso
Enthusiast
Enthusiast
Posts: 595
Joined: Wed Jul 20, 2022 10:09 am

Re: Finding cause of a crashing application

Post by Oso »

infratec wrote: Wed Feb 01, 2023 3:17 pm First enable the Purifier an use

Code: Select all

PurifierGranularity(1, 1, 1, 1)
Maybe you have a memory problem.
Thanks for this infratec, that seems to have found it...

[ERROR] Invalid memory access. (read error at address 10)

The following line of code was then highlighted in red.

Code: Select all

    result.i = SendNetworkString(clientid.i, senddata.s, #PB_Unicode) 
What appears to be happening, is that although the client has disconnected, the server's thread that is still running, is being caught out when it attempts to send its line of data to the client. Sometimes it's fine with that, but sometimes it fails.

Unless there is a way of getting PB to ignore that, I think I will need to trigger the thread to not send further data. That might not be easy, because there could be a small delay while a value is passed to the thread (perhaps via the memory structure).
User avatar
mk-soft
Always Here
Always Here
Posts: 5335
Joined: Fri May 12, 2006 6:51 pm
Location: Germany

Re: Finding cause of a crashing application

Post by mk-soft »

A server responds to a client request. A server should never send anything without a request. The clients must be managed in the server. If a client logs off, the server must also stop sending anything.
In the event of a physical failure of the client, it may be that no disconnect arrives at the server.
In this case, a timeout must be programmed for the client connection itself, which then disconnects the client from the administration and removes the data.

Otherwise it can happen, as in your case, that the server tries to use a resource that is no longer available.
My Projects ThreadToGUI / OOP-BaseClass / EventDesigner V3
PB v3.30 / v5.75 - OS Mac Mini OSX 10.xx - VM Window Pro / Linux Ubuntu
Downloads on my Webspace / OneDrive
Oso
Enthusiast
Enthusiast
Posts: 595
Joined: Wed Jul 20, 2022 10:09 am

Re: Finding cause of a crashing application

Post by Oso »

mk-soft wrote: Wed Feb 01, 2023 5:24 pm A server responds to a client request. A server should never send anything without a request. The clients must be managed in the server. If a client logs off, the server must also stop sending anything.
Thanks for the reply mk-soft and apologies for the detail in the below, but it's quite a tricky one to explain. The way the process works isn’t in that way. The clients connect to the server’s main process (at that point it’s not inside a thread) and then give an instruction or command to receive certain data. The server’s main process then starts a thread and passes the client’s instruction to that thread, along with the client id.

At that point, we have a new server thread that’s serving that particular client’s request. The server’s main process then goes back to wait for other client requests, leaving the thread to take care of the processing. When it has generated the data, it sends it to the client using SendNetworkString(clientid…) and deals with timeouts and delays successfully. This is all fine and the client receives the data. After the data is sent, the thread finishes gracefully.

What I’m testing for at the moment is against end-users who might close the client window (not logging off but just closing it), while the server is sending data to it. This is dealt with to some extent in the thread process, by code similar to Idle’s “retry and timeout” code at the link I included. It works well and if the sending is interrupted, it recognises that it hasn’t been successful for ‘n’ seconds and it signals to the thread to finish gracefully, which is working.

But what occasionally happens, is that the client connection is being closed so quickly that PB is losing the client resource, before my thread has an opportunity to recognise it. Maybe I have to find a way to get the disconnection status to the thread that’s handling that client’s request, to prevent it from even attempting to send anything. Or, what would be nice, is if there is a way in PB to get ‘isclient’ status, i.e. is the client still there? Not sure if that’s possible.
User avatar
mk-soft
Always Here
Always Here
Posts: 5335
Joined: Fri May 12, 2006 6:51 pm
Location: Germany

Re: Finding cause of a crashing application

Post by mk-soft »

This is sometimes difficult when the thead lasts longer and the connection to the client has been lost and unfortunately the resource no longer exists.
This has also happened to me.

Here is a solution that I have not yet fully tested.
With result NULL it should be ok

Code: Select all

#SOL_SOCKET = $FFFF
#SO_ERROR = $1007

Procedure GetSocketLastError(Socket)
  Protected error_code
  Protected error_code_size = SizeOf(error_code)
  
  r1 = getsockopt_(Socket, #SOL_SOCKET, #SO_ERROR, @error_code, @error_code_size)
  If r1 = 0
    r1 = error_code
  EndIf
  ProcedureReturn r1
EndProcedure

r1 = GetSocketLastError(ConnectionID(ClientID))
Last edited by mk-soft on Wed Feb 01, 2023 10:34 pm, edited 1 time in total.
My Projects ThreadToGUI / OOP-BaseClass / EventDesigner V3
PB v3.30 / v5.75 - OS Mac Mini OSX 10.xx - VM Window Pro / Linux Ubuntu
Downloads on my Webspace / OneDrive
Oso
Enthusiast
Enthusiast
Posts: 595
Joined: Wed Jul 20, 2022 10:09 am

Re: Finding cause of a crashing application

Post by Oso »

mk-soft wrote: Wed Feb 01, 2023 10:07 pm Here is a solution that I have not yet fully tested.
With result NULL it should be ok
This looks good mk-soft, thanks very much. I'll try this in the morning. I'll place the call after the attempt to send data, because I assume it needs to have first tried to send, in order to obtain the last error. I'll let you know how it goes :) thanks again.
User avatar
mk-soft
Always Here
Always Here
Posts: 5335
Joined: Fri May 12, 2006 6:51 pm
Location: Germany

Re: Finding cause of a crashing application

Post by mk-soft »

No,

Check before sending
My Projects ThreadToGUI / OOP-BaseClass / EventDesigner V3
PB v3.30 / v5.75 - OS Mac Mini OSX 10.xx - VM Window Pro / Linux Ubuntu
Downloads on my Webspace / OneDrive
Oso
Enthusiast
Enthusiast
Posts: 595
Joined: Wed Jul 20, 2022 10:09 am

Re: Finding cause of a crashing application

Post by Oso »

mk-soft wrote: Wed Feb 01, 2023 10:35 pm No, Check before sending
It works beautifully well mk-soft :D I've been testing it with multiple connections for a few hours, closing several client processes while they are receiving large sets of data. I can see it trigger error code r1 = 4294967295 every time. It's very robust. The main steps are as follows. Thanks for helping with this, it's appreciated.

Code: Select all

  Repeat
    r1 = GetSocketLastError(ConnectionID(ClientID))                     ; Check for socket error (client disconnection)
    If r1                                                               ; Non-zero value indicates error
      ; PrintN("Client " + Clientid.i + " - returned connection error (r1 = " + Str(r1) + ")")
      ProcedureReturn -1
    EndIf

    result.i = SendNetworkString(clientid.i, senddata.s, #PB_UTF8)
    .
    .
wayne-c
Enthusiast
Enthusiast
Posts: 335
Joined: Tue Jun 08, 2004 10:29 am
Location: Zurich, Switzerland

Re: Finding cause of a crashing application

Post by wayne-c »

@Fred maybe?

Should this check not be included in the SendNetworkString / SendNetworkData functions?
As you walk on by, Will you call my name? Or will you walk away?
Oso
Enthusiast
Enthusiast
Posts: 595
Joined: Wed Jul 20, 2022 10:09 am

Re: Finding cause of a crashing application

Post by Oso »

wayne-c wrote: Thu Feb 02, 2023 10:11 am @Fred maybe? Should this check not be included in the SendNetworkString / SendNetworkData functions?
It would certainly be good if the compiled code were to avoid crashing. Or a client/server status function would be nice.

When I was first trying to find out why my routine was crashing, I experimented with other possible causes, such as reading or writing to a file that has been closed already, because I suspected that might have been the reason. I found that the compiled executable ignores incorrect file handles. It seems that some problems don't affect the application's continued execution, but not for the network-related functions.
Post Reply