Exceptions and Pool

[ftrl]

In production, our application sometimes randomly starts to malfunction.
I've noticed that once a problem has occurred, the behavior is no longer stable.

Could it be that connections are being put back into the pool when an exception has occurred, leaving the connection in an incorrect state?
In the FirebirdClient sources, in the FbConnection Close[Async] functions, I see for example:

var broken = _innerConnection.Database.ConnectionBroken;
FbConnectionPoolManager.Instance.Release(_innerConnection, !broken);

But ConnectionBroken / IOFailed is only set to True on an IOException.
For example, in ReadAsync:

catch (IOException)
{
IOFailed = true;
throw;
}

Shouldn't other exceptions also be handled? In our case, it seems that OperationCanceledException is the one that occurs.

Thanks

[cincuranet]

Other exceptions should in general be OK, because the low level code can recover from it and then it is about what the developer is going to do next. But sending/receiving bytes should still work.

Do you have a repro?

[ftrl]

Alas no, impossible to reproduce.

The only thing that made me think of it was the following stackTrace:

StackTrace:
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
   at FirebirdSql.Data.Client.Managed.FirebirdNetworkHandlingWrapper.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadBytesAsync(Byte[] buffer, Int32 count, CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadInt32Async(CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.XdrReaderWriter.ReadOperationAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.Version11.GdsDatabase.ReadOperationAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.Version10.GdsDatabase.ReadSingleResponseAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.Version11.GdsDatabase.ProcessDeferredPacketsAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.Version11.GdsDatabase.ReadOperationAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.Version10.GdsDatabase.ReadSingleResponseAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.Version10.GdsDatabase.ReadResponseAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.Version10.GdsTransaction.BeginTransactionAsync(TransactionParameterBuffer tpb, CancellationToken cancellationToken)
   at FirebirdSql.Data.Client.Managed.Version10.GdsDatabase.BeginTransactionAsync(TransactionParameterBuffer tpb, CancellationToken cancellationToken)
   at FirebirdSql.Data.FirebirdClient.FbTransaction.BeginTransactionAsync(CancellationToken cancellationToken)
   at FirebirdSql.Data.FirebirdClient.FbConnectionInternal.BeginTransactionAsync(IsolationLevel level, String transactionName, CancellationToken cancellationToken)
   at LinqToDB.Async.AsyncFactory.WrapValue[TTransaction](ValueTask`1 transaction)
   at LinqToDB.Data.DataConnection.<>c.<<BeginTransactionAsync>b__1_1>d.MoveNext()
--- End of stack trace from previous location ---
   at LinqToDB.Data.DataConnection.TraceActionAsync[TContext,TResult](DataConnection dataConnection, TraceOperation traceOperation, Func`2 commandText, TContext context, Func`4 action, CancellationToken cancellationToken)
   at LinqToDB.Data.DataConnection.BeginTransactionAsync(CancellationToken cancellationToken)
   at DataAccess.TransactionManager.BeginTransactionAsync(CancellationToken token)
   at Business.Rentcar.Vehicules.VehiculesImpl.GetInfosAlertesAsync(Int32 id, DateTime dt, Boolean maj, CancellationToken token)
   at RentcarWeb.Api.Rentcar.Vehicules.VehiculesController.InfosAlerte(Int32 id, DateTime date, Boolean maj, CancellationToken token)
   at lambda_method1658(Closure , Object )

[cincuranet]

I don't see how this related to what you initially described... This looks like a regular exception when CancelationToken is used.

[ftrl]

Yes, but in this case, the connection is put back into the Pool.
And I thought it might be in an unstable state.

But if you don't think it's possible, okay.

[cincuranet]

Can you create repro that shows that the connection does not work after it is taken from the pool (and possibly the action that is going to happen is retried)? It is possible that there's a case that fails.

[ftrl]

Unfortunately, I can't see how to do this. Our application is very large and has many entry points (api). And errors occur without any apparent rule.
Firebird calls are made via an orm (linq2db).

I only get the exception globally, when it occurs (I no longer have the connection at that time).

But nevertheless, I've noticed that these problems we're experiencing are “recent” and concomitant with the systematic implementation of async functions with a CancellationToken. But here again, nothing is certain.

I know, it doesn't help!

[mrotteveel]

I can see how cancelling in the middle of an IO operation would leave the connection in an inconsistent state (what has been sent, what has been received?)

[cincuranet]

Ah right. Not every CancellationToken is registered and handled through server. I have to check the code, what was actually my idea in these cases. I no longer remember.

[ftrl]

Maybe I can just try to compile FirebirdClient (the version we are using) by replacing :

catch (IOException)
{
  IOFailed = true;
  throw;
}

by

catch
{
  IOFailed = true;
  throw;
}

and rapidly test the dll in production.