Duplicated Movements in Async Processes

BOS Duplicated Movements solution for asynchronous processes is not applied.

It is mandatory to find the way to skip this kind of behavior and implement a solution that prevents BOS to fail again given these circumstances.

Problem Description

BOS asynchronous processes are not protected against multiple objects' creation, this is causing the violation of several constraints that impact the balances. The synchronous processes are managed by the MovementChain but this context is not sent to the asynchronous processes.

Example of duplicated movement protection:

Duplicated movement protection sending the funds from a client to a trade (affb.clients.client.client_client.ClientClient.move_funds_from_client_to_trade):

money = Money(amount, ccy_symbol)
typed_deal = TradesFacade.get_trade_from_reference(trade_reference)
last_movement_chain_id = MovementChainFacade.get_last_movement_chain_id(typed_deal.client, typed_deal)
movement_chain_context = MovementChainContext((typed_deal.client, typed_deal, last_movement_chain_id))

assets_ids = MoneyWorkflowFacade.funds_from_client_to_trade(
    client_reference,
    trade_reference,
    money,
    user,
    movement_chain_context=movement_chain_context
)

The movement_chain_context is the information required to avoid the duplication and this information is not being sent to any asynchronous process. So this context is not used and it is resulting in the possibility to create duplicate movements in the system.

Example of asynchronous process:

asynchronous process create duplicate in asynchronous process

The second process creates a new movement_chain_context because as the first process has finished when the new context is created the changes of the first one are already in Data Base, the movement chain protection can not detect the duplicate.

Background

The Movement Chain is a way to protect the duplicates, it works as a chain, so at the beginning of the transaction BOS check the last link (for that origin and destination of the movement) and previously to create the movement, it creates the next link pointing to the previous one. As each link can be connected with only two links (the previous and the next one) if two transactions try to create a link pointing the same previous one, the last transaction will raise an error.

For more information please check the links below:

Solution

A way to avoid the duplication in synchronous processes is calculating the MovementChain.

The proposal is to add the movement_chain_context to the asynchronous processes and has to be provided by the "original" process.

Both processes using the same MovementChainContext

In case that both processes (synchronous and asynchronous) are using the movement_chain_context, the asynchronous process will fail unless the synchronous process commit in database before the asynchronous process starts because it will be use information that is not in database.

This should not happens because that means there is a dependency in the movement funds creation between both process and they should be not split in synchronous and asynchronous.

So to avoid this potential issue, the movement_chain_context needs to be flagged as updated when the context has been updated. This should affect to all workflows where the movement_chain_context is updated but this flag will be check only in asynchronous process, this flag should be by movement chain inside the context, to avoid overprotection.

For example, if the movement chain context has two movement chains (Trade->Client, Client->Payment) and one of them is used (Trade->Client), the flag should reflect the update only for this movement chain. In other case, we can have undesirable behavior in the async process because we have used the other movement chain of the context. In this case, the async process could use the (Client->Payment) chain and get an already used because the context is updated, but it is not true, it was the (Trade->Client) chain usage in the sync process.

same context in both processes

In Synchronous Processes

They should provide the movement_chain_context to the asynchronous processes called in the transaction. Also this context needs to be created as near to the beginning of the transaction as possible or received in the request.

In Asynchronous Processes

They need to be ready to manage a movement_chain_context in the case they are provided in the calling process, if not, the movement chain context need to be created. And provide this context in case of another asynchronous process will be called.

Also, it should check if the movement_chain_context has been updated (with the flag previously mentioned), in that case the actions if it is flagged are:

  • Ignore the movement_chain_context, this workflow will be not protected but it avoid integrity error.
  • Raise a notification informing that this workflow needs to be analysed with the goal of move all funds movement creation to the same process (synchronous or asynchronous).

The solution for the example:

proposal duplicated movement in asynchronous process

Positive points

  • The main logic of the process is already implemented.
  • Easy to implement.
  • Easy to maintain.
  • No concurrency control involved.

Negative points

  • Requires the identification and modification of every asynchronous task that is involved in the funds (internal or external) creation.
  • Hard to deal with the exceptions generated by the duplicate chain protection: Should be retried after N seconds?
  • This solution prevent the duplicated movements in great measure but it is not 100% reliable.

Alternatives

The MovementChain can be calculated at the beginning of the asynchronous process but as near is the MovementChain creation of the Object creation in database, greater will be the risk.

Positive points

  • The main logic of the process is already implemented.
  • Easy to implement.
  • Easy to maintain.
  • No concurrency control involved.
  • There is not change the parameters of the asynchronous processes.
  • It is already implemented in some asynchronous processes.

Negative points

  • Requires the identification and modification of every asynchronous task that is involved in the funds (internal or external) creation.
  • Hard to deal with the exceptions generated by the duplicate chain protection: Should be retried after N seconds?
  • The prevention of the duplicated movements decreases in great measure because the movement_chain_context could be calculated once the same funds are be already created by another process.
  • It is already implemented in Authorise Funds process and a duplicated movement was created.

Caveats

During the release process, updating the parameters of asynchronous tasks could affect to the tasks already queued that has not these new parameters, so in the development process we have to be aware that the process should be backward compatible with the previous version.

Operation

BOS users will not be affected, because this will affect to asynchronous process.

Security Impact

N/A

Performance Impact

The final user should not notice any performance impact as the only affected will be asynchronous task, but in these tasks:

  • Performance will be affected slightly because the new table will be queried every time a new movement is created.
  • Performance issues as a consequence of an increasing number of locks in the new table.

Developer Impact

  • Every time we implement a new asynchronous functionality involving a movement, we have to be aware of using the system controlling duplications.
  • Every time we implement the solution updating an asynchronous process, we have to be aware that the process should be backward compatible with the previous version to avoid affecting the releases.
  • Developer must be aware of the rollback in the cases covered by this new mechanism.

Data Consumer Impact

N/A

Deployment

Current BOS releases process.

Dependencies

RFC Duplicated movements should be implemented before this solution.

References