Ebury API Auth Login with Keycloak

This is a Request For Comments on a proposal to replace the login flow of ebury-api-auth with Keycloak to implement the 5 login attempts protection.

Scope

The scope of this document covers user authentication at the Ebury API (ebury-api-auth + ebury-api-webapp).

Background

The Ebury API implemented the OpenID Connect authorization code flow to give users access to the API. See more details at the Ebury API Auth docs.

This is not well maintained, it does not conform to the latest developments and recommendations of the OpenID Connect standards (e.g. use of asymmetric signature for tokens) and does not have the best security practices in mind (e.g. there is no throttling and no limitations of login attempts, insufficient/improper logging).

The Ebury API relevant parts for this RFC are:

  • Ebury API Auth: Handles authentication, token generation, contact impersonation and the equensWorldline integration (endpoints under /sca/).

  • Ebury API Webapp: Implements the actual API endpoints (e.g. list clients, etc). In this repository also lives the App proxy which does Authorization header validation using a lua module. Tokens generated by Keycloak will need to accepted by this proxy.

OpenID Connect client
A client is a piece of software that requests tokens either for authenticating a user or for accessing a resource (also often called a relying party or RP). A client must be registered with the OpenID Provider. Clients can be web applications, native mobile and desktop applications, etc.

The configured OpenID Connect clients of Ebury API Auth, among their id and secret, point to an specific backend which could be either our Ebury Partners BOS or the Ebury Mass Payments BOS. This means Ebury API Auth is serving users of both BOS instances and the Keycloak integration needs to take this into account.

The way users of the API authenticate against it is by doing the authorization code flow which requires the browser steps but then rely on the refresh tokens to gather new tokens programmatically. The implementation should take into account not only the initial access token generation but also the refresh mechanisms.

The login flow of Ebury API Auth does not only support 2FA in the general sense but also has some extra features built on top of it:

  • Some OpenID Connect clients can enforce 2FA (through the MANDATORY_2FA_CLIENT_IDS setting) and login will fail hard for users without MFA configured.

  • Some OpenID Connect clients can use fixed 2FA codes (this is only used by the Apple app submission process).

  • There's a 2FA Whitelist which allows some specific clients to skip 2FA requirements altogether.

  • But for the mentioned whitelist to work they must be part of the special MOBILE_CLIENT_ID.

The work proposed here focuses at Ebury API Auth, by ensuring the generated tokens are backwards compatible, all other validations at the Ebury API Webapp and proxies should remain working as usual.

Problem Description:

We want to Limit login attempts as noted by the PRD document. For this we'll start with the Ebury API, namely at the Ebury API Auth component.

We want to use the best industry standards available for all our services and avoid re-implementing authentication and authorization across Ebury.

Solution

Since Ebury API Auth implements OpenID Connect it is possible to put Keycloak in place as a drop-in replacement of the login flow and the token refresh mechanisms.

We will enable Keycloak brute force protection mechanism to limit failed login attempts.

Ebury API Auth

We will configure two realms in Keycloak, one for each BOS backends (Ebury Core and Ebury Mass Payments) Ebury API Auth talks to.

We will make requests to <ebury-api-auth>/authenticate redirect to <keycloak>/realms/<realm>/protocol/openid-connect/auth passing through the parameters to initiate the authorization code Browser flow.

Similarly, requests to <ebury-api-auth>/refresh endpoint will request refresh tokens to <keycloak>/realms/<realm>/protocol/openid-connect/token.

Ebury API Auth will decide the Keycloak realm based on the backend that the OpenID Client has.

For this to work we will duplicate/migrate all Ebury API Auth OpenID Connect configured clients to their equivalent representation on Keycloak.

In the initial phase, we'll keep the refresh token mechanism to detect whether the passed in token is a Keycloak based refresh token or a legacy one and we'll issue their refresh accordingly, meaning clients with legacy valid tokens will be able to keep refreshing them even after the initial Keycloak integration is on.

PoC references:

Keycloak

For this, because it's the first installment of Keycloak use for login of BOS contacts, we'll need to setup the initial custom user federation and 2FA flows through Verify that will then be available for use with other services like Ebury Online.

We'll also need to implement a custom ID token signature algorithm so signature uses the client secret and Keycloak access token response remains backwards compatible.

In addition, Keycloak needs to support both BOS backends (Ebury Core and Ebury Mass Payments), for this we'll use different realms, one for Ebury Mass Payments (emp) BOS and the other for Ebury Core BOS (ebp).

Additionally a new Ebury API Auth lookalike login theme will be available and set for these realms.

User federation

The users that log into the system (AKA contacts, AKA ClientUser in BOS models), live in the BOS database. Eventually, we want to move them into Keycloak but we'll need to make other services (e.g. EBO) migrate to Keycloak first before doing that.

So the plan is then to implement a custom Keycloak provider to do user authentication federated through BOS. For this Keycloak requires a set of endpoints to implement a custom UserStorageProvider that will query users against BOS instead of storing them locally. The endpoints will follow recommendations of the external Keycloak user migration plugin, the reason for this is to have the possibility of switching off at any time once we start to move users to the actual Keycloak database.

PoC reference: https://github.com/Ebury/ebury-keycloak/tree/FFF-2628-api-auth-poc-with-mfa/providers/ebury-keycloak-providers/src/main/java/com/ebury/keycloak/federation/bos

BOS Endpoints

All endpoints will use the current BOS authentication mechanism which requires users to start a session at <bos>/api/v1.0/session with client ID and secret and then pass the returned token like so in the authorization header Authorization: Token <token>. We would need BOS operations to set up an API client for the Keycloak integration. Keycloak custom code will take care of negotiating such token and making requests authenticated as required.

Another point worth clarifying is that Juan Carlos Gomez has confirmed users (even if present in different brands) have unique emails. Besides the BOS database does not enforce this constraint, most places where BOS works with contacts it enforces the presence of a single ClientUser with matching email. This will be key to the strategy of these endpoints as they will not be brand specific. Branding will be an attribute of the OpenID clients and the claims of generated tokens will reflect that (e.g. list of BOS clients, brand code). More details on this later.

The proposed endpoints are as follows:

GET users by email or AML ID
GET /api/v1.0/keycloak/contact/<email or aml_id>/
Authorization: Token :token

### Success Response

HTTP/1.1 200 OK
Allow: POST, OPTIONS, GET
Content-Type: application/json

{
  "username": "EBPCON00005",
  "groups": [],
  "id": "EBPCON00005",
  "firstName": "Ebury",
  "roles": [],
  "emailVerified": true,
  "lastName": "Demo 2",
  "enabled": true,
  "requiredActions": [],
  "attributes": {
    "language": [
      "en"
    ],
    "locale": [
      "en"
    ],
    "brandCodes": [
      "EBP"
    ],
    "timeZone": [
      "Europe/London"
    ]
  },
  "email": "eburydemo2@ebury.com"
}

### Failure Response

HTTP/1.1 404 NOT FOUND
Allow: POST, OPTIONS, GET
Content-Type: application/json

{}

PoC reference: https://github.com/Ebury/bos/blob/FFF-2628-api-auth-poc-with-mfa/api/views/keycloak.py#L58

POST Check user credentials
POST /api/v1.0/keycloak/contact/<user_email>/
Authorization: Token :token
Content-Type: application/json

{ "password": "user password" }

### Success Response

HTTP/1.1 200 OK
Allow: POST, OPTIONS, GET
Content-Type: application/json

{}

### Failure Response

HTTP/1.1 401 UNAUTHORIZED
Allow: POST, OPTIONS, GET
Content-Type: application/json

{}

PoC reference: https://github.com/Ebury/bos/blob/FFF-2628-api-auth-poc-with-mfa/api/views/keycloak.py#L166

2FA

We will implement 2FA at Keycloak with the Verify service as backend. This will integrate some of the features found at Ebury API Auth but with some simplifications as to make this implementation easier to understand, more secure, and potentially re-usable for integration with other services (e.g. EBO).

This integration will support:

  • Configuring clients with 2FA as mandatory (i.e. fail loudly if a user tried to login without a MFA device configured at verify).

  • Configuring a client with a fixed MFA code (only used for Apple app submission).

The main differences with Ebury API Auth approach are:

  • There will be no special ids to whitelist/skip MFA, clients need to be explicitly configured.

  • MFA is optional by default, under this scheme if a user has no MFA configured and tries to login it will go through without 2FA.

  • If a client enforces MFA, a user with no MFA configured will not be able to login.

  • There will be no "whitelisted" users, we want to avoid special cases as much as possible. This is currently not used in production.

PoC reference: https://github.com/Ebury/ebury-keycloak/blob/FFF-2628-api-auth-poc-with-mfa/providers/ebury-keycloak-providers/src/main/java/com/ebury/keycloak/authentication/authenticators/browser/VerifyOTPFormAuthenticator.java

OpenID Connect client configuration

Client's brand will live as a hardcoded claim protocol mapper exposed under the brand key. This will mean tokens created for this client will also have the brand embedded.

Client settings to control MFA behavior will live as attributes of the OpenID Connect client.

attribute type default
ebury.mfa.enforce bool false
ebury.mfa.fixed string

If ebury.mfa.enforce is present and true then login will require users to have MFA configured for login and will fail loudly if users have no MFA devices configured in Verify.

If ebury.mfa.fixed is present, then users will need to input exactly this string for MFA challenge. Ideally this would be a Verify method that's bound to an specific brand but we are going to follow Ebury API Auth approach for simplicity and handle this corner-case on the MFA verification logic of Keycloak itself.

PoC references:

BOS Endpoints

BOS stores Verify credentials, such data is brand specific. Since OpenID clients configured at Keycloak will bind against a brand through its brand code (e.g. EBP), we will require a simple endpoint that allows retrieving Verify credentials from a brand code.

Just the same as with the federation endpoints, this endpoint will use the current BOS API authentication mechanism.

GET Retrieve brand data by code

It will use the existing BOS get_brand_info serializer function to return the data. It will not encrypt the credentials with AES using the API_PRIVATE_KEY as it's found in a couple of places at the BOS API. The reasons for this are:

  • The encryption algorithm is custom based on an unmaintained python package and will not be trivial to re-implement in Java.

  • The algorithm uses PBKDF so that decryption is slow, this is for security reasons but make clients of the API pay a high price to query data. Got to say that I find it ironic we (unnecessarily) encrypt authenticated data responses better than user passwords.

  • The endpoint is behind API authentication requirements.

  • The endpoint will receive the requests through HTTPS, so only the requester will be able to see this data (i.e. the custom homemade encryption is not needed).

GET http://bos.localhost/api/v1.0/keycloak/brand/EBP/
Authorization: Token :token

### Success Response

HTTP/1.1 200 OK
Allow: OPTIONS, GET
Content-Type: application/json

{
  "website": "https://www.ebury.com",
  "code": "EBP",
  "name": "Ebury",
  "verify_password": "<password>",
  "verify_username": "<username>",
  "phone": "+44 (0) 207 197 2421",
  "master": true,
  "full_name": "Ebury Partners UK Ltd",
  "address": "3rd floor, 100 Victoria Street, Cardinal Place, London, SW1E 5JL, United Kingdom",
  "address_line_2": "London, SW1E 5JL, United Kingdom",
  "email": "info@ebury.com",
  "address_line_1": "3rd floor, 100 Victoria Street, Cardinal Place"
}

### Failure Response

HTTP/1.1 404 NOT FOUND
Allow: OPTIONS, GET
Content-Type: application/json

{}

PoC reference: https://github.com/Ebury/bos/blob/FFF-2628-api-auth-poc-with-mfa/api/views/keycloak.py#L187

Backwards compatible tokens

For Keycloak tokens to remain compatible, the ID Token signature will need to be HS256 with the key being the client secret of each OpenID Connect client. In order to achieve that, custom implementations of MacSecretSignatureProvider and ServerMacSignatureSignerContext will exist. Each Ebury API configured OpenID Connect client will set the ID token signature to use the custom providers (i.e. this setting is not Realm global) which means that for the future we could create new clients using the recommended signature algorithms and eventually migrate API customers to use ID tokens signed with the now recommended RS256 method.

For more info on HS256 and RS256 signature algorithms see: https://auth0.com/blog/rs256-vs-hs256-whats-the-difference/

It's important to note that Keycloak does have a built-in HS256 signature method but it uses a Realm generated key by default. This means our custom implementation will not be implementing any cryptography itself, it will just re-use Keycloak code and just select the OpenID Client secret as key instead.

The following document shows how the legacy and Keycloak access token responses look: https://docs.google.com/document/d/1IcX3p2m-M5NLcEvoZiXuPUmYqPqupDMgmzUzdPEoRlQ/edit#heading=h.pcpzw1xw8ucv

Claims

These are the required claims we'll map from the "legacy" Ebury API Auth tokens:

  • sub: will have the AML ID of the contact as the value.
  • clients: will have a list of BOS Client, with their ID and name.

Then we have an extra set of required claims that would help to avoid extra BOS queries at Ebury API Auth:

  • email: email address of the logged in contact.
  • entity ids: injected into the clients claim.

The reason to include these is to avoid extra queries to BOS when Ebury API Auth decodes the Keycloak token and stores it as a BearerToken representation but it will also open up the chance for the future to make the lua checks at the authenticating proxy of the Ebury API to use the access token itself (as it will be a JWT instead of a plain string) to make the checks and the Redis database would not used anymore for these.

PoC references:

The clients claim

To implement this claim, we'll implement a Script Mapper at Keycloak that will query the contact's client at BOS and represent them as they are today.

The BOS request will go to the existing <bos>/api/v1.0/keycloak/contact endpoint and the Script Mapper will handle the response and expose it the clients claim like so:

{
...
    "clients": [
        {
            "client_id": "EBPCLI00004",
            "client_name": "Ebury Demo",
            "client_entity": "b5adf317-35ea-447c-b547-758099919ce2"
        },
        {
            "client_id": "EBPCLI00007",
            "client_name": "Ebury Demo 2",
            "client_entity": "b5adf317-35ea-447c-b547-758099919ce2"
        }
    ]
}

The only addition to the former's clients claim is the client_entity but this is for convenience as it's needed in other places and will help to avoid extra queries to BOS.

PoC reference: https://github.com/Ebury/ebury-keycloak/blob/FFF-2628-api-auth-poc-with-mfa/providers/ebo-brand-scripts/bosClientsMapper.js

5 Login Attempts protection

Keycloak comes with a 5 login attempts protection mechanism built-in. We will enable this for all Realms. It will block users after the max attempts reaches and the operations team will need to unblock users as they do today after a max number of 2FA challenges fail.

For this to work, our user federation plugin will need to support marking a contact as blocked. Fortunately the endpoint to do so already exists in BOS at <bos>/api/v1.0/contact/change-status/.

Our UserAdapter will implement the setEnabled method by hitting the endpoint mentioned above.

PoC reference: https://github.com/Ebury/ebury-keycloak/blob/FFF-2628-api-auth-poc-with-mfa/providers/ebury-keycloak-providers/src/main/java/com/ebury/keycloak/federation/bos/UserAdapter.java#L101

Keycloak Brute Force protection configuration will be as follows: - Max login failures: 5 - Permanent Lockout: true - Quick login check milliseconds: 1000 - Minimum quick login wait: 60 seconds

PoC reference: https://github.com/Ebury/ebury-keycloak/blob/FFF-2628-api-auth-poc-with-mfa/keycloak-config/tf/contacts.tf#L31

Theme

We'll replicate the Ebury API Auth login theme and set it as the login theme for Keycloak. Both realms of this integration will use it.

PoC reference: https://github.com/Ebury/ebury-keycloak/tree/FFF-2628-api-auth-poc-with-mfa/themes/custom/login

OpenID Connect client migration

Keycloak will duplicate all clients present in Ebury API Auth via Terraform configuration. It'll mark secrets as sensitive and for the initial setup they'll come from an external source (either Vault or directly from the Ebury API Auth Redis DB if possible by infrastructure).

PoC reference: https://github.com/Ebury/ebury-keycloak/blob/FFF-2628-api-auth-poc-with-mfa/keycloak-config/tf/contacts.tf#L81

Caveats

  • We are not yet migrating contacts to Keycloak as other systems (e.g. EBO) depend on BOS.
  • Keycloak will still use Verify in the background as other systems depend on it.
  • Access tokens are longer (used to be 30 chars long) as now are JWT instead of plain strings. This means clients should be aware in case they are storing access tokens in a restricted storage.

Operation

  • We will migrate existing Ebury API Auth clients into Keycloak clients.
  • Ebury API Auth will depend on the KEYCLOAK_BASE_URL setting to enable the Keycloak integration.
  • Keycloak extensions and realms configuration can happen incrementally without downtime or affecting other realms (e.g. operations).

Security Impact

  • Increased security of login as it will block users after 5 failed attempts.
  • Increased quality of logging and auditing for authentication.
  • Unified login and MFA logic into the authorization server (Keycloak) that can be re-used in all other projects that require to log BOS contacts.
  • Migration path to recommended asymmetric signature algorithms for generated tokens.

Performance Impact

This implementation allows in the future to make the Redis database redundant by replacing the checks to use the signed data from the JWT access token.

Developer Impact

  • Developers will not have to implement brute force protection and detection in services that require BOS contacts to login.
  • Developers will be able to re-use login and MFA in other services that need to authenticate BOS contacts.
  • Developers will run ebury-keycloak when working on Ebury API Auth. The Readme will include updated instructions to do so.

Deployment

  • Terraform will manage Keycloak configuration for the new realms.
  • Terraform will manage migrated Ebury API Auth OpenID Clients into Keycloak.
  • Custom Keycloak extensions will deploy as part of the service image.
  • Vault will store BOS API secrets for each realm.
  • Once all is in place, we'll set KEYCLOAK_BASE_URL in Ebury API Auth to enable the integration.

Dependencies

Keycloak moved to the production publicfrontal cluster.

Future opportunities

This integration will be re-usable for other services and is the starting point to open new product offerings like:

  • Single Sign-On experience for all Ebury offerings.
  • Support for federated login via Google (and others like Facebook, Twitter, Github, LinkedIn, Microsoft and more).
  • Support for login with other devices/methods like mTLS, Security Hardware Keys (e.g. Yubikeys) or Passkeys.

All of these are available right away with simple configuration.

Reference Documents

Everything proposed here reflects in the PoC done as part of the research. Links to relevant pieces of code added in context.

Other documents of interest: