Reconcile payment status on Adyen webhook + accept externally tokenised cards

README.md

@@ -26,7 +26,10 @@ Version `0.8.0` of the plugin uses Adyen classic integration while version `0.9.
 
 ## Requirements
 
-The plugin needs a database. The latest version of the schema can be found [here](https://github.com/killbill/killbill-adyen-plugin/tree/master/src/main/resources).
+The plugin needs a database and does not create its tables automatically. Before starting the plugin for the first time, run the respective DDL on the Kill Bill database:
+
+* MySQL: run [ddl.sql](src/main/resources/ddl.sql).
+* PostgreSQL: run [ddl-postgresql.sql](src/main/resources/ddl-postgresql.sql) first to create the required domains, then run [ddl.sql](src/main/resources/ddl.sql) to create the tables.
 
 ## Build
 
@@ -80,13 +83,28 @@ org.killbill.billing.plugin.adyen.username=xxx ' \
 
 Where:
 * apiKey: API Key generated at step 3
-* returnUrl: URL set in Step 4c above. Typically `http://<KillBill_URL>/plugins/adyen-plugin/notification`
+* returnUrl: URL the shopper is sent back to after the Drop-in payment (passed to Adyen's `/sessions` as `returnUrl`). This is your application's URL, not the notification URL
 * merchantAccount: Merchant account created at step 2
 * hcmaKey: HMAC Key generated at step 4a
-* captureDelayHours: Desire capture delay in hours after Authorize , number must be between 0 - 168 hr
-* environment: Environment to use. Possible values are `TEST`/`LIVE`. default value is `TEST`
-* password: Password set at step 4b
-* username: Username set at step 4b 
+* captureDelayHours: desired capture delay in hours after authorisation, between 0 and 168
+* enviroment (spelled this way in the property name): environment to use. Possible values are `TEST`/`LIVE`. The default value is `TEST`
+* username / password: a Kill Bill user login (for example `admin` / `password`). The `/checkout` servlet uses these to authenticate against Kill Bill when it creates the first payment. They are not the webhook Basic Authentication credentials from step 4b; the plugin does not check those
+
+## Payment status after an Adyen notification
+
+Payments and refunds are recorded as `PENDING` when they are sent to Adyen. When Adyen's notification arrives (for example `AUTHORISATION` or `REFUND`), the plugin:
+
+1. Validates the HMAC signature and updates the matching transaction in its own tables.
+2. Stores the recurring token on the payment method, if Adyen sent one.
+3. For a successful notification, asks Kill Bill core to reconcile the payment right away, so the transaction moves from `PENDING` to `SUCCESS` without waiting for the Janitor.
+
+Failed notifications, and successful ones whose immediate reconciliation fails (the error is logged), are handled by the Kill Bill Janitor. Leaving failures to the Janitor keeps the payment retry schedule of invoice payments. By default the Janitor checks `PENDING` transactions 1 hour after the payment and again 1 day after that. The delays can be shortened with the `org.killbill.payment.janitor.pending.retries` property, for example:
+
+```
+org.killbill.payment.janitor.pending.retries=5m,15m,1h,1d
+```
+
+See the [payment user guide](https://docs.killbill.io/latest/userguide_payment.html) for details.
 
 ## Testing
 
@@ -95,19 +113,19 @@ Where:
 
 Since Adyen uses webhooks, the `http://<IP_ADDR>/plugins/adyen-plugin/notification` URL needs to be publicly accessible for Adyen to send notifications to it. Thus, testing can be done in one of the following ways:
 
-* **Using AWS**: Test on Kill Bill running on an AWS instance. In this case, you would need to specify the AWS URL (`http://<AWS_INSTANCE_ADDR>:8080/plugins/adyen-plugin/notification`) as the webhook server URL in Adyen and configure it as the `returnURL` in the Adyen plugin as explained above.
+* **Using AWS**: Test on Kill Bill running on an AWS instance. In this case, you would need to specify the AWS URL (`http://<AWS_INSTANCE_ADDR>:8080/plugins/adyen-plugin/notification`) as the webhook server URL in Adyen.
 * **Using ngrok**: If you would like to test on a local installation instead of AWS, you can use a tool like [ngrok](https://ngrok.com/) to create a temporary public DNS and redirect traffic from that DNS to the local Kill Bill server. Ngrok can be set up as follows:
     * Install ngrok from [here](https://ngrok.com/download). 
     * Open a terminal window and type `ngrok http 8080` (Since Kill Bill listens on port `8080`)
     * This will display a message like `Forwarding https://39dc-117-195-30-48.in.ngrok.io -> http://localhost:8080`
-    * Copy the URL displayed above (`https://39dc-117-195-30-48.in.ngrok.io`).  In this case, you would need to specify the ngrok URL (`<your-ngrok-url>/plugins/adyen-plugin/notification`) as the webhook server URL in Adyen and configure it as the `returnURL` in the Adyen plugin as explained above.
+    * Copy the URL displayed above (`https://39dc-117-195-30-48.in.ngrok.io`).  In this case, you would need to specify the ngrok URL (`<your-ngrok-url>/plugins/adyen-plugin/notification`) as the webhook server URL in Adyen.
 
 ### Testing
-1. Build, install and configure the plugin as explained above. Note that either your AWS URL or ngrok URL needs to be specified as the webhook server URL in Adyen and configured as the plugin `returnUrl`.
+1. Build, install and configure the plugin as explained above. Note that either your AWS URL or ngrok URL needs to be specified as the webhook server URL in Adyen.
 2. Test the application using the [Kill Bill Adyen Demo](https://github.com/killbill/killbill-adyen-demo) as explained [here](https://github.com/killbill/killbill-adyen-demo/#test).
 3. Verify that a new account is created in Kill Bill with a payment in `PENDING` status.
-4. If all goes well, Adyen would then send a notification to convert the `PENDING` payment status to `SUCCESS`. 
-5. Retrieve the payment using the `withPluginInfo` parameter to trigger the [Janitor](https://docs.killbill.io/latest/userguide_payment.html#_on_the_fly_janitor) as follows:
+4. If all goes well, Adyen would then send a notification and the plugin converts the `PENDING` payment status to `SUCCESS` in Kill Bill. 
+5. Retrieve the payment as follows. `withPluginInfo=true` also triggers the [on-the-fly Janitor](https://docs.killbill.io/latest/userguide_payment.html#_on_the_fly_janitor), which updates the payment status in Kill Bill if the plugin has processed the notification but the immediate update in Kill Bill failed. It does not query Adyen: if the notification never reached the plugin, the payment stays `PENDING` until Adyen delivers it again (Adyen retries webhooks it could not deliver, and they can be resent from the Customer Area):
 
 ```
 curl \
@@ -117,7 +135,7 @@ curl \
     -H 'Accept: application/json' \
     'http://127.0.0.1:8080/1.0/kb/payments/<paymentId>?withPluginInfo=true' 
 ```
-5. Verify that the payment status is converted to `SUCCESS`.
+6. Verify that the payment status is `SUCCESS`.
 
 ## Plugin Internals
 
@@ -131,7 +149,7 @@ The following steps need to be followed in order to use the Adyen plugin:
 
 2. [Create](https://killbill.github.io/slate/?shell#account-create-an-account) a Kill Bill account and save the `accountId` for further use.
 
-3. Create a Kill Bill Payment Method (Specify a `PluginProperty` corresponding to `enableRecurring` if this is going to be a recurring payment. The default value of this property is `false`):
+3. Create a Kill Bill Payment Method. Specify a `PluginProperty` corresponding to `enableRecurring` if this is going to be a recurring payment (the default value of this property is `false`). Add the `isDefault=true` query parameter if Kill Bill should use this payment method to pay invoices automatically:
 
 ```bash
 curl -v \
@@ -145,11 +163,9 @@ curl -v \
     -H "X-Killbill-Reason: demo" \
     -H "X-Killbill-Comment: demo" \
     -d '{
-  			"accountId": "2ad52f53-85ae-408a-9879-32a7e59dd03d",
+  			"accountId": "8785164f-b5d7-4da1-9495-33f5105e8d80",
   			"pluginName": "adyen-plugin",
-  			"isDefault": true,
   			"pluginInfo": {
-    			"isDefaultPaymentMethod": true,
     			"properties": [
       				{
         				"key": "enableRecurring",
@@ -159,9 +175,9 @@ curl -v \
     			]
   			}
 		}' \
-    	"http://127.0.0.1:8080/1.0/kb/accounts/8785164f-b5d7-4da1-9495-33f5105e8d80/paymentMethods" 
+    	"http://127.0.0.1:8080/1.0/kb/accounts/8785164f-b5d7-4da1-9495-33f5105e8d80/paymentMethods?isDefault=true" 
 ```
-4. Call `/plugins/adyen-plugin/checkout` to generate a session and create the first payment (Note that the amount needs needs to be specified in [minor units](https://docs.adyen.com/development-resources/currency-codes)):
+4. Call `/plugins/adyen-plugin/checkout` to generate a session and create the first payment (Note that the amount needs to be specified in [minor units](https://docs.adyen.com/development-resources/currency-codes)):
 
 ```bash
 curl -v \
@@ -179,12 +195,36 @@ curl -v \
 
 Note that this creates a payment in Kill Bill for the specified amount in `PENDING` status. 
 
-4. Set up a drop-in with the `sessionId` and `sessionData` obtained above as explained [here](https://docs.adyen.com/online-payments/web-drop-in#set-up).
+5. Set up a drop-in with the `sessionId` and `sessionData` obtained above as explained [here](https://docs.adyen.com/online-payments/web-drop-in#set-up).
+
+6. Collect customer's payment details via the drop-in. 
+
+7. If the payment is successful, Adyen sends a notification to Kill Bill; the plugin stores the recurring token (if `enableRecurring` was set) and moves the payment from `PENDING` to `SUCCESS`. Payments initiated by Kill Bill afterwards (for example for subscription invoices) use the stored token automatically, provided the payment method is the account's default (see step 3).
 
-5. Collect customer's payment details via the drop-in. 
+## Payment methods tokenised outside the plugin
 
-6. If the payment is successful, Adyen sends a notification to Kill Bill to convert the `PENDING` status to `SUCCESS`.
+If the card is tokenised in your own Adyen `/sessions` or `/payments` call (for example before the
+Kill Bill account exists), pass the token when creating the Kill Bill payment method:
+
+* `recurringDetailReference`: the Adyen token (`storedPaymentMethodId`)
+* `shopperReference`: the `shopperReference` used when Adyen stored the token
+
+```json
+"pluginInfo": {
+  "properties": [
+    { "key": "recurringDetailReference", "value": "K4C5QX3VQL9C5G65" },
+    { "key": "shopperReference", "value": "customer-42" }
+  ]
+}
+```
 
+The payment method is marked as recurring and payments initiated by Kill Bill (for example for
+subscription invoices) use the token right away; no first payment through `/checkout` is needed.
+To have Kill Bill pay invoices with it automatically, create it as the account's default payment
+method (`POST /1.0/kb/accounts/{accountId}/paymentMethods?isDefault=true`).
+Adyen only accepts a token together with the `shopperReference` it was stored under: without
+`shopperReference`, the plugin sends the Kill Bill account id and Adyen refuses the payment
+(`800 Contract not found`).
 
 
 ## Credits

pom.xml

@@ -321,7 +321,7 @@
         <dependency>
             <groupId>org.projectlombok</groupId>
             <artifactId>lombok</artifactId>
-            <version>1.18.22</version>
+            <version>1.18.34</version>
             <scope>provided</scope>
         </dependency>
 
@@ -361,7 +361,7 @@
                         <path>
                             <groupId>org.projectlombok</groupId>
                             <artifactId>lombok</artifactId>
-                            <version>1.18.22</version>
+                            <version>1.18.34</version>
                         </path>
                     </annotationProcessorPaths>
                 </configuration>

src/main/java/org/killbill/billing/plugin/adyen/api/AdyenPaymentPluginApi.java

@@ -64,6 +64,8 @@
 import org.killbill.clock.Clock;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
+import com.google.common.annotations.VisibleForTesting;
+
 
 public class AdyenPaymentPluginApi
     extends PluginPaymentPluginApi<
@@ -74,6 +76,10 @@ public class AdyenPaymentPluginApi
   public static final String SESSION_DATA = "sessionData";
   public static final String RECURRING_DATA = "recurring.recurringDetailReference";
   public static final String ENABLE_RECURRING = "enableRecurring";
+  // Token and shopper reference of a payment method tokenised outside the plugin (for example in
+  // the merchant's own Adyen /sessions call), passed as plugin properties on addPaymentMethod.
+  public static final String RECURRING_DETAIL_REFERENCE = "recurringDetailReference";
+  public static final String SHOPPER_REFERENCE = "shopperReference";
   protected static final ObjectMapper objectMapper = new ObjectMapper();
   private final AdyenConfigurationHandler adyenConfigurationHandler;
   private final AdyenDao adyenDao;
@@ -209,20 +215,28 @@ public void addPaymentMethod(
       throw new PaymentMethodException(
           "Missing one or more configuration properties (HMAC KEY/ Api Key / Merchant Account / Return URL) ");
     }
-    if (mergedProperties.get(ENABLE_RECURRING) != null
-        && mergedProperties.get(ENABLE_RECURRING).equals("true")) {
+    final String recurringDetailReference = mergedProperties.get(RECURRING_DETAIL_REFERENCE);
+    final boolean hasToken =
+        recurringDetailReference != null && !recurringDetailReference.trim().isEmpty();
+    if (hasToken
+        || (mergedProperties.get(ENABLE_RECURRING) != null
+            && mergedProperties.get(ENABLE_RECURRING).equals("true"))) {
       recurring = true;
     } else {
       recurring = false;
     }
     try {
+      // A payment method tokenised outside the plugin gets its token in the same insert, instead
+      // of waiting for an AUTHORISATION notification of a plugin-initiated payment. A single
+      // write means a failed creation leaves no row behind, so it can be retried with the same id.
       this.adyenDao.addPaymentMethod(
           kbAccountId,
           kbPaymentMethodId,
           mergedProperties,
           recurring,
           context.getTenantId(),
-          setDefault);
+          setDefault,
+          hasToken ? recurringDetailReference.trim() : null);
     } catch (SQLException e) {
 
       throw new PaymentMethodException("[addPaymentMethod] Error inserting payment method", e);
@@ -334,7 +348,12 @@ public PaymentTransactionInfoPlugin purchasePayment(
       formFields.add(
           new PluginProperty(SESSION_DATA, outputDTO.getAdditionalData().get(SESSION_DATA), false));
     } else {
-      input.setRecurringData(paymentMethodRecord.getRecurringDetailReference());
+      input.setRecurringData(getStoredToken(paymentMethodRecord));
+      // Adyen only accepts a stored token with the shopperReference it was stored under. For
+      // payment methods tokenised outside the plugin, that reference was given at creation;
+      // otherwise the plugin stored the token under the Kill Bill account id.
+      // Note: the processor sends input.getKbAccountId() to Adyen as the shopperReference.
+      input.setKbAccountId(getShopperReference(paymentMethodRecord, kbAccountId));
       outputDTO = gatewayProcessor.processOneTimePayment(input);
     }
 
@@ -590,7 +609,7 @@ public GatewayNotification processNotification(
           outputDTO.setStatus(PaymentPluginStatus.ERROR);
         }
         this.adyenDao.updateResponse(
-            UUID.fromString(record.getKbPaymentId()),
+            UUID.fromString(record.getKbPaymentTransactionId()),
             outputDTO,
             UUID.fromString(record.getKbTenantId()));
         this.adyenDao.addNotification(
@@ -608,6 +627,14 @@ public GatewayNotification processNotification(
               notificationItem.getAdditionalData().get(RECURRING_DATA));
         }
 
+        // The plugin tables are now up to date. On success, let Kill Bill core know right away
+        // instead of waiting for the Janitor (by default 1 hour later). See #138.
+        // Failures are left to the scheduled Janitor: it keeps the payment retry schedule of
+        // invoice payments, which an API-initiated refresh would skip.
+        if (notificationItem.isSuccess()) {
+          refreshPaymentInKillBill(UUID.fromString(record.getKbPaymentId()), tempContext);
+        }
+
       } else {
         logger.error("HMAC Key is not valid");
       }
@@ -617,6 +644,53 @@ public GatewayNotification processNotification(
     return new PluginGatewayNotification("[accepted]");
   }
 
+  /**
+   * Fetches the payment with plugin info, which runs the on-the-fly Janitor: Kill Bill core calls
+   * {@link #getPaymentInfo} and moves the PENDING transaction to SUCCESS based on the status the
+   * plugin has just recorded, then completes the invoice payment. A failure here is logged and not
+   * rethrown: the webhook must still be acknowledged, and the scheduled Janitor remains the
+   * fallback.
+   */
+  private void refreshPaymentInKillBill(final UUID kbPaymentId, final CallContext context) {
+    try {
+      this.killbillAPI
+          .getPaymentApi()
+          .getPayment(kbPaymentId, true, false, ImmutableList.<PluginProperty>of(), context);
+    } catch (final Exception e) {
+      logger.warn(
+          "Unable to refresh paymentId='{}' in Kill Bill after the Adyen notification,"
+              + " the Janitor will reconcile it later",
+          kbPaymentId,
+          e);
+    }
+  }
+
+  @VisibleForTesting
+  public String getShopperReference(
+      final AdyenPaymentMethodsRecord paymentMethodRecord, final UUID kbAccountId) {
+    final Map<String, String> data = getAdditionalDataMap(paymentMethodRecord.getAdditionalData());
+    final Object shopperReference = data.get(SHOPPER_REFERENCE);
+    final Object tokenAtCreation = data.get(RECURRING_DETAIL_REFERENCE);
+    final String storedToken = getStoredToken(paymentMethodRecord);
+    // Use the given reference only for the token it was given with; a token stored later by a
+    // plugin-initiated payment is always stored under the Kill Bill account id.
+    if (shopperReference != null
+        && !shopperReference.toString().trim().isEmpty()
+        && tokenAtCreation != null
+        && tokenAtCreation.toString().trim().equals(storedToken)) {
+      // Opaque key Adyen stored the token under: return it exactly as given (no trim).
+      return shopperReference.toString();
+    }
+    return kbAccountId.toString();
+  }
+
+  // recurring_detail_reference is a char(36) column: PostgreSQL returns it right-padded with
+  // spaces (MySQL strips them), so trim before sending it to Adyen or comparing it.
+  private static String getStoredToken(final AdyenPaymentMethodsRecord paymentMethodRecord) {
+    final String token = paymentMethodRecord.getRecurringDetailReference();
+    return token == null ? null : token.trim();
+  }
+
   public Map<String, String> getAdditionalDataMap(String additionalData) {
     if (additionalData == null) {
       return Collections.emptyMap();