dearsky/pico-hsm
1
0
Fork 0
Code Issues 17 Pull Requests 4 Actions Packages Projects Releases 26 Wiki Activity

EC P-256 + TLS 1.2 Client Authentication Fails with CKR_GENERAL_ERROR #123

New Issue
Closed
opened 2026-01-18 03:41:53 +08:00 by FotoFieber · 8 comments
FotoFieber commented 2026-01-18 03:41:53 +08:00 (Migrated from https://github.com/polhenarejos/pico-hsm)
Copy Link

Environment

  • Device: PicoHSM (SC-HSM based, RP2350)
  • Firmware: 6.2
  • PKCS#11 Library: OpenSC (opensc-pkcs11.so)
  • OS: Ubuntu 24.04 / WSL2
  • Java: 25.0.1 (SunPKCS11 provider)

When using an EC P-256 (prime256v1) key stored on PicoHSM for TLS 1.2 client certificate authentication, the PKCS#11 library returns CKR_GENERAL_ERROR during the TLS handshake signing operation.

The same key works correctly with TLS 1.3.

  1. Generate EC P-256 key on PicoHSM:
    pkcs11-tool --module /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
    --login --pin
    --keypairgen --key-type EC:prime256v1
    --label "client-ec256-a" --id 05
  2. Generate CSR and sign certificate with CA
  3. Import certificate to HSM:
    pkcs11-tool --module /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
    --login --pin
    --write-object cert.der --type cert --id 05 --label "client-ec256-a"
  4. Attempt TLS 1.2 client authentication using Java SunPKCS11 provider

Expected Behavior

TLS handshake succeeds and client is authenticated.

Actual Behavior

sun.security.pkcs11.wrapper.PKCS11Exception: CKR_GENERAL_ERROR

The error occurs during the signing operation in the TLS handshake.

Working Configuration

  • EC P-256 + TLS 1.3: Works correctly
  • RSA 3072 + TLS 1.2: Works correctly
  • RSA 4096 + TLS 1.2: Works correctly
  • RSA 3072 + TLS 1.3: Works correctly
  • RSA 4096 + TLS 1.3: Works correctly

Additional Notes

The same test with YubiHSM2 (using yubihsm_pkcs11.so) works correctly with EC P-256 + TLS 1.2, suggesting this is specific to the PicoHSM/OpenSC implementation.

The difference between TLS 1.2 and TLS 1.3 ECDSA signing:

  • TLS 1.2: Uses ECDSA with the hash algorithm determined by cipher suite
  • TLS 1.3: Uses ECDSA with explicit signature algorithm (e.g., ecdsa_secp256r1_sha256)
Environment - Device: PicoHSM (SC-HSM based, RP2350) - Firmware: 6.2 - PKCS#11 Library: OpenSC (opensc-pkcs11.so) - OS: Ubuntu 24.04 / WSL2 - Java: 25.0.1 (SunPKCS11 provider) When using an EC P-256 (prime256v1) key stored on PicoHSM for TLS 1.2 client certificate authentication, the PKCS#11 library returns CKR_GENERAL_ERROR during the TLS handshake signing operation. The same key works correctly with TLS 1.3. 1. Generate EC P-256 key on PicoHSM: pkcs11-tool --module /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so \ --login --pin <PIN> \ --keypairgen --key-type EC:prime256v1 \ --label "client-ec256-a" --id 05 2. Generate CSR and sign certificate with CA 3. Import certificate to HSM: pkcs11-tool --module /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so \ --login --pin <PIN> \ --write-object cert.der --type cert --id 05 --label "client-ec256-a" 4. Attempt TLS 1.2 client authentication using Java SunPKCS11 provider Expected Behavior TLS handshake succeeds and client is authenticated. Actual Behavior sun.security.pkcs11.wrapper.PKCS11Exception: CKR_GENERAL_ERROR The error occurs during the signing operation in the TLS handshake. Working Configuration - EC P-256 + TLS 1.3: Works correctly - RSA 3072 + TLS 1.2: Works correctly - RSA 4096 + TLS 1.2: Works correctly - RSA 3072 + TLS 1.3: Works correctly - RSA 4096 + TLS 1.3: Works correctly Additional Notes The same test with YubiHSM2 (using yubihsm_pkcs11.so) works correctly with EC P-256 + TLS 1.2, suggesting this is specific to the PicoHSM/OpenSC implementation. The difference between TLS 1.2 and TLS 1.3 ECDSA signing: - TLS 1.2: Uses ECDSA with the hash algorithm determined by cipher suite - TLS 1.3: Uses ECDSA with explicit signature algorithm (e.g., ecdsa_secp256r1_sha256)
polhenarejos commented 2026-01-18 05:41:21 +08:00 (Migrated from https://github.com/polhenarejos/pico-hsm)
Copy Link

Can you provide the steps with Java provider?
Is the problem only with P256+TLS1.2? specifically?
Did you test with another P256 key? Happens with always?

Can you provide the steps with Java provider? Is the problem only with P256+TLS1.2? specifically? Did you test with another P256 key? Happens with always?
FotoFieber commented 2026-01-19 01:22:48 +08:00 (Migrated from https://github.com/polhenarejos/pico-hsm)
Copy Link

Hi Pol, thank you for looking into this.

1. Java Provider Configuration

I'm using the SunPKCS11 provider with this configuration:
name = PicoHSM
library = /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
slot = 0

The TLS connection is established via standard Java SSLContext with the PKCS#11 KeyStore. I can provide a minimal reproducer if helpful.

2. Confirmation: Yes, specifically P256 + TLS 1.2

Key Type TLS 1.2 TLS 1.3
EC P-256 ❌ CKR_GENERAL_ERROR ✅ Works
RSA 3072 ✅ Works ✅ Works
RSA 4096 ✅ Works ✅ Works

3. Consistency: Yes, reproducible with multiple keys

I tested with freshly generated P-256 keys - same behavior every time.

Additional diagnostic information that may help:

The key difference between TLS 1.2 and 1.3 for ECDSA signing:

  • TLS 1.3: Client signs with explicit signature_algorithms (e.g., ecdsa_secp256r1_sha256) - the mechanism is clear
  • TLS 1.2: The hash algorithm is derived from the cipher suite, and the signing request may come differently through PKCS#11

Question: Could it be that TLS 1.2 is requesting a raw ECDSA signature (CKM_ECDSA) with pre-hashed data, while TLS 1.3 uses CKM_ECDSA_SHA256? If PicoHSM expects a specific mechanism or has constraints on input data length for CKM_ECDSA, that might explain why one works and the other doesn't.

I can enable PKCS#11 debug logging (OPENSC_DEBUG=9) to capture the exact C_Sign call parameters if that would help identify what mechanism/data is being passed.

Hi Pol, thank you for looking into this. **1. Java Provider Configuration** I'm using the SunPKCS11 provider with this configuration: name = PicoHSM library = /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so slot = 0 The TLS connection is established via standard Java SSLContext with the PKCS#11 KeyStore. I can provide a minimal reproducer if helpful. **2. Confirmation: Yes, specifically P256 + TLS 1.2** | Key Type | TLS 1.2 | TLS 1.3 | |----------|---------|---------| | EC P-256 | ❌ `CKR_GENERAL_ERROR` | ✅ Works | | RSA 3072 | ✅ Works | ✅ Works | | RSA 4096 | ✅ Works | ✅ Works | **3. Consistency: Yes, reproducible with multiple keys** I tested with freshly generated P-256 keys - same behavior every time. **Additional diagnostic information that may help:** The key difference between TLS 1.2 and 1.3 for ECDSA signing: - **TLS 1.3**: Client signs with explicit `signature_algorithms` (e.g., `ecdsa_secp256r1_sha256`) - the mechanism is clear - **TLS 1.2**: The hash algorithm is derived from the cipher suite, and the signing request may come differently through PKCS#11 **Question:** Could it be that TLS 1.2 is requesting a raw ECDSA signature (`CKM_ECDSA`) with pre-hashed data, while TLS 1.3 uses `CKM_ECDSA_SHA256`? If PicoHSM expects a specific mechanism or has constraints on input data length for `CKM_ECDSA`, that might explain why one works and the other doesn't. I can enable PKCS#11 debug logging (`OPENSC_DEBUG=9`) to capture the exact `C_Sign` call parameters if that would help identify what mechanism/data is being passed.
polhenarejos commented 2026-01-19 04:29:57 +08:00 (Migrated from https://github.com/polhenarejos/pico-hsm)
Copy Link

It could be. Perhaps also happens with P384 or P521. If you enable OPENSC_DEBUG=9 please provide the log around where it fails with APDU messages.

It could be. Perhaps also happens with P384 or P521. If you enable OPENSC_DEBUG=9 please provide the log around where it fails with APDU messages.
FotoFieber commented 2026-01-19 13:19:16 +08:00 (Migrated from https://github.com/polhenarejos/pico-hsm)
Copy Link

OPENSC_DEBUG=9 Log for EC P-256 TLS 1.2 Failure

Test scenario: EC P-256 key with TLS 1.2 (CKM_ECDSA_SHA256)

Summary

  • Mechanism: 0x1044 (CKM_ECDSA_SHA256)
  • Key type: 0x3 (EC)
  • Data to sign: 6039 bytes (TLS handshake data)
  • APDU sent: CLA:80, INS:68, P1:5, P2:73, data(6039)
  • Error: SCardTransmit/Control failed: 0x80100016 (SCARD_E_NOT_TRANSACTED)

Comparison with working EC P-384 TLS 1.2

  • Mechanism: 0x1045 (CKM_ECDSA_SHA384)
  • APDU sent: CLA:80, INS:68, P1:1B, P2:70, data(48) ← pre-hashed, only 48 bytes
  • Result: Success

The difference is that P-384 sends pre-hashed data (48 bytes), while P-256 sends the full data (6039 bytes) for internal hashing.

Detailed APDU trace around failure

06:10:41.801 mechanism.c:383:sc_pkcs11_sign_init: mechanism 0x1044, key-type 0x3
06:10:41.801 pkcs11-object.c:697:C_SignInit: C_SignInit() = CKR_OK
06:10:41.801 mechanism.c:570:sc_pkcs11_signature_update: data part length 6039
06:10:41.801 framework-pkcs15.c:4270:pkcs15_prkey_sign: Initiating signing operation, mechanism 0x1044.
06:10:41.802 apdu.c:367:sc_single_transmit: CLA:80, INS:68, P1:5, P2:73, data(6039)
06:10:41.802 reader-pcsc.c:325:pcsc_transmit: Outgoing APDU (6048 bytes)
06:10:41.879 reader-pcsc.c:272:pcsc_internal_transmit: SCardTransmit/Control failed: 0x80100016
06:10:41.888 apdu.c:380:sc_single_transmit: unable to transmit APDU: -1107 (Transmit failed)
06:10:41.888 card-sc-hsm.c:1128:sc_hsm_compute_signature: APDU transmit failed
06:10:41.970 pkcs11-object.c:809:C_SignFinal: C_SignFinal() = CKR_GENERAL_ERROR

Environment

  • PicoHSM (Pico Key with SmartCard-HSM firmware)
  • OpenSC 0.25.1
  • Java 25.0.1 with SunPKCS11 provider
  • WSL2 Linux 6.6.87.2-microsoft-standard-WSL2
  • pcscd running

Notes

  • EC P-256 with TLS 1.3 works (different signature scheme, pre-hashed data)
  • EC P-384 with TLS 1.2 works (P2=0x70, pre-hashed 48-byte data)
  • EC P-384 with TLS 1.3 works
  • RSA keys with TLS 1.2 and 1.3 both work

The issue appears to be specific to sending large payloads (6KB) to the HSM for internal hashing with EC P-256.

OPENSC_DEBUG=9 Log for EC P-256 TLS 1.2 Failure Test scenario: EC P-256 key with TLS 1.2 (CKM_ECDSA_SHA256) Summary - Mechanism: 0x1044 (CKM_ECDSA_SHA256) - Key type: 0x3 (EC) - Data to sign: 6039 bytes (TLS handshake data) - APDU sent: CLA:80, INS:68, P1:5, P2:73, data(6039) - Error: SCardTransmit/Control failed: 0x80100016 (SCARD_E_NOT_TRANSACTED) Comparison with working EC P-384 TLS 1.2 - Mechanism: 0x1045 (CKM_ECDSA_SHA384) - APDU sent: CLA:80, INS:68, P1:1B, P2:70, data(48) ← pre-hashed, only 48 bytes - Result: Success The difference is that P-384 sends pre-hashed data (48 bytes), while P-256 sends the full data (6039 bytes) for internal hashing. Detailed APDU trace around failure 06:10:41.801 mechanism.c:383:sc_pkcs11_sign_init: mechanism 0x1044, key-type 0x3 06:10:41.801 pkcs11-object.c:697:C_SignInit: C_SignInit() = CKR_OK 06:10:41.801 mechanism.c:570:sc_pkcs11_signature_update: data part length 6039 06:10:41.801 framework-pkcs15.c:4270:pkcs15_prkey_sign: Initiating signing operation, mechanism 0x1044. 06:10:41.802 apdu.c:367:sc_single_transmit: CLA:80, INS:68, P1:5, P2:73, data(6039) 06:10:41.802 reader-pcsc.c:325:pcsc_transmit: Outgoing APDU (6048 bytes) 06:10:41.879 reader-pcsc.c:272:pcsc_internal_transmit: SCardTransmit/Control failed: 0x80100016 06:10:41.888 apdu.c:380:sc_single_transmit: unable to transmit APDU: -1107 (Transmit failed) 06:10:41.888 card-sc-hsm.c:1128:sc_hsm_compute_signature: APDU transmit failed 06:10:41.970 pkcs11-object.c:809:C_SignFinal: C_SignFinal() = CKR_GENERAL_ERROR Environment - PicoHSM (Pico Key with SmartCard-HSM firmware) - OpenSC 0.25.1 - Java 25.0.1 with SunPKCS11 provider - WSL2 Linux 6.6.87.2-microsoft-standard-WSL2 - pcscd running Notes - EC P-256 with TLS 1.3 works (different signature scheme, pre-hashed data) - EC P-384 with TLS 1.2 works (P2=0x70, pre-hashed 48-byte data) - EC P-384 with TLS 1.3 works - RSA keys with TLS 1.2 and 1.3 both work The issue appears to be specific to sending large payloads (6KB) to the HSM for internal hashing with EC P-256.
polhenarejos commented 2026-01-19 15:31:27 +08:00 (Migrated from https://github.com/polhenarejos/pico-hsm)
Copy Link

Maximum payload is 2KB. 6KB seems a lot to be signed in one transaction. Not sure why tries to do a raw sign with P256 but not with P384.

Maximum payload is 2KB. 6KB seems a lot to be signed in one transaction. Not sure why tries to do a raw sign with P256 but not with P384.
FotoFieber commented 2026-01-19 17:15:09 +08:00 (Migrated from https://github.com/polhenarejos/pico-hsm)
Copy Link

The tests work fine with SoftCertificate and YubiHSM 2. Can we increase the maximum payload size?

The tests work fine with SoftCertificate and YubiHSM 2. Can we increase the maximum payload size?
polhenarejos commented 2026-01-26 00:16:47 +08:00 (Migrated from https://github.com/polhenarejos/pico-hsm)
Copy Link

What does pkcs11-tool -M returns with YubiHSM2?

What does `pkcs11-tool -M` returns with YubiHSM2?
polhenarejos commented 2026-01-26 01:21:12 +08:00 (Migrated from https://github.com/polhenarejos/pico-hsm)
Copy Link

After extensive debugging, the root cause is now clearly identified and confirmed.

When using OpenSC’s opensc-pkcs11.so with the sc-hsm driver, the module advertises combined digest+sign mechanisms such as CKM_ECDSA_SHA256. In TLS 1.2, OpenSSL is allowed to pass the full CertificateVerify input to PKCS#11 backends implementing such mechanisms. This input is not a hash, but the raw preimage, whose size depends on the handshake transcript and is effectively unbounded (in practice several kilobytes, but not guaranteed to stay small).

OpenSC forwards this buffer verbatim to the card using a single APDU. Even if the firmware supports APDU chaining, OpenSC does not fragment or stream this operation in this code path. As a result, small embedded devices must either allocate arbitrarily large buffers (unsafe and non-deterministic) or fail.

Increasing internal USB/APDU buffers makes the operation succeed temporarily, but this is not a robust solution: the input size is protocol- and implementation-dependent and may grow in future OpenSSL or TLS changes.

Disabling CKM_ECDSA_SHA* in OpenSC fixes the issue by forcing OpenSSL to hash on the host and call CKM_ECDSA with a fixed-size digest (32/48 bytes). However, requiring users to rebuild OpenSC is not acceptable.

Conclusion: this is not a firmware bug, nor a TLS bug, but a mismatch between OpenSC’s sc-hsm PKCS#11 model (designed for full SmartCard-HSM devices) and constrained embedded tokens.

The correct solution in this case is to use the embedded variant of the SmartCard-HSM stack, which exposes a PKCS#11 module tailored for embedded devices and does not advertise unsafe digest+sign mechanisms.

Please load and use:

libsc-hsm-pkcs11.so

(from https://github.com/CardContact/sc-hsm-embedded)

instead of opensc-pkcs11.so.

After extensive debugging, the root cause is now clearly identified and confirmed. When using OpenSC’s `opensc-pkcs11.so` with the `sc-hsm` driver, the module advertises combined digest+sign mechanisms such as `CKM_ECDSA_SHA256`. In TLS 1.2, OpenSSL is allowed to pass the *full CertificateVerify input* to PKCS#11 backends implementing such mechanisms. This input is **not a hash**, but the raw preimage, whose size depends on the handshake transcript and is effectively unbounded (in practice several kilobytes, but not guaranteed to stay small). OpenSC forwards this buffer verbatim to the card using a single APDU. Even if the firmware supports APDU chaining, OpenSC does not fragment or stream this operation in this code path. As a result, small embedded devices must either allocate arbitrarily large buffers (unsafe and non-deterministic) or fail. Increasing internal USB/APDU buffers makes the operation succeed temporarily, but this is not a robust solution: the input size is protocol- and implementation-dependent and may grow in future OpenSSL or TLS changes. Disabling `CKM_ECDSA_SHA*` in OpenSC fixes the issue by forcing OpenSSL to hash on the host and call `CKM_ECDSA` with a fixed-size digest (32/48 bytes). However, requiring users to rebuild OpenSC is not acceptable. **Conclusion:** this is not a firmware bug, nor a TLS bug, but a mismatch between OpenSC’s `sc-hsm` PKCS#11 model (designed for full SmartCard-HSM devices) and constrained embedded tokens. The correct solution in this case is to use the embedded variant of the SmartCard-HSM stack, which exposes a PKCS#11 module tailored for embedded devices and does not advertise unsafe digest+sign mechanisms. Please load and use: ``` libsc-hsm-pkcs11.so ``` (from https://github.com/CardContact/sc-hsm-embedded) instead of `opensc-pkcs11.so`.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something isn't working
 bug
Something isn't working
 dependencies
Pull requests that update a dependency file
 dependencies
Pull requests that update a dependency file
 documentation
Improvements or additions to documentation
 documentation
Improvements or additions to documentation
 duplicate
This issue or pull request already exists
 duplicate
This issue or pull request already exists
 enhancement
New feature or request
 enhancement
New feature or request
 good first issue
Good for newcomers
 good first issue
Good for newcomers
 help wanted
Extra attention is needed
 help wanted
Extra attention is needed
 invalid
This doesn't seem right
 invalid
This doesn't seem right
 question
Further information is requested
 question
Further information is requested
 wontfix
This will not be worked on
 wontfix
This will not be worked on
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dearsky
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dearsky/pico-hsm#123
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?