Skip to content

Fix sendpay failure status - #9731

Merged
nGoline merged 2 commits into
ElementsProject:masterfrom
daywalker90:fix-sendpay-failure-status
Nov 12, 2026
Merged

nGoline merged 2 commits into
ElementsProject:masterfrom
daywalker90:fix-sendpay-failure-status

Conversation

@daywalker90

Copy link
Copy Markdown
Collaborator

Fixes: #7561

and the flake in test_sendpay_notifications:

2026-11-11T16:51:55.7624353Z >       assert results['sendpay_failure'][0] == err.value.error
2026-11-11T16:51:55.7632579Z E       AssertionError: assert {'code': 204, 'message': 'failed: WIRE_UNKNOWN_NEXT_PEER (reply from remote)', 'data': {'created_index': 2, 'id': 2, 'payment_hash': 'e602bac89e9fbc14cc6f5233727c06fe7c3d36b10fd1c5bab43a098bd64cfe8f', 'groupid': 1, 'destination': '03cecbfdc68544cc596223b68ce0710c9e5d2c9cb317ee07822d95079acc703d31', 'amount_msat': 100000000, 'amount_sent_msat': 100001001, 'created_at': 1794415661, 'status': 'pending', 'erring_index': 1, 'failcode': 16394, 'failcodename': 'WIRE_UNKNOWN_NEXT_PEER', 'erring_node': '033845802d25b4e074ccfd7cd8b339a41dc75bf9978a034800444b51d42b07799a', 'erring_channel': '103x1x0', 'erring_direction': 0, 'raw_message': '400a'}} == {'code': 204, 'message': 'failed: WIRE_UNKNOWN_NEXT_PEER (reply from remote)', 'data': {'created_index': 2, 'id': 2, 'payment_hash': 'e602bac89e9fbc14cc6f5233727c06fe7c3d36b10fd1c5bab43a098bd64cfe8f', 'groupid': 1, 'updated_index': 2, 'destination': '03cecbfdc68544cc596223b68ce0710c9e5d2c9cb317ee07822d95079acc703d31', 'amount_msat': 100000000, 'amount_sent_msat': 100001001, 'created_at': 1794415661, 'completed_at': 1794415661, 'status': 'failed', 'erring_index': 1, 'failcode': 16394, 'failcodename': 'WIRE_UNKNOWN_NEXT_PEER', 'erring_node': '033845802d25b4e074ccfd7cd8b339a41dc75bf9978a034800444b51d42b07799a', 'erring_channel': '103x1x0', 'erring_direction': 0, 'raw_message': '400a'}}
2026-11-11T16:51:55.7640967Z E
2026-11-11T16:51:55.7641300Z E         Common items:
2026-11-11T16:51:55.7641886Z E         {'code': 204, 'message': 'failed: WIRE_UNKNOWN_NEXT_PEER (reply from remote)'}
2026-11-11T16:51:55.7642540Z E         Differing items:
2026-11-11T16:51:55.7644193Z E         {'data': {'amount_msat': 100000000, 'amount_sent_msat': 100001001, 'created_at': 1794415661, 'created_index': 2, ...}} != {'data': {'amount_msat': 100000000, 'amount_sent_msat': 100001001, 'completed_at': 1794415661, 'created_at': 1794415661, ...}}
2026-11-11T16:51:55.7645709Z E
2026-11-11T16:51:55.7646077Z E         Full diff:
2026-11-11T16:51:55.7646448Z E           {
2026-11-11T16:51:55.7646799Z E               'code': 204,
2026-11-11T16:51:55.7647208Z E               'data': {
2026-11-11T16:51:55.7647645Z E                   'amount_msat': 100000000,
2026-11-11T16:51:55.7648206Z E                   'amount_sent_msat': 100001001,
2026-11-11T16:51:55.7648742Z E         -         'completed_at': 1794415661,
2026-11-11T16:51:55.7649789Z E                   'created_at': 1794415661,
2026-11-11T16:51:55.7650308Z E                   'created_index': 2,
2026-11-11T16:51:55.7651384Z E                   'destination': '03cecbfdc68544cc596223b68ce0710c9e5d2c9cb317ee07822d95079acc703d31',
2026-11-11T16:51:55.7652223Z E                   'erring_channel': '103x1x0',
2026-11-11T16:51:55.7652742Z E                   'erring_direction': 0,
2026-11-11T16:51:55.7653218Z E                   'erring_index': 1,
2026-11-11T16:51:55.7654240Z E                   'erring_node': '033845802d25b4e074ccfd7cd8b339a41dc75bf9978a034800444b51d42b07799a',
2026-11-11T16:51:55.7654998Z E                   'failcode': 16394,
2026-11-11T16:51:55.7655472Z E                   'failcodename': 'WIRE_UNKNOWN_NEXT_PEER',
2026-11-11T16:51:55.7656007Z E                   'groupid': 1,
2026-11-11T16:51:55.7656390Z E                   'id': 2,
2026-11-11T16:51:55.7657018Z E                   'payment_hash': 'e602bac89e9fbc14cc6f5233727c06fe7c3d36b10fd1c5bab43a098bd64cfe8f',
2026-11-11T16:51:55.7657736Z E                   'raw_message': '400a',
2026-11-11T16:51:55.7658163Z E         -         'status': 'failed',
2026-11-11T16:51:55.7658564Z E         ?                    ^^ ^^^
2026-11-11T16:51:55.7658957Z E         +         'status': 'pending',
2026-11-11T16:51:55.7659572Z E         ?                    ^^^^ ^^
2026-11-11T16:51:55.7659971Z E         -         'updated_index': 2,
2026-11-11T16:51:55.7660354Z E               },
2026-11-11T16:51:55.7660819Z E               'message': 'failed: WIRE_UNKNOWN_NEXT_PEER (reply from remote)',
2026-11-11T16:51:55.7661346Z E           }
2026-11-11T16:51:55.7661518Z
2026-11-11T16:51:55.7661686Z tests/test_plugin.py:1839: AssertionError

@daywalker90 daywalker90 added this to the v26.12 milestone Nov 12, 2026
@daywalker90
daywalker90 requested a review from Andezion November 12, 2026 11:19

@Andezion Andezion left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from what i see renepay has /* FIXME shouldn't this be always SENDPAY_FAILED? */ (plugins/renepay/routetracker.c from line 556 to 567). It logs LOG_UNUSUAL "sendpay_failure notification returned status=%s" and forces the status to failed. Today this probably fires for every failed part, but after this pr it should never fire. so maybe remove the workaround in a follow up, or turn it into a plugin_err / LOG_BROKEN, so a regression gets noticed?

Comment thread lightningd/pay.c

/* Waiters and the notification need the updated status. */
payment = wallet_payment_by_hash(tmpctx, ld->wallet, payment_hash,
partid, groupid);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe add assert(payment);? same on 1557 line

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw that payment_succeeded() but i wasn't sure it could actually be NULL here? Anyways for consistency i added it here.

nGoline
nGoline previously approved these changes Nov 12, 2026

@nGoline nGoline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 0592b38

Checked that the new test fails without the fix.

Side effect worth knowing: for unparseable onion replies the reloaded row now includes erroronion in the notification and the in-flight waitsendpay error, which a late waitsendpay already returned. The schema already allows it.

Out of scope for this PR, but noticed while reading: wallet_payment_set_failinfo() matches on payment_hash and partid only, not groupid.

When an HTLC fails, payment_failed() reads the payment before marking
it failed, then hands that stale copy to the sendpay_failure notification
and to any waitsendpay already waiting.  Both report "status": "pending"
without completed_at or updated_index, while a waitsendpay issued
afterwards reads the database and says "failed".

This is the cause of the test_sendpay_notifications flake: whether its
waitsendpay lands before or after the failure decides which version it
gets, and the notification never matches the fresh one.

Reported-in: ElementsProject#7561
payment_failed() loads the payment, marks it failed in the database,
then passes the copy it loaded beforehand to the waiters and the
sendpay_failure notification.  So they saw "status": "pending" and no
completed_at or updated_index, unlike a later waitsendpay or listsendpays.

Reload the payment after updating it, as payment_succeeded() does; the
self-payment failure path had the same problem.

This also fixes the test_sendpay_notifications flake, where the
notification was compared against a waitsendpay that read the database.

Rusty proposed setting payment->status locally in 2024, but that never
landed and would still leave completed_at and updated_index out.

Fixes: ElementsProject#7561
Changelog-Fixed: Plugins: `sendpay_failure` notifications and `waitsendpay` errors now report `status` `failed` (with `completed_at` and `updated_index`), not `pending`.
@daywalker90

Copy link
Copy Markdown
Collaborator Author

from what i see renepay has /* FIXME shouldn't this be always SENDPAY_FAILED? */ (plugins/renepay/routetracker.c from line 556 to 567). It logs LOG_UNUSUAL "sendpay_failure notification returned status=%s" and forces the status to failed. Today this probably fires for every failed part, but after this pr it should never fire. so maybe remove the workaround in a follow up, or turn it into a plugin_err / LOG_BROKEN, so a regression gets noticed?

renepay is deprecated so i did not add changes to it to this PR and i don't intend to in another PR.

Out of scope for this PR, but noticed while reading: wallet_payment_set_failinfo() matches on payment_hash and partid only, not groupid.

Good catch, opened #9734 for that.

@daywalker90
daywalker90 requested a review from Andezion November 12, 2026 16:32

@nGoline nGoline left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack ad3ecda

@nGoline
nGoline merged commit 71076cb into ElementsProject:master Nov 12, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

status "pending" from "sendpay_failure"

3 participants