Skip to content

feat: read appended data - #24

Open
Abtektas wants to merge 1 commit into
PX4:mainfrom
Abtektas:Abtektas/appended-data
Open

Abtektas wants to merge 1 commit into
PX4:mainfrom
Abtektas:Abtektas/appended-data

Conversation

@Abtektas

Copy link
Copy Markdown
Contributor

A process that appends data to a ULog file (for example PX4's hardfault_log after a crash) sets the DATA_APPENDED flag and stores the file offset where the new data starts in appended_offsets. The file may have been cut in the middle of a message at that offset.

The reader had a TODO there: it reported "File contains appended offsets - this is not supported" and kept parsing through the offset. A message cut at the offset was then joined with the appended bytes. On pyulog's test/sample_appended.ulg (a real PX4 log) this gives one extra data sample with garbage values. With a cut message, appended messages can also be lost (in the new test, the appended logging message).

Changes

  • Reader::readChunk() now reads up to each appended offset separately. At the offset it drops the cut message (if any) and any recovery state, and continues with the appended data, which holds Data section messages.
  • Offsets are taken from the flag bits, sorted. An offset that points into the file header is ignored with a recoverable error.
  • The message loop moved to readMessages(); tryToRecover() calls it instead of readChunk().

Testing

  • New test "ULog parsing - appended data after a cut message": the log is cut 0 or 5 bytes into a data message, a logging message and a data message are appended, and the file is read in chunks of 1, 7 and all remaining bytes (after the header chunk). Without the reader change it fails (the "not supported" error; with a cut message the appended logging message is lost).
  • On pyulog's sample_appended.ulg and sample_appended_multiple.ulg, the data and logged messages now match pyulog's output exactly (before: one extra garbage sample in sample_appended.ulg).
  • On a corpus of 49 synthetic edge-case files, only the 3 appended-data files change, and their data and messages now match px4-ulog-rs.

On ubuntu:26.04 linux/amd64 (GCC 15.2.0, CMake 4.4.5, as on the CI runner): make -C build run-unit-tests passes 10/10 test cases, pre-commit run -a passes, and make -C build clang-tidy passes with clang-tidy 20.1.8 and 21.1.8. clang-tidy 22.1.2 reports the same 16 findings on main and on this branch, none in the changed lines.

This PR was prepared with AI assistance (Claude).

A process that appends data to a ULog file (for example PX4's
hardfault_log after a crash) stores the file offset where the new data
starts in the flag bits message. The log may have been cut in the
middle of a message at that offset.

The reader reported "File contains appended offsets - this is not
supported" and kept parsing through the offset. A cut message was then
joined with the appended bytes: on pyulog's sample_appended.ulg this
gives one data sample with garbage values, and appended messages after
a cut could be lost.

Read up to each appended offset separately, drop the cut message and
any recovery state there, and continue with the appended data. Offsets
that point into the file header are ignored with an error.

Assisted-by: Claude:claude-opus-5-5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant