FDL xlang mismatch between Rust and Python - deserialization failure

[RisinT96]

Search before asking

• I had searched in the issues and found no similar issues.

Version

0.16.0

Component(s)

Rust, Python, Other

Minimal reproduce step

Fory Cross-Language Mismatch Reproducer

This reproducer shows that Python and Rust serialize the same logical values to different Fory byte streams.

Files:

• repro.fdl: minimal schema for the repro
• python repro.in: Python serializer using pyfory 0.16.0
• rust/: tiny Rust crate using fory 0.16.0

Run:

uv venv .venv
uv pip install pyfory==0.16.0 fory-compiler==0.16.0 numpy==2.4.3
foryc --lang rust   --rust_out   rust/src/ repro.fdl
foryc --lang python --python_out python/   repro.fdl

python python/repro.py
cargo run --manifest-path rust/Cargo.toml

Files:

repro.fdl

package repro.fory_crosslang_mismatch;

message SomeType [id=101] {
    string some_string = 1;
    list<fixed_int32> some_ints = 2;
}

message OtherType [id=102] {
    list<SomeType> nested = 1;
}

rust/Cargo.toml

[package]
name = "fory-crosslang-mismatch"
version = "0.1.0"
edition = "2024"
publish = false

[dependencies]
fory = "0.16.0"
fory-core = "0.16.0"

rust/src/main.rs

#[allow(dead_code)]
mod repro_fory_crosslang_mismatch;

use repro_fory_crosslang_mismatch::{OtherType, SomeType};

fn hex_bytes(bytes: &[u8]) -> String {
    bytes
        .iter()
        .map(|byte| format!("{byte:02x}"))
        .collect::<String>()
}

fn main() {
    let nested = OtherType {
        nested: vec![
            SomeType {
                some_string: "0-96267bf9".to_string(),
                some_ints: vec![14783],
            },
            SomeType {
                some_string: "1-bfd05330".to_string(),
                some_ints: vec![26098],
            },
        ],
    };

    let nested_bytes = nested.to_bytes().expect("serialize OtherType");

    println!("rust_len={}", nested_bytes.len());
    println!("rust_hex={}", hex_bytes(&nested_bytes));
}

python/repro.py

from repro_fory_crosslang_mismatch import SomeType, OtherType

import numpy as np


def main() -> None:
    nested = OtherType(
        nested=[
            SomeType(
                some_string="0-96267bf9",
                some_ints=np.array([14783], dtype=np.int32),
            ),
            SomeType(
                some_string="1-bfd05330",
                some_ints=np.array([26098], dtype=np.int32),
            ),
        ]
    )

    nested_bytes = nested.to_bytes()
    print(f"pyth_len={len(nested_bytes)}")
    print(f"pyth_hex={nested_bytes.hex()}")


if __name__ == "__main__":
    main()

What did you expect to see?

I expect pyth_hex to be equal to rust_hex.

What did you see instead?

Mismatch for the same logical values:

pyth_len=71
rust_len=69

pyth_hex=02001c0006810170805ab647011c66c4167002091c0207811d0d145d6a1b021c65c415c82e0028302d393632363762663904bf3900000028312d626664303533333004f2650000
rust_hex=02001c0006c134383927b95e011c66c4167202081c020781e2f2eba29564021c65c415c82e2a302d393632363762663904bf3900002a312d626664303533333004f2650000

Decode failure of the python bytes:

let bytes = "02001c0006810170805ab647011c66c4167002091c0207811d0d145d6a1b021c65c415c82e0028302d393632363762663904bf3900000028312d626664303533333004f2650000";
let deserialized = OtherType::from_bytes(&hex::decode(bytes).expect("decode hex"));
println!("deserialized: {deserialized:#?}");

Results in:

deserialized: Err(
    BufferOutOfBound(
        39,
        40,
        71,
    ),
)

Anything Else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[RisinT96]

I used some AI tools to try and find a root cause for this, what I got is that the python serializer ignores the False in the generated

nested: List[Ref[SomeType, False]] = pyfory.field(id=1, default_factory=list)

And uses refcounting anyway as it's enabled globally for fory in the autogenerated:

def _create_fory() -> pyfory.Fory:
    fory = pyfory.Fory(xlang=True, ref=True, compatible=True)
    register_repro_fory_crosslang_mismatch_types(fory)
    return fory

You can take a look at this commit for the full analysis in the message and a proposed fix in the commit itself.

I have validated that the fix works locally, and generates a much smaller binary output for some reason:

pyth_len=52
pyth_hex=02001c0006810170805ab647011c66c41670020c28302d393632363762663904bf39000028312d626664303533333004f2650000

and rust deserializes it correctly:

let bytes = "02001c0006810170805ab647011c66c41670020c28302d393632363762663904bf39000028312d626664303533333004f2650000";
let deserialized = OtherType::from_bytes(&hex::decode(bytes).expect("decode hex"));
if let Ok(deserialized) = deserialized {
    println!("equal? {}", deserialized == nested);
]

I don't have the time to fully validate the changes, but you can take that commit as reference.

Locally I worked around the issue by explicitly setting ref=False in _create_fory.

Tal