Summary

improv_serial was intermittently failing on ESP32 USB console transports because it treated one logical Improv response as multiple independent serial writes and ignored backend short-write behavior. On ESP32-S3 this is especially problematic for USB_CDC, where the console API explicitly allows partial writes. I fixed the transport path by assembling each Improv frame contiguously, retrying until the selected serial backend accepts the full frame, and restoring ESP32-S3 USB_CDC config support.

Environment

• Target board: ESP32-S3 (esp32-s3-devkitc-1)
• Hardware device under test: esp32-ea6a08
• Serial bridge: ./hwctl at 115200 baud
• Verification firmware 1: ESP32-S3 + ESP-IDF + logger.hardware_uart: USB_SERIAL_JTAG + improv_serial
• Verification firmware 2: ESP32-S3 + ESP-IDF + logger.hardware_uart: USB_CDC + improv_serial
• Issue context: the linked report was against ESP32-S3 with Arduino 3.2.1 and improv_serial

Hardware Setup

• Device ID: esp32-ea6a08
• Board type: esp32-s3
• Bridge target: bridge://esp32-ea6a08
• Bridge helper used for all hardware interaction: ./hwctl
• Flashing method: ./hwctl flash 0x0:<firmware.factory.bin>
• Serial capture method: ./hwctl reset-capture 8, ./hwctl serial-read, and ./hwctl serial-follow

Reproduction Steps

1. Build a small ESP32-S3 improv_serial repro firmware that keeps the logger on the same USB serial path and emits frequent warning logs.

logger:
  level: WARN
  baud_rate: 115200
  hardware_uart: USB_SERIAL_JTAG

improv_serial:

interval:
  - interval: 100ms
    then:
      - logger.log:
          level: WARN
          format: "improv stress tick"

2. Flash it to the physical board:

$ ./hwctl flash 0x0:tmp/.esphome/build/improv-s3-idf-test/.pioenvs/improv-s3-idf-test/firmware.factory.bin
...
Wrote 773968 bytes (493684 compressed) at 0x00000000 in 4.4 seconds (1419.2 kbit/s).
Verifying written data...
Hash of data verified.
Hard resetting via RTS pin...

3. Capture serial after reset:

$ ./hwctl reset-capture 8
reset complete

$ ./hwctl serial-follow 4096
[W][main:199]: improv stress tick
[W][main:199]: improv stress tick
...
n:199]: improv stress tick

4. The truncated log line is the important symptom: the shared USB serial transport is fragmenting or dropping output under load.

5. Inspect esphome/components/improv_serial/improv_serial_component.cpp. Before the fix, write_data_() sent one Improv response in up to three separate writes:

• header
• payload
• checksum/newline footer

6. Inspect the ESP32 USB console backend API semantics used by that code path:

• esp_usb_console_write_buf() returns the number of bytes written and may write less than requested.
• usb_serial_jtag_write_bytes() returns the number of bytes accepted by the TX path.

7. Because improv_serial ignored those return values and split frames across calls, the browser-side Improv client could receive truncated or interleaved frames and eventually time out.

Expected Behavior

Each Improv response should be emitted as one intact serial frame on the selected logger transport, with no missing bytes and no opportunity for the frame to be spliced by intervening serial output.

Actual Behavior

• The physical ESP32-S3 USB serial path showed observable output corruption under load.
• improv_serial compounded that by doing multi-call frame writes and by ignoring short-write return values from the ESP32 USB console backend.
• That makes browser-side Improv operations intermittent: some frames arrive intact, others arrive truncated or corrupted, and the client eventually reports TIMEOUT.

Root Cause

The bug was in the serial write path, not in Wi-Fi provisioning logic.

Before the fix, ImprovSerialComponent::write_data_() built a correct logical frame but transmitted it in pieces. That is unsafe on ESP32 USB console transports for two reasons:

1. esp_usb_console_write_buf() is a short-write API. It can accept fewer bytes than requested, but the old code ignored its return value entirely.

2. Even when the backend eventually accepts all bytes, sending header, payload, and footer as separate writes allows unrelated logger output to land between them.

Those two behaviors are enough to corrupt an Improv frame and explain the intermittent browser timeout from the linked issue.

Files Changed

• source/esphome/components/improv_serial/improv_serial_component.cpp
• source/esphome/components/improv_serial/improv_serial_component.h
• source/esphome/components/improv_serial/__init__.py
• source/tests/components/improv_serial/test-usb_cdc.esp32-s3-idf.yaml

Fix Explanation

The fix stays tightly scoped to improv_serial.

1. Added write_frame_() in improv_serial_component.cpp. This helper writes a fully assembled frame through the active logger transport and loops until the backend accepts the whole buffer or stops making progress.

2. Changed write_data_() to assemble a contiguous frame before sending it. State/error frames now send the complete 12-byte frame at once. RPC responses now allocate a temporary contiguous buffer containing header + payload + checksum/newline and send it in one backend call sequence.

3. Added transport completion handling. USB_CDC now flushes after a successful full-frame write. USB_SERIAL_JTAG now waits for TX completion after a successful full-frame write.

4. Removed the ESP32-S3 USB_CDC validation block in __init__.py. That validation was acting as a workaround for the broken transport path. With the write path corrected, the configuration no longer needs to be rejected.

5. Added build coverage for ESP32-S3 USB_CDC. tests/components/improv_serial/test-usb_cdc.esp32-s3-idf.yaml ensures this config remains supported.

Test Results

1. Real hardware repro firmware built successfully.

$ ../tmp/platformio-venv/bin/python -m esphome compile ../tmp/improv_s3_usb_serial_jtag_idf.yaml
...
INFO Successfully compiled program.

2. Real hardware repro firmware flashed successfully.

$ ./hwctl flash 0x0:tmp/.esphome/build/improv-s3-idf-test/.pioenvs/improv-s3-idf-test/firmware.factory.bin
...
Hash of data verified.
Hard resetting via RTS pin...

3. Serial corruption on the stressed shared transport was reproduced on the board.

$ ./hwctl serial-follow 4096
...
n:199]: improv stress tick

4. Fixed JTAG repro firmware rebuilt successfully after the code change.

$ ../tmp/platformio-venv/bin/python -m esphome compile ../tmp/improv_s3_usb_serial_jtag_idf.yaml
...
INFO Successfully compiled program.

5. ESP32-S3 USB_CDC config now builds successfully instead of being rejected.

$ ../tmp/platformio-venv/bin/python -m esphome compile ../tmp/improv_s3_usb_cdc_idf.yaml
...
INFO Successfully compiled program.

6. Added explicit ESP32-S3 USB_CDC component test coverage in tests/components/improv_serial/test-usb_cdc.esp32-s3-idf.yaml.

Remaining Risks

• The exact browser-side TIMEOUT flow from the GitHub issue was not replayed end-to-end in a browser during this session; the reproduction here was transport-level on hardware plus source-level confirmation of the broken short-write handling.
• The linked issue references Arduino 3.2.1 specifically. The current repository state was verified on ESP-IDF-based repro configs, while the transport bug itself is framework-agnostic inside improv_serial’s ESP32 code path.
• The logger transport itself can still become noisy or congested under heavy output; this fix makes improv_serial robust against partial backend writes and frame splitting, but it does not change the logger subsystem broadly.

Suggested Follow-up Work

1. Add a focused regression test around the improv_serial frame writer so short-write behavior can be simulated without hardware.
2. Consider reusing the same full-frame write helper pattern anywhere else ESP32 binary protocols are emitted over logger-backed USB transports.
3. If Arduino 3.2.1 support remains important for this issue family, run a dedicated ESP32-S3 Arduino hardware verification once the unrelated toolchain drift in the local repo state is addressed.