hidapi documentation about number of bytes written and read

[mcuee]

Discusison here:

• HIDAPI testing device with known good firmware, including using Virtual HID device for CI Testing #478 (comment)
• HIDAPI testing device with known good firmware, including using Virtual HID device for CI Testing #478 (comment)

As @todbot mentioned, this has been an issue bugging him. Same for me.

Testing device: Circuit Python rawhid example, no report IDs.

• Generic HID device support adafruit/Adafruit_CircuitPython_HID#115 (comment)

boot.py

import usb_hid

RAWHID_REPORT_DESCRIPTOR = bytes((
    0x06, 0x00, 0xFF,  # Usage Page (Vendor Defined 0xFF00)
    0x09, 0x01,        # Usage (0x01)
    0xA1, 0x01,        # Collection (Application)
    0x09, 0x02,        #   Usage (0x02)
    0x15, 0x00,        #   Logical Minimum (0)
    0x26, 0xFF, 0x00,  #   Logical Maximum (255)
    0x75, 0x08,        #   Report Size (8)
    0x95, 0x40,        #   Report Count (64)
    0x81, 0x02,        #   Input (Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position)
    0x09, 0x03,        #   Usage (0x03)
    0x15, 0x00,        #   Logical Minimum (0)
    0x26, 0xFF, 0x00,  #   Logical Maximum (255)
    0x75, 0x08,        #   Report Size (8)
    0x95, 0x40,        #   Report Count (64)
    0x91, 0x02,        #   Output (Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position,Non-volatile)
    0xC0,              # End Collection
))

raw_hid = usb_hid.Device(
    report_descriptor=RAWHID_REPORT_DESCRIPTOR,
    usage_page=0xFF00,
    usage=0x01,
    report_ids=(0,),
    in_report_lengths=(64,),
    out_report_lengths=(64,),
)

usb_hid.enable((raw_hid,))

code.py

import usb_hid
import time

d = usb_hid.devices[0]

while True:
    report = bytearray(64)  # must be same size as specified in HID Report Descriptor in boot.py    

    report[0] = 1
    report[1] = 2
    report[2] = 3
    report[3] = 4
    report[63] = 64
    d.send_report(report)
    time.sleep(1)
    print(d.get_last_received_report())

hidapitester output under Windows, it says 65 bytes written

PS C:\work\hid\hidapitester> .\hidapitester --vidpid 2e8a:102e --open --length 64 --send-output 1,2,3,4,5,6,7,8
Opening device, vid/pid: 0x2E8A/0x102E
Writing output report of 64-bytes...wrote 65 bytes:
 01 02 03 04 05 06 07 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Closing device

hidapitester output under Linux (Ubuntu 20.04 x64): it says 64 bytes written.

mcuee@UbuntuSwift3:~/build/hid/hidapitester$ sudo ./hidapitester --vidpid 2e8a:102e --open --length 64 --send-output 1,2,3,4,5,6,7,8
Opening device, vid/pid: 0x2E8A/0x102E
Writing output report of 64-bytes...wrote 64 bytes:
 01 02 03 04 05 06 07 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Closing device

hidapitester output under macOS 13.4 (Mac Mini M1): it says 64 bytes written.

mcuee@mcuees-Mac-mini hidapitester % ./hidapitester --vidpid 2e8a:102e --open --length 64 --send-output 1,2,3,4,5,6,7,8
Opening device, vid/pid: 0x2E8A/0x102E
Writing output report of 64-bytes...wrote 64 bytes:
 01 02 03 04 05 06 07 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Closing device```

hidtest info about the device (using Linux to get the real HID report descriptors)

Device Found
type: 2e8a 102e
path: /dev/hidraw3
serial_number: DE6185100F4D6522
Manufacturer: VCC-GND Studio
Product: YD-RP2040
Release: 100
Interface: 3
Usage (page): 0x1 (0xff00)
Bus type: 1 (USB)

Report Descriptor: (34 bytes)
0x06, 0x00, 0xff, 0x09, 0x01, 0xa1, 0x01, 0x09, 0x02, 0x15,
0x00, 0x26, 0xff, 0x00, 0x75, 0x08, 0x95, 0x40, 0x81, 0x02,
0x09, 0x03, 0x15, 0x00, 0x26, 0xff, 0x00, 0x75, 0x08, 0x95,
0x40, 0x91, 0x02, 0xc0,

[mcuee]

Relevant discussion before, which fixed the reported Input Report number of reading bytes

• hid_get_input_report returns one byte more than hid_read, also how to deal with hid_get_feature_report #229
• hid_get_input_report: Correct number of bytes_returned #232

PR #232 is for sure correct, since the original code wrongly assumes the buffer does not contain the report ID.

Probably similar patch is requrired for Output report.

The Windows behavior is not wrong per se, Windows HID driver indeed adds the report ID 0 (no report ID) as the first byte in the buffer. However the report ID 0 is not really send on the USB bus. Since HIDAPI is a cross-platform library, I think we should make Windows to behave the same as Linux and macOS. Therefore we should not report 65 bytes in the above example for Windows, but rather 64 bytes which are the number of bytes the device really received.

When there is repot ID, Windows will still require the first byte to be the report ID, which is transmitted on the USB Bus.

Idealy we should always report the number of bytes transmitted on the USB bus, no matter there is a report ID or not. In that case, we will get the same number of bytes, for HID Input report, Output report and Feature report.

[mcuee]

Test using Jan Axelson's generic HID example here.
http://janaxelson.com/hidpage.htm

Windows reported 3 for Feature report writing and reading, one byter longer than the real device (2 bytes Input report, Output Report and Feature report, no report IDs). And same for Output report, Windows reports 3 bytes written.

Only for Input report, Windows is correct to report 2 bytes have been read, thanks to PR #232.

WIndows also always requires to send report ID 0 for the Output report for devices without report ID (and this causes the extra one byte length reported for the write), whereas Linux does not.

//Class specific descriptor - HID 
// Defines 2-byte Input, Output, and Feature reports with vendor-defined data.

ROM struct{BYTE report[HID_RPT01_SIZE];}hid_rpt01={
{
  	0x06, 0xA0, 0xFF,		// Usage page (vendor defined) 
  	0x09, 0x01,				// Usage ID (vendor defined)
  	0xA1, 0x01,				// Collection (application)

	// The Input report
       0x09, 0x03,     		// Usage ID - vendor defined
       0x15, 0x00,     		// Logical Minimum (0)
       0x26, 0xFF, 0x00,   	// Logical Maximum (255)
       0x75, 0x08,     		// Report Size (8 bits)
       0x95, 0x02,     		// Report Count (2 fields)
       0x81, 0x02,     		// Input (Data, Variable, Absolute)  

	// The Output report
       0x09, 0x04,     		// Usage ID - vendor defined
       0x15, 0x00,     		// Logical Minimum (0)
       0x26, 0xFF, 0x00,   	// Logical Maximum (255)
       0x75, 0x08,     		// Report Size (8 bits)
       0x95, 0x02,     		// Report Count (2 fields)
       0x91, 0x02,      	// Output (Data, Variable, Absolute)  

	// The Feature report
       0x09, 0x05,     		// Usage ID - vendor defined
       0x15, 0x00,     		// Logical Minimum (0)
       0x26, 0xFF, 0x00,   	// Logical Maximum (255)
       0x75, 0x08,			// Report Size (8 bits)
       0x95, 0x02, 			// Report Count	(2 fields)				
       0xB1, 0x02,     		// Feature (Data, Variable, Absolute)  

  	0xC0}					// end collection
};

PS C:\work\hid\hidapitester> .\hidapitester --vidpid 0925:7001 --open -l 3 --send-output 0,1,2 --read-input
Opening device, vid/pid: 0x0925/0x7001
Writing output report of 3-bytes...wrote 3 bytes:
 00 01 02
Reading 3-byte input report 0, 250 msec timeout...read 2 bytes:
 01 02 00
Closing device

PS C:\work\hid\hidapitester> .\hidapitester --vidpid 0925:7001 --open -l 2 --send-output 1,2 --read-input
Opening device, vid/pid: 0x0925/0x7001
Writing output report of 2-bytes...wrote 3 bytes:
 01 02
Reading 2-byte input report 0, 250 msec timeout...read 2 bytes:
 02 00
Closing device

PS C:\work\hid\hidapitester> .\hidapitester --vidpid 0925:7001 --open -l 3 --send-feature 0,1,2 --read-feature 0
Opening device, vid/pid: 0x0925/0x7001
Writing 3-byte feature report...wrote 3 bytes:
 00 01 02
Reading 3-byte feature report, report_id 0...read 3 bytes:
 00 01 02
Closing device

But in this case, hidapitester under Linux has the same behavior in terms of feature report as Windows.

And we have already a commit similar to PR #232 for Feature report. So maybe the device really has a three bytes Feature report.

6fcb0bb

mcuee@UbuntuSwift3:~/build/hid/hidapitester$ sudo ./hidapitester  --vidpid 0925:7001 --open -l 3 --send-output 0,1,2 --read-input
Opening device, vid/pid: 0x0925/0x7001
Writing output report of 3-bytes...wrote 3 bytes:
 00 01 02
Reading 3-byte input report 0, 250 msec timeout...read 2 bytes:
 01 02 00
Closing device

mcuee@UbuntuSwift3:~/build/hid/hidapitester$ sudo ./hidapitester  --vidpid 0925:7001 --open -l 2 --send-output 1,2 --read-input
Opening device, vid/pid: 0x0925/0x7001
Writing output report of 2-bytes...wrote 2 bytes:
 01 02
Reading 2-byte input report 0, 250 msec timeout...read 2 bytes:
 01 02
Closing device

mcuee@UbuntuSwift3:~/build/hid/hidapitester$ sudo ./hidapitester  --vidpid 0925:7001 --open -l 3 --send-feature 0,1,2 --read-feature 0
Opening device, vid/pid: 0x0925/0x7001
Writing 3-byte feature report...wrote 3 bytes:
 00 01 02
Reading 3-byte feature report, report_id 0...read 3 bytes:
 00 01 02
Closing device

[mcuee]

So I am pretty sure we need to fix number of bytes written Output Report under Windows, with all my test results in #478.

I am not so sure about Feature report yet since Linux behaves the same as Windows.

Current implementation under Windows. It mentions that Windows expects the number of bytes which are in the longest report (plus one for the report number) bytes even if the data is a report which is shorter than that.

I think we need to report the length minus one, at least for the case when there is no report ID.

int HID_API_EXPORT HID_API_CALL hid_write(hid_device *dev, const unsigned char *data, size_t length)
{
	DWORD bytes_written = 0;
	int function_result = -1;
	BOOL res;
	BOOL overlapped = FALSE;

	unsigned char *buf;

	if (!data || !length) {
		register_string_error(dev, L"Zero buffer/length");
		return function_result;
	}

	register_string_error(dev, NULL);

	/* Make sure the right number of bytes are passed to WriteFile. Windows
	   expects the number of bytes which are in the _longest_ report (plus
	   one for the report number) bytes even if the data is a report
	   which is shorter than that. Windows gives us this value in
	   caps.OutputReportByteLength. If a user passes in fewer bytes than this,
	   use cached temporary buffer which is the proper size. */
	if (length >= dev->output_report_length) {
		/* The user passed the right number of bytes. Use the buffer as-is. */
		buf = (unsigned char *) data;
	} else {
		if (dev->write_buf == NULL)
			dev->write_buf = (unsigned char *) malloc(dev->output_report_length);
		buf = dev->write_buf;
		memcpy(buf, data, length);
		memset(buf + length, 0, dev->output_report_length - length);
		length = dev->output_report_length;
	}

	res = WriteFile(dev->device_handle, buf, (DWORD) length, NULL, &dev->write_ol);

	if (!res) {
		if (GetLastError() != ERROR_IO_PENDING) {
			/* WriteFile() failed. Return error. */
			register_winapi_error(dev, L"WriteFile");
			goto end_of_function;
		}
		overlapped = TRUE;
	}

	if (overlapped) {
		/* Wait for the transaction to complete. This makes
		   hid_write() synchronous. */
		res = WaitForSingleObject(dev->write_ol.hEvent, 1000);
		if (res != WAIT_OBJECT_0) {
			/* There was a Timeout. */
			register_winapi_error(dev, L"hid_write/WaitForSingleObject");
			goto end_of_function;
		}

		/* Get the result. */
		res = GetOverlappedResult(dev->device_handle, &dev->write_ol, &bytes_written, FALSE/*wait*/);
		if (res) {
			function_result = bytes_written;
		}
		else {
			/* The Write operation failed. */
			register_winapi_error(dev, L"hid_write/GetOverlappedResult");
			goto end_of_function;
		}
	}

end_of_function:
	return function_result;
}

[mcuee]

@JoergAtGithub

I think you have very good understanding of the issue and the necessary test devices. Please take a look at this issue as well when you got the chance.

[mcuee]

@Youw

I think for Feature report, Linux hidraw implemenation is kind of the same as Windows. So that probably explains why I see the same behavior under Windows and Linux, since we just report the buffer length (without minus one for the case when there is no repor ID) and not the data on the USB bus.

Commits 6fcb0bb is for sure correct, since the original code wrongly assues the buffer does not contain the Feature Report ID.

Now the question is whether we need to minus 1 for the case of no Report ID, to reflect the data on the USB Bus, and not the OS driver.

https://www.kernel.org/doc/Documentation/hid/hidraw.txt

HIDIOCSFEATURE(len): Send a Feature Report
This ioctl will send a feature report to the device. Per the HID
specification, feature reports are always sent using the control endpoint.
Set the first byte of the supplied buffer to the report number. For devices
which do not use numbered reports, set the first byte to 0. The report data
begins in the second byte. Make sure to set len accordingly, to one more
than the length of the report (to account for the report number).

HIDIOCGFEATURE(len): Get a Feature Report
This ioctl will request a feature report from the device using the control
endpoint. The first byte of the supplied buffer should be set to the report
number of the requested report. For devices which do not use numbered
reports, set the first byte to 0. The report will be returned starting at
the first byte of the buffer (ie: the report number is not returned).

[mcuee]

As of now, hidapitester using hidapi-libusb (custom build) behaves pretty same as hidapi-hidraw. So that is a good thing.

mcuee@UbuntuSwift3:~/build/hid/hidapitester$ sudo ./hidapitester_libusb  --vidpid 0925:7001 --open -l 3 --send-output 0,1,2 --read-input
Opening device, vid/pid: 0x0925/0x7001
Writing output report of 3-bytes...wrote 3 bytes:
 00 01 02
Reading 3-byte input report 0, 250 msec timeout...read 2 bytes:
 01 02 00
Closing device

mcuee@UbuntuSwift3:~/build/hid/hidapitester$ sudo ./hidapitester_libusb  --vidpid 0925:7001 --open -l 2 --send-output 1,2 --read-input
Opening device, vid/pid: 0x0925/0x7001
Writing output report of 2-bytes...wrote 2 bytes:
 01 02
Reading 2-byte input report 0, 250 msec timeout...read 2 bytes:
 01 02
Closing device

mcuee@UbuntuSwift3:~/build/hid/hidapitester$ sudo ./hidapitester_libusb  --vidpid 0925:7001 --open -l 3 --send-feature 0,1,2 --read-feature 0
Opening device, vid/pid: 0x0925/0x7001
Writing 3-byte feature report...wrote 3 bytes:
 00 01 02
Reading 3-byte feature report, report_id 0...read 3 bytes:
 00 01 02
Closing device

[Youw]

  	@returns
  		This function returns the number of bytes read plus
  		one for the report ID (which is still in the first
  		byte), or -1 on error.
  		Call hid_error(dev) to get the failure reason.
  */
  int HID_API_EXPORT HID_API_CALL hid_get_feature_report(hid_device *dev, unsigned char *data, size_t length);

As per HIDAPI documentation, we should include the 0 report ID in the return. Is it included?

[JoergAtGithub]

0 is a placeholder and not a Report ID. This is because the HID class specification specifies "If a Report ID tag was used in the Report descriptor, all reports include a single byte ID prefix. If the Report ID tag was not used, all values are returned in a single report and a prefix ID is not included in that report." and further about the representation in the ReportDescriptor "Set Report ID to 0 (zero) if Report IDs are not used."

Therefore only values from 1 to 255 can be Report IDs. The value 0 can be used in the ReportDescriptor, to define that the reports do not contain a ReportID.

Windows implements it as specified in the class specification and adds the Report ID prefix byte only for devices that use Report IDs.

I know that we need to keep backward compatibility, but from a clean API I would expect a clear seperation between address and data in two arguments.

[mcuee]

I know that we need to keep backward compatibility, but from a clean API I would expect a clear seperation between address and data in two arguments.

@JoergAtGithub

Just wondering what you mean by the above. Are you proposing some new APIs? I think that will be okay if there are real benefits. We can always keep the older APIs (and in the future to deprecate them in HIDAPI 1.x version if necessary).