pubsub interprocess comm is fast but does it scale?

[littlebee]

We have a new integration test that tests our interprocess communication through central_hub with a single publisher and a single subscriber. It's not exactly slow, but how does it scale to say something more complicated like with the vision service running and publishing {"recognition": [...big ol array of objects...]} 30 times per second. What is the latency and throughput of two threads communicating via central_hub state pub/sub while more is happening? Maybe add in the compass provider or simulator that updates a key with a single float value in addition to the recognition updates?

This can be a field test, or maybe better a src/basic_bot/debug test that we ship with the pip package. I don't think it needs to tested with every PR?

I also want to see if there are any standard benchmarks for pub/sub. I'd love to know how central_hub compares to Kafka, but honestly, it will probably not be necessary to have more than a dozen publishers and subscribers that connect to a single central_hub instance. Scatbot for example starts nine pub/sub clients connected to central_hub:

You can see the console output for pytest in real-time using the -s switch:

python -m pytest -s -vv tests/integration_tests/test_hub_state_monitor.py -k test_round_trip_latency

see also #45

On Pi5 Bookworm 64:

all messages received
average latency: 0.0007216763496398925
self.recv_latencies=[0.0008289813995361328, 0.0010194778442382812, 0.0008957386016845703, 0.0008783340454101562, 0.00086212158203125, 0.0008575916290283203, 0.0008361339569091797, 0.0008237361907958984, 0.0008296966552734375, 0.0007958412170410156, 0.0007848739624023438, 0.0007877349853515625, 0.0007660388946533203, 0.0007910728454589844, 0.0007596015930175781, 0.0007121562957763672, 0.000705718994140625, 0.000789642333984375, 0.0007441043853759766, 0.0007364749908447266, 0.0008144378662109375, 0.0007860660552978516, 0.0006957054138183594, 0.0006661415100097656, 0.0007336139678955078, 0.0006830692291259766, 0.0007183551788330078, 0.0006592273712158203, 0.0006759166717529297, 0.0006520748138427734, 0.0005984306335449219, 0.0006811618804931641, 0.0007076263427734375, 0.0006840229034423828, 0.0007219314575195312, 0.0006577968597412109, 0.0006799697875976562, 0.0006382465362548828, 0.0006778240203857422, 0.0006353855133056641, 0.0009431838989257812, 0.0008220672607421875, 0.0007941722869873047, 0.0007522106170654297, 0.0007681846618652344, 0.0007736682891845703, 0.0007233619689941406, 0.0006749629974365234, 0.0006814002990722656, 0.0006546974182128906, 0.0007724761962890625, 0.0007703304290771484, 0.0006961822509765625, 0.0006494522094726562, 0.0006892681121826172, 0.0007128715515136719, 0.0008020401000976562, 0.0007498264312744141, 0.0007169246673583984, 0.0006749629974365234, 0.0006635189056396484, 0.0007524490356445312, 0.0007443428039550781, 0.0007021427154541016, 0.0006868839263916016, 0.000667572021484375, 0.0007433891296386719, 0.0007290840148925781, 0.0007550716400146484, 0.0006866455078125, 0.0006649494171142578, 0.0007143020629882812, 0.0007114410400390625, 0.0006327629089355469, 0.0006318092346191406, 0.0006270408630371094, 0.0006392002105712891, 0.0006949901580810547, 0.0006940364837646484, 0.0006985664367675781, 0.0006742477416992188, 0.0007250308990478516, 0.0007448196411132812, 0.0008056163787841797, 0.0010347366333007812, 0.0008366107940673828, 0.0007450580596923828, 0.0007030963897705078, 0.0006422996520996094, 0.0006015300750732422, 0.0006456375122070312, 0.0006256103515625, 0.0006923675537109375, 0.0007131099700927734, 0.0006575584411621094, 0.0005812644958496094, 0.0006134510040283203, 0.0006201267242431641, 0.0004036426544189453, 0.0004673004150390625]
overall_duration=0.05681633949279785
p90 latency: 0.0007033374283339951
throughput: 1760.0570697215753 messsages per second