Unable to join larger rooms

[lifeofguenter]

It is impossible currently to join larger external rooms - after a long time there is an error.

The VM is fairly small (4 vCPU / 6GB RAM / SSD) and also located in Asia, however, neither is there any high load.

I do not run docker/containers, and I also already run unbound to help with DNS cache.

Memory usage is fairly low:

Services:
federation_mutex: 1
bad_event_ratelimiter: 0 (0 B)
roomid_spacehierarchy_cache: 3
state_mutex: 1
stateinfo_cache: 35 93 (1.0 MiB)
insert_mutex: 0

Database:
Memory buffers: 94.18 MiB
Pending write: 94.18 MiB
Table readers: 0.35 MiB
Row cache: 3.16 MiB
Shared cache: 6.45 MiB
eventid_outlierpdu cache: 1.45 MiB
eventid_pduid cache: 0.07 MiB
eventid_shorteventid cache: 2.12 MiB
servername_destination cache: 0.09 MiB
servername_override cache: 0.12 MiB
servernameevent_data cache: 0.06 MiB
shorteventid_authchain cache: 0.00 MiB
shorteventid_eventid cache: 0.17 MiB
shortstatekey_statekey cache: 1.12 MiB
statekey_shortstatekey cache: 1.21 MiB

Allocator:
allocated: 172.64 MiB
active: 180.78 MiB
mapped: 258.23 MiB
metadata: 4.52 MiB
resident: 249.62 MiB
retained: 27.77 MiB

but increased the following values just in case:

db_cache_capacity_mb = 2048.0
db_write_buffer_capacity_mb = 256.0

bottom line: expectations for joining large room (10k+) would be 100% CPU + OOM, but nothing is happening and client eventually errors out. Smaller rooms up to 1k are fine.

[jevolk]

Are these matrix.org rooms?

[lifeofguenter]

Are these matrix.org rooms?

Yes, but not only. E.g. the symptom is the same joining any larger room: long times, often errors when joining, sometimes it works after couple of tries - but CPU/RAM/Net/Disk usage is close to zero.

Maybe its running too few threads?

[jevolk]

This is probably a duplicate of #128. What kind of times are you experiencing for the join? When there is a failure, what is the error/code seen in either the server logs or client?

The majority of time spent joining large rooms is from acquiring server keys. That also relies on matrix.org in many cases. Either way Synapse is very slow to serve room joins for large rooms, even for servers other than matrix.org. The reason you are not seeing any CPU/IO activity is because Tuwunel is waiting for Synapse to respond.

There is nothing we can do about this. I recommend joining large rooms via other Tuwunel servers.

[jevolk]

My sense is that this is still a duplicate of #128 which has now been resolved. If this is still a persistent issue for non-matrix.org rooms, especially when the remote server is not Synapse, I would be happy to re-evaluate it as necessary.