# RNS 1.5.0 Testing - Traffic Prioritization & Stability Improvements _General · started by Mark on Mon, Aug 17, 2026 8:14 PM_ --- ## Original post **Mark** · Mon, Aug 17, 2026 8:14 PM The latest source version of RNS on Aleph has completely reworked inbound traffic processing in the Transport core. By default, RNS now uses four different prioritization levels of packets, which very effectively alleviates increased latency for actual traffic under the insane levels of PR/announces DoS attempts going on lately. Bad traffic from ingress-limited interfaces are now placed in a low-priority queue, which only drains once higher-priority traffic is done. There's much more detail in the source itself, if you want to have a look, but I'll keep it short here. There's also a long list of various bugfixes. I'd appreciate any help testing this out before release, it's a pretty big restructuring of how inbound traffic works, so would be good to give it a bit of a spin first. Also, *what the actual ?!?* is up with these insane levels of PR/announce spam going on? For example, on one node right now, I'm seeing ~30 clients just connecting and dumping *nothing* but path requests at around 75 per second. Then there's all the DoS attempts from TOR exit nodes, attempting more or less the same, but mostly just connecting for a few seconds, dumping a couple thousand PRs or announces, and then disconnecting again. I'm not sure if whoever wrote that script missed the fact that they're getting silently blocked. But shhh, don't tell them :P Let em keep playing around, it's great for testing stuff out. *Some* of this definitely seems to come from malicious intent, but some of it is also just so weird, that I'm wondering how much of it is just completely broken implementations running wild and doing crazy stuff. I haven't looked at any of the new `matmul`-generated "implementations" lately, but I'm wondering if any of them actual have any sort of filtering, de-duplication of announces/PRs, rate-limiting, et cetera, et cetera, or if they're partly responsible for this mess by just passing whatever is handed to them happily to the entire network. --- ## Reply 1 **Mark** · Mon, Aug 17, 2026 8:16 PM Also, this lays the groundwork for fully customizable traffic prioritization and QoS functionality, so certain traffic can go to a real-time queue, and so on. That won't be in the next release though, let's get the foundations well tested and optimized first. --- ## Reply 2 **Anonymous** · Mon, Aug 17, 2026 8:44 PM DoS? Who sees rns as the enemy? --- ## Reply 3 **qbit** · Mon, Aug 17, 2026 9:13 PM Updated my main nodes. Things seem stable. Watchin the queue pressure meow. --- ## Reply 4 **K8** · Mon, Aug 17, 2026 9:37 PM This looks great, awesome work! I'll do what I can to test this out. I have been having a hell of a time keeping `rns.kin.earth` stable recently, even with very aggressive ingress limiting for announces and PRs (and egress limiting to not pass any bullshit downstream). It's not like it's been particularly stable to begin with so I'm inclined to just throw this release up there and see how it goes. The node does ok for a while (maybe for a few days to a week) but then I start seeing RNS peg a core to 100% and hang (unrelated to discovery stamps). Packets stop making it through, links have issues establishing, and even `rnstatus` can hang for 10-60s at times. I really have no idea the root cause at this point. I just restart RNS when things start having issues and it works ok for a few more days. The behavior kind of pattern matches to getting stuck spending too much time in garbage collection but I haven't verified (I was going to experiment by having `Transport._jobs()` run garbage collection less frequently instead of every 0.25s, or maybe having it decide when to GC based on the fraction of the period that `_jobs()` actually spent doing work or something like that). This is all on a 4-core VPS with 8GB RAM and fast network. I think part of it is possibly that I have a high stamp value, which puts me at or near the top of the list for auto-connection. A result of that, as far as I can tell, is that a ton of transport nodes out there auto-connect to my node. One of the patterns I've observed is the node getting pummeled by duplicate path requests (sometimes 50+) all at once, whenever a path request happens on the network. Filtering dupes is normal behavior but the extent of it was pretty extreme, and points to the node being over-connected (aka a *nexus of bullshit* 😂). Inbound/outbound queues with prioritization should help a lot. I'm working on collecting more in-depth statistics from the instance to see if I can spot any other patterns. **Edit:** looking good so far. --- ## Reply 5 **p1ld7a** · Tue, Aug 18, 2026 5:20 AM @K8: Sharing the same issue with 1.4.2. After a while, the service stop working. Here's the last one: ``` aoû 18 07:15:06 apollo systemd[1]: rnsd.service: Main process exited, code=exited, status=255/EXCEPTION aoû 18 07:15:06 apollo systemd[1]: rnsd.service: Failed with result 'exit-code'. aoû 18 07:15:06 apollo systemd[1]: rnsd.service: Consumed 3min 14.919s CPU time over 11min 36.317s wall clock time, 223.4M memory peak, 69M read from disk, 28.8M written to disk, 29.3M incoming IP traffic, 28.2M outgoing IP traffic. ``` I won't be able to test 1.5.0 sadly, but I hope it will fix this. --- ## Reply 6 **bergie** · Tue, Aug 18, 2026 5:37 AM **Mark** wrote: > I haven't looked at any of the new `matmul`-generated "implementations" lately, but I'm wondering if any of them actual have any sort of filtering, de-duplication of announces/PRs, rate-limiting, et cetera, et cetera, or if they're partly responsible for this mess by just passing whatever is handed to them happily to the entire network. So, I checked. Since reticulum-js is currently leaf node only (no transport capability), we're not retransmitting announces or path requests. However, since we probably want to have `transport: true` in the future, I'll try to make sure we handle these things as close to Python as is reasonable (given difference in event model, programming language, etc). De-duplication of announces is already in place, and the max announce rate allowed is once per minute (30min default, though this is JS and such limits can be overridden). What's missing is at least rate limiting. I'll also see what we can do to ensure JS apps can't accidentally spam path requests etc. --- ## Reply 7 **Mark** · Tue, Aug 18, 2026 10:08 AM Pushed some more tweaks for this. Queue sizes are now configurable with: ```ini [reticulum] qlen_in_data = 4096 qlen_in_announce = 256 qlen_in_pr = 256 qlen_in_il = 128 ``` Also tuned the defaults, the previous one were unnecessarily large. You can most likely drop the IL queue even further, running it at 48 pkts right now, and seems to be a good spot for my particular node. --- ## Reply 8 **K8** · Wed, Aug 19, 2026 1:26 AM I've been running on `503bd6c8` for about 10 hours. Right now I have around 200 clients, and I'm having a hard time actually using the node. Can't even browse Nomad sites that are hosted on the same machine. Queue stats: ``` Totals : ↑3.45 GB 201.76 Kbps ↓3.14 GB 293.58 Kbps Qu. Pressure : 0.4% total, 19 pkts, 12341 dropped 0.0% data, 0 pkts 0.0% announce, 9 pkts, 1386 dropped 0.0% path request, 1 pkts 7.0% ingress limiter, 9 pkts, 10955 dropped ``` The dropped counts don't seem to be raising particularly quickly right now. Node has the same symptoms I described above with constant high CPU usage by RNS on one core, and `rnstatus` is back to taking ~20s to print anything, so the queues don't seem to be helping with whatever is causing all that. Logs are completely full of path requests as usual. --- ## Reply 9 **KenAKAFrosty** · Wed, Aug 19, 2026 2:08 AM Have you noticed anything else about the patterns? Like time of day, or the window of time they're connecting for? And are the PRs/announces you're seeing all for the same destination(s), or varied? Just asking cuz that might help me dig for specifics. But in the aggregate I haven't seen any sign of that. Currently connected to ~60 public nodes as a transport node to stress test and look for this kind of stuff, and while the data flow fluctuates a lot like you'd expect, and the link counts fluctuate a lot, the inbound announce flow is pretty steady and doesn't show signs of that. Even over a 24 hour period it's mostly a steady flow with a large number of expected-ignored ones ![image.png](/storage/forum/gHB5ak7w0wgcwPcFhY0zc9fPM96y1ss7dx3wHvom.png) *(also just in case anyone's concerned, the seemingly-large number of outbound announces here is just because this is total announces on all interfaces. Each accepted inbound announce from an interface relays out across the rest)* There's one specific interface that had a few spikes compared to the pack, but even then they're extremely brief and only about 20/s during that small window. Doesn't quite seem to fit. ![image.png](/storage/forum/obhoPwkPgKMbT8o2n3ROKtXPxgti7X6sV7CQKLUd.png) Of course, I can't see what kind of traffic those public nodes are getting directly. And it might be something only happening directly (like your experience, direct connection + immediate spam). But it seems these public nodes are, at least, indeed properly filtering any excess bursts before passing qualified traffic along. --- ## Reply 10 **weird_bleks** · Wed, Aug 19, 2026 7:10 AM I've had the same experience as @K8. I've updated my public node to test it and about 12h ago(after running for close to 10h) i've had connectivity issues. Not sure if it's related but i suspect it is. Most nomad pages failed to establish link. Some worked suprisingly, but most didn't. One particularity i observed was when i tried to connect to rns.recipes via Columba, that the request failed again and again but i didn't receive a new announce/path, it was still showing '4h ago'. This makes me think the path request was blocked somehow, and it never got back to me. This aligns with the latest changes, and i think somewhere a queue was full so my PRs never got accepted. Maybe this happened when the network was being spammed like Mark said, maybe not, who knows. It did happen with the default settings after Mark decreased the sizes, so there's that, maybe i should have used higher values for queues. Unfortunately i didn't actually debug, just observed passively. After restarting the node it worked better. --- ## Reply 11 **Mark** · Wed, Aug 19, 2026 4:34 PM Thanks for the testing and feedback everyone! I just pushed another round of fixes and improvements to Aleph, latest on `master` now. There was a few mistakes relating to PR processing, and a range of other bugs fixed as well. I think some of the core pegging might have stemmed from a stupid bug in the way pending PRs were processed, and that's fixed now. @K8 (and others), can you try the latest version (and also try dropping loglevel below `LOG_PATHING`)? Under insane announce/PR loads, logging all of that *is* going to have a significant performance hit, especially on IOPS constrained servers. I recently refactored all the logging so `LOG_DEBUG` should now be sufficient for mostly everything, unless you actually want to see all that path/announce log spam. --- ## Reply 12 **KenAKAFrosty** · Wed, Aug 19, 2026 10:51 PM Dug into it some more, findings might be helpful. Saw a notable, short-term burst after all. It was evenly spread across most of these public nodes, but struck me as odd that there'd be a unified burst like that. Kept looking and saw that right around that same time, one of the public nodes flapped. Also noted about 70% of the destinations are for "lxmf.delivery" and "lxst.telephony", (~50% the former, ~20% the latter). It starts to make a bit more sense looking at Sideband on android. The default-on start_announce config setting means it will re-announce on all interfaces after one drops. Also, when an interface comes back up, it re-announces again. The rate limiting there looks like just a single 3.5s cooldown kind of thing. So it looks *highly* plausible that what's happening is as follows: Android Sideband apps are connected to a public tcp/backbone server. Often a few of them. Then, one of the public nodes flaps, for whatever reason. When it goes down, all those android Sideband instances re-announce on all interfaces, including those other public nodes. Genuine, fresh announces which is why they're also forwarded. After that, ~5s later, when their interfacce to the flapped node reconnects, it fires an announce on all interfaces again. The reconnect period is longer than those reactive announces' 3.5s cooldown, so this would happen pretty reliably. In fact, if that same flapping public node was itself a client to one of your nodes, then the behavior makes a lot of sense with what you experienced: If that other public node flapped for reasons that take that client interface offline for a blip as well (seems likely), then it would be reconnecting to your node around the same time as all the android Sideband users reconnect to *it*, and it forwards those once-again-fresh-legitimate-announces. So for each user, a flap can trigger up to 4 announces per user it happens to. There are the 2 destinations of lmxf.delivery and sometimes lxst.telephony, with the 2nd wave occasionally getting eaten by the 3.5s cooldown, just depending on where exactly the timing windows land. Since these are categorically already known destinations by the time all this happens, the rush doesn't enter the burst ingress jail. It's only subject to the 2% limit. Considering the flaps don't happen super often, the recently-added defaults wouldn't likely stop it either, since it's a 5 announce grace on the per-hour limit of announces per destination. Even the TCP default bitrate guess of 10Mbps would allow for ~100 transportd announces/second. It would only take like 40-70 android SB users on the other side of the public node to feel this. Very reasonable that's already happening Also, a lot of the steady baseline announce flow was app data like this `{"v":1,"n":"Maps","c":0}`, meshchatx's map pack beacon thing. It's on by default for all users and fires one every 15 minutes, even for users with no map packs published. But a recent commit is changing that setting to be off by default and to not do the announce unless at least 1 map is published, so that's apparently coming down the pipeline and should help too. Anyway, haven't narrowed down the path requests as much, but considering a lot of active Links could be open on those same interfaces when the flap happens, there's a good chance that's related, too. The burst I saw also showed a burst of PRs shortly *before* the announces, which lines up. --- ## Reply 13 **Mark** · Wed, Aug 19, 2026 11:27 PM Thanks for looking into it Ken, but it's not so much the announce waves happenning as clients reconnect that I find weird. Those make pretty decent sense, and it's usually just one or two announces per sub-interface. They do tend to come in a wave, but usually only around a couple of hundred at a time, not really a problem. And *active* links that go down wouldn't cause new path requests when they drop, not in RNS at least. It's only attempted links that were pending but never established (and only at the initiator and penultimate hop that would occur, not in any way over the entire path) that trigger a path request under some conditions. The thing I find weird is more the clients that are connecting, and then pumping out path requests or announces at rates between 10 to 100 Hz, often sustained for 5-30 seconds. Some of these clients connect for exactly 5 seconds (with a jitter of only ~5-15ms), and start spamming as soon as they're connected, then disconnect (presumably to rotate IPs). Most of that kind of "traffic" comes from TOR exit nodes. Another class of them *stay* connected, and will wait for varying intervals until they start dumping. They then dump their (primarily PR) spam for destinations that has never existed on the network, keep it up for 10-30 seconds, go silent again and wait until the next cycle. Sometimes it coincides with several clients doing it at the same time, which can easily compound into 10k+ path requests being received in around 30 seconds. --- ## Reply 14 **KenAKAFrosty** · Wed, Aug 19, 2026 11:52 PM **Mark** wrote: > Thanks for looking into it Ken, but it's not so much the announce waves happenning as clients reconnect that I find weird. Those make pretty decent sense, and it's usually just one or two announces per sub-interface. They do tend to come in a wave, but usually only around a couple of hundred at a time, not really a problem. > > And *active* links that go down wouldn't cause new path requests when they drop, not in RNS at least. It's only attempted links that were pending but never established (and only at the initiator and penultimate hop that would occur, not in any way over the entire path) that trigger a path request under some conditions. > > The thing I find weird is more the clients that are connecting, and then pumping out path requests or announces at rates between 10 to 100 Hz, often sustained for 5-30 seconds. Some of these clients connect for exactly 5 seconds (with a jitter of only ~5-15ms), and start spamming as soon as they're connected, then disconnect (presumably to rotate IPs). Most of that kind of "traffic" comes from TOR exit nodes. > > Another class of them *stay* connected, and will wait for varying intervals until they start dumping. They then dump their (primarily PR) spam for destinations that has never existed on the network, keep it up for 10-30 seconds, go silent again and wait until the next cycle. > > Sometimes it coincides with several clients doing it at the same time, which can easily compound into 10k+ path requests being received in around 30 seconds. Oh right sorry to clarify, I mean in LXMF, the rtt x 6 timeout. Assuming for these kinds of links that's like 0.5-2s. With maybe a few seconds of delay from the polling timing. But anyway, yeah what you're describing definitely sounds like it's a different breed. Will keep an eye on that pattern going forward. For now at the very least it's one small lil data point that I'm currently not seeing any sign of that traffic pattern from my position. Thanks! --- ## Reply 15 **KenAKAFrosty** · Thu, Aug 20, 2026 12:01 AM Actually, might've spoken too soon; gonna dig in more on specifically looking for the volume of destinations that i've not heard an announce for myself and work backward from there. If anything fruitful comes of it I'll share with ya --- ## Reply 16 **K8** · Thu, Aug 20, 2026 12:22 AM **Mark** wrote: > Thanks for the testing and feedback everyone! > > I just pushed another round of fixes and improvements to Aleph, latest on `master` now. There was a few mistakes relating to PR processing, and a range of other bugs fixed as well. I think some of the core pegging might have stemmed from a stupid bug in the way pending PRs were processed, and that's fixed now. > > @K8 (and others), can you try the latest version (and also try dropping loglevel below `LOG_PATHING`)? Under insane announce/PR loads, logging all of that *is* going to have a significant performance hit, especially on IOPS constrained servers. I recently refactored all the logging so `LOG_DEBUG` should now be sufficient for mostly everything, unless you actually want to see all that path/announce log spam. Cool, I'm deployed up to the latest HEAD now, we'll see how it goes. I sent you a patch to move logging into its own thread, that should help some with that issue (previously it was also reopening and closing the logfile on every single log write, that certainly wasn't helping anything either). I've thought about the logging perf before, but my server has NVMe backed storage that I've tested to be plenty fast so I don't think that is the issue I'm seeing. I've still reduced the log level for now to see if it helps. So far it doesn't seem to have changed the CPU usage patterns at all. More stats from the last version I deployed: ```text Totals : ↑11.30 GB 1.51 Mbps ↓9.99 GB 1.01 Mbps Qu. Pressure : 0.0% total, 0 pkts, 85120 dropped 0.0% data, 0 pkts 0.0% announce, 0 pkts, 40479 dropped 0.0% path request, 0 pkts, 290 dropped 0.0% ingress limiter, 0 pkts, 44351 dropped Uptime is 20h, 49m and 56.32s, 573 entries in link table ``` --- ## Reply 17 **Mark** · Thu, Aug 20, 2026 8:10 AM Thanks for the patch @K8, got it fetched locally here. I'll look at that after 1.5.0 is out, there's already plenty of new stuff for this one. To be honest, I'm a bit on the fence about adding the buffering and and non-blocking log writes. Call it a work-related injury, but I don't like log-files that lag ;) But let's see what we can do to improve it. Yeah, if it's a physical server it shouldn't be much of a problem, but on some VPSes on shared storage backend, all the individual, short and small file open/write/flush ops can bog things down. If you're on the latest source version, running "vanilla" with no local patches, it's very odd that the sustained CPU spikes are still there. I'd really love to know what on earth is causing that. Not seeing anything like it here on any of my nodes anymore. It is the `rnsd` process itself doing it, right? Not some other RNS program? If it's completely opaque, it might be worth adding a sprinkling of `with RNS.Profiler:` contexts to various parts of Transport.py to capture a profiling trace and guage where things are going wonky. --- ## Reply 18 **K8** · Thu, Aug 20, 2026 8:50 AM Fair enough about log buffering! I have it set up to notify the writer thread using more or less the same kind of structure as the new inbound queues, and I have the file buffering mode set to flush after every line, so in most cases it should be pretty responsive. But I'm sure there's a middle ground solution to be found. At the very least there are some more edge cases to handle like making sure that the log queue is fully flushed before exiting. Yup, just the `rnsd` process. Speaking of the Profiler, I just spent part of the day reworking it so that it's possible to get live results through `rnstatus`, which is way way more usable when trying to profile a big running node like this where the problems don't usually start early on. It's still a bit shit and bogs down the node when processing a ton of samples, so take these numbers (especially the maximums) with a grain of salt: --- ## Reply 19 **K8** · Thu, Aug 20, 2026 8:50 AM ```text Profiling : Transport._inbound Samples : 1000000 Total : 696s 647ms 188.38µs Mean | Median | Min | Max | St. dev Stats : ( 696.65µs | 381.61µs | 11.09µs | 940ms 672.49µs | 7ms 175.09µs ) 1m : ( 716.2µs | 362.31µs | 14.6µs | 870ms 693.37µs | 8ms 878.46µs ) 5m : ( 691.25µs | 373.71µs | 14.6µs | 884ms 220.88µs | 8ms 735.54µs ) 30m : ( 718.95µs | 381.5µs | 11.09µs | 940ms 672.49µs | 8ms 418.49µs ) Transport._inbound.0qget Samples : 1000000 Total : 2467s 848ms 6.58µs Mean | Median | Min | Max | St. dev Stats : ( 2ms 467.85µs | 9.84µs | 1.31µs | 1s 66ms 644.86µs | 14ms 587.09µs ) 1m : ( 1ms 710.88µs | 5.99µs | 1.42µs | 894ms 199.92µs | 15ms 219.51µs ) 5m : ( 2ms 77.09µs | 7.15µs | 1.42µs | 894ms 199.92µs | 16ms 111.42µs ) 30m : ( 2ms 547.86µs | 9.58µs | 1.31µs | 992ms 174.0µs | 16ms 830.31µs ) Transport._inbound.0qput Samples : 1000064 Total : 18s 481ms 312.09µs Mean | Median | Min | Max | St. dev Stats : ( 18.48µs | 13.08µs | 1.28µs | 80ms 85.21µs | 185.64µs ) 1m : ( 33.14µs | 7.77µs | 1.36µs | 31ms 200.31µs | 542.1µs ) 5m : ( 20.51µs | 9.6µs | 1.36µs | 31ms 200.31µs | 283.69µs ) 30m : ( 18.87µs | 12.95µs | 1.28µs | 80ms 85.21µs | 199.28µs ) Transport._inbound.1preamble Samples : 1000000 Total : 12s 72ms 308.7µs Mean | Median | Min | Max | St. dev Stats : ( 12.07µs | 10.26µs | 2.23µs | 26ms 467.84µs | 35.06µs ) 1m : ( 10.14µs | 7.94µs | 2.23µs | 5ms 276.61µs | 31.41µs ) 5m : ( 10.82µs | 8.97µs | 2.23µs | 5ms 276.61µs | 18.34µs ) 30m : ( 12.01µs | 10.24µs | 2.23µs | 9ms 16.73µs | 21.21µs ) Transport._inbound.2broadcast Samples : 1000000 Total : 1s 769ms 213.77µs Mean | Median | Min | Max | St. dev Stats : ( 1.77µs | 1.48µs | 0.51µs | 8ms 963.64µs | 14.58µs ) 1m : ( 1.51µs | 1.25µs | 0.54µs | 111.46µs | 2.32µs ) 5m : ( 1.62µs | 1.37µs | 0.54µs | 666.38µs | 3.59µs ) 30m : ( 1.8µs | 1.49µs | 0.51µs | 8ms 963.64µs | 19.05µs ) Transport._inbound.3routing Samples : 1000000 Total : 356s 111ms 111.75µs Mean | Median | Min | Max | St. dev Stats : ( 356.11µs | 252.79µs | 0.84µs | 847ms 270.25µs | 3ms 31.4µs ) 1m : ( 427.17µs | 246.73µs | 0.85µs | 763ms 679.62µs | 5ms 979.5µs ) 5m : ( 379.65µs | 267.45µs | 0.85µs | 763ms 679.62µs | 4ms 596.7µs ) 30m : ( 342.67µs | 221.09µs | 0.85µs | 847ms 270.25µs | 3ms 352.37µs ) Transport._inbound.4announce Samples : 191543 Total : 159s 10ms 113.93µs Mean | Median | Min | Max | St. dev Stats : ( 830.15µs | 370.23µs | 1.03µs | 1s 6ms 391.2µs | 10ms 522.28µs ) 1m : ( 2ms 566.51µs | 349.36µs | 179.35µs | 1s 6ms 391.2µs | 28ms 595.39µs ) 5m : ( 1ms 524.79µs | 358.61µs | 179.35µs | 1s 6ms 391.2µs | 20ms 627.92µs ) 30m : ( 960.59µs | 376.37µs | 179.31µs | 1s 6ms 391.2µs | 13ms 753.1µs ) 60m : ( 846.51µs | 369.09µs | 5.64µs | 1s 6ms 391.2µs | 10ms 910.17µs ) Transport._inbound.5linkreq Samples : 139 Total : 656.32µs Mean | Median | Min | Max | St. dev Stats : ( 4.72µs | 4.16µs | 2.26µs | 39.11µs | 3.61µs ) Transport._inbound.6localdata Samples : 1000000 Total : 215s 707ms 890.47µs Mean | Median | Min | Max | St. dev Stats : ( 215.71µs | 3.47µs | 0.8µs | 989ms 968.77µs | 5ms 657.0µs ) 1m : ( 745.63µs | 2.83µs | 0.87µs | 989ms 968.77µs | 11ms 821.01µs ) 5m : ( 379.16µs | 3.16µs | 0.87µs | 989ms 968.77µs | 8ms 654.97µs ) 30m : ( 292.53µs | 3.52µs | 0.87µs | 989ms 968.77µs | 7ms 393.89µs ) 60m : ( 220.61µs | 3.47µs | 0.87µs | 989ms 968.77µs | 5ms 766.42µs ) Transport._inbound.7proofs Samples : 15031 Total : 8s 489ms 946.9µs Mean | Median | Min | Max | St. dev Stats : ( 564.83µs | 534.0µs | 1.09µs | 402ms 697.03µs | 3ms 379.42µs ) 1m : ( 483.94µs | 438.67µs | 1.44µs | 14ms 398.25µs | 808.8µs ) 5m : ( 518.79µs | 505.57µs | 1.44µs | 14ms 398.25µs | 655.32µs ) 30m : ( 564.51µs | 545.23µs | 1.3µs | 38ms 306.9µs | 780.41µs ) 60m : ( 567.58µs | 535.1µs | 1.09µs | 402ms 697.03µs | 3ms 473.32µs ) Transport._outbound Samples : 480707 Total : 378s 887ms 972.18µs Mean | Median | Min | Max | St. dev Stats : ( 788.19µs | 198.96µs | 8.4µs | 3s 894ms 165.77µs | 12ms 22.41µs ) 1m : ( 4ms 821.53µs | 208.63µs | 34.94µs | 2s 440ms 169.09µs | 57ms 900.53µs ) 5m : ( 2ms 242.8µs | 207.01µs | 30.96µs | 3s 894ms 165.77µs | 22ms 632.99µs ) 30m : ( 1ms 83.75µs | 205.03µs | 30.96µs | 3s 894ms 165.77µs | 14ms 789.18µs ) 60m : ( 800.34µs | 199.09µs | 8.4µs | 3s 894ms 165.77µs | 12ms 149.74µs ) Transport.jobs Samples : 14014 Total : 493s 468ms 695.57µs Mean | Median | Min | Max | St. dev Stats : (35ms 212.55µs | 44.41µs | 14.11µs | 1s 781ms 862.98µs | 148ms 344.6µs ) 1m : (73ms 288.13µs | 43.7µs | 15.73µs | 1s 781ms 862.98µs | 257ms 256.36µs) 5m : ( 61ms 75.99µs | 41.55µs | 14.81µs | 1s 781ms 862.98µs | 226ms 485.52µs) 30m : (50ms 433.39µs | 43.63µs | 14.11µs | 1s 781ms 862.98µs | 193ms 717.94µs) 60m : (37ms 646.16µs | 43.82µs | 14.11µs | 1s 781ms 862.98µs | 154ms 789.88µs) Transport.jobs.gc Samples : 759 Total : 371s 241ms 487.19µs Mean | Median | Min | Max | St. dev Stats : (489ms 119.22µs | 576ms 382.83µs | 35ms 737.43µs | 1s 78ms 71.96µs | 240ms 796.72µs) 1m : (820ms 979.27µs | 807ms 622.03µs | 685ms 695.3µs | 1s 78ms 71.96µs | 93ms 50.71µs ) 5m : (776ms 645.01µs | 755ms 954.83µs | 631ms 438.14µs | 1s 78ms 71.96µs | 77ms 317.52µs ) 30m : (700ms 853.22µs | 695ms 284.06µs | 545ms 27.43µs | 1s 78ms 71.96µs | 76ms 137.25µs ) 60m : (524ms 311.13µs | 605ms 417.47µs | 92ms 458.45µs | 1s 78ms 71.96µs | 216ms 533.47µs) Transport.preprocess_inbound Samples : 1000113 Total : 602s 916ms 166.08µs Mean | Median | Min | Max | St. dev Stats : ( 602.85µs | 403.73µs | 0.79µs | 1s 105ms 270.83µs | 3ms 959.36µs ) 1m : ( 1ms 207.85µs | 386.93µs | 0.82µs | 1s 105ms 270.83µs | 11ms 616.07µs ) 5m : ( 837.96µs | 397.57µs | 0.81µs | 1s 105ms 270.83µs | 7ms 457.0µs ) 30m : ( 642.77µs | 405.44µs | 0.79µs | 1s 105ms 270.83µs | 4ms 638.01µs ) Transport.transmit Samples : 1952690 Total : 557s 122ms 313.25µs Mean | Median | Min | Max | St. dev Stats : ( 285.31µs | 206.69µs | 0.62µs | 851ms 483.43µs | 2ms 697.23µs ) 1m : ( 244.3µs | 201.96µs | 0.62µs | 851ms 483.43µs | 2ms 168.61µs ) 5m : ( 252.74µs | 202.99µs | 0.62µs | 851ms 483.43µs | 2ms 430.83µs ) 30m : ( 273.97µs | 203.25µs | 0.62µs | 851ms 483.43µs | 2ms 652.99µs ) 60m : ( 285.32µs | 206.7µs | 0.62µs | 851ms 483.43µs | 2ms 697.33µs ) ``` --- ## Reply 20 **Mark** · Thu, Aug 20, 2026 9:08 AM Wow, that is a gift from heaven @K8 :) Very nice! Been wanting something like that forever, but just couldn't muster the will to get it done ;) It would be awesome to get this merged in as well. Please do let me know when you feel it's ready for review. Is it on your git node already? Would love to try it out. And yeah, I do agree with the logging being better as non-blocking in almost all cases. We can probably use your patch more or less as is. Maybe I can just add a blocking mode switch or something for the paranoid :) --- ## Reply 21 **K8** · Thu, Aug 20, 2026 9:43 AM It's on the `live_profiling` branch! This was fueled by equal parts frustration and obsession so there's still plenty of work to do to make it nice and fast. I'm still resisting the urge to bring in `numpy` as an optional dependency... --- ## Reply 22 **Mark** · Thu, Aug 20, 2026 1:07 PM No numpy! Take it as a challenge! lol :) Thanks! Gonna have a play with that :) --- ## Reply 23 **K8** · Fri, Aug 21, 2026 11:29 PM More stats with the much improved live profiling: https://pastebin.com/PmUuUhnW Check out the 1-3s spikes in `_outbound()`! All the interfaces on this node are Backbones (plus one thing on the shared instance, which should also be using epoll), so IO shouldn't be blocking. Barring the entire server just stalling out (which I would expect to see some sign of in other places, not just in `_outbound()`), and given the other things that the function does, this seems to leave lock contention as the likely culprit. Also notice that `jobs()` has spikes of nearly 1s, and as I've been watching these stats over time, I consistently notice that `jobs()` (as well as the GC that it sometimes runs) takes more and more time to complete the longer the node has been running. Jobs takes out a lot of big locks, and if it holds them for a long time, it could cause stalls in many other locations throughout Transport. I know I was going on about this a while back, and the new inbound queues can enable a lot of improvements, but there is some low hanging fruit that could help improve things. For example, every instance of this pattern (example from `_outbound()`): ```python with Transport.path_table_lock: if not packet.destination_hash in Transport.path_table: RNS.log(f"Dropped packet since path table entry disappeared during outbound processing", RNS.LOG_WARNING) return False else: path_entry = Transport.path_table[packet.destination_hash] # ... work with path_entry outside the lock ... ``` Can be replaced with this: ```python path_entry = Transport.path_table.get(packet.destination_hash) if path_entry is None: RNS.log(f"Dropped packet since path table entry disappeared during outbound processing", RNS.LOG_WARNING) return False # ... work with path_entry ... ``` `dict.get(...)` is atomic with the GIL, and both lock-free and atomic with free-threading, so you don't need to hold `path_table_lock`. Can't necessarily do it in every instance, depending on what else needs to happen with what you get from the dict, but there are many cases where it's totally fine (and, at the very least, equivalent to the locking logic that's already there, just without the lock). --- ## Reply 24 **Mark** · Sat, Aug 22, 2026 1:56 PM Thanks K8, yeah that one is definitely an obvious contender :) I changed that one now, but the full trawling through everything I'll start looking at after 1.5.0 is out. There's enough changes already, so right now I'm wrapping that one up and getting it out. Once it is, I'll merge in your live profiling patches, and start looking at those improvements in a more focused manner. Thanks for the feedback and testing everyone! --- ## Reply 25 **RaspiSpoon** · Sun, Aug 23, 2026 9:29 PM Hi, don't know if it is intentional, but rnprobe in version 1.5.0 does not show RSSI and SNR. Tested on same client host with LoRa as the only interface and RNS in separated python virtual environments. Destination is transport node version 1.5.0. $ rnprobe --version rnprobe 1.5.0 $ rnprobe rnstransport.probe 1ba5.... Sent probe 1 (16 bytes) to <1ba5....> Valid reply from <1ba5....> Round-trip time is 696.223 milliseconds over 1 hop [Link Quality 100.0%] Sent 1, received 1, packet loss 0.0% $ rnprobe --version rnprobe 1.4.2 $ rnprobe rnstransport.probe 1ba5.... Sent probe 1 (16 bytes) to <1ba5....> Valid reply from <1ba5....> Round-trip time is 618.224 milliseconds over 1 hop [RSSI -63 dBm] [SNR 10.0 dB] [Link Quality 100.0%] Sent 1, received 1, packet loss 0.0% Edit: formatting --- ## Reply 26 **Mark** · Sun, Aug 23, 2026 10:38 PM Thanks @RaspiSpoon, that's definitely not intentional. I'll look into it. @K8, I think I might have found the cause of your CPU spikes. You're running IFAC on the public interface of your transport node, right? Well, I knew that the IFAC implementation was slow - I never really designed it with high throughput in mind, but as an extremely bandwidth efficient and entirely *passive* authentication system, waaay back. But, I hadn't really looked at *how* slow it actually was compared to the rest of RNS at this point. Turns out I had a couple of O(n2) party crashers in that, so we were running at quadratic complexity, which *massively* hurts on high-MTU packets, like most backbone interfaces use now. I optimized it. It's now around 90x faster, lol :) Source is on Aleph if you want to try it out. --- ## Reply 27 **K8** · Mon, Aug 24, 2026 4:56 AM That's great, I'll give it a try. Thanks! I was wondering about that too and already added some profiling to check yesterday. It was actually not as bad as I was expecting, though all optimization helps. I'll post up some more stats tomorrow. It also looks like the `path_table_lock` in `_outbound` was maybe also not the culprit (I missed it before, but there's another `path_table_lock` in `next_hop_interface` which doesn't seem to be taking much time at all. I see two other big sources of spikes in outbound handling. First is creating the Timer thread for queued announces. Averages low but spikes very high, and it can happen in a loop for every interface. Granted that particular branch doesn't run all that often. Second is pretty much the entire branch for broadcasting on all interfaces. I was able to trace the spikes down through `transmit` and to `BackboneInterface.tx_ready` which is really only doing `epoll.modify`. It would just take a few interfaces seeing a spike to add up to the bigger spikes in `_outbound`. I'm not sure what to make of that epoll call being sometimes slow. Anyways I sent you a message about some more improvements to live profiling that I pushed. --- ## Reply 28 **Anonymous** · Mon, Aug 24, 2026 6:31 AM **Mark** wrote: > Thanks @RaspiSpoon, that's definitely not intentional. I'll look into it. > > @K8, I think I might have found the cause of your CPU spikes. You're running IFAC on the public interface of your transport node, right? Well, I knew that the IFAC implementation was slow - I never really designed it with high throughput in mind, but as an extremely bandwidth efficient and entirely *passive* authentication system, waaay back. But, I hadn't really looked at *how* slow it actually was compared to the rest of RNS at this point. Turns out I had a couple of O(n2) party crashers in that, so we were running at quadratic complexity, which *massively* hurts on high-MTU packets, like most backbone interfaces use now. > > I optimized it. It's now around 90x faster, lol :) Source is on Aleph if you want to try it out. Wow! That's impressive. Is it possible to see the average MTU or similar for an interface? --- ## Reply 29 **p1ld7a** · Mon, Aug 24, 2026 8:20 AM Hello, I have been testing version 1.5.0 on my public node recently. After a few days, I can say that the issue I had with 1.4.2 is still present. For some reason, the daemon stop working, here's the log of my `Apollo RNS` node: ``` ❯ sudo systemctl status rnsd.service × rnsd.service - Reticulum Network Stack Daemon Loaded: loaded (/etc/systemd/system/rnsd.service; enabled; preset: ignored) Active: failed (Result: exit-code) since Mon 2026-08-24 06:23:04 CEST; 3h 51min ago Duration: 1d 9h 24min 3.221s Invocation: 6fad41208f42476e9358f1c97f781885 Process: 121784 ExecStartPre=/nix/store/3746cb7f5j9apgmgnj7735w5dczwnyh8-unit-script-rnsd-pre-start/bin/rnsd-pre-start (code=exited, status=0/SUCCESS) Process: 121798 ExecStart=/nix/store/hwk2777va14slqqlcj9b0y6217r418vf-python3.14-rns-1.5.0/bin/rnsd --config $STATE_DIRECTORY (code=exited, status=255/EXCEPTION) Main PID: 121798 (code=exited, status=255/EXCEPTION) IP: 4.7G in, 5.4G out IO: 700K read, 2.3G written Mem peak: 405.6M CPU: 4h 59min 8.631s aoû 23 13:52:57 apollo rnsd[121798]: [2026-08-23 12:12:28] [Warning] No-outbound return on link packet from BackboneInterface[Client on rns.not-a-number.io/69.30.146.224:53448] aoû 23 13:52:57 apollo rnsd[121798]: [2026-08-23 12:12:28] [Warning] No-outbound return on link packet from BackboneInterface[Client on rns.not-a-number.io/69.30.146.224:53448] aoû 23 13:52:57 apollo rnsd[121798]: [2026-08-23 12:12:28] [Warning] No-outbound return on link packet from BackboneInterface[Client on rns.not-a-number.io/69.30.146.224:53448] aoû 23 13:52:57 apollo rnsd[121798]: [2026-08-23 12:12:28] [Warning] No-outbound return on link packet from BackboneInterface[Client on rns.not-a-number.io/69.30.146.224:53448] aoû 23 13:52:57 apollo rnsd[121798]: [2026-08-23 12:12:28] [Warning] No-outbound return on link packet from BackboneInterface[Client on rns.not-a-number.io/69.30.146.224:53448] aoû 23 13:52:57 apollo rnsd[121798]: [2026-08-23 12:12:28] [Warning] No-outbound return on link packet from BackboneInterface[Client on rns.not-a-number.io/69.30.146.224:53448] aoû 24 06:23:04 apollo rnsd[121798]: [2026-08-23 12:12:28] [Warning] No-outbound return on link packet from BackboneInterface[Client on rns.not-a-number.io/69.30.146.224:53448] aoû 24 06:23:04 apollo systemd[1]: rnsd.service: Main process exited, code=exited, status=255/EXCEPTION aoû 24 06:23:04 apollo systemd[1]: rnsd.service: Failed with result 'exit-code'. aoû 24 06:23:04 apollo systemd[1]: rnsd.service: Consumed 4h 59min 8.631s CPU time over 1d 9h 24min 3.256s wall clock time, 405.6M memory peak, 700K read from disk, 2.3G written to disk, 4.7G incoming IP traffic, 5.4G outgoing IP traffic. ``` You can see the instability of the server on `rns.fyi`: ![image.png](/storage/forum/Sn9mytEEuWhgjK1LaevFi8SNP3BtapoGdagcyNlv.png) What you can also clearly see is this: ![image.png](/storage/forum/o8iFMq6NBIQ1m4yTT34pBqVaEIEXhJBNZ6DBXVWL.png) Before Point 1: very stable service, before version 1.4.2. Point 1 represent the day I updated the server and switched to version 1.4.2. I was manually watching the server from time to time and restarted it when it was down. At some point I gave up, waiting for the next release. Point 2 represent the day I updated the server and switch to version 1.5.0. Same pattern unfortunately. Is there anything I do to help understanding what is going on here ? --- ## Reply 30 **K8** · Mon, Aug 24, 2026 12:53 PM **p1ld7a** wrote: > Hello, > > I have been testing version 1.5.0 on my public node recently. After a few days, I can say that the issue I had with 1.4.2 is still present. > > For some reason, the daemon stop working, here's the log of my `Apollo RNS` node: > > [...] > > You can see the instability of the server on `rns.fyi`: > > [image] > > What you can also clearly see is this: > > [image] > > Before Point 1: very stable service, before version 1.4.2. > > Point 1 represent the day I updated the server and switched to version 1.4.2. I was manually watching the server from time to time and restarted it when it was down. At some point I gave up, waiting for the next release. > > Point 2 represent the day I updated the server and switch to version 1.5.0. Same pattern unfortunately. > > Is there anything I do to help understanding what is going on here ? Can you share the `logfile` from the reticulum config directory after a crash? --- ## Reply 31 **K8** · Mon, Aug 24, 2026 1:44 PM Stats! This is running on `956d688e` and the node is currently still stable and responsive. https://pastebin.com/raw/nABJ0uAU Looks like `next_hop_interface` does actually see some big spikes after all. There's some more interesting data in there too. `Transport._outbound.no_known_path_all` is the entire broadcast loop over all interfaces. `Transport._outbound.no_known_path_transmit` is just the `if should_transmit:` branch at the end of that loop. --- ## Reply 32 **p1ld7a** · Mon, Aug 24, 2026 2:09 PM **K8** wrote: > Can you share the `logfile` from the reticulum config directory after a crash? The systemd service is not running with `--service` flag, so there's no `logfile`. I will modify the systemd service tonight and restart it with the `--service` flag. Will keep you posted. --- ## Reply 33 **Anonymous** · Mon, Aug 24, 2026 3:09 PM **p1ld7a** wrote: > **K8** wrote: > > Can you share the `logfile` from the reticulum config directory after a crash? > > The systemd service is not running with `--service` flag, so there's no `logfile`. I will modify the systemd service tonight and restart it with the `--service` flag. Will keep you posted. I believe they mean when you manually restart it, before you do you copy the logfile file out. (Why does Reticulum not use file extensions anyway?) --- ## Reply 34 **Mark** · Tue, Aug 25, 2026 9:06 AM @p1ld7a are you getting any `oom-killer` messages in your syslog/journal? Looking at the systemd output, I have a suspicion those crashes might be coming from memory exhaustion. How much RAM does the system have? The queue length defaults are rather high for low-memory devices right now, actually. Especially if you have interfaces with high MTUs. --- ## Reply 35 **Mark** · Tue, Aug 25, 2026 9:18 AM I've tuned the queue lengths and auto MTU configurations down a bit. My initial estimates at appropriate sizes were honestly waaay too optimistic, and I didn't properly consider the effect on low-RAM nodes. I've tuned this to be much more reasonable now, and the default settings should bound queue memory consumption to 34 megabytes in the default configuration (assuming a backbone interface at the default 100 Mbps rate). All of this is pushed to Aleph, so if you want to have a go at trying it out on your node @p1ld7a, I'd be interested to hear the results. --- ## Reply 36 **Mark** · Tue, Aug 25, 2026 9:44 AM Good data @K8! There's some very interesting stuff in there, including the path request and next-hop interface spikes, and the `handle_tunnel` spike is very strange. I assume this is captured the from before `Merge branch 'optimize'` merge I did last night, right? Btw, there's now also a (very janky, but functional) transport throughput benchmarker in `tests/`. Obviously, it's a very synthetic setup with only a few interfaces, so it does not give a full picture of how things look on a live node with 400+ interfaces, but the intention is to be able to get a baseline of the raw transport core packet throughput on various systems. Here's some sample numbers. The direct/drainer numbers are packets per second. **On a decently fast laptop, with hardware accelerated crypto:** ``` scenario direct drainer ---------------------------------------------------------------------- transit_single_475 244,722 159,597 929.94 Mbps / 606.47 Mbps transit_single_16384 81,574 70,271 10.69 Gbps / 9.21 Gbps transit_link_475 260,089 224,753 988.34 Mbps / 854.06 Mbps transit_link_1024 233,364 197,047 1.91 Gbps / 1.61 Gbps transit_link_16384 83,830 72,889 10.99 Gbps / 9.55 Gbps transit_link_32768 50,834 48,087 13.33 Gbps / 12.61 Gbps announce_ingress 12,290 12,166 16.42 Mbps / 16.25 Mbps outbound_path 862,222 - 931.20 Mbps / - ``` **On the slow, single-core/1G RAM VPS that Bern is running on:** ``` scenario direct drainer ---------------------------------------------------------------------- transit_single_475 77,817 57,980 295.70 Mbps / 220.32 Mbps transit_single_16384 26,178 18,095 3.43 Gbps / 2.37 Gbps transit_link_135 79,952 71,570 86.35 Mbps / 77.30 Mbps transit_link_475 83,273 68,537 316.44 Mbps / 260.44 Mbps transit_link_1024 70,163 55,815 574.78 Mbps / 457.24 Mbps transit_link_16384 26,703 18,559 3.50 Gbps / 2.43 Gbps transit_link_32768 13,211 12,576 3.46 Gbps / 3.30 Gbps announce_ingress 2,243 2,006 3.00 Mbps / 2.68 Mbps outbound_path 103,497 - 111.78 Mbps / - ``` --- ## Reply 37 **Mark** · Tue, Aug 25, 2026 9:46 AM Ok, those text code blocks render weirdly on the web version. They're fine in nomadnet though :) --- ## Reply 38 **K8** · Wed, Aug 26, 2026 1:54 AM **Mark** wrote: > Good data @K8! There's some very interesting stuff in there, including the path request and next-hop interface spikes, and the `handle_tunnel` spike is very strange. I assume this is captured the from before `Merge branch 'optimize'` merge I did last night, right? Yeah, those stats are with the IFAC optimizations but not the `optimize` branch. I just got that pushed to the node, we'll see how it goes. Also, bwahahaha, maybe this server is a bit shit after all: ``` ------------------------------------------------------------------------ Transport Throughput, median of runs scenario direct drainer ---------------------------------------------------------------------- transit_single_135 33,181 22,286 35.84 Mbps / 24.07 Mbps transit_single_475 32,692 22,714 124.23 Mbps / 86.31 Mbps transit_single_1024 26,358 20,476 215.93 Mbps / 167.74 Mbps transit_single_16384 8,902 7,932 1.17 Gbps / 1.04 Gbps transit_single_final 31,166 20,131 33.66 Mbps / 21.74 Mbps transit_link_135 33,725 22,856 36.42 Mbps / 24.68 Mbps transit_link_475 31,449 23,337 119.51 Mbps / 88.68 Mbps transit_link_1024 33,203 19,041 272.00 Mbps / 155.98 Mbps transit_link_16384 9,505 7,777 1.25 Gbps / 1.02 Gbps transit_link_32768 5,551 5,099 1.46 Gbps / 1.34 Gbps terminus_single 5,467 4,688 5.90 Mbps / 5.06 Mbps announce_ingress 1,549 1,554 2.07 Mbps / 2.08 Mbps outbound_path 49,500 - 53.46 Mbps / - ``` Compared to my laptop (M4 Pro, should be fast but also won't be using epoll): ``` ------------------------------------------------------------------------ Transport Throughput, median of runs scenario direct drainer ---------------------------------------------------------------------- transit_single_135 167,307 123,479 180.69 Mbps / 133.36 Mbps transit_single_475 153,314 116,829 582.59 Mbps / 443.95 Mbps transit_single_1024 147,699 113,189 1.21 Gbps / 927.25 Mbps transit_single_16384 75,494 37,650 9.90 Gbps / 4.93 Gbps transit_single_final 157,222 117,739 169.80 Mbps / 127.16 Mbps transit_link_135 165,470 122,134 178.71 Mbps / 131.91 Mbps transit_link_475 155,451 117,212 590.71 Mbps / 445.41 Mbps transit_link_1024 151,818 114,917 1.24 Gbps / 941.40 Mbps transit_link_16384 78,048 36,115 10.23 Gbps / 4.73 Gbps transit_link_32768 49,320 34,483 12.93 Gbps / 9.04 Gbps terminus_single 30,107 28,772 32.52 Mbps / 31.07 Mbps announce_ingress 7,539 7,477 10.07 Mbps / 9.99 Mbps outbound_path 401,622 - 433.75 Mbps / - ``` 4 cores and they're all sad 😂. ---