Rendered at 19:39:29 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
qdotme 2 days ago [-]
Hi HN - Tom here, I built scsipub.
The short version: it's iSCSI targets on the public internet. Pick
an image, get a block device. The free tier doesn't need a signup
at all - iscsiadm -m discovery -t sendtargets -p scsipub.com and
--login to iqn.2025-01.pub.scsipub:blank lands you a 64 MB
scratch disk. There's a small catalog of OS images you can mount
the same way.
The paid tier is where it gets less hobby-shaped: sessions survive
disconnects, a single target can expose multiple LUNs, and SCSI-3
Persistent Reservations work end-to-end (REGISTER / RESERVE /
RELEASE round-trip clean against sg_persist). That last bit is
the cluster-storage primitive — Pacemaker, ESXi HA, and Windows
MSCS all use PR for fencing — so you can actually back a 2-node
failover cluster off a target on the public internet.
The post linked in the submission is the architectural decision
log: Ranch 2.x listeners, a BEAM process per session, COW overlays
with per-sector bitmaps, Caddy-managed Let's Encrypt for the
iSCSI-TLS port without restarting the listener, and the four
open-iscsi quirks that each cost me few hours. There's a section on
what we're deliberately not solving (multi-region, RDMA, etc.)
so you know the scope.
Two companion projects ship as embedded sub-sites on the front
page — one turns an ESP32-S3 into a wireless iSCSI-to-USB bridge,
one lets a Raspberry Pi 3/4/5 netboot directly from a target. Both
linked from the landing page under "Hardware initiators".
Happy to answer any questions about the protocol, the deployment,
or the BEAM-side design choices.
100ms 8 hours ago [-]
I dislike neg comments but really curious - I can see the how but absolutely clueless about the why. Running a block device over a high latency WAN link seems like a terrible idea, what's the use case?
qdotme 6 hours ago [-]
I've answered some down the tree a bit for the inspirational use case for it.
Since I built it, I've started seeing it as a hammer for many nail-like problems - I think that would die down over time;
but.. I have my ESP32 "pendrive" that's net-synced. I have used it to install OS through UEFI-built-in initiator. I have added iSCSI targets to my windows laptop machine (and VMs) - while you need to deal with disconnects and reconnects, it actually works well enough.
It is a terrible idea, that doesn't sound as terrible for odd use-cases. But yes, the ESP32 over 2.4GHz over 3G internet is slow as molasses (20-30kB/s) - but when the alternative is 0.. or walking over there with a laptop, it works OK.
mbreese 8 hours ago [-]
I don’t have a use case, but I was thinking the same thing. But then I realized that the WAN speeds available now are equal to or faster than the LAN speeds I had when I had reason to use iSCSI. And things worked out decently well then, so I can see this being useful.
Apparently, exposing small USB sticks to industrial equipment that uses it for loading/saving configs and screenshots and being able to 'network' it with shared iSCSI drives.
"The scope writes screen_001.png to “USB”; the file appears in a directory on my desktop, in the iSCSI overlay. Combined with a dropbox-style sync I no longer need to walk over and pull the stick out."
Quite brilliant and clever, if you ask me.
I'm wondering now about using an ESP32 stick and an iSCSI image of Windows install media - that could make for some fun in-house computer imaging setups.
qdotme 7 hours ago [-]
That was indeed one of the main drivers for it! ESP32 (especially with 2.4GHz WiFi latencies) is not super well suited for OS installs, but... many UEFI firmwares (and some network drivers!) will let you boot iSCSI directly.
The other one is the Raspberry Pi{3,4,5} iSCSI shim linked there as well - I have a bunch of them for a bunch of paying clients CI/CD kinds of work, and I wanted these to boot from network, not from microSD.
Both of these projects could've benefited from a public demo iSCSI endpoint, we have http://example.com and whateveryouwant@mailinator.com - why not iSCSI
kotaKat 1 hours ago [-]
Ah, yeah, drat. I forgot entirely about the moonshot that becomes streaming several GB through the ESP... I was just thinking of an easier solution that avoids UEFI networking - wireless devices, tablets, odd things like that ;)
Then again this might still be useful yet - a small 64MB thumb drive with an autounattend.xml streamed to it is also an equally powerful tool for some Windows shenanigans.
qdotme 40 minutes ago [-]
The Pi4 shim actually exposes USB device as well. This works way, way better (and IMHO mostly because wired network is better than wireless for latency, ESP32’s feeble CPU aside)
chaz6 2 hours ago [-]
Do you support multi-pathing, for example, connecting using both ipv4 and ipv6?
qdotme 1 hours ago [-]
That's a tricky one. Sort of - we just didn't have enough reason to implement it (nor IPv6 but this is a low lift, and we can get it done quickly).
We do suport VPD 0x83 and advertise consistent NAA/WWID, so linux will support multiple iscsiadm sessions to the same device, and it will be stitched across sessions as paths to the same disk.
We currently hardcode MC/S to 1 as part of login negotiations, advertise single portal and dropping a path will require a re-login.
So - theoretically yes, you can support multipath and it won't fall on its face, but without any practical benefits of it (no bandwidth aggregation and no failover - no ALUA) - at this point it's a single boring target.
But the underlying plumbing can support it - if you have a real usecase for it.
futune 8 hours ago [-]
I saw the mention of BEAM in the article, and immediately wanted to know more. But I don't have any specific questions unfortunately...
somat 1 hours ago [-]
I think the biggest problem is the various filesystem layers assume the system will be more reliable than it is and suffer. I have a fun weekend experiment running, encrypted remote block storage where I put softraid crypto device(yes I like obsd) on a iscsi target from across the internet. I was vaguely surprised it works at all. But it tends to lock up often as something in the raid layer fails to understand iscsi hiccups.
A fun thought experiment here. Is an encrypted transport layer needed? (ipsec, wiregaurd, tls, etc...) I mean, sure, it probably should be plumbed in I bet a raw iscsi stream exposes a ton of metadata. But does it need to? Would a system designed to encrypt blocks in storage provide protection for those blocks in transit?
qdotme 1 hours ago [-]
Well, sort of. I think a big portion about it for me was figuring out the TaskAttribute and ACA handling. Basically, many of these assumptions are (well?!) communicated to the block layer by the SCSI protocol (not even iSCSI, that is a thin layer) - but configuring it on the target side was always poorly understood.
We run "Ordered" - without queueing - so we essentially are trading off some /more/ performance for reliability in this situation. The block layer tends to handle it well in most OS.. I'll be game to test what softraid crypto would do to it (luks actually handles it reasonably!)
In terms of encrypted transport - the value is mostly in HMAC layer. That might die oddly if the ISPs decide to mess around with these packets because they look too cute - but for encrypted FS, it shouldn't be accidental, only malicious. (Accidentally, I've seen ISPs get upset over MPEG streams over netcat!)
luckman212 9 hours ago [-]
I wish there was built in iSCSI initiator support on macOS. All of the halfway decent third-party ones either broke many OS versions ago (GlobalSAN) or cost a small fortune ($250 for Atto Xtend)
qdotme 6 hours ago [-]
That was another reason for the RPi4 iSCSI shim (https://scsipub.com/initiators/PI4-iSCSI-shim/) - not over scsipub.com itself if you need performance (latency over WAN) - works "OK" but not great.
You can demo it out on scsipub without having to provision local SAN, and I came up with scsipub idea as a consequence of lack of any public iSCSI hosts to try things out.
Yes, it's ridiculous, but I have plugged my M1 Macbook Air to (LAN based) iSCSI through a Pi4. Works remarkably well.
jollymonATX 9 hours ago [-]
I should reevaluate my feeling about iscsi I developed around the md1000 era.
EvanAnderson 8 hours ago [-]
If my experience with the MD1000 was like yours you developed the feeling for good reason. It has gotten better but I'll still take fibre channel over iSCSI every day.
justsomehnguy 4 hours ago [-]
Over the Internet? Doubtful.
And iSCSI as a protocol is a mature solution - it's always the implementation what can add some surprises. But otherwise it works and even survives the network switch stack split-brain sigh.
jetbalsa 3 hours ago [-]
always wanted to see if this was doable using NBD or even RBD over the internet. but I do love the ESP32 device they came up with.
sensarts 2 days ago [-]
[flagged]
LargoLasskhyfv 2 days ago [-]
> This is the kind of post that makes me wish HN had bookmarks.
You could 'abuse' favorite for that. Works for whole threads, or just single comments.
doublerabbit 2 days ago [-]
Click the "minutes ago" and then click on "favorite". Basic but it works.
qdotme 2 days ago [-]
Thanks! Let me know if you have any questions - I've long wanted to write something "system-level" in Elixir.
kajman 54 minutes ago [-]
I've bounced off learning (hobby-tier) Elixir a few times now. For me, the gradual type system combined with the functional flow has been hardest to wrap my head around when it comes to solving problems.
I'm curious about how you approached handling a lower-level protocol in Elixir. When you began implementation, did you start thinking at the level of packets as structs, or maybe first as just maps that flowed through functions?
qdotme 6 minutes ago [-]
Pretty much functions with pattern matching all-across, passing around bitstrings, relatively dynamically typed.
Grab a buffer, make sure it's at least as long as BHS, shove into BHS, ok, make sure it's at least as long as what the BHS requires...
Pattern matching is the hidden power of Elixir that's probably the hardest to get fluent with, it's IMHO harder than Rust's equivalent (because of gradual typing!) - but when it works it blows Pythons' match syntax way, way out.
The short version: it's iSCSI targets on the public internet. Pick an image, get a block device. The free tier doesn't need a signup at all - iscsiadm -m discovery -t sendtargets -p scsipub.com and --login to iqn.2025-01.pub.scsipub:blank lands you a 64 MB scratch disk. There's a small catalog of OS images you can mount the same way.
The paid tier is where it gets less hobby-shaped: sessions survive disconnects, a single target can expose multiple LUNs, and SCSI-3 Persistent Reservations work end-to-end (REGISTER / RESERVE / RELEASE round-trip clean against sg_persist). That last bit is the cluster-storage primitive — Pacemaker, ESXi HA, and Windows MSCS all use PR for fencing — so you can actually back a 2-node failover cluster off a target on the public internet.
The post linked in the submission is the architectural decision log: Ranch 2.x listeners, a BEAM process per session, COW overlays with per-sector bitmaps, Caddy-managed Let's Encrypt for the iSCSI-TLS port without restarting the listener, and the four open-iscsi quirks that each cost me few hours. There's a section on what we're deliberately not solving (multi-region, RDMA, etc.) so you know the scope.
Two companion projects ship as embedded sub-sites on the front page — one turns an ESP32-S3 into a wireless iSCSI-to-USB bridge, one lets a Raspberry Pi 3/4/5 netboot directly from a target. Both linked from the landing page under "Hardware initiators".
Happy to answer any questions about the protocol, the deployment, or the BEAM-side design choices.
Since I built it, I've started seeing it as a hammer for many nail-like problems - I think that would die down over time;
but.. I have my ESP32 "pendrive" that's net-synced. I have used it to install OS through UEFI-built-in initiator. I have added iSCSI targets to my windows laptop machine (and VMs) - while you need to deal with disconnects and reconnects, it actually works well enough.
It is a terrible idea, that doesn't sound as terrible for odd use-cases. But yes, the ESP32 over 2.4GHz over 3G internet is slow as molasses (20-30kB/s) - but when the alternative is 0.. or walking over there with a laptop, it works OK.
Apparently, exposing small USB sticks to industrial equipment that uses it for loading/saving configs and screenshots and being able to 'network' it with shared iSCSI drives.
"The scope writes screen_001.png to “USB”; the file appears in a directory on my desktop, in the iSCSI overlay. Combined with a dropbox-style sync I no longer need to walk over and pull the stick out."
Quite brilliant and clever, if you ask me.
I'm wondering now about using an ESP32 stick and an iSCSI image of Windows install media - that could make for some fun in-house computer imaging setups.
The other one is the Raspberry Pi{3,4,5} iSCSI shim linked there as well - I have a bunch of them for a bunch of paying clients CI/CD kinds of work, and I wanted these to boot from network, not from microSD.
Both of these projects could've benefited from a public demo iSCSI endpoint, we have http://example.com and whateveryouwant@mailinator.com - why not iSCSI
Then again this might still be useful yet - a small 64MB thumb drive with an autounattend.xml streamed to it is also an equally powerful tool for some Windows shenanigans.
We do suport VPD 0x83 and advertise consistent NAA/WWID, so linux will support multiple iscsiadm sessions to the same device, and it will be stitched across sessions as paths to the same disk.
We currently hardcode MC/S to 1 as part of login negotiations, advertise single portal and dropping a path will require a re-login.
So - theoretically yes, you can support multipath and it won't fall on its face, but without any practical benefits of it (no bandwidth aggregation and no failover - no ALUA) - at this point it's a single boring target.
But the underlying plumbing can support it - if you have a real usecase for it.
A fun thought experiment here. Is an encrypted transport layer needed? (ipsec, wiregaurd, tls, etc...) I mean, sure, it probably should be plumbed in I bet a raw iscsi stream exposes a ton of metadata. But does it need to? Would a system designed to encrypt blocks in storage provide protection for those blocks in transit?
We run "Ordered" - without queueing - so we essentially are trading off some /more/ performance for reliability in this situation. The block layer tends to handle it well in most OS.. I'll be game to test what softraid crypto would do to it (luks actually handles it reasonably!)
In terms of encrypted transport - the value is mostly in HMAC layer. That might die oddly if the ISPs decide to mess around with these packets because they look too cute - but for encrypted FS, it shouldn't be accidental, only malicious. (Accidentally, I've seen ISPs get upset over MPEG streams over netcat!)
You can demo it out on scsipub without having to provision local SAN, and I came up with scsipub idea as a consequence of lack of any public iSCSI hosts to try things out.
Yes, it's ridiculous, but I have plugged my M1 Macbook Air to (LAN based) iSCSI through a Pi4. Works remarkably well.
And iSCSI as a protocol is a mature solution - it's always the implementation what can add some surprises. But otherwise it works and even survives the network switch stack split-brain sigh.
You could 'abuse' favorite for that. Works for whole threads, or just single comments.
I'm curious about how you approached handling a lower-level protocol in Elixir. When you began implementation, did you start thinking at the level of packets as structs, or maybe first as just maps that flowed through functions?
Grab a buffer, make sure it's at least as long as BHS, shove into BHS, ok, make sure it's at least as long as what the BHS requires...
Pattern matching is the hidden power of Elixir that's probably the hardest to get fluent with, it's IMHO harder than Rust's equivalent (because of gradual typing!) - but when it works it blows Pythons' match syntax way, way out.