Okies, but be warned the purpose of this repo was to get hands dirty with building llvm and zig compiler variants, and getting more into asm. Eventually it might turn into a nice Atari toolkit, but rough edges as of today
Then did my own experiments and notes to get llvm with m68k backend enabled. https://github.com/zigster64/atari.zig. That gets you a bare bones basic zig toolchain on MacOS, and duplicates the demo from the other repo.
At that point, the repo is pure hacky time experiments from there on ! Its exploration code in its current state, not a polished toolkit (yet).
Then I used some DeepSeek tokens to ask an LLM to vibeslop me a space invaders clone against that provided api, to prove itβs usable and expose a tonne of shortcomings and other bugs.
Note that the LLM claims βmiscompilationsβ with llvm .. which it did manage to provide a solution for, but it may or may not have made that up ?
.. so now I need to deep dive on llvm source code to see if the m68k backend is really buggy or not, then fix it if true.
Working on learning more about Zig and lower-level programming with tes3tool: an attempt at replacing tes3cmd for working with ESM/ESP files for The Elder Scrolls III: Morrowind.
I learned programming and software development mostly on my own, and Iβve worked mostly in web development and, honestly, Iβm trying to get out of web development. Generally Iβve worked with and taught myself with higher level languages: Common Lisp, Ruby, Python, Elixir, C#/F#, Java/Kotlin, Javascript/Typescript. While I do genuinely like some of those languages, Iβve found myself wanting to actually learn βlower-levelβ languages like C/C++ and Zig.
tes3tool has been quite the project, especially since most people donβt take much issue with tes3cmd. I really just started it because I couldnβt run the giant Perl script that is tes3cmd on my musl Linux desktop (Chimera Linux) for whatever reason. If youβve been modding Morrowind in recent years, you probably noticed the modding scene has had a lot of more modern tools come about. tes3cmd isβ¦not exactly modern. I wanted to create a static binary that could be a drop-in-place alternative to tes3cmd, more performant, and easier to extend. I also wanted to make a tool that could be compiled to work on all, if not more, of the OSes and architectures that OpenMW can run on.
At first I had AI help me start the project, and then at some point I had to stop because I wasnβt learning anything. Since then, I found myself really liking Zig. I like that it forces me to be correct and clear about what the program is doing more than C and C++ do. I also really like not having to link against libc.
tes3tool works on raw binary slices: a Record is just an in-memory slice into the file buffer, and the format specs arenβt applied until a command actually needs the structured/decode form (e.g. dump). tes3cmd decodes every record into an object model during the read pass.
It doesnβt have 1-to-1 parity with tes3cmd yet, but Iβm working on it.
Iβd been working on a simple file inspector as a fun way to get deeper into stuff Iβd always hand-waived at (learning how file types are determined, for instance). After writing some code to stick comma separators into large numbers when displaying byte sizes, I decided to working on the file inspector and instead start porting go-humanize to Zig (zig-humanize), which is now nearly done! (still learning the Ziggy way to approach things, so the APIs etc might change)
Iβve enjoyed thinking through the tradeoffs between convenience and accuracy, determining when to stay close to that libraryβs design choices vs going out on my own, etc. Itβs also been a nice opportunity to read more of the standard library. Even got myself rickrolled in the process!
Made some big progress on our MIDI playing robot since I last posted about it in May or June. My buddy finished up the fixture for our v2 prototype and hand soldered some sweet looking custom PCBs. We have two esp32s (one on each hand) based off this esp-idf zig sample code! Built a half baked solver that will try converting MIDI events into a sequence of move commands and solonoid actuations. Thereβs some basic βfuture predictionβ baked in so far that tries to move the hand a little over if more notes will be covered by that position in the future.
Next steps are to polish up the solver to try and do some crazier solving strategies and to clean up the firmware to host a small web server for up the MIDI files instead of baking the image into the build. Video of the two hands working together is in the readme if youβre interested to see a musical demo!
Iβve been working on Zova this month which I previously posted on here, and Iβve managed to release 1.0.0 version today, Iβm proud to carry it to this far! It is an SQLite-backed database that can store vectors,graphs and objects, if you want to check it out or give some recommendations Iβd be glad to read, here is the repo: GitHub - ata-sesli/zova: SQLite-backed local storage for records, objects, vectors, and graphs. Β· GitHub
Itβs running in a WebGL canvas, autogenerates swedish crosswords from so far english and german bundled in databases, and supports serverless Multiplayer using WebRTC and WebTorrent trackers for signaling.
Entire, scalable database in-memory nosql. High-performance with many clients and it implements batch/pipeline as object, not βabstract network protocolβ.
I started a few days ago a little crazy project, related to my earlier work with the DOS lemmings. The idea is to generate random levels for a screensaver and let the lemmings (dimwits as they are) try to get to the exit themselves. First random level Lots of work to doβ¦
Recently Iβve released v1.0.0 of Meioziz, self-hosted analytics for games and apps.
I use it in production for more than a month β for eight my projects. And it does what I expect.
Product wise Iβve added cleanup of old daily aggregates. For example, I keep data for the last 30 days only.
Code wise I simplified html-writing functions: use *std.Io.Writer instead of html: *std.ArrayList(u8), allocator: std.mem.Allocator as parameters. And simplified working with allocator β it is defined in fn handleRequest().
This sounds like the perfect place to use Wave Function Collapse. You model the constraints each piece has, like what can go beside what, then start at a random spot and let randomness loose within the constraints of your system.
Iβve not made a lot of showcasable progress on my game, mainly bein working on the boring stuff, and sharpening my axe stuff. But at work I got the time to start working on an alternative to libcamera written purely in zig
src ) cloc .
62 text files.
61 unique files.
4 files ignored.
github.com/AlDanial/cloc v 2.06 T=0.03 s (2003.6 files/s, 637841.6 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
Zig 59 1585 1157 16610
JSON 2 0 0 67
-------------------------------------------------------------------------------
SUM: 61 1585 1157 16677
-------------------------------------------------------------------------------
src ) tree
ξΏ .
βββ ξΏ hardware
β βββ ξ© Camera.zig
β βββ ξ© controls.zig
β βββ ξ© core.zig
β βββ ξ© Encoder.zig
β βββ ξ© FrameBuffer.zig
β βββ ξ© FrameLease.zig
β βββ ξ© .zig
β βββ ξΏ platform
β β βββ ξΏ debug
β β β βββ ξ© Camera.zig
β β β βββ ξ© module.zig
β β βββ ξΏ failing
β β β βββ ξ© Camera.zig
β β β βββ ξ© module.zig
β β βββ ξΏ ffmpeg
β β β βββ ξ© Camera.zig
β β β βββ ξ© H264Stream.zig
β β β βββ ξ© module.zig
β β β βββ ξΏ testdata
β β β βββ ο½ sample_1280.mp4
β β β βββ ο
tiny.ppm
β β βββ ξΏ v4l2
β β β βββ ξ© Device.zig
β β β βββ ξ© dma.zig
β β β βββ ξ© Encoder.zig
β β β βββ ξ© h264.zig
β β β βββ ξΏ imx477
β β β β βββ ξ© Camera.zig
β β β β βββ ξ© Embedded.zig
β β β β βββ ξ© Encoder.zig
β β β β βββ ξ© Hardware.zig
β β β β βββ ξΏ ipa
β β β β β βββ ξ© Aaa.zig
β β β β β βββ ξ© algorithms.zig
β β β β β βββ ξ© alsc.zig
β β β β β βββ ξ© awb.zig
β β β β β βββ ξ© ClockRecovery.zig
β β β β β βββ ξ© DelayedControls.zig
β β β β β βββ ξ© hist.zig
β β β β β βββ ξ© module.zig
β β β β β βββ ξ© options.zig
β β β β β βββ ξ© Sync.zig
β β β β β βββ ξ© sync_runner.zig
β β β β β βββ ξ© test_hardware.zig
β β β β β βββ ξΏ testdata
β β β β β β βββ ξ flat.json
β β β β β β βββ ξ tuning_min.json
β β β β β βββ ξ© tuning.zig
β β β β βββ ξ© Isp.zig
β β β β βββ ξ© lens_shading.zig
β β β β βββ ξ© metadata.zig
β β β β βββ ξ© module.zig
β β β β βββ ξ© options.zig
β β β β βββ ξ© peek.zig
β β β β βββ ξ© Sensor.zig
β β β β βββ ξ© simd.zig
β β β β βββ ξ© stats.zig
β β β β βββ ξ© topology.zig
β β β β βββ ξ© types.zig
β β β β βββ ξ© Unicam.zig
β β β βββ ξ© module.zig
β β βββ ξΏ virtual
β β βββ ξ© Camera.zig
β β βββ ξ© module.zig
β β βββ ξΏ testdata
β β βββ ο
tiny.ppm
β β βββ ο
tiny.y4m
β βββ ξ© Request.zig
β βββ ξ© stream.zig
βββ ξ© hardware.zig
βββ ξ© root.zig
βββ ξ© runtime_config.zig
βββ ξΏ ustream
β βββ ξ© h264.zig
β βββ ξ© Packetizer.zig
β βββ ξ© rtp.zig
βββ ξ© ustream.zig
Itβs going well, my idea is to fix what I donβt like about libcamera, while also being a drop in replacement, the goal is to give the user the full power even racy API exist where you can if you want do stuff in the ring buffer in DMA, but the clock is ticking and if your processing is slower than the camera, too bad. The idea is centered around an Allocator like API, and the library is expected to be a BYOB (bring your own backend). Where basically if you want something you just implement it, expose the Camera, Encoder, Processor interface and you can reuse the rest of the code. Thanks to the interface, Iβve implemented a Virtual camera which uses ffmpeg to loopback video, Iβve implemented a Debug camera which records/logs issues like the DebugAllocator, And I have a Failing camera for testing. This is far from finish and will still need a lot of time and polish.
But the end result should be a library with an C api if needed, thatβs minimal, doesnβt use 3 threads out of the box, gives full power, still uses and parses the calibration.json file that camera ships with, and gives you everything that you need in order to use, and manage cameras efficiently with Zero copy. I also integrated my ustream project to it, in order to send the h264 through the network in order to visually debug it with ffplay.
Anyway still a work in progress, no repo to show for it
On the note of uniformity, I havenβt seen people mess enough with using a different algorithm to generate more interesting initial constraints! Something I really want to try (but is forever on my backlog of projects)
Iβm a total beginner when it comes to web dev, so please excuse my question if itβs naive: I see you have no html or css files in your codebase. To my understanding, all of it is generated on the fly in server.zig. What are the advantages of this technique? Is this a standard way of building an application?
Thatβs a great question. No, itβs usually not a standard approach. Common practice is to use templates and embed them into binary or put near the binary.
Inlining HTML could be used for very small and not changing frequently code.
I chose the second approach because Zig standard library doesnβt provide templating and I wanted to keep external dependencies minimal.
I think it would be more clear to move all of rendering code into separate html.zig, so server.zig would be compact. Alas, in current version some structs from DB-layer are involved and I find it okayish now, but not appropriate for use in html.zig.
I am trying to create a tool to query/stat CIM XML. CIM is used a lot in the (European) (energy) industry and the performance of the existing tools is pretty bad.
So far, I have only implemented a single feature: making a histogram of the classes in the xml.
For the architecture, I generate the counting struct during compile time (based on the RDF schema of CIM), plus some slack for extensions and user-defined classes. The (zipped) XML is then streamed in blocks of 512kB and I use SIMD patterns to match elements in the blocks.
I am getting around 5-10GB/s performance on raw XML on my laptop, depending on the order of the elements. It is optimal when elements are placed contiguously per class. Zipped XML is doing around 1.2 GB/s. I am very happy with the result, because I think this streaming design could process even a terabyte of xml with consistent performance.
Now I am trying to crack a harder problem. I want to make a histogram of references between elements.
Something like this (this is mock data)
$ cmd relate data/*.xml CommonAndBoundaryData/CommonData_and_Boundary_merged.zip
SOURCE | PROPERTY | TARGET | COUNT
Terminal | Terminal.ConnectivityNode | ConnectivityNode | 3536
Terminal | Terminal.TopologicalNode | TopologicalNode | 3536
SvPowerFlow | SvPowerFlow.Terminal | Terminal | 2409
Terminal | Terminal.ConductingEquipment | Disconnector | 2062
ConnectivityNode | ConnectivityNode.TopologicalNode | TopologicalNode | 1192
ConnectivityNode | ConnectivityNode.ConnectivityNodeContainer | VoltageLevel | 1186
Disconnector | Equipment.EquipmentContainer | VoltageLevel | 1031
SvStatus | SvStatus.ConductingEquipment | Disconnector | 1031
SvSwitch | SvSwitch.Switch | Disconnector | 1031
Terminal | Terminal.ConductingEquipment | Breaker | 878
CurrentLimit | OperationalLimit.OperationalLimitSet | OperationalLimitSet | 482
...
PowerTransformerEnd | PowerTransformerEnd.connectionKind | WindingConnection | 106
...
FullModel | Model.DependentOn | ? | 2
Hereβs what the input data looks like:
<cim:Terminal rdf:ID="_a1b2c3">
<cim:Terminal.ConnectivityNode rdf:resource="#_f00d01"/>
<cim:Terminal.ConductingEquipment rdf:resource="#_ZZZ"/>
<cim:Terminal.sequenceNumber>1</cim:Terminal.sequenceNumber>
</cim:Terminal>
...(could be millions of elements later, in any order)...
<cim:ConnectivityNode rdf:ID="_f00d01">
<cim:IdentifiedObject.name>CN_123</cim:IdentifiedObject.name>
</cim:ConnectivityNode>
The problem is that each element references the ID of another element, not its class, so we cannot directly count from here. While streaming through the XML, we do not have a map yet which class each ID belongs to. So, we have to build that map while we go through the XML.
One solution would be to make a conservative guess of how many bytes per class in the text (e.g. 64 bytes per instance), and allocate memory based on the file size stat during initialization, but I felt that this would allocate gigantic amounts of memory in an imprecise way (the mapping would be from u128 to u16, so (if I am correct) 16 + 2 = 18 bytes per 64 bytes, so a pretty large fraction of the original size of the XML.
But something irks me in this design as it could potentially lead to gigantic memory allocations when files are very large. One idea is to write this to disk to reduce memory pressure, which perhaps is even worse.. so Iβm still tinkering with this!
Iβm currently working on a small WebAssembly runtime on Zig.
Now itβs something like a hybrid of runtime and compiler. I implement the part itself related to WASM on my own, and I use LLVM for experiments with native code generation and separate parts of the compilation pipeline.