Edgar Luque - Blog

Intro to LLVM and MLIR with Rust and Melior

2023-11-27T00:00:00+00:00

If you haven't heard about MLIR</a> yet, it is a novel project born within LLVM, also what powers MOJO</a>, the Torch-MLIR project</a>, the high level IR of flang</a>, iree</a> and more</a>.

But in case you don't know much about LLVM yet, I'll try to explain a bit.

A primer on LLVM</h1>

LLVM, as their web</a> says, is a collection of modular and reusable compiler and toolchain technologies. But in this post I will focus on the LLVM Core libraries, which focus on providing a source, target independent optimizer with code generation for many CPUs.

LLVM basis is the LLVM IR, a Static single-assignment form</a> (SSA) language, that looks like pseudo assembly. The main property of SSA is that each variable is assigned exactly once and defined before it is used.

It is what LLVM works on to apply all the optimization passes, it being SSA is one of the major enablers of the optimizations it can do (and what makes it easier to implement them), to name a few (from wikipedia):

Constant propagation: conversion of computations from runtime to compile time, e.g. treat the instruction a=3*4+5; as if it were a=17;</li>

Value range propagation: precompute the potential ranges a calculation could be, allowing for the creation of branch predictions in advance.</li>

Sparse conditional constant propagation: range-check some values, allowing tests to predict the most likely branch.</li>

Dead-code elimination: remove code that will have no effect on the results.</li>

Global value numbering: replace duplicate calculations producing the same result.</li>

Partial-redundancy elimination: removing duplicate calculations previously performed in some branches of the program.</li>

Strength reduction: replacing expensive operations by less expensive but equivalent ones, e.g. replace integer multiply or divide by powers of 2 with the potentially less expensive shift left (for multiply) or shift right (for divide).</li>

Register allocation: optimize how the limited number of machine registers may be used for calculations.</li> </ul> If you want to learn LLVM IR in detail, you should look at the LLVM Language Reference Manual</a>. If you know assembly already it shouldn't be too hard, one interesting property is that LLVM has infinite registers, so you don't need to worry about register allocation. It has a simple type system, you can work with integers of any bit size (i1, i32, i1942652), although if you don't use a recent version (17+) you will find some bugs using big integers.
Probably the biggest wall you will hit is learning about GEPs (Get Element Ptr), it's often misunderstood how it works, so they even have a entire documentation page for it: https://llvm.org/docs/GetElementPtr.html</a>. Another thing that may need attention are PHI nodes</a>, which are how LLVM selects a value that comes from control flow branches due to the nature of SSA.
The API to build such IR have the following structure:

Context: Opaquely owns and manages the global data of the LLVM infrastructure, including types, uniquing tables, etc.</li>
Module: The top level container of all other IR objects, it is akin to a compile unit, holds a list of global variables, functions, libraries or other modules it depends on, a symbol table, and target data such as the layout.</li>
Builder: Provides a uniform API for creating instructions and inserting them into a basic block. Blocks contain a list of sequential instructions, and you can jump using blocks, they are like assembbly labels.</li> </ul>
If you want to use LLVM with Rust in a type safe manner, I recommend the really well done inkwell</a> crate. Check out their README to see how the previous mentioned structures are used.
MLIR</h1>
So what is MLIR? It goes a level above, in that LLVM IR itself is one of it's dialects.
MLIR is kind of a IR of IRs, and it supports many of them using "dialects". For example, you may have heard of NVVM IR (CUDA), MLIR supports modeling it through the NVVM</a> dialect (or ROCDL</a> for AMD), but there is also a more generic and higher level GPU</a> dialect.
Those dialects define conversion passes</a> between them, meaning for example, you can convert IR code using the GPU dialect to the NVVM dialect.
They also may define dialect passes, for example the -gpu-map-parallel-loops</code> which greedily maps loops to GPU hardware dimensions.
Some notable dialects:
Builtin</a>: The builtin dialect contains a core set of Attributes, Operations, and Types that have wide applicability across a very large number of domains and abstractions. Many of the components of this dialect are also instrumental in the implementation of the core IR.</li> Affine</a>: This dialect provides a powerful abstraction for affine operations and analyses.</li> Async</a>: Types and operations for async dialect This dialect contains operations for modeling asynchronous execution.</li> SCF</a>: The scf (structured control flow) dialect contains operations that represent control flow constructs such as if and for. Being structured means that the control flow has a structure unlike, for example, gotos or asserts.</li> CF</a>: This dialect contains low-level, i.e. non-region based, control flow constructs. These constructs generally represent control flow directly on SSA blocks of a control flow graph.</li> LLVM</a>: This dialect maps LLVM IR into MLIR by defining the corresponding operations and types. LLVM IR metadata is usually represented as MLIR attributes, which offer additional structure verification.</li> GPU</a>: This dialect provides middle-level abstractions for launching GPU kernels following a programming model similar to that of CUDA or OpenCL.</li> Arith</a>: The arith dialect is intended to hold basic integer and floating point mathematical operations. This includes unary, binary, and ternary arithmetic ops, bitwise and shift ops, cast ops, and compare ops. Operations in this dialect also accept vectors and tensors of integers or floats.</li> TOSA</a>: TOSA was developed after parallel efforts to rationalize the top-down picture from multiple high-level frameworks, as well as a bottom-up view of different hardware target concerns (CPU, GPU and NPU), and reflects a set of choices that attempt to manage both sets of requirements.</li> Func</a>: This dialect contains operations surrounding high order function abstractions, such as calls.</li> </ul> You can also make your own dialect</a>, useful to make a domain specific language for example, in this dialect you can define transformations to other dialects, passes, etc. All these dialects can exist in your MLIR code at the same time, but at the end, you want to execute your code, for this there are Targets, one is LLVM IR itself. In this case, you would need to use passes to convert all dialects to the LLVM dialect, and then you can make the translation from MLIR to LLVM IR</a>. The structure of MLIR is recursive as follows: Region -> Block(s) -> Operation(s) -> Region(s) </code></pre> The top level module is also a operation, which holds a single region with a single block. A region can have 1 or more blocks, each block can have one or more operations, a operation can use 1 or more regions. Operations</h2> These provides the functionality, and what make up the bulk of MLIR. A operation has the following properties: Name: The name of the operation, a unique identified within MLIR. Operations live within a dialect, so they are refered to using dialect.operation</code>, for example arith.add</code></li> Traits: A operation can have a set of traits that affect the syntax or semantics, for example, whether it has side effects, whether it's a block terminator, etc.</li> Constraints: These are used when verifying a operation is correct, for example whether the operands have matching shape and types.</li> Arguments: There are 2 types of arguments: operands, which are runtime values, and attributes, which are compile time constant values.</li> Regions: A operation can accept regions, which contain blocks within.</li> Results: The results of the operation, which can be more than 1. For example arith.add</code> has 1 result, defined by the type of it's arguments.</li> Successors: When a operation is a terminator it needs successors, for example cf.br</code> which is a unconditional jump and thus a branching operation, accepts a single successor (block).</li> </ul> You can read more about operations in the Operation Definition Specification</a>. To use MLIR with Rust, I recommend melior</a>, here is a snippet making a function that adds 2 numbers: use melior::{ Context, dialect::{arith, DialectRegistry, func}, ir::{*, attribute::{StringAttribute, TypeAttribute}, r#type::FunctionType}, utility::register_all_dialects, }; // We need a registry to hold all the dialects let registry = DialectRegistry::new(); // Register all dialects that come with MLIR. register_all_dialects(&registry); // The MLIR context, like the LLVM one. let context = Context::new(); context.append_dialect_registry(&registry); context.load_all_available_dialects(); // A location is a debug location like in LLVM, in MLIR all // operations need a location, even if its "unknown". let location = Location::unknown(&context); // A MLIR module is akin to a LLVM module. let module = Module::new(location); // A integer-like type with platform dependent bit width. (like size_t or usize) // This is a type defined in the Builtin dialect. let index_type = Type::index(&context); // Append a `func::func` operation to the body (a block) of the module. // This operation accepts a string attribute, which is the name. // A type attribute, which contains a function type in this case. // Then it accepts a single region, which is where the body // of the function will be, this region can have // multiple blocks, which is how you may implement // control flow within the function. // These blocks each can have more operations. module.body().append_operation(func::func( &context, // accepts a StringAttribute which is the function name. StringAttribute::new(&context, "add"), // A type attribute, defining the function signature. TypeAttribute::new( FunctionType::new(&context, &[index_type, index_type], &[index_type]).into() ), { // The first block within the region, blocks accept arguments // In regions with control flow, MLIR leverages // this structure to implicitly represent // the passage of control-flow dependent values without the complex nuances // of PHI nodes in traditional SSA representations. let block = Block::new(&[(index_type, location), (index_type, location)]); // Use the arith dialect to add the 2 arguments. let sum = block.append_operation(arith::addi( block.argument(0).unwrap().into(), block.argument(1).unwrap().into(), location )); // Return the result using the "func" dialect return operation. block.append_operation( func::r#return( &[sum.result(0).unwrap().into()], location) ); // The Func operation requires a region, // we add the block we created to the region and return it, // which is passed as an argument to the `func::func` function. let region = Region::new(); region.append_block(block); region }, &[], location, )); assert!(module.as_operation().verify()); </code></pre> Here is a more complex function, using the SCF</code> dialect, which allows us to use a while</code> loop: let context = Context::new(); load_all_dialects(&context); let location = Location::unknown(&context); let module = Module::new(location); let index_type = Type::index(&context); let float_type = Type::float64(&context); module.body().append_operation(func::func( &context, StringAttribute::new(&context, "foo"), TypeAttribute::new(FunctionType::new(&context, &[], &[]).into()), { let block = Block::new(&[]); let initial = block.append_operation(arith::constant( &context, IntegerAttribute::new(0, index_type).into(), location, )); block.append_operation(scf::r#while( &[initial.result(0).unwrap().into()], &[float_type], { let block = Block::new(&[(index_type, location)]); let condition = block.append_operation(arith::constant( &context, IntegerAttribute::new(0, IntegerType::new(&context, 1).into()) .into(), location, )); let result = block.append_operation(arith::constant( &context, FloatAttribute::new(&context, 42.0, float_type).into(), location, )); block.append_operation(scf::condition( condition.result(0).unwrap().into(), &[result.result(0).unwrap().into()], location, )); let region = Region::new(); region.append_block(block); region }, { let block = Block::new(&[(float_type, location)]); let result = block.append_operation(arith::constant( &context, IntegerAttribute::new(42, Type::index(&context)).into(), location, )); block.append_operation(scf::r#yield( &[result.result(0).unwrap().into()], location, )); let region = Region::new(); region.append_block(block); region }, location, )); block.append_operation(func::r#return(&[], location)); let region = Region::new(); region.append_block(block); region }, &[], location, )); assert!(module.as_operation().verify()); </code></pre> This code generates the following MLIR IR: module { func.func @foo() { %c0 = arith.constant 0 : index %0 = scf.while (%arg0 = %c0) : (index) -> f64 { %false = arith.constant false %cst = arith.constant 4.200000e+01 : f64 scf.condition(%false) %cst : f64 } do { ^bb0(%arg0: f64): %c42 = arith.constant 42 : index scf.yield %c42 : index } return } } </code></pre> There is way more to MLIR, but this is meant to be a small introduction.

Implementing a simple Hashmap in Rust 2023-07-10T00:00:00+00:00 Have you ever wondered how a hash map works internally? The idea is quite simple: you map a key to a value, but how do you do that efficiently? This is where hashes come into play. You get a key, then hash it, which gives you an integer value, but we want to map that value into an index inside the backing storage, so you need to calculate the number modulo the current capacity of the backing storage. Since hashing is involved, and we modulo the hash result to the capacity, there will be collisions. One way to deal with collisions is called Open addressing</a>. Storage</h2> We will use an array as the backing storage for our hashmap. The slots of this array will either be nothing or a key value pair. So in Rust it would be a Option<(K, V)></code>. #[derive(Debug, Clone)] pub struct MyHashmap<K, V, S = RandomState> { // Our storage storage: Vec<Option<(K, V)>>, // Save the length for quick querying len: usize, // The hasher. state: S, // Whether to use quadratic or linear probing. quadratic: bool, } </code></pre> Here state</code> is a structure that will help to hash the keys. We add some basic utility methods: impl<K, V, S> MyHashmap<K, V, S> { pub fn new(hasher: S, quadratic: bool) -> Self { Self { storage: (0..8).map(|_| None).collect(), len: 0, state: hasher, quadratic } } pub const fn len(&self) -> usize { self.len } pub const fn is_empty(&self) -> bool { self.len == 0 } pub const fn is_quadratic(&self) -> bool { self.quadratic } fn should_resize(&self) -> bool { (self.len as f64 / self.storage.len() as f64) > 0.7 } } </code></pre> You may wonder why there are no trait bounds in this impl block, like K: Hash</code>, and this is because it's usually better to only put the trait bounds in impl blocks where you use them. The standard library does this extensively (from what I have seen). Load factor</h2> For the hashmap to perfom well, it needs to keep a good load factor. The load factor is defined as: load factor = number of entries / number of slots</code> According to some papers</a> when the load factor aproaches 0.7-0.8 it should be resized. Probing / Searching</h2> When searching for a slot, either at insertion or search, we will need to do something when a hash collision happens: Linear probing</h3> Let x</code> be the value of hash(key) (mod capacity)</code>. Starting with i = 0</code>: If the slot x + i (mod capacity)</code> is unused, use it.</li> If the slot x + i (mod capacity)</code> is used, check if the key matches:</li> If the key matches, use it.</li> </ul> </li> If the key doesn't match, repeat the probing with i += OFFSET</code>, offset usually being 1.</li> </ul> </li> </ul> impl<K, V, S> MyHashmap<K, V, S> where K: Eq + Hash, S: BuildHasher, { // Should be called with a sensible load factor. fn find_slot_linear(&self, key: &K) -> &Option<(K, V)> { let mut hasher = self.state.build_hasher(); key.hash(&mut hasher); let start_idx = hasher.finish() as usize; let mut iter_idx = 0; let len = self.storage.len(); let slot_idx = loop { let idx_mod: usize = (start_idx + iter_idx) % len; match &self.storage[idx_mod] { Some(kv) if kv.0.eq(key) => break idx_mod, None => break idx_mod, _ => { iter_idx += 1; assert!( iter_idx <= len, "find_slot called without a matching key and full storage!" ); } } }; &self.storage[slot_idx] } } </code></pre> Quadratic probing</h3> It works by taking successive values of an arbitrary quadratic polynomial</a>. We will use hash(key) + (i + i^2) / 2</code>. Let x</code> be the value of hash(key)</code>. Starting with i = 0</code>: If the slot x + (i + i^2) / 2 (mod capacity)</code> is unused, use it.</li> If the slot x + (i + i^2) / 2 (mod capacity)</code> is used, check if the key matches:</li> If the key matches, use it.</li> </ul> </li> If the key doesn't match, repeat the probing with i += OFFSET</code>, offset usually being 1.</li> </ul> </li> </ul> impl<K, V, S> MyHashmap<K, V, S> where K: Eq + Hash, S: BuildHasher, { // Should be called with a sensible load factor. fn find_slot_quadratic(&self, key: &K) -> &Option<(K, V)> { let mut hasher = self.state.build_hasher(); key.hash(&mut hasher); let start_idx = hasher.finish() as usize; let mut iter_idx: usize = 0; let len = self.storage.len(); let slot_idx = loop { let idx_mod: usize = (start_idx + (iter_idx + iter_idx.pow(2)) / 2) % len; match &self.storage[idx_mod] { Some(kv) if kv.0.eq(key) => break idx_mod, None => break idx_mod, _ => { iter_idx += 1; assert!( iter_idx <= len, "find_slot called without a matching key and full storage!" ); } } }; &self.storage[slot_idx] } } </code></pre> A mutable version of both linear and quadratic methods should be made for insertion. impl<K, V, S> MyHashmap<K, V, S> where K: Eq + Hash, S: BuildHasher, { fn find_slot_linear_mut(&mut self, key: &K) -> &mut Option<(K, V)> { /// ... same code &mut self.storage[slot_idx] // mut ref } } </code></pre> Inserting</h2> To insert a key value pair, we need to do the following: Check if the load factor is good, otherwise resize the backing structure.</li> Search a slot using one of the probing / search methods.</li> </ul> impl<K, V, S> MyHashmap<K, V, S> where K: Eq + Hash, S: BuildHasher, { pub fn insert(&mut self, key: K, value: V) -> Option<V> { if self.should_resize() { self.resize(); } let slot = if self.quadratic { self.find_slot_quadratic_mut(&key) } else { self.find_slot_linear_mut(&key) }; let new_slot = Some((key, value)); let old_slot = std::mem::replace(slot, new_slot).map(|kv| kv.1); if old_slot.is_none() { self.len += 1; } old_slot } } </code></pre> Resizing</h3> Resizing involves allocating the new storage, and rehashing all entries into the new storage. impl<K, V, S> MyHashmap<K, V, S> where K: Eq + Hash, S: BuildHasher, { fn resize(&mut self) { let new_storage: Vec<Option<(K, V)>> = (0..self.storage.len() * 2).map(|_| None).collect(); // Replace the storage with the new one, so we can use self.x methods to rehash. let old_storage = std::mem::replace(&mut self.storage, new_storage); self.len = 0; for (k, v) in old_storage.into_iter().flatten() { self.insert(k, v); } } } </code></pre> Searching</h3> To search a key, we need to do the following: Search a slot using one of the probing / search methods.</li> </ul> impl<K, V, S> MyHashmap<K, V, S> where K: Eq + Hash, S: BuildHasher, { pub fn get(&self, key: &K) -> Option<&V> { let slot = if self.quadratic { self.find_slot_quadratic(key) } else { self.find_slot_linear(key) }; slot.as_ref().map(|kv| &kv.1) } pub fn get_mut(&mut self, key: &K) -> Option<&mut V> { let slot = if self.quadratic { self.find_slot_quadratic_mut(key) } else { self.find_slot_linear_mut(key) }; slot.as_mut().map(|kv| &mut kv.1) } } </code></pre> Conclusion</h1> And that's a way to implement a hash map using open addressing with linear or quadratic probing. I benchmarked both and they were pretty close. linear probing 10 time: [255.08 ns 256.75 ns 258.66 ns] quadratic probing 10 time: [252.66 ns 252.96 ns 253.25 ns] linear probing 100 time: [3.7110 µs 3.7169 µs 3.7236 µs] quadratic probing 100 time: [3.6763 µs 3.6783 µs 3.6803 µs] linear probing 1000 time: [32.555 µs 32.703 µs 32.906 µs] quadratic probing 1000 time: [32.002 µs 32.046 µs 32.126 µs] linear probing 10000 time: [464.28 µs 464.39 µs 464.53 µs] quadratic probing 10000 time: [453.38 µs 454.40 µs 455.71 µs] linear probing 100000 time: [6.4169 ms 6.4213 ms 6.4267 ms] quadratic probing 100000 time: [6.3506 ms 6.3548 ms 6.3601 ms] </code></pre> You can find all the code here: https://gist.github.com/edg-l/7842a445367bfdf2a89ad8c5f70348d4</a> UI Code in DDraceNetwork 2023-05-19T00:00:00+00:00 This is some kind of guide into how the UI code works in DDNet. Probably can be considered part 1, in case I continue it. Foremost, since this is a game, the UI is rendered using immediate mode</a>, which means, every render tick, the logic on whether something is hovered, rendered, clicked, etc is done. There are some optimizations around this, DDNet uses text containers to cache the rendered text texture for example. UI Client Class</h2> The UI client class defined in src/game/client/ui.h</code> holds most of the related methods aiding at rendering the UI, for example bool MouseInside(const CUIRect *pRect) const;</code> to know whether the mouse is inside the passed CUIRect</code>. Talking about CUIRect</code>, they are the most important structure, using them, elements are placed on the screen, by using a combination of methods to "split" the rect into smaller rects where UI components will be placed, this can be done with the methods the CUIRect</code> class has. As the name suggests, it's quite a simple class, holding the x</code>, y</code>, w</code> and h</code> values, corresponding to the coordinates of the top left point of the rectangle, and the width and height. HSplitMid and variants</h2> /** * Splits 2 CUIRect inside *this* CUIRect horizontally. You can pass null pointers. * * @param pTop This rect will end up taking the top half of this CUIRect. * @param pBottom This rect will end up taking the bottom half of this CUIRect. * @param Spacing Total size of margin between split rects. */ void HSplitMid(CUIRect *pTop, CUIRect *pBottom, float Spacing = 0.0f) const; </code></pre> As the method explains, it places the pTop</code> rect at the top half of the this</code> instance of rect (the method caller), and the pBottom</code> at the bottom half, with the given spacing. CUIRect MyTopButton; CUIRect MyBottomButton; MainView.HSplitMid(&MyTopButton, &MyBottomButton); static CButtonContainer s_MyTopButton; if(DoButton_Menu(&s_MyTopButton, Localize("Top Button"), 0, &MyTopButton, 0, IGraphics::CORNER_ALL, 5.0f, 0.0f, vec4(0.0f, 0.0f, 0.0f, 0.9f), vec4(0.0f, 0.0f, 0.0f, 0.7f))) { dbg_msg("tutorial", "top button pressed!"); } static CButtonContainer s_MyBottomButton; if(DoButton_Menu(&s_MyBottomButton, Localize("Bottom Button"), 0, &MyBottomButton, 0, IGraphics::CORNER_ALL, 5.0f, 0.0f, vec4(0.0f, 0.0f, 0.0f, 0.9f), vec4(0.0f, 0.0f, 0.0f, 0.7f))) { dbg_msg("tutorial", "bottom button pressed!"); } </code></pre> With VSplitMid</code> instead: In addition to (V|H)SplitMid</code> there is also (V|H)Split(Bottom|Top)</code>, they allow making one of the split rects take a different size, for example, given a rect of height 400, using HSplitBottom</code> and giving Cut</code> a value of 100, will make the bottom rect have a height of 100, and the top rect have a height of 300. Knowing this, we can easily add another button to the start menu in menu_start.cpp:129</code>: // Using HSplitBottom this way, we reduce the Menu rect height by 100. Menu.HSplitBottom(100.0f, &Menu, 0); // our new button, will have a height of 40, and reduces the height of Menu by 40. Menu.HSplitBottom(40.0f, &Menu, &Button); static CButtonContainer s_MyMenuButton; if(DoButton_Menu(&s_MyMenuButton, Localize("My Button!"), 0, &Button, 0, IGraphics::CORNER_ALL, Rounding, 0.5f, vec4(0.0f, 0.0f, 0.0f, 0.5f), vec4(0.0f, 0.0f, 0.0f, 0.25f))) { // do something once pressed } // add a bit of margin, i.e, reduce the height of Menu by 5 Menu.HSplitBottom(5.0f, &Menu, 0); // little space // this code already existed, adding it for reference Menu.HSplitBottom(40.0f, &Menu, &Button); static CButtonContainer s_SettingsButton; if(DoButton_Menu(&s_SettingsButton, Localize("Settings"), 0, &Button, g_Config.m_ClShowStartMenuImages ? "settings" : 0, IGraphics::CORNER_ALL, Rounding, 0.5f, vec4(0.0f, 0.0f, 0.0f, 0.5f), vec4(0.0f, 0.0f, 0.0f, 0.25f)) || CheckHotKey(KEY_S)) NewPage = PAGE_SETTINGS; </code></pre> Result: More to come soon. Gentoo as a daily driver 2023-05-15T00:00:00+00:00 I've been using GNU/Linux for quite a while now, I don't remember exactly what my first distro was, probably Ubuntu or Debian. I eventually switched to Arch Linux and stayed with it for a long time, I really enjoyed it, but I have a thing for trying new stuff, and eventually delved into it. Filesystem</h1> Coming from an Arch Linux installation using LVM on LUKS, this time I decided it wasn't that worth it encrypting my whole disk, so I went a simpler way. This is my /etc/fstab</code> file: # boot UUID=8E37-E91C /boot vfat defaults,noatime 0 2 # root, a 500gb NVME m2 ssd UUID=6739c2cc-4a15-4a30-b223-bafcdff6688f / ext4 noatime 0 1 # a extra 2tb ssd partition UUID=c598fbd0-b87c-4e69-9fb6-8b2fd0624f24 /data1 ext4 defaults,noatime 0 2 # 1tb hdd, which i don't really use UUID=978e8a4d-a200-42ff-b501-ee25baf195d4 /data2 ext4 defaults,noatime 0 2 # swap UUID=d2359c6c-99c0-4f4f-9225-5605bed37399 none swap sw 0 0 /dev/cdrom /mnt/cdrom auto noauto,user 0 0 # tmpfs (RAM file system) for temp files and portage build files. tmpfs /tmp tmpfs rw,nosuid,noatime,nodev,size=8G,mode=1777 0 0 tmpfs /var/tmp/portage tmpfs size=14G,uid=portage,gid=portage,mode=775,nosuid,noatime,nodev 0 0 </code></pre> As you can see it's just a traditional setup, using tmpfs for portage stuff, this way i save on SSD disk writes. I should note though that I have 32gb of RAM so I can afford this. For example, firefox requires the tmpfs to be atleast 4.5gb. You can read more here</a>. No Systemd</h1> Since all the distros I used so far used systemd, I decided to try to avoid it this time, which I did for everything but udev</code>, which I may replace for eudev</code> someday. OpenRc is really intuitive so far, I can't speak about writing a service file since i haven't had the need to do so yet. Desktop</h1> I'm currently on X11, using i3 as my tiled window manager, alacritty as my terminal emulator and pipewire for the sound server. No problems whatsoever. Updating</h1> I used to use the testing kernel, but I encountered some issues and the updates where pretty frequent, so I decided to use the non masked kernel. To update my kernel I made a simple script: sudo make -j16 sudo make install sudo make modules_install sudo emerge @module-rebuild sudo dracut # sudo grub-install --efi-directory=/boot/efi sudo grub-mkconfig -o /boot/grub/grub.cfg </code></pre> Configuring the kernel to my liking wasn't hard either, you just need some time to read through the options you need, you mostly will find out what you need by reading the packages wiki for the software you install, since they often list the kernel requirements. To update my packages, I have two aliases: alias update='sudo emaint -a sync' alias upgrade='sudo emerge -avuDN @world' </code></pre> Creating a package</h1> To my surprise, creating a ebuild is quite easy, specially if the software you package uses a common build system. I am currently maintaining the ddnet</a> ebuild, which uses a mix of cpp, rust, python, managed by cmake. https://gitweb.gentoo.org/repo/proj/guru.git/tree/games-action/ddnet?h=dev</a> Further thoughts</h1> Something that I like about Gentoo is that, in my case, due to long compile times for (possibly bloated) software, you tend to avoid those, so it forces you to try to minimize dependencies, find software that just does what you want, most probably without any GUI. It's certainly a different experience compared to binary distributed distros. You can also enjoy the small performance benefits of building everything with your native ISA</a>. Creating a bencode parser with nom 2022-09-08T00:00:00+00:00 Since long I wanted try out nom, at first I boldly started parsing PDFs</a> but after realizing the scope of such project, I put it off and started with a way smaller idea: a bencode parser. If you have never delved into the BitTorrent protocol you probably don't know what bencoding is so let me explain it. The Bencode Spec</h1> Bencode</a> is the encoding used by the BitTorrent protocol to store data, .torrent</code> files are encoded using this. It's a pretty simple</a> encoding: Strings are length-prefixed base ten followed by a colon and the string. For example 4:spam</code> corresponds to spam</code>.</li> Integers are represented by an i</code> followed by the number in base 10 followed by an e</code>. For example i3e</code> corresponds to 3</code> and i-3e</code> corresponds to -3</code>. Integers have no size limitation. i-0e</code> is invalid. All encodings with a leading zero, such as i03e</code>, are invalid, other than i0e</code>, which of course corresponds to 0</code>.</li> Lists are encoded as an l</code> followed by their elements (also bencoded) followed by an e</code>. For example l4:spam4:eggse</code> corresponds to ['spam', 'eggs']</code>.</li> Dictionaries are encoded as a 'd' followed by a list of alternating keys and their corresponding values followed by an 'e'. For example, d3:cow3:moo4:spam4:eggse</code> corresponds to {'cow': 'moo', 'spam': 'eggs'}</code> and d4:spaml1:a1:bee</code> corresponds to {'spam': ['a', 'b']}</code>. Keys must be strings and appear in sorted order (sorted as raw strings, not alphanumerics).</li> </ul> Now, I only saw it mentioned on Wikipedia</a> but the Strings are more like byte strings, since they don't require any proper encoding, and they are used as such. So what is nom?</h1> It's a parser combinators library, which essentially means that from some really basic functions you create more complex ones and keep building on top of it, until you have one final function that parses everything. It comes with some basic combinators, you can find them in the "choosing a combinator"</a> docs. For example, there is a parser function digit1</code> which returns one or more digits. fn parser(input: &str) -> IResult<&str, &str> { digit1(input) } assert_eq!(parser("21c"), Ok(("c", "21"))); assert_eq!(parser("c1"), Err(Err::Error(Error::new("c1", ErrorKind::Digit)))); assert_eq!(parser(""), Err(Err::Error(Error::new("", ErrorKind::Digit)))); </code></pre> All parsers are build around IResult</code>: pub type IResult<I, O, E = Error> = Result<(I, O), Err<E>>; </code></pre> Basically, when a parser correctly finishes, it returns a tuple with a slice starting where the parser ended and the parsed content. It seamlessly works with &str</code> and &[u8]</code> so most of the time you don't need to do any allocation when parsing. So digit1("123therest")</code> returns a tuple with ("therest", "123")</code>. Handling errors</h1> When parsing bencode there can be possible errors outside of what the combinators may find, such as leading 0's on integers, we need to create our own error struct: #[derive(Debug, thiserror::Error)] pub enum Error { // When the integer is invalid: e.g leading 0's #[error("invalid integer: {0:?}")] InvalidInteger(I), // When the byte string length is invalid, e.g it's negative. #[error("invalid bytes length: {0:?}")] InvalidBytesLength(I), // When there is an error parsing the ascii integer to a i64. #[error("parse int error: {0:?}")] ParseIntError(#[from] ParseIntError), // Errors from the combinators itself. #[error("nom parsing error: {0:?}")] NomError(#[from] nom::error::Error), } impl From<Error> for nom::Err<Error> { fn from(e: Error) -> Self { nom::Err::Error(e) } } impl From<nom::Err<Error>> for Error { fn from(e: nom::Err<Error>) -> Self { e.into() } } impl ParseError for Error { fn from_error_kind(input: I, kind: nom::error::ErrorKind) -> Self { Self::NomError(nom::error::Error { input, code: kind }) } fn append(_: I, _: nom::error::ErrorKind, other: Self) -> Self { other } } </code></pre> We will also define the type alias BenResult, so we don't need to type as much everytime: type BenResult<'a> = IResult<&'a [u8], Value<'a>, Error<&'a [u8]>>; </code></pre> We use &[u8]</code> since thats the type of data our parsers will be dealing with. Representing all the possible bencode value types</h1> To hold all the value types bencode has, an enum like this will do: #[derive(Debug, Clone)] pub enum Value<'a> { Bytes(&'a [u8]), Integer(i64), List(Vec<Self>), Dictionary(HashMap<&'a [u8], Self>), } </code></pre> Parsing the byte string</h1> Lets start with the easiest one, byte strings, as you can recall, they are made up of the a textual integer, a colon and the data: 4:spam</code> Since the data can be non UTF-8 encoded, we will build the parser around &[u8]</code> instead of &str</code>. fn parse_bytes(start_inp: &'a [u8]) -> BenResult<'a> { todo!() } </code></pre> We can use the digit1</a> combinator to get the length of the byte string and then the char</a> to consume the colon: let (inp, length) = digit1(start_inp)?; // We don't need the colon so we just discard it with '_'. let (inp, _) = char(':')(inp)?; </code></pre> Now we need to convert the length (which is a number in ASCII) to an integer and check if it's 0, which would be an error: // SAFETY: digit1 always returns ASCII numbers, which are always valid UTF-8. let length = unsafe { std::str::from_utf8_unchecked(length) }; let length: u64 = length.parse().map_err(Error::ParseIntError)?; if length == 0 { Err(Error::InvalidBytesLength(start_inp))? } </code></pre> Then we use the take</a> parser to take an exact amount of elements and finally return it, resulting in the complete function like this: fn parse_bytes(start_inp: &'a [u8]) -> BenResult<'a> { let (inp, length) = digit1(start_inp)?; let (inp, _) = char(':')(inp)?; // SAFETY: digit1 always returns ASCII numbers, which are always valid UTF-8. let length = unsafe { std::str::from_utf8_unchecked(length) }; let length: u64 = length.parse().map_err(Error::ParseIntError)?; if length == 0 { Err(Error::InvalidBytesLength(start_inp))? } let (inp, characters) = take(length)(inp)?; Ok((inp, Value::Bytes(characters))) } </code></pre> Parsing integers</h1> Format: i10e</code>, i-30e</code> Integers can come alone or with the positive and negative symbol, we also need to handle the invalid -0</code> and they can't have leading 0</code>s like 002</code>. Here the power of combinators can come to light, we will use the following parsers and combine them: delimited</a>: Matches an object from the first parser and discards it, then gets an object from the second parser, and finally matches an object from the third parser and discards it.</li> char</a>: Recognizes and consumes a single character.</li> alt</a>: Tests a list of parsers one by one until one succeeds.</li> recognize</a>: If the child parser was successful, return the consumed input as produced value.</li> pair</a>: Gets an object from the first parser, then gets another object from the second parser.</li> </ul> With this we can handle the following example numbers: 1,+1,-1,10,0,50,-62</code> fn parse_integer(start_inp: &'a [u8]) -> BenResult<'a> { let (inp, value) = delimited( char('i'), alt(( recognize(pair(char('+'), digit1)), recognize(pair(char('-'), digit1)), digit1, )), char('e'), )(start_inp)?; // SAFETY: This will always be a valid UTF-8 sequence. let value_str = unsafe { std::str::from_utf8_unchecked(value) }; if value_str.starts_with("-0") || (value_str.starts_with('0') && value_str.len() > 1) { Err(Error::InvalidInteger(start_inp))? } else { let value_integer: i64 = value_str.parse().map_err(Error::ParseIntError)?; Ok((inp, Value::Integer(value_integer))) } } </code></pre> Parsing lists</h1> Format: li2ei3ei4ee</code>, l4:spam4:eggsi22eli1ei2eee</code> A list can hold any type of value, including dictionaries and list themselves. Now that we can parse numbers and byte strings, parsing lists is just a matter of using those parsers. We will use the following new nom parsers: many_till(f, g)</a>: Applies the parser f until the parser g produces a result. Returns a pair consisting of the results of f in a Vec and the result of g.</li> </ul> We will apply the parser alt</code> until the char</code> parser recognizes the end character e</code>: // Self here is the enum Value fn parse_list(start_inp: &'a [u8]) -> BenResult<'a> { let (inp, value) = preceded( char('l'), many_till( alt(( Self::parse_bytes, Self::parse_integer, Self::parse_list, Self::parse_dict, )), char('e'), ), )(start_inp)?; Ok((inp, Value::List(value.0))) } </code></pre> Parsing dictionaries</h1> Format: d3:cow3:moo4:spam4:eggse</code> Parsing dictionaries is nearly identical to parsing lists, but we need to parse the keys too, which are byte strings: fn parse_dict(start_inp: &'a [u8]) -> BenResult<'a> { let (inp, value) = preceded( char('d'), many_till( pair( Self::parse_bytes, alt(( Self::parse_bytes, Self::parse_integer, Self::parse_list, Self::parse_dict, )), ), char('e'), ), )(start_inp)?; let data = value.0.into_iter().map(|x| { // Keys are always a byte string if let Value::Bytes(key) = x.0 { (key, x.1) } else { unreachable!() } }); let map = HashMap::from_iter(data); Ok((inp, Value::Dictionary(map))) } </code></pre> The parser</h1> And finally, we can make the final parser function which parses all the possible values: pub fn parse(source: &[u8]) -> Result<Vec<Value>, Error<&[u8]>> { let (_, items) = many_till( alt(( Value::parse_bytes, Value::parse_integer, Value::parse_list, Value::parse_dict, )), eof, )(source)?; Ok(items.0) } </code></pre> An example using the parser: use nom_bencode::Value; let data = nom_bencode::parse(b"d3:cow3:moo4:spam4:eggse").unwrap(); let v = data.first().unwrap(); if let Value::Dictionary(dict) = v { let v = dict.get(b"cow").unwrap(); if let Value::Bytes(data) = v { assert_eq!(data, b"moo"); } let v = dict.get(b"spam").unwrap(); if let Value::Bytes(data) = v { assert_eq!(data, b"eggs"); } } </code></pre> You can find the full source code here: https://github.com/edg-l/nom-bencode</a> Parsing compressed files efficiently with Rust 2022-01-06T00:00:00+00:00 I recently wanted to create a tool to create plots showing concurrent players each day on the open-source game DDraceNetwork</a> (DDNet for short). DDNet hosts an HTTP "master server", which is what the game client uses to fetch information about game servers they can join. Thankfully they keep online the master server status of previous days</a>. Each .tar.zstd</code> file contains a JSON file every 5 seconds starting from 00:00 to 23:59 which has information about all servers and players within those servers at that current time. The problem</h2> These files, while compressed use only about ~8mb, but they are very efficiently compressed, when decompressed they take about 7gb. So if we don't want to use a lot of disk space or memory we need to parse the data in a streaming way. The libraries</h2> We will use the following libraries to achieve this: tar</a>: To read the entries of the tar archive.</li> zstd</a>: To decompress the files.</li> ureq</a>: To get the archives.</li> </ul> Fetching the data</h2> With ureq we can fetch the data easily: let resp = ureq::get("https://ddnet.tw/stats/master/2022-01-04.tar.zstd").call()?; </code></pre> Processing the data</h2> In Rust i/o operations are modeled around 2 traits: Read</a> and Write</a>, thanks to this it's really ergonomic to use both libraries (tar and zstd) together. Now we convert the response into a Reader and pass it to the zstd Decoder</a>, which takes anything that implements Read</a>, it also wraps it around a BufReader</a> for buffered reading. let decoder = zstd::stream::Decoder::new(resp.into_reader())?; </code></pre> Now we need to pass this decoder</code> to tar to get its entries: let mut archive = tar::Archive::new(decoder); // Loop over the entries for entry in archive.entries()? { let entry = entry.unwrap(); let path = entry.path().unwrap(); let filename = path.file_name().expect("exist"); // process each entry } </code></pre> Here entry implements Read too, in our case each entry is a json file, we could parse it this way, for example using serde</code> and simd_json</code>: let data: ServerList = simd_json::from_reader(entry).expect("parse json"); </code></pre> This way, we are parsing each file efficiently while using almost no memory thanks to the streaming nature of these operations. This all fits really well thanks to the design of Read</a> and Write</a>. The tool</h2> Here is the source code of the tool: https://github.com/edg-l/teemasterparser</a> And an image of the result: Discussion on reddit.</a> The Rust dbg! macro 2021-07-25T00:00:00+00:00 The dbg!</code> macro is a useful macro to debug, and I think a not well known one, not to be confused with debug logs using format strings, this macro is useful when you are about to put println</code> calls everywhere in your code to know if it reached a path, what value a variable has, etc. It uses the Debug</code> trait implementation of the type of the given expression. Since Rust is an expression oriented language, you can use this macro nearly everywhere: fn factorial(n: u32) -> u32 { if dbg!(n <= 1) { dbg!(1) } else { dbg!(n * factorial(n - 1)) } } fn main() { dbg!(factorial(4)); } </code></pre> This outputs the following: [src/main.rs:2] n <= 1 = false [src/main.rs:2] n <= 1 = false [src/main.rs:2] n <= 1 = false [src/main.rs:2] n <= 1 = true [src/main.rs:3] 1 = 1 [src/main.rs:5] n * factorial(n - 1) = 2 [src/main.rs:5] n * factorial(n - 1) = 6 [src/main.rs:5] n * factorial(n - 1) = 24 [src/main.rs:10] factorial(4) = 24 </code></pre> Using this macro without any argument will print the current file and line number. One thing to be aware is that this macro moves the input, in cases where this is a problem it's useful to borrow. Sources: https://doc.rust-lang.org/std/macro.dbg.html</a></li> </ul> Wrapping errors in Rust 2021-01-24T00:00:00+00:00 While I was developing a rust crate (paypal-rs</a>) I noticed my error handling was pretty bad. In that crate I had to handle 2 different types of errors: HTTP related errors, in this case reqwest::Error</code></li> Paypal API errors, which I represent with my own struct PaypalError</code>.</li> </ul> Initially I used anyhow</a> but then I found out this is pretty much only good to be used on binary applications, not in libraries. The way to make this nice and clean for the library consumers is to wrap the errors. Wrapping the errors</h2> First we need to know which errors need to be wrapped, in my case I have PaypalError</code>: /// A paypal api response error. #[derive(Debug, Serialize, Deserialize)] pub struct PaypalError { // ... } // implement Error and Display for PaypalError... </code></pre> And then an error from the reqwest library: reqwest::Error</code>. First we create an enum to represent all possible errors in our library: #[derive(Debug)] pub enum ResponseError { /// paypal api error. ApiError(PaypalError), /// http error. HttpError(reqwest::Error) } </code></pre> And as with any error, we have to implement Error</code> and fmt::Display</code>: impl fmt::Display for ResponseError { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { match self { ResponseError::ApiError(e) => write!(f, "{}", e), ResponseError::HttpError(e) => write!(f, "{}", e), } } } impl Error for ResponseError { // Implement this to return the lower level source of this Error. fn source(&self) -> Option<&(dyn Error + 'static)> { match self { ResponseError::ApiError(e) => Some(e), ResponseError::HttpError(e) => Some(e), } } } </code></pre> Now we should make all the functions that return a Result use the new Error type, this will also give consistency to all our functions. However, right now we can't use ?</code> directly on our wrapped Errors: /// func_call_returns_error() returns a Result<(), reqwest::Error> pub fn some_func() -> Result<(), ResponseError> { // Won't work, because the error returned is not ResponseError and has no From implementation! // func_call_returns_error()? // However we can map it. func_call_returns_error().map_err(ResponseError::HttpError)?; Ok(()) } </code></pre> Implementing From on the wrapped errors</h2> To solve this, we have to implement From<PaypalError></code> and From<reqwest::Error></code>: impl From<PaypalError> for ResponseError { fn from(e: PaypalError) -> Self { ResponseError::ApiError(e) } } impl From<reqwest::Error> for ResponseError { fn from(e: reqwest::Error) -> Self { ResponseError::HttpError(e) } } </code></pre> And now our code becomes like this: pub fn some_func() -> Result<(), ResponseError> { func_call_returns_error()?; Ok(()) } </code></pre> The library to skip all this process</h2> There is a library called thiserror</a>, which implements macros to make this process a breeze, here is how our code ends up if we use this library: use thiserror::Error; #[derive(Error, Debug)] pub enum ResponseError { /// A paypal api error. #[error("api error {0}")] ApiError(#[from] PaypalError), /// A http error. #[error("http error {0}")] HttpError(#[from] reqwest::Error) } </code></pre> And that's all the code we need! This is equal (or maybe even better) than our previous code, the best is that it is entirely transparent, the library consumers won't even know thiserror</code> was used, quoting their github: Thiserror deliberately does not appear in your public API. You get the same thing as if you had written an implementation of std::error::Error by hand, and switching from handwritten impls to thiserror or vice versa is not a breaking change. </blockquote> Implementing a chat command in DDraceNetwork 2020-12-04T00:00:00+00:00 This is the part 3 of my series of articles about coding in DDraceNetwork, you can find the first article here</a>. We will implement a command that shows info about our player: /aboutme <times></code> Times will be how many times we print this info. Go to src/game/server/ddracechat.h</code> Here you can see all the chat commands, they are created using a macro. The chat command macro</h2> The macro looks like this: #define CHAT_COMMAND(name, params, flags, callback, userdata, help) </code></pre> The first field is the name of the command, we will create a command called "aboutme". The second field is the command parameters, it uses a special syntax: Add a ?</code> if it's optional. A i</code> for integers, s</code> for a string, r</code> for "everything else". You can give a hint by using []</code>, like r[player name]</code> or possible values ?i['0'|'1'|'2']</code>. The next field are the flags, for server-side chat commands they are always: CFGFLAG_CHAT | CFGFLAG_SERVER </code></pre> Then comes the name of our method, usually prefixed by Con: ConRules</code>. Next field is userdata, always pass this</code>. Then comes the help text, put whathever you see fit. We will use this command definition: CHAT_COMMAND("aboutme", "?i[times]", CFGFLAG_CHAT | CFGFLAG_SERVER, ConAboutMe, this, "Show info about yourself"); </code></pre> Adding the static method</h2> We added ConAboutMe</code> on the CHAT_COMMAND macro, now we need to implement it. First go to src/game/server/gamecontext.h</code> and under the last static Con command you see add it: // ... static void ConFreezeHammer(IConsole::IResult *pResult, void *pUserData); static void ConUnFreezeHammer(IConsole::IResult *pResult, void *pUserData); // Here! static void ConAboutMe(IConsole::IResult *pResult, void *pUserData); </code></pre> Implementing the chat command</h2> Then to implement it we go to src/game/server/ddracechat.cpp</code> and add it to the end: void CGameContext::ConAboutMe(IConsole::IResult *pResult, void *pUserData) { /// The following code will be added here. } </code></pre> As you have seen, ConAboutMe is a static method, so in order to get hold of a CGameContext instance which lets us access all the information, we need to get it from pUserData</code>. CGameContext *pSelf = (CGameContext *)pUserData; </code></pre> Here pResult</code> holds information about the caller client id, the number of arguments and how to get them. Just to be on the safe side, we check that the ClientID we got is actually valid: if(!CheckClientID(pResult->m_ClientID)) return; </code></pre> When you are new to the ddnet codebase, a really useful thing to do is to check how similar things you are doing are implemented, in our case there are a lot of other chat commands and just by looking at their implementations we can learn lot of things, like checking the client id, how to print to the console, chat, etc... </blockquote> Now we will handle our optional argument "times", which tells us how many times we will print this information. int Times = 1; if(pResult->NumArguments() > 0) Times = pResult->GetInteger(0); if(Times < 1) Times = 1; </code></pre> As you can see pResult</code> has some handy methods, aside from those 2 seen in the snippet, you can also get a float, string and a color. You can find out more here</a>. In our command we want to get the following information: name, client version, team and position. The player is always present while the character is only present if the player has a physical body (the tee). const char *pName = pSelf->Server()->ClientName(pResult->m_ClientID); int ClientVersion = pSelf->GetClientVersion(pResult->m_ClientID); int Team = pSelf->GetDDRaceTeam(pResult->m_ClientID); CPlayer *pPlayer = pSelf->m_apPlayers[pResult->m_ClientID]; // Check if it's null if(!pPlayer) { dbg_msg("chat-about-me", "Player not found!"); return; } CCharacter *pChar = pPlayer->GetCharacter(); // Check if it's null if(!pChar) { dbg_msg("chat-about-me", "Character not found! Player may be dead or spectating."); return; } </code></pre> You can see we use dbg_msg</code> to print debug messages, an alternative is to print to console, but I recommend using dbg_msg</code>. Some methods to get information may not be on the class CGameContext, for example with the player name, we had to access the IServer class with the method Server()</code>. That said, now we send the chat message: char aBuf[256]; str_format(aBuf, sizeof(aBuf), "Name: %s | " "ID: %d | " "Version: %d | " "Team: %d | " "Current position: %f, %f", pName, pResult->m_ClientID, ClientVersion, Team, pChar->m_Pos.x, pChar->m_Pos.y); for(int i = 0; i < Times; i++) { pSelf->SendChatTarget(pResult->m_ClientID, aBuf); } </code></pre> We format our message with str_format</code> and send it with the method SendChatTarget</code> and the client id we get from pResult</code>. And we can test it in the game: More to come</h2> Implementing a rcon command</li> Adding configuration options to the client.</li> Modify the menus to show a label and a button/checkbox/slider for the previously added options.</li> Add a new network packet.</li> Add a new map tile.</li> Any other idea I may get in the future.</li> </ul> Rust Iterators: Fibonacci series 2020-12-01T00:00:00+00:00 In this article we will implement an iterator that generates Fibonacci numbers, where each number is the sum of the preceding ones, starting with 0 and 1. First we define our data structure: struct Fibonacci { a: u64, b: u64, } impl Fibonacci { fn new() -> Self { Fibonacci { a: 1, b: 0 } } } </code></pre> Here a and b represent the preceding numbers. To implement the iterator we need to implement the std::iter::Iterator</code> trait. trait Iterator { type Item; fn next(&mut self) -> Option<Self::Item>; } </code></pre> Thus, we have to import it: use std::iter::Iterator; </code></pre> We need to define the type Item and implement next()</code>. impl Iterator for Fibonacci { type Item = u64; fn next(&mut self) -> Option<Self::Item> { let r = self.b; self.b = self.a; self.a += r; Some(r) } } </code></pre> Since fibonacci series are infinite, we always return Some()</code>, but if you implement a non-infinite iterator you will have to return None at some point. And then to see how it works: fn main() { let fib = Fibonacci::new(); // Take 20 fibonacci numbers and put them into a vector. let result: Vec<u64> = fib.take(20).collect(); println!("{:?}", result); } </code></pre> Which outputs: [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181] </code></pre> Code conventions in DDraceNetwork 2020-11-28T00:00:00+00:00 This is the part 2 of my series of articles about coding in DDraceNetwork, you can find the previous one here</a>. What are coding conventions?</h2> They are a set of rules that dictate how the code should be written, so that the code style is consistent among the codebase. DDNet naming conventions</h2> Note: There is an ongoing discussion about variable naming, find out more here</a>. Currently, this is how we name things: Classes and structs</h2> They are prefixed with C</code> and followed by a capital letter, like CController</code>, if the class is meant to be an interface it is prefixed by I</code>. Enum constants</h2> They must be all screaming snake case like: MAX_PLAYERS</code>. Variable naming</h2> The name is divided in 3 parts: qualifier, prefix and name. Common qualifiers: m</code> for member variables, s</code> for static variables. There is also g</code> for global variables with external linkage. If, the qualifier is not empty it is followed by an underscore. Example: ms_YourVariable</code>. There are 2 common type prefixes: p</code> for pointers, a</code> for arrays and fn</code> for functions, note that you can stack the prefixes for example in the case of a pointer to a pointer. Example: pMyPointer</code>, aBuf</code>, ppArgs</code>, m_pCharacter</code>, m_pfnMyCallback</code>, m_papfnMyPointerToArrayOfCallbacks</code> The first letter of the variable must be uppercase. Common idioms.</h2> The following snippet is very common in ddnet code: char aBuf[128]; str_format(aBuf, sizeof(aBuf), "number: %d", 2); </code></pre> This is how we format strings, and it's used everywhere (str_format is defined in system.h). I will add more here when I find/remember more. More to come</h2> Implementing a chat command</a></li> Implementing a rcon command</li> Adding configuration options to the client.</li> Modify the menus to show a label and a button/checkbox/slider for the previously added options.</li> Add a new network packet.</li> Add a new map tile.</li> Any other idea I may get in the future.</li> </ul> Creating precompiled headers with cmake 2020-11-12T00:00:00+00:00 What are precompiled headers?</h2> They are a partially processed version of header files, this speeds up compilation because it doesn't have to repeatedly parse the original header. How to use it</h2> The way to do it is to pass the header files you want precompiled to the target_precompile_headers</code> command. Imagine this folder structure: . ├── CMakeLists.txt └── src ├── header.h └── main.cpp 1 directory, 3 files </code></pre> Now we create a very basic CMakeLists.txt: cmake_minimum_required(VERSION 3.16) project(MyProject) set(HEADER_FILES src/header.h) set(SOURCE_FILES src/main.cpp ${HEADER_FILES} ) include_directories(src) add_executable(MyProject ${SOURCE_FILES}) </code></pre> Notice we require a minimum cmake version of 3.16, this is due to target_precompile_headers</code> being added in that version. We add the command after the add_executable</code> instruction to the private scope. It's recommended to use the private scope to prevent precompiled headers appearing in an installed target, consumers should decide whether they want to use precompiled headers or not. target_precompile_headers(MyProject PRIVATE ${HEADER_FILES}) </code></pre> When compiling you can see now that it indeed compiles the headers: Scanning dependencies of target MyProject [ 33%] Building CXX object CMakeFiles/MyProject.dir/cmake_pch.hxx.gch [ 66%] Building CXX object CMakeFiles/MyProject.dir/src/main.cpp.o [100%] Linking CXX executable MyProject [100%] Built target MyProject </code></pre> What's new in Python 3.9 2020-11-10T00:00:00+00:00 Python 3.9</h2> Python 3.9 has been released on October 5, 2020. Add Union Operators To dict (PEP 584)</h2> This allows the union operation to be performed on dicts: >>> a = {'x': 1, 'y': 2, 'z': 3} >>> e = {'w': 'hello world'} >>> a | e {'x': 1, 'y': 2, 'z': 3, 'w': 'hello world'} </code></pre> And also: >>> x = a >>> x {'x': 1, 'y': 2, 'z': 3} >>> x |= e >>> x {'x': 1, 'y': 2, 'z': 3, 'w': 'hello world'} </code></pre> Type Hinting Generics In Standard Collections (PEP 585)</h2> This feature enables type hinting using the standard collections without having to rely on the typings</code> module. Previously to type hint a list you would do: from typings import List def somefunc(a: List[int]): pass </code></pre> Now you can use the standard type: def somefunc(a: list[int]): pass </code></pre> From this version, importing collections from typings</code> is deprecated, and they will be removed in 5 years. Flexible function and variable annotations (PEP 593)</h2> This feature adds a new type Annotated</code> which allows us to extend type annotations with metadata. This allows a type T</code> to be annotated with metadata x</code> like so: T1 = Annotated[T, x] # E.g UnsignedShort = Annotated[int, struct2.ctype('H')] SignedChar = Annotated[int, struct2.ctype('b')] # Multiple type annotations are supported T2 = Annotated[int, ValueRange(3, 10), ctype("char")] </code></pre> The metadata can then be used for static or runtime analysis with tools such as mypy</a> This feature allows authors to introduce new data types with graceful degradation, for example if mypy doesn't know how to parse X Annotation it should just ignore its metadata and use the annotated type. Relaxing Grammar Restrictions On Decorators (PEP 614)</h2> Python currently requires that all decorators consist of a dotted name, optionally followed by a single call. This PEP proposes removing these limitations and allowing decorators to be any valid expression. An expression here means "anything that's valid as a test in if, elif, and while blocks". Basically this: button_0 = buttons[0] @button_0.clicked.connect def spam(): pass </code></pre> Can now be: @buttons[0].clicked.connect def spam(): pass </code></pre> Support for the IANA Time Zone Database in the Standard Library</h2> This feature adds a new module zoneinfo</code> that provides a concrte time zone implementation supporting the IANA time zone database. You can find more about this module here: zoneinfo</a> Example: >>> from zoneinfo import ZoneInfo >>> from datetime import datetime, timedelta >>> dt = datetime(2020, 10, 31, 12, tzinfo=ZoneInfo("America/Los_Angeles")) >>> print(dt) 2020-10-31 12:00:00-07:00 >>> dt.tzname() 'PDT' </code></pre> String methods to remove prefixes and suffixes</h2> Adds two new methods, removeprefix()</a> and removesuffix()</a>, to the APIs of Python's various string objects. Modernize your linux workflow with Rust 2020-11-06T00:00:00+00:00 exa, a replacement for 'ls'</h2> A replacement for ls which features better defaults and more features. github.com/ogham/exa</a> bat, a cat(1) clone with wings.</h2> Bat supports syntax highlighting, git integration, paging and more. If you use (neo)vim with fzf.vim and have ripgrep and bat installed your search will have a preview window with highlighted code: fd, a simple, fast and user-friendly alternative to find</h2> github.com/sharkdp/fd</a> It's faster, more convenient, ignores hidden files and gitignore files by default and more. Starship, a customizable cross-shell prompt</h2> starship.rs</a> An intro to the DDraceNetwork game source code 2020-11-03T00:00:00+00:00 What is DDraceNetwork?</h2> It's an open source mod of teeworlds</a> which was released on steam</a> not long ago. The language used is C++ along with some python scripts to generate the network protocol, it uses CMake</a> for building. It's made on top of SDL2 with a custom OpenGL renderer. The source can be located here: github.com/ddnet/ddnet</a> My story on DDNet</h2> This mod, also called ddnet was a breaking point on my path towards learning and improving my programming skills, since it introduced me to a codebase quite larger than what I was used to, if I'm honest, the first time I tried to do something in it, I was overwhelmed, but after some hardships and help from a friend</a> I got more used to it. The basic layout</h2> The code of the game is located in the src</code> directory, here I highlight the important directories it has: - src/base</code></h3> This contains the system.h and system.c files, which is kind of an abstraction layer over the standard library, it also contains most platform specific code. Here you can find functions for string formatting, memory and thread management and some cryptography stuff. You will mostly never touch these files, but they are used a lot. There is a subfolder in base named "tl", it contains an implementation for algorithm and array, but we are currently planning to move away from this. - src/engine</code></h3> Here lives the code for anything that is not game specific (or kinda, it's not crystal clear actually), you can find the implementation for the graphics backend, sound, input, notifications, networking, console, SQL, etc. If your objective is to make a mod, you will probably not touch this folder either. - src/game</code></h3> Here you will find the code that implements the client and the server. In the base folder you can find: collision.cpp</code>: Collision related code. ddracecommands.h</code>: Rcon commands. gamecore.cpp</code>: Implements the physics, changing this means the community will cry at you because physics bugs are used in actual maps (so as we say, our physics have no bugs only features). layers.cpp</code>: Map layers. mapbugs.cpp</code>: A try into fixing the physics "bugs", what it does is preserve the bug in specific maps that use them and fix them for new maps. Currently, the only "bug" preserved with this is a double grenade explosion bug used in the map binary</a>. - src/game/client</code></h3> Client specific code, the client is made up of components, each component is a class that extends CComponent</code>, with this, you can access</a> the kernel, graphics, text rendering, sound, console; basically most of the stuff implemented in the engine. Components may also implement methods which will be called on a particular event</a>, such as OnRender, OnInit, OnInput. The file gameclient.cpp</code> implements the game client, which has all the logic behind handling the components, receiving network packets</a> and more. - src/game/server</code></h3> Server specific code, it keeps track of everything, players and all the entities. In gamecontext.cpp</code> CGameContext</code> is implemented, it's the heart of the server, it handles lots of stuff like chat, clients, map changes, network messages, commands, moderation, etc. CGameContext</code> is not only implemented in this file, it also has some parts implemented in ddracecommands.cpp</code> for example. The gamecontroller.cpp</code> handles player spawns, some map entities, it also sends the gameinfo packet. Thegameworld.cpp</code> tracks all the entities in the game. In player.cpp</code> you can find the class CPlayer</code> that keeps track of a player, it stores any information related to the player that is not related to physics. One of the most important entities is CCharacter</code>, it keeps track of the physical representation of the player (we call them "tee(s)"), it's destroyed on death and created when spawning by CPlayer</code>. There are also more entities like pickups, laser, projectile, etc... More to come</h2> There is lot more stuff in here, and the best way to learn it is by actually implementing things. I plan on making this a series of posts implementing stuff to the game (useful or not), here is a sneak peak of whats to come: Code conventions and basic stuff</a>.</li> Implementing a chat command</a></li> Implementing a rcon command</li> Adding configuration options to the client.</li> Modify the menus to show a label and a button/checkbox/slider for the previously added options.</li> Add a new network packet.</li> Add a new map tile.</li> Any other idea I may get in the future.</li> </ul> An intro to MiniUPNP 2020-04-14T00:00:00+00:00 If you have a software that requires port forwarding, you may want to implement UPnP to make things easy for your end users. When I wanted to implement it, the first library I found was MiniUPnP</a>, sadly it doesn't have much documentation but a quick look at the header files and some examples on the internet I managed to make it work, here is how: First the required include directives we need: #include <miniupnpc/miniupnpc.h> #include <miniupnpc/upnpcommands.h> #include <miniupnpc/upnperrors.h> #include <stdio.h> </code></pre> The first thing we need to do is discover the UPnP devices on the network, this is done using upnpDiscover</code>: int main() { struct UPNPDev *upnp_dev = 0; int error = 0; upnp_dev = upnpDiscover(2000, NULL, NULL, 0, 0, 2, &error); if(error != 0) { printf("error discovering upnp devices: %s\n", strupnperror(error)); return 1; } } </code></pre> Then we need to retrieve the Internet Gateway Device we discovered: struct UPNPUrls upnp_urls; struct IGDdatas upnp_data; char aLanAddr[64]; const char *pPort = "8000"; // Retrieve a valid Internet Gateway Device int status = UPNP_GetValidIGD( upnp_dev, &upnp_urls, &upnp_data, aLanAddr, sizeof(aLanAddr) ); printf("status=%d, lan_addr=%s\n", status, aLanAddr); </code></pre> We check if we got the correct status and then map the port. We also declare the port we want to use, in this case I use the same number for the internal and external ports: if (status == 1) { printf("found valid IGD: %s\n", upnp_urls.controlURL); error = UPNP_AddPortMapping( upnp_urls.controlURL, upnp_data.first.servicetype, pPort, // external port pPort, // internal port aLanAddr, "My Application Name", "UDP", 0, // remote host "0" // lease duration, recommended 0 as some NAT // implementations may not support another value ); if (error) { printf("failed to map port\n"); printf("error: %s\n", strupnperror(error)); } else printf("successfully mapped port\n"); } else printf("no valid IGD found\n"); </code></pre> Now the port is mapped, at this point you can open a socket bound to the internal port and external clients will be able to connect using the external port. Then we do some cleanup: Note: The port here is the external one. error = UPNP_DeletePortMapping(upnp_urls.controlURL, upnp_data.first.servicetype, pPort, "UDP", 0); if (error != 0) { printf("port map deletion error: %s\n", strupnperror(error)); } FreeUPNPUrls(&upnp_urls); freeUPNPDevlist(upnp_dev); return 0; </code></pre> Complete source code: gist</a> Compiled using: cc -lminiupnpc upnp.c</code> Setting Up SDL2 with CMake 2019-04-27T00:00:00+00:00 Installing CMake</h2> Most common distributions have cmake available on their package manager repostories: # Debian based sudo apt install cmake # Arch pacman -S cmake </code></pre> Install SDL2 libraries</h2> I only know about the debian based ones, if you are on another distro you should look them up. sudo apt install libsdl2-dev libsdl2-image-dev libsdl2-mixer-dev libsdl2-net-dev libsdl2-ttf-dev libsdl2-gfx-dev </code></pre> Directory Structure</h2> Here is a common directory structure when using cmake to find packages: ├── cmake │ ├── FindSDL2.cmake │ ├── FindSDL2_mixer.cmake │ └── FindSDL2_net.cmake ├── CMakeLists.txt └── src └── main.cpp </code></pre> You can find the cmake files to find SDL2 and it's components here</a> Or you can use this simple command: cd cmake wget https://raw.githubusercontent.com/aminosbh/sdl2-cmake-modules/master/FindSDL2{,_gfx,_image,_mixer,_net,_ttf}.cmake </code></pre> Creating the CMakeLists.txt file</h2> cmake_minimum_required(VERSION 3.13) project(MyProject) # Needed so that cmake uses our find modules. list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake) find_package(SDL2 REQUIRED) find_package(SDL2_net REQUIRED) find_package(SDL2_mixer REQUIRED) find_package(SDL2_image REQUIRED) find_package(SDL2_gfx REQUIRED) find_package(SDL2_ttf REQUIRED) set(SOURCE_FILES src/main.cpp ) include_directories(src) add_executable(MyProject ${SOURCE_FILES}) target_link_libraries(MyProject SDL2::Main SDL2::Net SDL2::Mixer SDL2::Image SDL2::TTF SDL2::GFX) </code></pre> And thats it! Now you can remove the SDL2 components you don't want to use. Building</h2> Following the standard cmake procedure: mkdir build cd build cmake .. make -j${nproc} </code></pre>

The tool</h2>
Here is the source code of the tool: https://github.com/edg-l/teemasterparser</a></p>
And an image of the result:</p>
Discussion on reddit.</a></p>

DDNet naming conventions</h2>
Note: There is an ongoing discussion about variable naming, find out more here</a>.</em></p>
Currently, this is how we name things:</p>

String methods to remove prefixes and suffixes</h2>
Adds two new methods, removeprefix()</a> and removesuffix()</a>, to the APIs of Python's various string objects.</p>

exa, a replacement for 'ls'</h2>
A replacement for ls which features better defaults and more features.</p>
github.com/ogham/exa</a></p>
</p>

fd, a simple, fast and user-friendly alternative to find</h2>
github.com/sharkdp/fd</a></p>
It's faster, more convenient, ignores hidden files and gitignore files by default and more.</p>
</p>

Starship, a customizable cross-shell prompt</h2>
starship.rs</a></p>
</p>

Building</h2>
Following the standard cmake procedure:</p>
`mkdir</span> build </span>cd</span> build </span>cmake</span> .. </span>make</span> -j$</span>{nproc} </span></code></pre>`

Edgar Luque - Blog

Intro to LLVM and MLIR with Rust and Melior

Implementing a simple Hashmap in Rust

UI Code in DDraceNetwork

Gentoo as a daily driver

Creating a bencode parser with nom

Parsing compressed files efficiently with Rust

The Rust dbg! macro

Wrapping errors in Rust

Implementing a chat command in DDraceNetwork

Rust Iterators: Fibonacci series

Code conventions in DDraceNetwork

Creating precompiled headers with cmake

What are precompiled headers?</h2>
They are a partially processed version of header files, this speeds up compilation because it doesn't have to repeatedly parse the original header.</p>

What's new in Python 3.9

Python 3.9</h2>
Python 3.9 has been released on October 5, 2020.</p>

Modernize your linux workflow with Rust

An intro to the DDraceNetwork game source code

An intro to MiniUPNP

Setting Up SDL2 with CMake

Edgar Luque - Blog

Intro to LLVM and MLIR with Rust and Melior

Operations</h2> These provides the functionality, and what make up the bulk of MLIR.</p> A operation has the following properties:</p>

Implementing a simple Hashmap in Rust

Probing / Searching</h2> When searching for a slot, either at insertion or search, we will need to do something when a hash collision happens:</p> Linear probing</h3> Let x</code> be the value of hash(key) (mod capacity)</code>.</p> Starting with i = 0</code>:</p>

Linear probing</h3> Let x</code> be the value of hash(key) (mod capacity)</code>.</p> Starting with i = 0</code>:</p>

UI Code in DDraceNetwork

Gentoo as a daily driver

Filesystem</h1> Coming from an Arch Linux installation using LVM on LUKS, this time I decided it wasn't that worth it encrypting my whole disk, so I went a simpler way.</p> This is my /etc/fstab</code> file:</p>

Updating</h1> I used to use the testing kernel, but I encountered some issues and the updates where pretty frequent, so I decided to use the non masked kernel.</p> To update my kernel I made a simple script:</p>

Creating a package</h1> To my surprise, creating a ebuild is quite easy, specially if the software you package uses a common build system.</p>

Creating a bencode parser with nom

Parsing the byte string</h1> Lets start with the easiest one, byte strings, as you can recall, they are made up of the a textual integer, a colon and the data:</p> 4:spam</code></p> Since the data can be non UTF-8 encoded, we will build the parser around &[u8]</code> instead of &str</code>.</p>

Parsing dictionaries</h1> Format: d3:cow3:moo4:spam4:eggse</code></p> Parsing dictionaries is nearly identical to parsing lists, but we need to parse the keys too, which are byte strings:</p>

The parser</h1> And finally, we can make the final parser function which parses all the possible values:</p>

Parsing compressed files efficiently with Rust

The libraries</h2> We will use the following libraries to achieve this:</p>

Fetching the data</h2> With ureq we can fetch the data easily:</p>

The Rust dbg! macro

Wrapping errors in Rust

Implementing From on the wrapped errors</h2> To solve this, we have to implement From<PaypalError></code> and From<reqwest::Error></code>:</p>

Implementing a chat command in DDraceNetwork

Adding the static method</h2> We added ConAboutMe</code> on the CHAT_COMMAND macro, now we need to implement it.</p> First go to src/game/server/gamecontext.h</code> and under the last static Con command you see add it:</p>

Implementing the chat command</h2> Then to implement it we go to src/game/server/ddracechat.cpp</code> and add it to the end:</p>

Rust Iterators: Fibonacci series

Code conventions in DDraceNetwork

Creating precompiled headers with cmake

What are precompiled headers?</h2> They are a partially processed version of header files, this speeds up compilation because it doesn't have to repeatedly parse the original header.</p>

What's new in Python 3.9

Python 3.9</h2> Python 3.9 has been released on October 5, 2020.</p>

Add Union Operators To dict (PEP 584)</h2> This allows the union operation to be performed on dicts:</p>

Type Hinting Generics In Standard Collections (PEP 585)</h2> This feature enables type hinting using the standard collections without having to rely on the typings</code> module.</p> Previously to type hint a list you would do:</p>

Support for the IANA Time Zone Database in the Standard Library</h2> This feature adds a new module zoneinfo</code> that provides a concrte time zone implementation supporting the IANA time zone database.</p>

Modernize your linux workflow with Rust

exa, a replacement for 'ls'</h2> A replacement for ls which features better defaults and more features.</p>

An intro to the DDraceNetwork game source code

An intro to MiniUPNP

Setting Up SDL2 with CMake

Building</h2> Following the standard cmake procedure:</p> mkdir</span> build </span>cd</span> build </span>cmake</span> .. </span>make</span> -j$</span>{nproc} </span></code></pre>

What are precompiled headers?</h2>
They are a partially processed version of header files, this speeds up compilation because it doesn't have to repeatedly parse the original header.</p>

Python 3.9</h2>
Python 3.9 has been released on October 5, 2020.</p>

exa, a replacement for 'ls'</h2>
A replacement for ls which features better defaults and more features.</p>
github.com/ogham/exa</a></p>
</p>

Building</h2>
Following the standard cmake procedure:</p>
`mkdir</span> build </span>cd</span> build </span>cmake</span> .. </span>make</span> -j$</span>{nproc} </span></code></pre>`