From a user to a contributor. A Linux Kernel Mentorship Journey

I have been a linux user for couple of years now and at the same time, I’m into systems / low level programming and have always been interested in operating systems especially the linux kernel. So when the opportunity showed itself, I wasted no time in taking it.

I first heard about the linux kernel mentorship in the techleef community  (techleef is a tunisian & arab community of many talented learners which hosts multiple free online lessons about embedded systems, embedded linux, yocto and many more related subject all of which is offered by highly experienced engineers in those fields. It was founded by a yocto contributor and developer and our fellow tunisian Talel from a member. We then both got into applying to the mentorship with very high hopes of making the most out of it.

Prerequisite tasks and acceptance

So I have applied to enter the program in the 17th of july, which we then got assigned a lot of tasks to complete in order for the application to be reviewed at the end of the application window. The tasks were well structured and ordered by deadlines. One of the first task was to complete the LFD103 course,a beginner guide to linux kernel development, Which built the foundation for completing other Prerequisite tasks. All of the required tasks were helping any new kernel developer for building a good mental model of what is the kernel development workflow. This included many things like how to build & boot the kernel, how to make custom modules, running selftests, decoding the stack trace, working with git to send email and make patches and many other similar tasks that lay the bases for what’s to come in the mentorship.We also are required to make some simple doc-related patches and send them to the linux kernel community to practice that too.So these tasks are more of a preparing phase rather than a show me your skill testing phase which makes this program a lot more approachable for many people that are unfamiliar with the kernel space of linux like myself. After a duration of 40 days approximately, this phase was over and all of the applicants would have submitted all of their prerequisite tasks which I also managed to do. The 5th of September, I have received an email from LFX informing me of my acceptance to the mentorship program.

Initial phase

After being accepted, we also have been invited to a regular meeting that was called office hours held every Wednesday. This wasn’t our only means of communication but office hours was our official meeting with our mentors which was really helpful in getting us to start on the right track and also guiding us along the mentorship based on our individual progress. In the first couple of meetings,the mentors were introducing us mentees to the structure of the program, recommending to us the pace that we should work in, showing us what to work on and what to avoid,revealing some tools that should help us and also receiving and answering our questions.Our mentors quickly realized that the number of mentees in this program are way higher than usual which consequently led to them requesting another mentor for help. After those first meetings, each new meeting was more of a tracking what the mentees were doing and recommending what to do and what not to do while also answering our questions and helping us reviewing our patches. Shuah, one of our mentors, was adding tasks accordingly from the start of the mentorship which help us fill some of the lacking areas of our knowledge. It was also recommended to us from the start to choose 2 subsystem at max to narrow our focus and have solid patches in those subsystems by the end of the mentorship. For me, initially I wanted to work on the bpf  (Berkeley Packet Filter subsystem since earlier this year I got a job offer in that area which I rejected. So I have spent the first few weeks in exploring the bpf subsystem with multitude of resources and also experimenting with syzbot bugs that are relative to bpf. Quickly after some failed patches and the initial exploration of the bpf subsystem, it was apparent to me that it’s not the best choice as it’s a very complex subsystem  (All kernel subsystems are complex but relatively speaking, the bpf subsystem is one of the harder ones to get into given the restricted period of 3 months to get multiple patches in and it’s also very active and rapidly changing.

Strategic call

After the first couple of weeks of the mentorship while tinkering with the bpf subsystem and sending some related patches to that, I got caught up more in my university studies which required me to think of a strategic plan to follow. I opted to do some simpler refactoring patches based on the deprecated document to get some patches accepted into the kernel so that I get closer to the minimal graduation requirements and also give myself more time by the end of the mentorship period to focus on more substantial patches with less or no time pressure. I managed to get 4 patches accepted here across different subsystems which are the mm  (Memory management subsystem,btrfs  (B-tree filesystem subsystem, the block subsystem and the drm  (Direct Rendering Manager subsystem. This process took couple of weeks and I still didn’t manage to narrow my focus on a specific subsystem. This is my next goal following my plan.

Reading focused phase

In the last month of the mentorship, I wanted to make some more meaningful patches following my plan after the 4 accepted patches. But since I didn’t have a focus on a particular subsystem, I needed to sort that out first. I was inclined to focus on more stable subsystems like the mm and filesystems. To do so, I needed to have a more depth of knowledge on the general mechanisms of each subsystem. I then chose to read the OSTEP  (Operating Systems, Three Easy Pieces book. I took my time reading most of it and taking notes to help me understand better the kernel code. After completing that, I was reading to jump back into making patches for those subsystems.

Pulled an ace by the end

Jumping back to the code, I stumbled upon an improvement to the kernel selftests subsystem for the mm userfaultfd. I spun up a patch for that and it quickly got accepted. Since I have hit the minimal requirements for graduation, I then focused on solving a syzbot bug related to my new area of focus  (mm and fs subsystems . I have attempted many bugs all during the mentorship but they were a bit scattered. During this final run, I came close to a lot of fixes only to be beaten by some other contributors in terms of time to send a complete patch. Until I finally was the first one to fix a bug. I have made a blog post talking about this syzbot bug which was causing a memory leak in the HFS subsystem and how I solved in more details.I highly recommend reading that post.Although it wasn’t some crazy concurrency bug, but it led to me building good connection with maintainers and to discovering the presence of the same bug in multiple other filesystems which I’m currently working on and already have a patch for the HFS+ subsystem.

Current merged patches

Currently, the following are my patches that have been merged to mainline and upstreamed which will be available in the next linux kernel release of version 6.19 while my HFS/HFS+ patches have been reviewed and are awaiting merge.

2346b966c66a btrfs: refactor allocation size calculation in alloc_btrfs_io_context()
1ec5d5810b6f selftests/mm/uffd: remove static address usage in shmem_allocate_area()
f0c74b6cb91d mm/vmalloc: use kmalloc_array() instead of kmalloc()
e5a82249d88c blk-mq: use struct_size() in kmalloc()
603525846316 drm/gud: Use kmalloc_array() instead of kmalloc()

Final remarks && Tips for future mentees

This journey was tough but very fulfilling. It helped bridge the gap between my linux userspace knowledge and linux kernel space. It’s needless to say that this mentorship has helped me acquired a lot of skills and deepen my knowledge on a lot of different tools all related to systems programming and linux kernel development and workflow. I wanted to share some of the knowledge that I learned during this mentorship from my and my fellow mentee’s mistakes and from the experience:

  1. Take the LFD103 course seriously: When doing the prerequisite tasks, don’t just rush through the LFD103 course. Take your time understanding it’s contents and taking notes. It will help you in this mentorship and also help mentors by not asking questions that are already answered in the course.
  2. Focus on the why not the how: This is regarding your commit message. It has become more evident to my and my fellow mentees that maintainers value the reason behind your patch more than anything. When writing your commit message, Think about describing why you are making this patch, what does it solve, why does it have to add that functionnality , etc.. Because even if your code is totally valid and reasonable, Without a strong reason for the patch, it will be hard for you to convince the maintainers to accept such a patch.On the other hand, If you do have a strong why but lack in your technical skills / implementation of your solution, the community and maintainers are very happy to guide you to the right implementation.
  3. Testing,Testing,Testing: This should be clear as day, that any patch that you send should be carefully tested beforehand even if the change is very minimal and simple. But during my mentorship, Many of my fellow mentees have sent untested patches not knowing that testing is mandatory meanwhile maintainers have assumed testing of those patches. At the time of merge, those patches have caused a lot of regressions and many maintainers weren’t happy and contacted shuah to ensure that this point is communicated to the mentees.
  4. Respect the community resources: when sending a patch to the kernel community, keep in mind that each patch requires community resources from time reviewing to testing and merging. So don’t just rush to send many incomplete patches that would waste those resources.This will only hurt your reputation among the kernel community.Instead,Take your time to ensure that everything is well tested and within the kernel mailing list guidelines and your patch isn’t missing anything then you can send your patch.
  5. We are humans: In your mentorship journey, You may expect that when you send a patch some maintainer should respond immediately.Well that’s not the case. Most of the maintainers and contributors are part of the linux kernel community on their own time except a few. So many have other jobs, life responsibility and the list goes on. So give the community time to review and respond to your patch. And also, don’t expect all maintainers of different subsystems to have the same opinion to similar patches.Even within the same subsystem maintainers may differ on if the patch is good and worthwhile or not. So keep that in mind.
  6. Bonus: I have discovered at the end of my mentorship about a very useful website to start exploring the kernel source code in a more structure and guided way.It may not be up-to-date with the latest mainline kernel, but if you’re starting to hack your way into the kernel, I think it’s a very good starting point and I highly recommend giving it a try.

Finally, I would like to say thank you to my mentors shuah,david and khalid and congratulate them for their work too because it was not easy for them at all to manage more than 100 applicants across two mentorships at the same time. AndThank you mokhtar for sharing with us this amazing opportunity in the techleef community.

Happy Hacking Everyone!