Skip to content

Solved my dramatiq puzzle, two times!

It took me some time and a push into the right direction to solve my puzzle.

My first solution is a fix for my all actors in one worker. The fix here was to remove the dramatiq setup call from within each actor and to centralize it in the one worker file before importing the actors per context.

My second solution is a fix for my one worker per context. The fix here was to add a state to the dramatiq setup to prevent creating brokers instances after broker instances.

With my code, you are now able to create python project that uses dramatiq and has multiple actors defined in multiple worker files.

Sounds simple but took me more than an week. Hope it prevents at least one to waste that time!

Lets write a cli application in python using dramatiq and spend a week chasing issues

I am currently developing a cli application using typer.

As far as possible, I try to have strict separated bounded context. I ended up build two problems I need to solve. One problem was to create a adapter to enable each context to exchange well defined messages or data with another context. The second issue was a pure scale issue. There are moments were part of the application just iterates over an result set of gazillion (or "millausend" as the German's are saying) entries. The runtime was ramping minutes and this sucks.

I ended up evaluating task queue libraries in python. Based on my gut-feeling, I wanted to avoid celery since it is to complex and offers way more than I was search for. rq was an option but the mighty internet told me that it has its issues and is kind of outdated. taskq was almost the choice but tends to be a tool I would use but not for the area it was invented for. dramatiq was the choice. It was looking simple and claims to be written for python 3 from a maintainer of celery and focused on delivering 20 percent of the code base to fit in 80 percent of the celery use cases. Not feature complete but I must admit that I did not spend time to have a look on huey or arq.

Adding the first actor was super satisfying since I solved my scaling problem so hard that I need to reduce the number of workers to not break one of the other systems my code works with. Adding a second actor was also working fine since it was defined in the same context workers file.

As I've written down earlier, my code lives in one application but I have strong bounded contexts. The "fun" had started when I've written a third actor but this time in a second workers file. Still until today, I am not able to separate the actors into dedicated files and still run the dramatiq workers process while it is processing all queued messages.

I've tried a lot of example projects out there. Each example project just defines one actor, almost always the one actor mentioned in the documentation which was boring. I've updated dependencies to make an example still "workable" today. I found an issue that periodiq is not supporting python 3.12. And I've asked the maintainer of one example project if he has an idea why multiple actors are not working as expected.

At the end, I've created my own dramatiq example project. But this is an example project with multiple actors. Furthermore, this project shows the issues since it has multiple branches. Each branch shows a way to use dramatiq and lists the downside of this approach. I've used this example project to ask my question in the official dramatiq group. Let's see if there will be an answer on it.

I still hope that there is either an understanding issue on my side (like I use the tool simply wrong) or that I make a mistake in the setup and worker process creation.

Your takeaway is, that there is now at least one dramatiq example with multiple actors.

"a service you can count on"? - free as in freedom unique number repository (web service and client) available

Every needed a service that takes care of ensures each number is unique?
The unique number repository is such a service. Since it is "free as in freedom", you can own your data (why? because it is your data of course).
What is it? A simple REST based server and an example command line client.
It is using an authentication token to prevent the service is used by unauthorized persons. Feel free to create a pull request for a more sophisticated authentication method.

Convert VCF File into a CSV File

I needed to convert a VCF file into a CSV file by fetching the important and readable informations.
Thanks to the bazzline component it was a quick win/no brainer. You can find the result here. It is written under GPL-3 License.

How to use it?

  • open a terminal / command line
  • git clone https://github.com/stevleibelt/incubator
  • cd incubator/general/vcf_to_csv_converter
  • ./converter