artodeto's blog about coding, politics and the world

I am currently developing a cli application using typer.

As far as possible, I try to have strict separated bounded context. I ended up build two problems I need to solve. One problem was to create a adapter to enable each context to exchange well defined messages or data with another context. The second issue was a pure scale issue. There are moments were part of the application just iterates over an result set of gazillion (or "millausend" as the German's are saying) entries. The runtime was ramping minutes and this sucks.

I ended up evaluating task queue libraries in python. Based on my gut-feeling, I wanted to avoid celery since it is to complex and offers way more than I was search for. rq was an option but the mighty internet told me that it has its issues and is kind of outdated. taskq was almost the choice but tends to be a tool I would use but not for the area it was invented for. dramatiq was the choice. It was looking simple and claims to be written for python 3 from a maintainer of celery and focused on delivering 20 percent of the code base to fit in 80 percent of the celery use cases. Not feature complete but I must admit that I did not spend time to have a look on huey or arq.

Adding the first actor was super satisfying since I solved my scaling problem so hard that I need to reduce the number of workers to not break one of the other systems my code works with. Adding a second actor was also working fine since it was defined in the same context workers file.

As I've written down earlier, my code lives in one application but I have strong bounded contexts. The "fun" had started when I've written a third actor but this time in a second workers file. Still until today, I am not able to separate the actors into dedicated files and still run the dramatiq workers process while it is processing all queued messages.

I've tried a lot of example projects out there. Each example project just defines one actor, almost always the one actor mentioned in the documentation which was boring. I've updated dependencies to make an example still "workable" today. I found an issue that periodiq is not supporting python 3.12. And I've asked the maintainer of one example project if he has an idea why multiple actors are not working as expected.

At the end, I've created my own dramatiq example project. But this is an example project with multiple actors. Furthermore, this project shows the issues since it has multiple branches. Each branch shows a way to use dramatiq and lists the downside of this approach. I've used this example project to ask my question in the official dramatiq group. Let's see if there will be an answer on it.

I still hope that there is either an understanding issue on my side (like I use the tool simply wrong) or that I make a mistake in the setup and worker process creation.

Your takeaway is, that there is now at least one dramatiq example with multiple actors.