So, around the new year I disabled the feeds during an operation to shut off staff eyeballs for one of the networks that’s been targeting the users running the IRCTHULU runners.
I didn’t. I built a new tool called Leptin, which, like synapse, pulls from the MQ, only it dumps to disk instead of the database. Leptin will eventually be integrated into Synapse.
This was to give the runners a break while I focused on some work stuff without having to worry about someone finding something in the logs to identify the runners again.
Unfortunately during the development of Leptin I had to drop about 80,000 messages over a pretty stupid typo so we lost a couple days of logs. I’ve got some safeguards in the code now that will prevent that from even being possible in the future.
As for Leptin, what’s especially cool about the design for this part is you can use the existing tool, nerve, to replay it back into the queue, and it’ll slurp it up.
Leptin itself is a bottleneck though right now. I should have written it to be asynchonous like synapse as Pika in python is very, very slow.